CN113791740B - Method for recording object storage bucket statistics and counting - Google Patents
Method for recording object storage bucket statistics and counting Download PDFInfo
- Publication number
- CN113791740B CN113791740B CN202111325910.3A CN202111325910A CN113791740B CN 113791740 B CN113791740 B CN 113791740B CN 202111325910 A CN202111325910 A CN 202111325910A CN 113791740 B CN113791740 B CN 113791740B
- Authority
- CN
- China
- Prior art keywords
- bucket
- key
- sub
- thread
- keys
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for recording the statistical count of an object storage bucket, which belongs to the field of data storage and comprises the following steps: after the object storage gateway RGW finishes the write operation, synchronously updating the value of the key for counting the storage bucket to which the target object belongs; dividing a key for counting the target storage bucket into a plurality of sub-keys, and dividing the whole key space into subspaces with appointed number of pieces; the task thread reads the value of the target key from the cache, and stores and persists the value to the TIKV to which the sub-key of the target key belongs after modification; after receiving the request for reading the statistic count of the target bucket, the RGW splits the processing request for the plurality of sub-keys and sends the processing request to the TIKV at the rear end by combining the buckets corresponding to the request; and S5, traversing the sub-keys of the statistical count of the target storage bucket by TIKV, merging in the TIKV node, and returning the merging result to RGW for secondary merging. Unnecessary waiting time consumption caused by request queuing under a high-concurrency scene and the problems of writing conflict and hot spot caused by modifying the same Key are avoided.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a method for recording the statistical count of an object storage bucket.
Background
Object storage, i.e., object-based storage, is a generic term used to describe methods of resolving and processing discrete units, referred to as objects. Like a file, an object contains data, but unlike a file, an object no longer has a hierarchy in one hierarchy. Each object is in the same level of a flat address space called a storage pool, and an object does not belong to the next level of another object.
Both files and objects have metadata related to the data they contain, but objects are characterized by extended metadata. Each object is assigned a unique identifier, allowing a server or end-user to retrieve the object without knowing the physical address of the data. This approach is helpful for automating and simplifying data storage in a cloud computing environment. Object storage is often compared with parking in a high-grade restaurant, and when a customer needs to park in a hotel, the customer gives a key to others and changes a receipt; the customer does not have to know where his car is parked, nor how many times the attendant will move his car while he is eating. In this metaphor, a unique identifier for a stored object represents the customer's receipt.
An object storage gateway (RGW) is a service provided by object storage that enables clients to access object storage clusters using standard object storage APIs, while the object storage gateway accesses the underlying storage engine through an internal interface.
The object storage gateway generally provides a standard restful interface for web services, and can perform user right control, protocol conversion, data analysis and processing and the like in an object gateway layer, so that the use scene of object storage is greatly enriched.
The container used to store the storage objects (objects) is called a storage space or Bucket (Bucket), which may be referred to simply as a "Bucket," and each storage Object must be subordinate to a Bucket. Different buckets have different configuration attributes including region, access rights, storage type, etc. Users can create buckets of different attributes based on the requirements of the storage objects in actual production.
The statistical count of the bucket belongs to the metadata information of the bucket, and is mainly used for storing the object statistical information of the bucket, including the total number of objects, the total size of the objects and the like, and each bucket has a unique statistical count. In distributed storage Ceph, the statistical count is stored in Omap (single-machine Key-Value storage system), and a Key (Key) named rgw _ bucket _ dir _ header is corresponded to a Key/Value structure.
Each bucket has a unique statistical count, and in a high concurrency scene, if a method for synchronously updating the statistical count is adopted, the phenomenon of blocking I/O (input/output) can occur due to queuing of requests in an OSD (object storage engine), so that the mode of recording the bucket statistical count by many manufacturers at present is asynchronous. When the updating operation of the relevant statistical counting is carried out, the content needing to be updated is changed into a message to be sent to a third-party component, the message is consumed by a background, and the statistical counting is asynchronously updated. The asynchronous statistical scheme depends on a third-party component, and has the advantages that the write operation I/O flow is not blocked due to updating of statistical information, the performance is less affected, and a conflict scene can be avoided when a plurality of clients or a plurality of threads operate the same bucket simultaneously; however, the disadvantages are also apparent, and are shown in the following two aspects:
1) the maintenance cost is increased by depending on a third-party message queue, and when the message is asynchronously sent to the queue, the message needs to be persisted to prevent the message loss caused by power failure from causing that the statistical count cannot be correctly updated;
2) due to the adoption of the asynchronous updating scheme, after the operation on the storage object is completed, a certain delay exists for the updating of the statistical count, so that when the operation of the asynchronous updating is not completed, the reading of the statistical information is inaccurate.
Another scheme for recording the bucket statistics count is synchronous statistics, in which, when the object storage gateway receives a write request every time and completes Put/Delete operations on a storage object in a bucket, the object storage gateway synchronously updates the bucket statistics count, and the update of the statistics count also belongs to a part of the I/O flow of the current modification operation. The scheme of synchronizing statistics may increase the time consumption of write requests, but may not be inaccurate for reading the statistics due to the synchronization operation. However, in a high concurrency scenario, when multiple threads in multiple clients operate on a storage object in the same bucket, all threads update the statistical count of the bucket, and due to the uniqueness of the statistical count, the request is queued in OSD and executed serially in Ceph, which may result in performance degradation.
Disclosure of Invention
In the scheme of synchronously updating the statistical count, due to the self framework of the distributed storage Ceph, only one thread updates the statistical count of the storage bucket to which the target object belongs at the same time under a high concurrency scene, so that the problem of I/O blockage caused by request queuing exists, and extra queuing waiting time is increased for a request processing flow. In view of this, the present invention provides a method for recording statistics and counts of object buckets, and aims to solve unnecessary waiting time consumption caused by request queuing in a high concurrency scenario by replacing a backend storage component on the basis of a synchronous statistics flow, and avoid a write collision problem and a write hot spot problem caused by modifying the same Key in the high concurrency scenario.
A method of recording a statistical count of object buckets, comprising the steps of: s1, after completing the write operation, the object storage gateway synchronously updates the value of the key of the statistical count of the storage bucket of the target object; s2, dividing the key of the statistic count of the storage bucket of the target object into a plurality of sub-keys, and dividing the whole key space into sub-spaces with appointed number of pieces; s3, the task thread reads and modifies the value of the key of the statistical count of the bucket to which the target object belongs from the cache, stores the modified value into the cache of the task thread again, and persists to the distributed transaction key value database to which the sub-key of the statistical count key of the bucket to which the target object belongs; s4, after receiving the request for reading the statistic count of the bucket to which the target object belongs, the object storage gateway splits the processing request for the plurality of sub-keys by combining the bucket corresponding to the request and sends the processing request to the distributed transaction key value database at the back end; and S5, traversing the sub-keys of the statistic count of the storage bucket of the target object by the distributed transaction key value database, merging the sub-keys in the nodes of the distributed transaction key value database, and returning the merged result to the object storage gateway for secondary merging.
Further, in step S1, after completing the self-write request, each thread of the object storage gateway generates a key for counting the buckets to which the target object belongs according to the bucket to which the object belongs and the self-thread ID.
Further, when the client sends the write request to the object storage gateway, the object storage gateway sends the write request to a plurality of threads for processing.
Further, after the statistically counted keys are split in step S2, when a write operation is performed, the thread performing the write operation first splices the keys to be modified according to the pre-agreed format.
Further, the thread executing the write operation splices the keys to be modified according to a pre-agreed format, including: the thread executing the write operation splices the sub-keys of the keys needing to be modified according to the format of < (UUID% shared _ num) + Bucket ID + separate char + UUID >; wherein, (UUID% shrard _ num) is formed by performing mould-taking and splicing on the UUID according to the number of fragments shrard _ num; bucket ID represents Bucket ID; the separate char is a separator; the UUID is a universally unique identifier generated from the object store gateway ID and the thread ID.
Further, when the key of the statistical count is split in step S2, each object storage gateway is assigned with a unique ID, and the thread of each object storage gateway also has an independent ID in the object storage gateway to which it belongs; when the statistical count is updated, each thread independently completes the updating of the key of the thread.
Further, when the statistically counted key is split in step S2, the sub-key is configured to: firstly, the ID of the storage bucket to which the target object belongs is obtained, then a separator for distinguishing the target object from other sub-keys is obtained, and then a universal unique identification code is generated by using the ID of the object storage gateway and the ID of the thread and is spliced at the tail end.
Further, sub-keys formed by the shards are scattered on different distributed transaction key value database nodes.
Further, in step S5, when performing the read operation, traversing the statistically counted sub-keys of the buckets to which the target objects belong in the distributed transaction key value database by < shard number + Bucket ID >, and merging, where the Bucket ID refers to a Bucket ID.
Further, the step S5 executing the read operation includes: when a client sends a read request for a storage bucket to which a target object belongs to an object storage gateway, the object storage gateway forwards the read request to each distributed transaction key value database node at the back end; traversing the sub-key prefixes of the storage Bucket to which the target object belongs by each node, wherein the sub-key prefixes are the sub-key storage gateway IDs after fragmentation plus the thread IDs and then are spliced with the Bucket IDs, at the moment, the results are not returned, but are pushed down to a distributed transaction key value database for co-processing, the traversal results are merged in the distributed transaction key value database, and the different sub-key suffixes < object storage gateway IDs + thread IDs >; and each distributed transaction key value database node returns the merging result to the object storage gateway, and secondary merging is carried out in the object storage gateway.
The technical scheme of the invention has the beneficial effects that: on the basis of the flow of synchronous statistics, unnecessary waiting time consumption caused by request queuing in a high-concurrency scene is solved by replacing a back-end storage component (the original single-machine key value storage system is replaced by a distributed transaction key value database in a metadata storage mode of a storage bucket), and queued serial requests are changed into parallel requests; moreover, a counting key is redesigned, and the unique counting key of the original storage barrel is divided into a plurality of sub-keys, so that writing conflict caused by modification of the same key in a high-concurrency scene is avoided; in addition, the sub-keys are fragmented, so that the problem of hot spots can be avoided when the sub-keys are stored in a back-end distributed database.
Drawings
Fig. 1 is a block diagram of an implementation of the method for recording a statistical count of object buckets according to an embodiment of the present invention.
FIG. 2 is a diagram of the sub-key structure of the statistical counting according to the embodiment of the present invention.
FIG. 3 is a flow chart illustrating a recording statistic counting method according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a synchronous statistical count reading process according to an embodiment of the present invention.
Detailed Description
The invention is further described with reference to the following figures and detailed description of embodiments.
The specific embodiment of the invention provides a method for recording the statistical count of an object bucket, which comprises the following steps: and S1, after the object storage gateway finishes the write operation, synchronously updating the value of the key of the statistical count of the bucket to which the target object belongs. S2, dividing the key of the statistic count of the storage bucket of the target object into a plurality of sub-keys, and dividing the whole key space into sub-spaces with appointed number of pieces; after the keys with the statistical count are split, when the write operation is executed, the thread executing the write operation firstly splices the keys needing to be modified according to a predetermined format. S3, the task thread reads and modifies the value of the key of the statistical count of the bucket to which the target object belongs from the cache, stores the modified value into the cache of the task thread again, and persists to the distributed transaction key value database to which the sub-key of the statistical count key of the bucket to which the target object belongs. And S4, after receiving the request for reading the statistic count of the bucket to which the target object belongs, the object storage gateway splits the processing request for the plurality of sub-keys by combining the buckets corresponding to the request and sends the processing request to the distributed transaction key value database at the back end. And S5, traversing the sub-keys of the statistic count of the storage bucket of the target object by the distributed transaction key value database, merging the sub-keys in the nodes of the distributed transaction key value database, and returning the merged result to the object storage gateway for secondary merging.
Different from the storage mode of the distributed storage Ceph for the metadata, the method of the embodiment of the present invention replaces the metadata storage mode of the bucket with the original OMAP (single machine key value storage system) to the distributed TIKV (distributed transaction key value database), which improves the read/write performance of the metadata to a certain extent, i.e., reduces the extra I/O time consumption generated by the synchronization statistics. In addition, because the back-end storage is replaced, the request is not sent to an OSD (object storage engine), and in a high concurrency scene, the modification of a certain bucket statistic count Key (Key) does not need to be queued, namely serial processing is changed into parallel processing.
However, in a highly concurrent scenario, when a write request is executed, the write request frequently encounters a conflict problem due to the uniqueness of the statistical count, and common solutions include two kinds, namely locking a resource and retrying a conflict. When a resource is locked, there may be a waiting time for acquiring the lock, and the I/O may be blocked during the waiting process, which affects the performance. Retry of a conflict may re-execute the conflicting flow if the resource conflicts, and may also incur additional memory overhead in addition to the blocking of normal I/O flows. Therefore, the method and the device optimize the synchronous counting scheme, split the counting key of the storage barrel into a plurality of sub-keys, and bind the updating of the counting to the corresponding thread, thereby effectively avoiding the writing conflict.
Splitting the key with statistical count can effectively avoid writing conflict, but because prefixes of all sub-keys after splitting are the same, the common writing hot spot problem of a distributed system exists, and the problem can cause that the pressure of I/O can not be distributed to each node, so that a new technical problem is generated(scope of the bond)By passingHashThe sub-spaces divided into a specified number of pieces, the sub-keys belonging to the same sub-space fall into the same range or adjacent ranges when written.
When a write request is processed, because an original single statistical counting key is split into a plurality of sub-keys, when a read request for statistical counting is executed, the split statistical counting sub-keys need to be read in batches, and then merging processing is performed to obtain a final statistical counting result. However, the number of keys is increased due to the splitting of the statistically counted keys, and can reach up to 10W magnitude, and if the key values are all read into the RGW (object storage gateway), the service pressure is increased dramatically. Therefore, the invention pushes down part of the operation of the keys for merging the statistical counts to each TIKV node, each TIKV node traverses the statistical count information related to the storage bucket to which the target object belongs, carries out primary merging statistics in the TIKV node, and then transmits the result back to the RGW for secondary merging, so that the number of the keys merged in the RGW is equal to the number of the TIKV nodes, the magnitude of the merged keys is reduced exponentially, and the network overhead and the service pressure of the RGW layer are effectively reduced.
Referring to fig. 1, a block diagram of an implementation of the method for recording the statistical count of object buckets according to the present invention is shown. As shown in fig. 1, the client 10 issues an object write request to an object storage gateway (e.g., gateway 1, gateway 2, etc.). When a write request is received, the object storage gateway will dispatch the request to multiple threads, such as: when receiving a write request issued by a client 10, the gateway 1 distributes the write request to a plurality of threads (such as a thread 11, a thread 12, a thread 13, etc.) belonging to the gateway; when receiving a write request issued by the client 10, the gateway 2 distributes the write request to a plurality of threads (e.g., thread 21, thread 22, thread 23, etc.) belonging to it. And after each thread of the object storage gateway finishes the self writing request, generating a key for counting the storage bucket of the target object according to the storage bucket of the object and the self thread ID. In the object storage gateway layer, in order to fragment the original Key of the statistical count, a unique ID needs to be allocated to each RGW, and the thread in each RGW also has an independent ID in the corresponding RGW, and when the statistical count is updated, each thread independently completes the updating of the Key related to itself.
In order to solve the writing conflict under the scene of high synchronous statistics, the invention redesigns the unique statistical counting Key of the storage bucket and splits the statistical counting Key into a plurality of sub-keys. The structure design of the sub-Key is as follows: firstly, adding a special separator into the ID of a target storage space Bucket, and then generating a UUID (universal unique identifier) by using the ID of an object storage gateway and the ID of a thread and splicing the UUID at the tail end. Since the back end is stored as the distributed database TIKV, there is a problem of writing hot spots, it is necessary to perform fragment processing on these keys, perform modular extraction on the UUID generated in the previous step according to the shard _ num (number of fragments), and splice into the prefix of the sub-Key generated in the previous step. This operation breaks up the statistical count Key, so that it falls into the divided pieces and is distributed to different TIKV nodes. Therefore, the 64-bit UUID generated by the object storage gateway ID and the thread ID in the structure after the statistical count Key is split is scattered according to the number of the fragments, and the ID of the corresponding Bucket, the special separator for distinguishing from other keys and the UUID are connected, as shown in fig. 2, the format is < (UUID% guard _ num) + Bucket ID + separate char + UUID >, wherein (UUID% guard _ num) is a prefix, and the Bucket ID represents the ID of the storage Bucket to which the target object belongs; the separate char is a special delimiter distinguished from other child keys; the UUID is a universal unique identification code generated by the object storage gateway ID and the thread ID and is positioned at the tail end. When the write operation is executed, the thread executing the write operation splices the sub-keys of the key needing to be modified according to the format of < (UUID% shared _ num) + Bucket ID + separate char + UUID >.
On the basis of adopting the TIKV as a back-end storage component and splitting and Key space fragmentation processing of bucket statistics and counting, a write request flow of recording statistics and counting of the present invention is shown in fig. 3, and includes:
1.1 the client sends a plurality of Put/Delete requests for different objects to the RGW, and the RGW distributes the requests to a plurality of threads (such as Thread 1, Thread 2, …, Thread N) and processes the requests;
1.2 after each thread completes the Object write request, generating a Bucket statistical count Key to which the target Object belongs according to the Bucket to which the Object belongs and the thread ID of the thread;
1.3 each thread reads the bucket statistics count Key to which the target object belongs from its own statistics count cache and modifies (for example, modifies the capacity, number, etc.), and writes the result into the TIKV after modification.
When the statistical count of the storage bucket is read, because the statistical count Key is split into a plurality of sub keys, the sub keys need to be read in batches when the statistical count Key is read, and the result needs to be merged. Referring to fig. 4, the specific reading process includes:
2.1 when the client sends a read request to the storage bucket to which the target object belongs to the object storage gateway, the object storage gateway forwards the read request to each distributed transaction key value database node at the back end;
2.2, each node traverses the sub-key prefixes of the storage Bucket to which the target object belongs, wherein the sub-key prefixes are the partitioned object storage gateway ID plus the thread ID and then are spliced with the Bucket ID, the result is not returned at the moment, but is pushed down to the distributed transaction key value database for co-processing, the traversed result is merged in the distributed transaction key value database, and the merged different sub-key suffixes < object storage gateway ID + thread ID >;
and 2.3, each distributed transaction key value database node returns the merging result to the object storage gateway, and secondary merging is carried out in the object storage gateway.
In conclusion, the invention modifies the synchronous counting scheme, realizes the parallel recording counting under the high concurrency scene, reduces the queuing time of a plurality of writing requests, improves the writing performance, optimizes the influence possibly caused by the reading requests and lightens the service pressure of the reading counting requests.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.
Claims (10)
1. A method of recording a statistical count of object buckets, comprising the steps of:
s1, after completing the write operation, the object storage gateway synchronously updates the value of the key of the statistical count of the storage bucket of the target object;
s2, dividing the key of the statistic count of the storage bucket of the target object into a plurality of sub-keys, and dividing the whole key space into sub-spaces with appointed number of pieces;
s3, the task thread reads and modifies the value of the key of the statistical count of the bucket to which the target object belongs from the cache, stores the modified value into the cache of the task thread again, and persists to the distributed transaction key value database to which the sub-key of the statistical count key of the bucket to which the target object belongs;
s4, after receiving the request for reading the statistic count of the bucket to which the target object belongs, the object storage gateway splits the processing request for the plurality of sub-keys by combining the bucket corresponding to the request and sends the processing request to the distributed transaction key value database at the back end;
and S5, traversing the sub-keys of the statistic count of the storage bucket of the target object by the distributed transaction key value database, merging the sub-keys in the nodes of the distributed transaction key value database, and returning the merged result to the object storage gateway for secondary merging.
2. The method of claim 1, wherein in step S1, each thread of the object storage gateway generates a key for the statistical count of the bucket to which the target object belongs based on the bucket to which the object belongs and the thread ID of the thread after completing the write request of the thread.
3. The method of recording object bucket statistics counts of claim 2, wherein when a client issues a write request to the object storage gateway, the object storage gateway dispatches the write request to multiple threads for processing.
4. The method for recording the statistical count of the object buckets according to claim 1, wherein after the step S2 is performed to split the statistically counted keys, when performing a write operation, a thread performing the write operation firstly splices the keys to be modified according to a pre-agreed format.
5. The method of recording object bucket statistics counts of claim 4, wherein the thread performing the write operation concatenates the keys that need to be modified according to a pre-agreed format, comprising:
the thread executing the write operation splices the sub-keys of the keys needing to be modified according to the format of < (UUID% shared _ num) + Bucket ID + separate char + UUID >; wherein, (UUID% shrard _ num) is formed by performing mould-taking and splicing on the UUID according to the number of fragments shrard _ num; bucket ID represents Bucket ID; the separate char is a separator; the UUID is a universally unique identifier generated from the object store gateway ID and the thread ID.
6. The method of claim 1, wherein when the key of the statistical count is split in step S2, each object storage gateway is assigned a unique ID, and each object storage gateway has a thread with an ID independent in the object storage gateway; when the statistical count is updated, each thread independently completes the updating of the key of the thread.
7. The method of recording a statistical count for an object bucket as claimed in claim 1, wherein when the statistically counted key is split in step S2, the sub-key is structured such that: firstly, the ID of the storage bucket to which the target object belongs is obtained, then a separator for distinguishing the target object from other sub-keys is obtained, and then a universal unique identification code is generated by using the ID of the object storage gateway and the ID of the thread and is spliced at the tail end.
8. The method of recording object bucket statistical counts of claim 7, wherein the sharded child keys are distributed across different distributed transaction key value database nodes.
9. The method of claim 1, wherein in step S5, when performing a read operation, a sub-key of the statistical count of the Bucket to which the target object belongs in the distributed transaction key value database is traversed by < slice number + Bucket ID > and merged, and the Bucket ID refers to a Bucket ID.
10. The method of recording object bucket statistical counts of claim 9, wherein the step S5 of performing a read operation includes:
when a client sends a read request for a storage bucket to which a target object belongs to an object storage gateway, the object storage gateway forwards the read request to each distributed transaction key value database node at the back end;
traversing the sub-key prefixes of the storage Bucket to which the target object belongs by each node, wherein the sub-key prefixes are the sub-key storage gateway IDs after fragmentation plus the thread IDs and then are spliced with the Bucket IDs, at the moment, the results are not returned, but are pushed down to a distributed transaction key value database for co-processing, the traversal results are merged in the distributed transaction key value database, and the different sub-key suffixes < object storage gateway IDs + thread IDs >;
and each distributed transaction key value database node returns the merging result to the object storage gateway, and secondary merging is carried out in the object storage gateway.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111325910.3A CN113791740B (en) | 2021-11-10 | 2021-11-10 | Method for recording object storage bucket statistics and counting |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111325910.3A CN113791740B (en) | 2021-11-10 | 2021-11-10 | Method for recording object storage bucket statistics and counting |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113791740A CN113791740A (en) | 2021-12-14 |
| CN113791740B true CN113791740B (en) | 2022-02-18 |
Family
ID=78955119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111325910.3A Active CN113791740B (en) | 2021-11-10 | 2021-11-10 | Method for recording object storage bucket statistics and counting |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113791740B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114385090B (en) * | 2022-03-23 | 2022-06-07 | 深圳市杉岩数据技术有限公司 | Data automatic processing method and device based on object storage site synchronization mechanism |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9411862B1 (en) * | 2013-12-03 | 2016-08-09 | Google Inc. | Systems and methods for dynamic sharding of hierarchical data |
| CN106909556A (en) * | 2015-12-23 | 2017-06-30 | 中国电信股份有限公司 | The storage equalization methods and device of main memory cluster |
| US10318491B1 (en) * | 2015-03-31 | 2019-06-11 | EMC IP Holding Company LLC | Object metadata query with distributed processing systems |
| CN111008185A (en) * | 2019-10-29 | 2020-04-14 | 厦门网宿有限公司 | Data sharing method, system and equipment |
| CN111857890A (en) * | 2020-07-24 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN111881133A (en) * | 2020-07-24 | 2020-11-03 | 苏州浪潮智能科技有限公司 | Storage bucket management method and device, computer equipment and readable storage medium |
| CN113064859A (en) * | 2021-03-26 | 2021-07-02 | 山东英信计算机技术有限公司 | Metadata processing method and device, electronic equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10891264B2 (en) * | 2015-04-30 | 2021-01-12 | Vmware, Inc. | Distributed, scalable key-value store |
| US11386065B2 (en) * | 2017-01-31 | 2022-07-12 | Salesforce.Com, Inc. | Database concurrency control through hash-bucket latching |
-
2021
- 2021-11-10 CN CN202111325910.3A patent/CN113791740B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9411862B1 (en) * | 2013-12-03 | 2016-08-09 | Google Inc. | Systems and methods for dynamic sharding of hierarchical data |
| US10318491B1 (en) * | 2015-03-31 | 2019-06-11 | EMC IP Holding Company LLC | Object metadata query with distributed processing systems |
| CN106909556A (en) * | 2015-12-23 | 2017-06-30 | 中国电信股份有限公司 | The storage equalization methods and device of main memory cluster |
| CN111008185A (en) * | 2019-10-29 | 2020-04-14 | 厦门网宿有限公司 | Data sharing method, system and equipment |
| CN111857890A (en) * | 2020-07-24 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Service processing method, system, device and medium |
| CN111881133A (en) * | 2020-07-24 | 2020-11-03 | 苏州浪潮智能科技有限公司 | Storage bucket management method and device, computer equipment and readable storage medium |
| CN113064859A (en) * | 2021-03-26 | 2021-07-02 | 山东英信计算机技术有限公司 | Metadata processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113791740A (en) | 2021-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11093468B1 (en) | Advanced metadata management | |
| US11449507B2 (en) | Database engine | |
| CN109710388B (en) | Data reading method and device, electronic equipment and storage medium | |
| CN109710190B (en) | Data storage method, device, equipment and storage medium | |
| US20190340171A1 (en) | Data Redistribution Method and Apparatus, and Database Cluster | |
| US20160026660A1 (en) | Distribution of an object in volatile memory across a multi-node database | |
| CN111159252A (en) | Transaction execution method and device, computer equipment and storage medium | |
| CN112199427A (en) | Data processing method and system | |
| CN112789606A (en) | Data redistribution method, device and system | |
| CN112307119A (en) | Data synchronization method, device, equipment and storage medium | |
| US11288254B2 (en) | Method of and system for processing request in distributed database | |
| CN112445626A (en) | Data processing method and device based on message middleware | |
| JP2023541298A (en) | Transaction processing methods, systems, devices, equipment, and programs | |
| CN108694230B (en) | Management of unique identifiers in a database | |
| CN113791740B (en) | Method for recording object storage bucket statistics and counting | |
| US11741081B2 (en) | Method and system for data handling | |
| EP3377970B1 (en) | Multi-version removal manager | |
| CN111221773B (en) | Data storage architecture method based on RDMA high-speed network and skip list | |
| CN111459913B (en) | Capacity expansion method and device of distributed database and electronic equipment | |
| US11210236B2 (en) | Managing global counters using local delta counters | |
| US11138231B2 (en) | Method and system for data handling | |
| US20230385263A1 (en) | Improved execution of operations on partitioned tables | |
| EP3686751A1 (en) | Method and system for data handling | |
| US11227016B2 (en) | Scalable locking techniques | |
| CN110716814B (en) | Performance optimization method and device for inter-process large-data-volume communication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |