WO2024021488A1

WO2024021488A1 - Metadata storage method and apparatus based on distributed key-value database

Info

Publication number: WO2024021488A1
Application number: PCT/CN2022/141807
Authority: WO
Inventors: 胡爱存; 侯飞; 梁成武; 陈玉鹏; 张翼
Original assignee: 天翼云科技有限公司
Priority date: 2022-07-29
Filing date: 2022-12-26
Publication date: 2024-02-01
Also published as: CN115454994A

Abstract

The present invention relates to the field of data processing. Disclosed are a metadata storage method and apparatus based on a distributed key-value database. The method comprises: determining metadata of objects stored in buckets, and determining preset types and key-value pairs of the metadata; and storing the metadata in corresponding hash tables and ordered lists on the basis of the determined preset types and key-value pairs of the metadata, wherein each bucket in a distributed storage system stores metadata by means of at least one hash table and at least one ordered list, the hash table is used for storing metadata, and the ordered list is used for storing retrieval information of metadata. The present invention reduces the complexity of I/O operations, and increases the storage scale of a single bucket while ensuring efficiency, thereby achieving efficient and low-space-utilization retrieval suitable for metadata across multiple ordered lists.

Description

A metadata storage method and device based on distributed key-value database

Technical field

The invention relates to the field of data processing, and in particular to a metadata storage method and device based on a distributed key-value database.

Background technique

The distributed object storage system maintains an index table for each bucket, which stores the mapping relationship between the bucket and the metadata of all objects in the bucket. When a user accesses an object in a bucket, the specific data of the object is obtained through the bucket index. , and when the number of objects stored in the bucket is too large, oversized indexes will cause performance and reliability problems.

technical problem

In order to solve the limitation of the size of a single bucket object, the existing technology adopts a dynamic sharding mechanism by creating multiple index tables. However, when bucket sharding is performed, the object metadata in the bucket will be rebalanced. If the bucket is stored With a large number of objects, the data rebalancing operation during sharding will be very time-consuming, blocking front-end writing and giving users a very unfriendly experience.

Therefore, how to solve the business blocking problems caused by the limited single-bucket object storage scale and the sharding mechanism in distributed object storage systems is an important issue that needs to be solved urgently in the industry.

Technical solutions

In view of this, embodiments of the present invention provide a metadata storage method and device based on a distributed key-value database to solve the business congestion caused by the limited single-bucket object storage scale and the sharding mechanism in the distributed object storage system. question.

According to a first aspect, an embodiment of the present invention provides a metadata storage method based on a distributed key-value database. The method is applied to a server, and the server is installed with a distributed object storage system. The method includes:

Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata;

Based on the determined preset type and key-value pair of the metadata, the metadata is saved into the corresponding hash table and ordered list; each bucket in the distributed storage system passes through at least one hash The hash table and at least one ordered list store metadata, the hash table is used to store metadata, and the ordered list is used to store retrieval information of metadata.

With reference to the first aspect, in the first embodiment of the first aspect, the preset types include first, second, third and fourth types, the metadata of the first type is basic metadata, and the metadata of the second type The data is object attribute metadata, the third type of metadata is index metadata, and the fourth type of metadata is index sequence metadata;

The step of saving the metadata into the corresponding hash table and ordered list based on the determined preset type and key-value pair of the metadata specifically includes:

Based on the determined preset type, the metadata is stored in a container group corresponding to the preset type; each preset type of metadata corresponds to one of the container groups, and each of the container groups corresponds to at least One hash slot, and the number of hash slots corresponding to the container group is equal to each other;

Based on the key-value pair of the metadata, determine the CRC16 value of the metadata, and determine the hash slot corresponding to the metadata based on the total number of hash slots and the determined CRC16 value;

Map and store the metadata to the hash table corresponding to the determined hash slot; each hash table corresponds to at least one hash slot;

Assign a corresponding score value to the metadata, determine the ordered list corresponding to the metadata based on the assigned score value, and store the metadata in the corresponding ordered list; The metadata of each item in the ordered list is sorted in order according to the score value.

With reference to the first implementation of the first aspect, in the second implementation of the first aspect, the number of hash tables is determined based on a preset number and the total number of hash slots, and the number of hash tables exceeds The preset number is a factor of the total number.

In conjunction with the first aspect, in the third implementation of the first aspect, the method further includes the following steps:

Determine the client's metadata retrieval request; the metadata retrieval request includes retrieval information, and the retrieval information includes the index order and number of index elements of each of the ordered lists;

Based on the retrieval information, retrieve the corresponding metadata from each of the ordered lists;

The metadata is stored in a preset map table, and based on the preset map table, the metadata stored in the preset map table is sequentially sorted, and all steps required to complete the storage and sorting of the metadata are The above preset map table is returned to the client.

In conjunction with the first aspect, in the fourth implementation manner of the first aspect, the method further includes the following steps:

Determine the client's metadata retrieval request; the metadata retrieval request includes retrieval information, and the retrieval information includes reference metadata and the number of index elements;

Determine the storage location of the baseline metadata in each of the ordered lists;

Based on the retrieval information and the storage location, retrieve the corresponding metadata from each of the ordered lists;

With reference to the third implementation or the fourth implementation of the first aspect, in the fifth implementation of the first aspect, the metadata is stored in a preset map table, and based on the preset map table, the The metadata stored in the preset map table is sequentially sorted, and the preset map table that completes the storage and sorting of the metadata is returned to the client, which also includes:

It is determined that the preset map expression has reached the storage upper limit and there is unstored metadata. Based on the preset map table, it is determined that the unstored metadata and the last element in the preset map table are The map table sorting value of the data;

Determine that the map table sorting value corresponding to the tailmost metadata exceeds the map table sorting value corresponding to the unstored metadata, delete the tailmost metadata, and store the unstored metadata. The data is stored in the preset map table.

According to the second aspect, an embodiment of the present invention also provides a metadata storage device based on a distributed key-value database, the device is applied to a server, and the server is installed with a distributed object storage system, and the Devices include:

The first determination module is used to determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pair of the metadata;

A data storage module, configured to save the metadata into the corresponding hash table and ordered list based on the determined preset type and key-value pair of the metadata; each in the distributed storage system Each bucket stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store retrieval information of metadata.

According to a third aspect, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above is implemented. The steps of any one of the metadata storage methods based on a distributed key-value database.

According to a fourth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the distributed key-based storage medium is implemented as described above. Steps for the metadata storage method for value databases.

According to a fifth aspect, embodiments of the present invention further provide a computer program product, including a computer program that, when executed by a processor, implements any one of the above-mentioned metadata storage methods based on a distributed key-value database. A step of.

beneficial effects

The metadata storage method and device based on the distributed key-value database provided by the present invention stores the metadata of the object in the form of key-value pairs in the distributed key-value cluster, and adopts the method of sub-tables to use multiple Hash tables and multiple ordered lists are used to carry the metadata of objects in a bucket. The storage of metadata through hash tables reduces the complexity of I/O operations and increases the storage scale of a single bucket while ensuring efficiency. The storage of metadata in ordered lists provides an interface for adding, deleting, modifying, and querying object metadata. Without increasing storage costs, it achieves high efficiency and low space utilization suitable for retrieval of metadata across multiple ordered lists. .

Description of drawings

The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way, in which:

Figure 1 shows a schematic flow chart of a metadata storage method based on a distributed key-value database provided by the present invention;

Figure 2 shows a schematic flow chart of step S20 in the metadata storage method based on a distributed key-value database provided by the present invention;

Figure 3 shows a schematic structural diagram of a metadata storage device based on a distributed key-value database provided by the present invention;

Figure 4 shows a schematic structural diagram of the electronic device provided by the present invention.

Embodiments of the invention

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present invention.

At present, the distributed object storage system based on the distributed storage architecture has become a preferred solution for cloud computing, and its file storage has the advantages of being shareable and low-priced. Object metadata in a distributed object storage system is divided into two parts for storage: one part is the index metadata of the object, which is called omap, and omap refers to object. Map is an object used to save key-value pair map data. In distributed storage, omap objects play a very important role. In the file and object services provided by distributed storage, the performance of omap directly affects the cluster. Storage performance performance, omap is stored in an independent key-value (key-value) storage system outside the local file system, which is levelDB when using filestore, and rocksDB when using bluestore; the other part is the extended attributes of the object, It is called xattr and is usually used to save the version information of the object. xattr is stored in the RADOS (Reliable Autonomic Distributed Object Store) object of the bucket. The RADOS object is stored in the local file system. Its size is affected by the file system. Limitation, resulting in a limit on the number of objects it can carry. This results in the need to read related data in two I/O paths when reading the metadata of an object, corresponding to the local file system and the key-value storage system respectively.

The distributed object storage system maintains an index table for each bucket, which stores the mapping relationship between the bucket and the metadata of all objects in the bucket. When a user accesses an object in a bucket, the specific data of the object is obtained through the bucket index. , and when the number of objects stored in the bucket is too large, oversized indexes will cause performance and reliability problems. In order to solve the limitation of the size of a single bucket object, the existing technology uses a dynamic sharding mechanism, that is, multiple RADOS objects are used to maintain the bucket index, and multiple index tables are created to solve the problem that a single index object may be too large and cannot meet the data growth. and other business needs. However, when bucket sharding is performed, the object metadata in the bucket will be rebalanced, and the index data in the old RADOS object will be recalculated, organized and migrated to the new RADOS object. Then, if a large number of objects are stored in the bucket, the data rebalancing operation during sharding will be very time-consuming, blocking front-end writes, and giving users a very unfriendly experience.

In response to the above problems, the metadata storage method based on the distributed key-value database of the present invention is described below with reference to Figure 1. This method is designed to solve the problems caused by the limited single-bucket object storage scale and the fragmentation mechanism in native distributed object storage. To solve the problem of business blocking, this method is applied to the server side. The server side is installed with a distributed object storage system. The method includes:

S10. Determine the metadata of the objects stored in the bucket, and determine the preset type (type) and key-value pairs of the metadata. Specifically, determine the metadata stored in each bucket on the server, as well as the preset type of metadata and key-value pairs.

In this application, the preset types include first, second, third and fourth types. Among them, the first type of metadata is basic metadata (object metadata), basic metadata saves the basic information of the latest version of the object, including size, instance and last version; the second type of metadata is object attribute metadata (xattrs), which is the attribute information of the object itself; the third type of metadata Metadata is index metadata (omap), which stores bucket object index information, which is omap information; the fourth type of metadata is index sequence metadata (omap order), the index order element stores the sequence list of all omaps of the objects in the bucket.

S20. Based on the determined preset type and key-value pair of the metadata, save the metadata into the corresponding hash table (hash table) and ordered list (zset). In this application, each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list. Specifically, in this application, the hash table is used to store metadata, and the ordered list is used to store metadata. Stores retrieval information for metadata.

In this application, a single-bucket multi-table metadata storage model is constructed based on the data structure of distributed key-value data. The native distributed object storage system only includes object gateway and back-end data storage. On this basis, the present invention adds A proprietary distributed key-value module is used to store object metadata and realize separate storage of object data and metadata. At the same time, this application reorganizes the object's metadata, removes redundant data types, and reclassifies the metadata into four types of metadata. Since dozens of types of metadata in the native distributed object storage system are not simultaneously operated during object operations, the object metadata granularity can be reduced, and metadata read and write operations are more flexible and efficient.

For the four preset types of metadata in the same bucket, this method will use at least one hash table and at least one ordered list for storage, and when storing metadata, it will also ensure that it is inside the container group (container). It is evenly distributed. How to achieve uniform distribution of metadata in the container group will be explained below.

In this application, the metadata of the object is stored in the distributed key-value cluster in the form of key-value pairs. This method uses a table partitioning method and uses multiple hash tables to carry the metadata of the object in a bucket. Each A hash table can store up to 4.2 billion key-value pairs. Multiple hash tables can easily support the metadata of tens of billions of objects by uniformly storing the metadata of objects in a distributed key-value hash table. , the time complexity of its read and write operations is 0 or 1, which reduces the complexity of I/O operations and increases the scale of single-bucket storage while ensuring efficiency. There is no need to load RADOS object data from the local file system.

The metadata storage method based on a distributed key-value database provided by this application stores the object's metadata in the form of key-value pairs in a distributed key-value cluster, and uses multiple tables in a split-table manner. Hash tables and multiple ordered lists are used to carry the metadata of objects in a bucket. The storage of metadata through hash tables reduces the complexity of I/O operations and increases the storage scale of a single bucket while ensuring efficiency. Sequence lists store metadata and provide an interface for adding, deleting, modifying, and querying object metadata. Without increasing storage costs, it achieves high efficiency and low space utilization suitable for metadata retrieval across multiple ordered lists.

The metadata storage method based on the distributed key-value database of the present invention will be described below with reference to Figure 2. Step S20 specifically includes:

S21. Based on the determined preset type, store the metadata in the container group corresponding to the preset type. In this application, each preset type of metadata corresponds to a container group, and each container group corresponds to at least one hash slot (hash slot). slot), and the number of hash slots corresponding to the container group is equal to each other, that is, the number of hash slots corresponding to each container group remains consistent.

S22. Based on the key-value pair of the metadata, determine the CRC16 value of the metadata, and determine the hash slot corresponding to the metadata based on the total number of hash slots and the determined CRC16 value.

Specifically, first calculate the CRC16 value of the metadata based on the key of the metadata, and then take the modulo of the total number of hash slots to obtain the hash slot corresponding to the metadata. slot).

S23. Map and store the metadata to the hash table corresponding to the determined hash slot. In this method, each hash table corresponds to at least one hash slot.

In order to ensure that metadata is evenly distributed in each hash table of the container group, in this application, each hash table needs to be allocated a slot partition in advance. For each container group, the total number of slots in the entire container group is 16384. We configure the number of hash tables to exceed the preset number (for example, 5) and be a factor of 16384. After that, these 16384 slots are ) are evenly distributed among the hash tables in the container group.

Therefore, the hash slot corresponding to the metadata can be determined in the following way:

HASH_SLOT = CRC16(key) mod 16384

The HASH_SLOT algorithm can be used to map the metadata of the bucket object to the corresponding hash table, so that the metadata can be evenly distributed in the corresponding container group.

S24. Assign a corresponding score value (score) to the metadata. Based on the assigned score value, determine the ordered list corresponding to the metadata, and store the metadata in the corresponding ordered list. In this method, the ordered list Each item of metadata in the list is sorted sequentially according to the score value. For example, based on the score value assigned/assigned by the metadata, the metadata in a single ordered list is sorted in an orderly manner according to the score value from low to high.

In this embodiment, a score value is assigned to the key of the metadata.

Compared with the process of uploading, downloading, deleting and other single object metadata operations, the ordered object list business is undoubtedly more complicated. In this application, multiple ordered lists are used to maintain the ordered metadata (omap key) of the objects in the bucket. , these omap keys are ordered within a single ordered list, but in order to evenly distribute metadata, the omap keys across multiple ordered lists cannot be arranged in an orderly manner. In order to solve the low efficiency of data retrieval problem, in the existing technology, metadata will be separately stored on high-performance disks, such as solid-state drives (Solid-state drives). State Disk, SSD), however, although the efficiency of reading and writing objects has improved, the required storage cost has also increased significantly.

In some possible embodiments, the method also includes the following steps, aiming to achieve high efficiency and low space utilization suitable for retrieval of metadata across multiple ordered lists without increasing storage costs:

A10. Determine the client's metadata retrieval request. In these embodiments, the metadata retrieval request contains retrieval information, and the retrieval information includes the index order and number of index elements of each ordered list, etc.

In this embodiment, the index order and number of index elements of different ordered lists are consistent. For example, the metadata retrieval request is to retrieve the top 10 elements in each ordered list, that is, the index order is based on the highest order. The front-end elements start to be retrieved in a continuous sequence, and the number of index elements is 10.

A20. Based on the retrieval information, retrieve the corresponding metadata from each ordered list. For example, take the consecutive n elements (metadata) from the frontmost, lastmost, i-th to i+n-1th elements from each ordered list.

A30. Store the metadata in the preset map table, sort the metadata stored in the preset map table based on the preset map table, and return the preset map table that has completed metadata storage and sorting to the client.

Since the map table itself can sort elements based on their letters, when metadata is stored in the preset map table, the preset map table can sort the stored metadata sequentially. The preset map table is a list in which the retrieved metadata is sorted again. Based on the preset map table, users can perform high-efficiency, low space utilization metadata retrieval across multiple ordered lists.

In some possible embodiments, the method further includes the following steps:

A40. Determine the client's metadata retrieval request. Similarly, similar to step A10, in these embodiments, the metadata retrieval request also contains retrieval information. The difference is that the retrieval information includes reference metadata and the number of index elements. The reference metadata Data is start key.

A50. Determine the storage location of the baseline metadata in each ordered list.

It should be noted that the baseline metadata (start key) is metadata that is definitely present in the bucket. It can be understood that the baseline metadata is and will only be stored in one of the ordered lists. Therefore, in step A50, the specific storage location of the benchmark metadata in an ordered list corresponding to the storage will be obtained, which is the storage sequence, that is, the storage element number/serial number in the ordered list; for those that do not store benchmark metadata In ordered lists, the storage location of the baseline metadata stored in these ordered lists will first be determined, that is, the storage order in the ordered lists. For example, based on the letters of the benchmark metadata and the letters of the metadata that have been stored in each other ordered list, the pre-storage position of the benchmark metadata in the ordered list where the benchmark metadata is not stored is determined.

A60. Based on the retrieval information and storage location. The storage location includes the real storage location and the pre-storage location, and retrieve the corresponding metadata from each ordered list. For example, starting from the pre-storage location, fetch consecutive n elements (metadata) from an ordered list of unstored baseline metadata.

A70. Store the metadata in the preset map table, sort the metadata stored in the preset map table based on the preset map table, and return the preset map table that has completed metadata storage and sorting to the client.

The default map table has a storage limit, which can be set by the user. For example, the storage limit is N elements. When the storage limit of the default map table is not reached and metadata needs to be stored, the metadata will be processed directly. Storage. When the preset map expression reaches the upper limit of storage, a corresponding retrieval algorithm needs to be designed. Therefore, steps A30 and A70 will also include:

A80. Determine that the preset map expression reaches the storage upper limit and there is unstored metadata. For example, the storage upper limit N is less than the total number of metadata taken out S. Based on the preset map table, determine the unstored metadata and the preset map table. In this embodiment, the sorting value can be understood as the storage location/storage order.

A90. Determine that the map table sorting value corresponding to the last/most metadata exceeds the map table sorting value corresponding to the unstored metadata, delete the last metadata, and store the unstored metadata to the preset Set up a map table. Use the above method to determine whether each unstored metadata needs to be stored in the default map table. It should be noted that when the last element is deleted and the new metadata is stored in the default map table, the default All stored metadata in the map table are reordered according to the map table sorting value, ensuring that the map table sorting value corresponding to the metadata at the end of the default map table is the maximum value among the map table sorting values of all elements.

The metadata storage device based on the distributed key database provided by the present invention is described below. The metadata storage device based on the distributed key database described below can be used with the metadata storage method based on the distributed key database described above. mutual reference.

The metadata storage device based on the distributed key-value database of the present invention is described below with reference to Figure 3. This device is designed to solve the problems of limited single-bucket object storage scale and business blocking caused by the sharding mechanism in native distributed object storage. , the device is applied to the server, and the server is installed with a distributed object storage system. The device includes:

The first determination module 10 is used to determine the metadata of the objects stored in the bucket, and determine the preset type (type) and key-value pairs of the metadata. Specifically, determine the metadata stored in each bucket on the server, as well as the preset type of metadata and key-value pairs.

The data storage module 20 is used to save the metadata to the corresponding hash table (hash table) based on the determined preset type and key-value pair of the metadata. table) and ordered list (zset). In this application, each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list. Specifically, in this application, the hash table is used to store metadata, and the ordered list is used to store metadata. Stores retrieval information for metadata.

For the four preset types of metadata in the same bucket, the device will use at least one hash table and at least one ordered list to store, and when storing metadata, it will also ensure that it is within the container group (container). It is evenly distributed. How to achieve uniform distribution of metadata in the container group will be explained below.

In this application, the metadata of the object is stored in the distributed key-value cluster in the form of key-value pairs. The device adopts a sub-table method and uses multiple hash tables to carry the metadata of the object in a bucket. Each A hash table can store up to 4.2 billion key-value pairs. Multiple hash tables can easily support the metadata of tens of billions of objects by uniformly storing the metadata of objects in a distributed key-value hash table. , the time complexity of its read and write operations is 0 or 1, which reduces the complexity of I/O operations and increases the scale of single-bucket storage while ensuring efficiency. There is no need to load RADOS object data from the local file system.

The metadata storage device based on a distributed key-value database provided by this application stores the object's metadata in the form of key-value pairs in a distributed key-value cluster, and uses multiple tables in a split-table manner. Hash tables and multiple ordered lists are used to carry the metadata of objects in a bucket. The storage of metadata through hash tables reduces the complexity of I/O operations and increases the storage scale of a single bucket while ensuring efficiency. Sequence lists store metadata and provide an interface for adding, deleting, modifying, and querying object metadata. Without increasing storage costs, it achieves high efficiency and low space utilization suitable for metadata retrieval across multiple ordered lists.

Figure 4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 4, the electronic device may include: a processor (processor) 810, a communications interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 complete communication with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a metadata storage method based on a distributed key-value database, which method includes:

In addition, the above-mentioned logical instructions in the memory 830 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Various media that can store program code, such as Memory), magnetic disks or optical disks.

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Execute the metadata storage method based on the distributed key-value database provided by each of the above methods, which method includes:

In another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by the processor to execute the elements based on the distributed key-value database provided by the above methods. Data storage method, which includes:

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disc, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be used Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A metadata storage method based on a distributed key-value database, characterized in that the method is applied to a server, and the server is installed with a distributed object storage system. The method includes:

Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata;

Based on the determined preset type and key-value pair of the metadata, the metadata is saved into the corresponding hash table and ordered list; each bucket in the distributed storage system passes through at least one hash The hash table and at least one ordered list store metadata, the hash table is used to store metadata, and the ordered list is used to store retrieval information of metadata.
The metadata storage method based on a distributed key-value database according to claim 1, characterized in that the preset types include first, second, third and fourth types, and the first type of metadata is basic Metadata, the second type of metadata is object attribute metadata, the third type of metadata is index metadata, and the fourth type of metadata is index sequence metadata;

The step of saving the metadata into the corresponding hash table and ordered list based on the determined preset type and key-value pair of the metadata specifically includes:

Based on the determined preset type, the metadata is stored in a container group corresponding to the preset type; each preset type of metadata corresponds to one of the container groups, and each of the container groups corresponds to at least One hash slot, and the number of hash slots corresponding to the container group is equal to each other;

Based on the key-value pair of the metadata, determine the CRC16 value of the metadata, and determine the hash slot corresponding to the metadata based on the total number of hash slots and the determined CRC16 value;

Map and store the metadata to the hash table corresponding to the determined hash slot; each hash table corresponds to at least one hash slot;

Assign a corresponding score value to the metadata, determine the ordered list corresponding to the metadata based on the assigned score value, and store the metadata in the corresponding ordered list; The metadata of each item in the ordered list is sorted in order according to the score value.
The metadata storage method based on a distributed key-value database according to claim 2, wherein the number of hash tables is determined based on a preset number and the total number of hash slots. The number of tables exceeds the preset number and is a factor of the total number.
The metadata storage method based on a distributed key-value database according to claim 1, characterized in that the method further includes the following steps:

Determine the client's metadata retrieval request; the metadata retrieval request includes retrieval information, and the retrieval information includes the index order and number of index elements of each of the ordered lists;

Based on the retrieval information, retrieve the corresponding metadata from each of the ordered lists;

The metadata is stored in a preset map table, and based on the preset map table, the metadata stored in the preset map table is sequentially sorted, and all steps required to complete the storage and sorting of the metadata are The above preset map table is returned to the client.
The metadata storage method based on a distributed key-value database according to claim 1, characterized in that the method further includes the following steps:

Determine the client's metadata retrieval request; the metadata retrieval request includes retrieval information, and the retrieval information includes reference metadata and the number of index elements;

Determine the storage location of the baseline metadata in each of the ordered lists;

Based on the retrieval information and the storage location, retrieve the corresponding metadata from each of the ordered lists;

The metadata is stored in a preset map table, and based on the preset map table, the metadata stored in the preset map table is sequentially sorted, and all steps required to complete the storage and sorting of the metadata are The above preset map table is returned to the client.
The metadata storage method based on a distributed key-value database according to claim 4 or 5, characterized in that the metadata is stored in a preset map table, and based on the preset map table, the metadata is stored in a preset map table. The metadata stored in the preset map table is sequentially sorted, and the preset map table that completes the storage and sorting of the metadata is returned to the client, which also includes:

It is determined that the preset map expression has reached the storage upper limit and there is unstored metadata. Based on the preset map table, it is determined that the unstored metadata and the last element in the preset map table are The map table sorting value of the data;

Determine that the map table sorting value corresponding to the tailmost metadata exceeds the map table sorting value corresponding to the unstored metadata, delete the tailmost metadata, and store the unstored metadata. The data is stored in the preset map table.
A metadata storage device based on a distributed key-value database, characterized in that the device is applied to a server, and the server is installed with a distributed object storage system. The device includes:

The first determination module is used to determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pair of the metadata;

A data storage module, configured to save the metadata into the corresponding hash table and ordered list based on the determined preset type and key-value pair of the metadata; each in the distributed storage system Each bucket stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store retrieval information of metadata.
An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the program, it implements claims 1 to 6 The steps of any one of the metadata storage methods based on distributed key-value database.
A non-transitory computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, it implements the distributed key-value database based on any one of claims 1 to 6. Metadata storage method steps.
A computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the metadata storage method based on a distributed key-value database according to any one of claims 1 to 6 are implemented.