WO2022267508A1

WO2022267508A1 - Metadata compression method and apparatus

Info

Publication number: WO2022267508A1
Application number: PCT/CN2022/077759
Authority: WO
Inventors: 高蒙; 潘浩; 宋雨恒
Original assignee: 华为技术有限公司
Priority date: 2021-06-25
Filing date: 2022-02-24
Publication date: 2022-12-29

Abstract

The present application provides a metadata compression method and apparatus, related to the field of storage and used for reducing storage resources occupied by metadata. The method comprises: acquiring n pieces of metadata, where n is a positive integer greater than 1. One piece of metadata comprises a key-value pair, the key-value pair comprising a keyword and a value. The keyword is used for indicating an identifier of data corresponding to the metadata, and the value is used for indicating an actual address where the data is stored. Then, m pieces of data corresponding to at least some of the n pieces of metadata are processed to obtain n target values corresponding to the n pieces of metadata which conform to a set pattern, where m is a positive integer less than or equal to n. The n target values are then compressed.

Description

Metadata compression method and device

This application is required to be submitted to the State Intellectual Property Office on June 25, 2021, the application number is 202110710701.4, and the application name is "Metadata Index Processing Method" and submitted to the State Intellectual Property Office on August 17, 2021, the application number is 202110944078.9. The priority of the Chinese patent application entitled "Metadata Compression Method and Device", the entire content of which is incorporated in this application by reference.

technical field

The present application relates to the field of storage, in particular to a metadata compression method and device.

Background technique

At present, it is necessary to store metadata recording addresses of user data in the storage system, so as to access the user data stored in the storage system according to the metadata. As the amount of data in the storage system increases, more storage resources are required to store metadata.

Therefore, how to reduce the storage resources occupied by metadata is a problem that needs to be solved at present.

Contents of the invention

The present application provides a method and device for compressing metadata, which solves the problem that metadata occupies more storage resources.

In order to achieve the above object, the application adopts the following technical solutions:

In a first aspect, the present application provides a metadata compression method, which includes: acquiring n pieces of metadata, where n is a positive integer greater than 1. Wherein, a piece of metadata includes a key-value pair, and the key-value pair includes a keyword and a value. Wherein, the keyword is used to indicate the identification of the data corresponding to the metadata, and the value is used to indicate the actual address of the data storage. Afterwards, m data corresponding to at least part of the n metadata are processed to obtain n target values corresponding to the n metadata conforming to the set rules, where m is a positive integer less than or equal to n. Then, compress the n target values.

Wherein, the n pieces of metadata in this application may specifically be n pieces of metadata stored in the storage system. For example, in a scenario where metadata is stored in a tree-structured data structure, the n pieces of metadata may include n pieces of metadata in one or more nodes in the tree structure. Specifically, in the scenario where the data structure of the LSM tree is used to organize and store metadata, the n pieces of metadata may include n pieces of metadata in one or more ordered string tables (SStable) in a storage layer in the LSM tree. In addition, there is no limit to the processing method of m data in this application. For example, processing m data may include migrating m data to change the actual address of m data, and then obtain n data that conforms to the set rule. n target values corresponding to the metadata; for another example, data may not be migrated to make the actual address of the data appear regular, and then n target values corresponding to n metadata that conform to the set rule can be obtained. In addition, the present application does not have to limit the law that the n target values conform to, as long as the target value can be compressed according to the law. For example, the above n target values conforming to the set rule may refer to the n actual addresses indicated by the n target values as continuous actual addresses; for another example, the above n target values conforming to the set rule may refer to n Among the n actual addresses indicated by the target value, there is a storage space of the same size between each adjacent two actual addresses; for another example, the above n target values conforming to the set rule can refer to the n indicated by the n target values. The size of the storage space between every two adjacent real addresses in one real address changes regularly and so on.

In the above-mentioned method of the present application, by processing the data corresponding to the m metadata in the n metadata, and then obtaining n target values corresponding to the n metadata that conform to the set rule, in this way, according to the above design According to a certain rule, the n target values are compressed. Therefore, the problem that the value in the metadata cannot be compressed due to irregularity is solved, thereby achieving the effect of reducing storage resources occupied by the metadata.

In an implementation manner, the n actual addresses indicated by the above n target values conforming to the set rule are continuous.

In the above implementation manner, by making the n actual addresses indicated by the n target values continuous, it is convenient to compress the target values in the metadata. For example, when compressing n target values, it is possible to record n target values by recording the first target value as the start bit among the n target values and the offset between other target values and the first target value. effect, so as to realize the compression of n target values.

In an implementation manner, the above-mentioned m data corresponding to at least part of the metadata are processed to obtain n target values corresponding to the n metadata conforming to the set rules, including: migrating the m data to The data corresponding to the n pieces of metadata is stored in a storage space with continuous actual addresses. Afterwards, the actual addresses where n data are stored in the continuous storage space are saved as n target values.

In the above implementation, the method of migrating m data is adopted, so that the data corresponding to n metadata is stored in a storage space with continuous actual addresses, and then the n data is stored in the continuous storage space. The actual address is stored as n target values, so that the n target values can be made regular (that is, conform to the set rule), thereby facilitating the compression of the n target values.

In an implementation manner, the method further includes: selecting n pieces of metadata from the pieces of metadata according to the hotness and coldness of the data corresponding to the pieces of metadata included in the metadata set. The data corresponding to the n pieces of metadata is cold data.

In the above implementation, considering that the hot data in the storage system is likely to be modified, the possibility of changing the value of the metadata of the hot data is high. Compressing the metadata of hot data in one way will lead to low compression efficiency. Especially in the scenario where user data is written by means of additional write or redirect-on-write (ROW), each time the user data is modified, the modified data content of the user data will be Deposit a new physical address, so the above problem will be more obvious. For example, in the scenario where user data is written in append or ROW mode, it may happen that the actual address of the user data is just Because the user data is modified and changed, this is equivalent to the process of compressing the value in the metadata of the user data, which is meaningless. Therefore, select n pieces of metadata corresponding to cold data from the multiple pieces of metadata contained in the metadata set through the above implementation method, and then compress the n target values of the n pieces of metadata according to the above method, so that One is to only process and compress the values corresponding to the cold data, and avoid processing and compressing the values corresponding to the hot data, thereby improving the efficiency of metadata compression. Wherein, the metadata set may be any set including multiple metadata. For example, in the case where the above n pieces of metadata are metadata in the first storage layer (which can be any storage layer) in the LSM tree, the metadata set can be , a set of metadata included in the storage layer above the first storage layer. For another example, in the case where the above n pieces of metadata are metadata in the first storage layer in the LSM tree, the metadata collection can be a collection of multiple metadata in the first storage layer. In this application, the metadata collection is in The range in practical application may not be limited.

In an implementation manner, the n pieces of metadata are metadata in the first storage layer in the LSM tree. Wherein, the LSM tree is used to store metadata, and the LSM tree includes multiple storage layers, and the multiple storage layers include the above-mentioned first storage layer.

In the above implementation manner, the metadata compression method provided in the present application can be applied to any storage layer in the LSM tree, so as to achieve the effect of compressing the metadata value in the storage layer.

In an implementation manner, the above key and value are stored in two data entries respectively.

In the above implementation, by storing the keyword and the value in two data entries, the keyword data entry may not be affected when the data entry storing the value (called the value data entry) is modified. In this way, after compressing the value in the metadata to obtain the compressed value, you only need to update the compressed value to the value data entry, and it will not affect the keyword data entry, that is, no need to update the keyword data entry Processing, thereby improving the compression efficiency of metadata.

In an implementation manner, the method further includes: detecting a data change amount of the metadata set; the metadata set is used to record metadata of multiple pieces of data. Acquiring n pieces of metadata includes: after determining that the amount of data change in the metadata set exceeds a change threshold, acquiring n pieces of metadata included in the metadata set.

In the above implementation manner, it is considered that with the operation of the storage system, even if the values in the metadata have been compressed before, new metadata will be continuously stored in the metadata collection afterwards. Therefore, in the above implementation, the data change amount of the metadata set is detected, and after it is determined that the data change amount exceeds the change threshold, the acquisition of n pieces of metadata in the metadata set is triggered, so that the n pieces of metadata can be analyzed according to the method of the present application. This way of compressing the value can achieve the effect of compressing the metadata in the metadata set after new metadata is stored in the metadata set, so as to reduce the storage resources occupied by the metadata set. Wherein, the metadata set refers to any set including multiple metadata.

In an implementation manner, the method further includes: acquiring a degree of dispersion of actual addresses of data corresponding to the n pieces of metadata. In addition, the above-mentioned processing of m data corresponding to at least part of the metadata in the n metadata includes: after determining that the degree of dispersion is greater than the discrete threshold, processing the m data corresponding to at least part of the metadata in the n metadata to process.

In the above implementation, considering that when the discreteness of the actual addresses of the data corresponding to the n pieces of metadata is small, it means that the n actual addresses themselves have certain regularity, so the data is not processed at this time, and the Values in metadata are compressed to some extent. Therefore, in the above implementation manner, when it is determined that the discreteness of the actual addresses of the data corresponding to the n metadata is sufficiently large, then the m data corresponding to at least part of the n metadata are processed. That is to say, when the degree of discreteness is relatively small, the value of the metadata may be directly compressed instead of processing the data. In this way, the amount of data processing in the metadata compression process is reduced, write amplification is reduced, and compression efficiency is improved.

In one implementation, the method can be applied to a centralized storage system. Specifically, the method can be executed by an engine in the centralized storage system.

In an implementation manner, the method can be applied to a distributed storage system. Specifically, the distributed storage system includes multiple storage servers, and the foregoing method may be executed by one or more storage servers among the multiple storage servers.

In a second aspect, the present application provides a metadata compression device. The metadata compression device may be a hardware device for managing metadata in a storage system. For example, the metadata compression device may be an engine in a centralized storage system or a part of hardware devices in an engine, or the metadata compression device may be a storage server in a distributed storage system or a part of hardware devices in a storage server. Specifically, the metadata compression device may include: an acquisition unit, configured to acquire n pieces of metadata, one piece of metadata includes a key-value pair, the key-value pair includes a keyword and a value, and the keyword is used to indicate The identifier of the data corresponding to the metadata, the value is used to indicate the actual address of the data storage, and the n is a positive integer greater than 1. A processing unit, configured to process m data corresponding to at least some of the n metadata, to obtain n target values corresponding to the n metadata conforming to a set rule, where m is less than or equal to n positive integer of . A compression unit, configured to compress the n target values.

In an implementation manner, the processing unit is configured to process the m data corresponding to the at least part of the metadata, and obtain the n target values corresponding to the n metadata conforming to the set rules, including: The processing unit is specifically configured to migrate the m pieces of data, so as to store the data corresponding to the n pieces of metadata in a storage space with continuous actual addresses. The processing unit is specifically configured to save the actual addresses of the n data stored in the continuous storage space as the n target values.

In an implementation manner, the processing unit is further configured to select the n pieces of metadata from the multiple pieces of metadata according to the hotness and coldness of the data corresponding to the multiple pieces of metadata, and the n pieces of metadata The corresponding data is cold data.

In an implementation manner, the processing unit is further configured to detect a data change amount of a metadata set; the metadata set is used to record metadata of multiple pieces of data. The acquisition unit is configured to acquire n pieces of metadata, including: the acquisition unit is specifically configured to acquire the metadata included in the metadata set after determining that the amount of data change in the metadata set exceeds a change threshold. n metadata.

In an implementation manner, the obtaining unit is further configured to obtain a degree of dispersion of actual addresses of data corresponding to the n pieces of metadata. The processing unit is configured to process m pieces of data corresponding to at least part of the metadata in the n pieces of metadata, including: the processing unit is specifically configured to, after determining that the degree of dispersion is greater than a dispersion threshold, process m data corresponding to at least part of the n metadata are processed.

In one implementation, the metadata compression device is located in an engine in the centralized storage system.

In an implementation manner, the metadata compression device is located in a storage server in a distributed storage system.

In a third aspect, a storage device is provided, including: a memory and a processor, the memory is used to store computer instructions, and the processor is used to call and execute computer instructions from the memory, so as to realize the first aspect or the implementations in the first aspect The method provided by any of the methods.

In a fourth aspect, a storage system is provided, including an engine and a plurality of hard disks, the plurality of hard disks are used to store data, and the engine is used to execute the method provided in any one of the above-mentioned first aspect or each implementation manner of the first aspect . Specifically, the storage system may be a centralized storage system.

In a fifth aspect, a storage system is provided, including a plurality of storage servers, the plurality of storage servers are used to store data, and the first server among the plurality of storage servers is used to perform the above-mentioned first aspect or in each implementation manner of the first aspect Either of the methods provided. Specifically, the storage system may be a distributed storage system. Wherein, the first server may be a storage server capable of managing metadata in the distributed storage system.

In a sixth aspect, there is provided a chip, including a memory and a processor, the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory, so as to implement the first aspect or The method provided by any one of the implementations in the first aspect.

In the seventh aspect, there is provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned first aspect or each implementation manner in the first aspect can be realized. method provided by the item.

In an eighth aspect, there is provided a computer program product, the computer program product includes instructions, and when the instructions are run on a processor, the above-mentioned first aspect or any one of the implementations in the first aspect is implemented. method.

The above-mentioned beneficial effects of the second aspect to the eighth aspect can participate in the beneficial effects of the first aspect and each implementation manner in the first aspect, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic structural diagram of a storage system provided by the present application;

FIG. 2 is a schematic flow chart of writing data to a storage system provided by the present application;

FIG. 3 is a schematic flow diagram of a metadata compression method provided by the present application;

FIG. 4A is _one of the flow diagrams for merging metadata from layer L1 to layer L2 in the LSM tree provided by the present application _;

FIG. 4B is the _second schematic flow diagram _of merging metadata from L1 layer to L2 layer in the LSM tree provided by the present application;

FIG. 5A is one of the schematic diagrams for data migration provided by the present application;

Figure 5B is a second schematic diagram of data migration provided by this application;

Figure 6A is the third schematic diagram of data migration provided by this application;

FIG. 6B is a fourth schematic diagram of data migration provided by this application;

FIG. 7 is one of the schematic structural diagrams of a keyword data entry and a value data entry provided by the present application;

Fig. 8 is the second structural diagram of a keyword data entry and a value data entry provided by the present application;

FIG. 9 is the third schematic diagram of the structure of a keyword data entry and a value data entry provided by this application;

FIG. 10 is one of the structural schematic diagrams of a metadata compression device provided by the present application;

FIG. 11 is the second structural schematic diagram of a metadata compression device provided by the present application.

detailed description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Among them, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. Those skilled in the art can understand that words such as "first" and "second" do not limit the number and execution order, and words such as "first" and "second" do not necessarily limit the difference. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. To be precise, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner for easy understanding.

In order to facilitate the understanding of the embodiments of the present application, firstly, the application scenarios of the technical solutions provided by the embodiments of the present application are introduced:

Exemplarily, FIG. 1 is a schematic diagram of a network architecture provided by an embodiment of the present application. In the application scenario shown in Figure 1, user data can be accessed by running an application program. Wherein, the computer running the application program may be referred to as an "application server". The application server 100 may be a physical machine or a virtual machine. The application server 100 includes, but is not limited to, desktop computers, servers, notebook computers, and mobile devices. The application server accesses the storage system 120 through the switch 110 to access user data. However, the switch 110 is only an optional device, and the application server 100 can also directly communicate with the storage system 120 through the network. Alternatively, the switch 110 can also be replaced with an Ethernet switch, an InfiniBand switch, a RoCE (RDMA over Converged Ethernet) switch, and the like.

Wherein, the storage system 120 is a device or a device cluster for storing user data. Specifically, in an actual application process, the storage system 120 may be a centralized storage system. A centralized storage system is characterized by a unified entrance, and all data from external devices such as application servers must pass through this entrance.

As shown in FIG. 1 , the entrance of the centralized storage system may specifically be the engine 121 of the centralized storage system. Wherein, the engine 121 may include one or more controllers, and one controller 122 is taken as an example in FIG. 1 for illustration. In addition, when there are multiple controllers in the engine 121, multiple controllers can be used as backups for each other through mirroring channels. When one of the controllers fails, other controllers can take over the business of the faulty controller, thereby Avoid hardware failures leading to the unavailability of the entire storage system.

In addition, the engine 121 may further include a front-end interface 125 and a back-end interface 126 , wherein the front-end interface 125 is used to communicate with the application server 100 to provide storage services for the application server 100 . The backend interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, the engine 121 can be connected with more hard disks 134, thereby forming a very large storage resource pool.

In addition, the controller 122 may include a processor 123 and a memory 124 . Processor 112 may be a central processing unit (central processing unit, CPU), used to process data access requests from outside the storage system (such as application servers or other storage systems), and also used to process requests generated inside the storage system. Exemplarily, when the CPU 123 receives the write data requests sent by the application server 100 through the front-end interface 125, it will temporarily store the user data in these write data requests in the memory 124. When the total amount of user data in the internal memory 124 reaches a certain threshold, the CPU 123 sends the user data stored in the internal memory 124 to the hard disk 134 for persistent storage through the back-end interface.

The memory 124 is an internal memory for directly exchanging data with the processor. It can read and write data at any time, and the reading and writing speed is fast. It can be used as a temporary data storage for an operating system or other running programs. The memory 124 may include various types of memory, for example, the memory may be a random access memory or a read-only memory (Read Only Memory, ROM). For example, the random access memory is Dynamic Random Access Memory (Dynamic Random Access Memory, DRAM) or Storage Class Memory (Storage Class Memory, SCM). DRAM is a semiconductor memory, which, like most Random Access Memory (RAM), is a volatile memory device. SCM is a composite storage technology that combines the characteristics of traditional storage devices and memory. SCM can provide faster read and write speeds than hard disks, but slower access speeds than DRAM and cheaper than DRAM. However, DRAM and SCM are exemplary illustrations in this embodiment, and the memory may also include other random access memories, such as Static Random Access Memory (Static Random Access Memory, SRAM) and the like. As for the read-only memory, for example, it may be a programmable read-only memory (Programmable Read Only Memory, PROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), and the like. In addition, the memory 124 can also be a dual in-line memory module or a dual in-line memory module (Dual In-line Memory Module, DIMM for short), that is, a module composed of DRAM, or a solid state disk (Solid State Disk, SSD) . In practical applications, multiple memories 124 and different types of memories 124 may be configured in the controller 0 . This embodiment does not limit the quantity and type of the memory 124 . In addition, the memory 124 can be configured to have a power saving function. The power saving function means that the data stored in the internal memory 124 will not be lost when the system is powered off and then powered on again. Memory with a power saving function is called non-volatile memory.

It should be noted that only one engine 121 is shown in FIG. 1 , but in practical applications, the storage system may include two or more engines 121 , and redundancy or load balancing is performed among the multiple engines 121 . In addition, in an implementation manner, the engine 121 may also include hard disk slots. In this case, the hard disk 134 can be directly deployed in the engine 121, and the back-end interface 126 is an optional configuration. When the storage space of the system is insufficient, you can More hard disks or hard disk enclosures are connected through the back-end interface 126 .

In addition, it should be noted that FIG. 1 only provides a schematic structural diagram of a centralized storage system as an example. In other application scenarios, the storage system 120 may be composed of multiple independent storage servers, where the storage servers may communicate with each other. Wherein, each storage server may respectively include hardware components such as a processor, a memory, a network card, and a hard disk. Among them, the processor and memory are used to provide computing resources; the processor is used to process data access requests from outside the storage server; the memory is used to directly exchange data with the processor's internal memory, which can read and write data at any time, and the speed is very fast. Can be used as temporary data storage for the operating system or other running programs. A hard disk is used to provide storage resources, such as storing data, and it can be a magnetic disk or other types of storage media, such as solid-state hard disks or shingled magnetic recording hard disks. In addition, the storage server may also include a network card for communicating with the application server.

During the operation of the storage system 120 , the actual address of the storage space provided by the hard disk 134 is generally not directly exposed to the application server 100 for use. Specifically, the storage system 120 stores metadata recording actual addresses of user data. Specifically, when the application server 100 writes data into the hard disk 134, the metadata of the user data is added to the metadata file to record the actual address of the data. When the application server 100 needs to read the user data stored in the hard disk 134 in the storage system 120, the actual address of the user data can be determined by searching the metadata of the user data in the metadata file recording the above metadata.

It should be noted that, in order to distinguish metadata from data described by metadata, the data described by metadata is referred to as "user data" in this embodiment of the application. The user data mentioned in the embodiment of this application can be understood as the data stored in the storage system provided by the application server to provide related services, and the metadata is the data used to describe these user data (data that describes other data), including but It is not limited to the actual address of the user data storage, the mapping relationship between the logical address and the actual address, the attribute of the user data and other information. In a specific implementation, user data may also be called "data" or other names, which may not be limited in this embodiment of the present application.

Exemplarily, in the scenario where the application server 100 reads user data from the storage system 120, the application server may send a read data request carrying the identifier of the user data to the storage system 120, where the identifier of the user data may be the application server 100 The logical address of this user data used in , etc. After the storage system 120 receives the read data request, the CPU 123 finds the actual address of the user data from the metadata file stored in the internal memory 124 or the hard disk 134 according to the identifier of the user data, wherein the actual address of the user data The address may be the physical address of the bottom layer of the user data in the storage system 120 or the logical address of the middle layer. Then CPU 123 reads the user data in the above-mentioned actual address of hard disk 134 through back-end interface 126, and feeds back to application server 100 through front-end interface 125.

Further, the mapping relationship between the identifier of the user data and the actual address of the user data is recorded in the metadata. Specifically, the mapping relationship may be stored in the form of a key-value pair (KV pair). As shown in Fig. 1, a metadata file including metadata is stored in the internal memory 124. In the metadata file, the identification of user data (identification 1-5 in the figure) can be used as the keyword (key) of the key-value pair , using the actual address of the data (addresses 1-5 in the figure) as the value of the key-value pair, so as to establish a mapping relationship between the identifier of the user data and the actual address of the data in the form of a key-value pair. It should be noted that the content of the key-value pairs in the metadata is only shown in the form of a list as an example in Figure 1, and the key-value pairs are also stored in other forms (such as using a tree structure) in practical applications. For the key-value pairs The storage form may not be limited in this application.

Among them, key-value pairs are stored as a representative of non-relational databases, which abandons the strict field structure of data tables in relational databases and the relationship restrictions between tables. The data stored in key-value correspondence adopts a simplified data model, so that key-value pair storage has the following advantages: First, high scalability, because there is no strict field structure of the data table and the relationship between tables, the key-value pair Distributed applications can be easily deployed on multiple servers, thereby improving the scalability of the entire system and making it more convenient and flexible. Second, mass storage and high throughput capacity to meet the needs of cloud computing. Key-value pair storage can well meet the flexible needs of users for scalability in the cloud computing environment. Therefore, key-value pair storage is increasingly becoming the mainstream storage method.

Further, in order to facilitate metadata management, structured forms such as binary search tree, balanced tree (B tree), B+ tree or structured merge tree (log structured merge tree, LSM tree) are usually used. data is stored. The following takes the LSM tree as an example to introduce the storage structure of metadata:

The LSM tree is one of the commonly used storage structures in a log-based database system, and the LSM tree is a multi-layer framework. In the scenario where the LSM tree is used to manage metadata, the LSM tree is mainly stored in memory. In some scenarios, the metadata in all or some nodes of the LSM tree can also be temporarily stored in the hard disk. When these nodes need to be read When the metadata in the node is copied to the memory.

Exemplarily, in a scenario where the storage system 120 uses the LSM tree framework to manage metadata. As shown in Figure 2, the hard disk 134 includes an ordered string table area 1341 for storing an ordered string table (sorted string table, SStable) and a data storage area 1342 for storing user data, wherein the ordered string table The table area 1341 and the data storage area 1342 are generally logically divided storage areas in the hard disk 134 .

When the storage system 120 receives a data write request for writing user data X into the storage system, as shown in FIG. Corresponding key-value pairs) are written into memory table (memtable) 1241 in order. In addition, what is not shown in the figure is that in practical applications, after receiving a data write request for data X, the storage system 120 can also record this write operation through a write-ahead logging (WAL) in the log file for failover.

The memory table 1241 is located in the memory 124 , and when the memory table 1241 exceeds a certain threshold, it will be frozen in the memory and switched to an immutable memory table (immutable memtable) 1242 . At this time, in order not to block the write operation, a new memory table will be regenerated in the memory of the storage system 120 to continue to provide services. Afterwards, the non-modifiable memory table 1242 will be written into the ordered string table area 1341 in batches. The ordered string table area is located on one or more hard disks 134 . Wherein, the ordered character string table area 1341 includes _a multi _- storage layer structure, such as the _L0 layer, L1 layer and L2 layer as shown in FIG. The larger the storage space, each storage layer may include one or more SSTables, and further one or more SSTables may be stored in the form of structural data. When the non-modifiable memory table 1242 is written into the ordered character string table area 1341, the non-modifiable memory table 1242 will first be written into the top-level storage layer, such as layer _L0 in FIG. 2 . When the amount of data in the L ₀ layer reaches the threshold, the SSTable in the L ₀ layer will be merged into the L ₁ layer, and when the data amount in the L ₁ layer reaches the threshold, the SSTable in the L ₁ layer will be merged (merge) to L2 layer, and so _on , so that old metadata can be continuously deleted and new data can be continuously written. In the above example, the LSM tree for storing metadata is introduced by taking the three-layer storage layer as an example. It can be understood that in practical applications, the LSM tree can be composed of more or fewer layers of storage layers. Do limit.

Continuing with the storage system 120 receiving a data write request for writing user data X into the storage system as an example, the storage system 120 first searches the metadata of the user data X in the memory table 1241; if found, then according to the actual address in the metadata , to access the data storage area 1342 . If there is no metadata of user data X in the memory table 1241, then search down in turn, specifically first look up the metadata of user data X in the non-modifiable memory table 1242, if it is determined that there is no user data X in the non-modifiable memory table 1242 metadata of user data X in L ₀ layer; if it is determined that there is no metadata of user data X in L ₀ layer, then search for metadata of user data X in L ₁ layer, and so on, Until the metadata of the user data X is found, then access the data storage area 1342 according to the actual address in the metadata.

It can be seen that in the above application scenarios, in order to successfully access the data in the storage system 120, it is necessary to establish a complete metadata index for storing and querying the metadata of each user data in the hard disk. The key in the key-value pair is the identifier of the user data, and the value in the key-value pair is the actual address of the user data storage. In the scenario where the storage system 120 does not have or does not enable the deduplication function, logical unit number (logical unit number ID, LUN ID), snapshot number (snap ID), logical block address (logical block address, LBA) can be used One or more of them are used as keywords to uniquely identify user data. If the storage system 120 has a file system, the key may also be one or more items of a version number (Version), a file name and an offset within the file, or a hash value of the file name and the offset within the file.

In this embodiment, in order to distinguish the address indicated by the value in the metadata from other addresses, the address corresponding to the user data identifier in the metadata is called the "actual address". Type produces restrictions. The identification and actual address of user data can be understood as a relative relationship. Compared with the actual address of user data, the identification of user data is closer to the upper application, and the actual address of user data is closer to the underlying hardware. . For example, the actual address of the user data can be the address in the logical chunk group (chunk group) corresponding to the user data in the LUN. For example, the actual address of the user data can include the chunk group ID of the chunk group where the user data is located and the user The offset of the data in the chunk group, or the actual address of the user data can be the address of the user data in the physical block (chunk) in the hard disk. For example, the actual address of the user data can include the physical chunk ID and the physical chunk of the physical chunk where the user data is located. The offset in the physical chunk. The actual address of the user data can also be the physical address where the user data is stored in the hard disk. Assuming that the hard disk is a solid-state hard disk, the physical address is the block ID and page ID where the user data is located in the solid-state hard disk. This application may not limit the specific form of the identification and actual address of the user data. For the convenience of description, this embodiment is described by taking the physical address of the user data as an example in which the value in the key-value pair is used.

Take the physical address (physical_address) as an example to construct a key-value pair. The size of the key-value pair depends on the LUN ID, LBA, version number, or the number of bytes in the physical address. In this case, the key-value pair The number of sections is generally 24 to 32 bytes. If calculated according to the data block size of 8K, the storage capacity occupied by metadata and the storage capacity occupied by user data account for about 0.3%.

As the capacity of the storage system increases, the amount of metadata in the storage system also increases, which means that more memory and hard disk resources are needed to save metadata, and due to the hot data in the storage system There are more changes, and more metadata is swapped in and out of memory.

In order to reduce the amount of metadata, in a possible design, a prefix compression method may be used to compress the keys in the key-value pairs. Specifically, considering that in the metadata index, the key part of multiple metadata (such as multiple metadata in a storage layer in an LSM tree or in one or more SSTables in a storage layer) usually has a common part (such as LUN ID or snap ID), then the common part of these keys can be extracted, and then the key part of each metadata only records the difference with other keys. This enables compression of metadata.

However, in the above design, only the key part of the key-value pair is compressed, and the value part of the key-value pair usually accounts for 40% to 50% of the data volume of the key-value pair. It can be seen that the above design can compress The amount of data is limited.

In view of the above-mentioned problem that metadata occupies too much resources in the storage system, this application considers that in related technologies, only the key field in the key-value pair is compressed, because the value field is generally the actual address of user data or Fingerprint, in which the actual address is related to the space allocated when user data is written, so the actual address usually has no regularity; in addition, the fingerprint of user data depends on the content of the user data itself, and it is difficult to have regularity, so it is difficult to determine the value Some content is compressed.

Therefore, the embodiment of the present application provides a technical solution for metadata compression. In this technical solution, when the value used to indicate the actual address of user data contained in multiple metadata does not have regularity, it can By processing the user data corresponding to the plurality of metadata, for example, the user data is migrated to multiple continuous actual addresses. Therefore, the value in the metadata of these user data has regularity, thereby facilitating the compression of the value.

Below in conjunction with the accompanying drawings, the technical solutions provided by the embodiments of the present application are described in detail:

Specifically, the embodiment of the present application provides a metadata compression method. In some scenarios, the method may be implemented by the engine 121 in the storage system 120 in FIG. 1 , specifically, by a controller in the engine 121 . In the controller, the central processing unit 123 invokes the program instructions in the memory for execution. The memory may be the memory 124 in FIG. 1 , or a cache located in the central processing unit 123 . In some other scenarios, the method may also be implemented by other hardware devices used to manage metadata in the storage system. For example, when the storage system is a distributed storage system, the method may be implemented by a storage server capable of managing metadata in the distributed storage system or by some hardware in the storage server.

Taking the running process of the engine 121 in the scenarios of FIG. 1 and FIG. 2 as an example, the metadata compression method is introduced below: as shown in FIG. 3 , in the process of writing data to the storage system, the method includes:

S301. Receive a data write request from the application server 100.

Wherein, the write data request carries user data requested to be written into the storage system.

S302. Write the user data into the memory 124 according to the data write request, and store metadata of the user data in the memory table 1241.

Among them, metadata includes key-value pairs. Wherein, the key of the key-value pair is used to indicate the identity of the user data, and the value is used to indicate the actual address of the user data storage. It should be noted that, for the sake of simplicity of description, the keywords in the key-value pairs included in the above-mentioned metadata are referred to as "keywords in the metadata"; the values in the key-value pairs included in the above-mentioned metadata , referred to as "values in metadata", unless otherwise specified, can be understood as above for "keywords in metadata" and "values in metadata", which will not be repeated below.

It should be noted that as the engine 121 continues to receive data write requests, more and more user data is stored in the memory 124. When the accumulated user data in the memory 124 reaches a certain threshold, the engine 121 will store the user data in the memory 124 The data is written into the hard disk 134 for persistent storage. The address where the user data is stored in the hard disk 134 is the actual address where the user data is stored in this embodiment. The write data request received by the engine 121 not only carries the user data, but also includes the logical address of the user data. The logical address is an address presented to the application server 100, and is used to enable the application server 100 to access the user data. After storing the user data, the engine 121 may use the logical address or the hash value corresponding to the logical address as a key in the metadata, and use the actual address of the user data stored in the hard disk 134 as the metadata In the value, save the corresponding relationship between the keyword and the value as a key-value pair.

S303. When the amount of metadata in the memory table 1241 exceeds the threshold, the engine 121 switches the content in the memory table 1241 to an unmodifiable memory table 1242 and generates a new memory table.

In the actual application process, switching the content in the memory table 1241 to the non-modifiable memory table 1242 may mean that the engine 121 modifies the attributes of the memory table 1241 so that the modified memory table (that is, the non-modifiable memory table) no longer receives new data.

S304 , when the amount of data included in the non-modifiable memory table 1242 exceeds the threshold, the engine 121 transfers the content in the non-modifiable memory table 1242 to the hard disk 134 .

Wherein, the threshold in S304 may be the same as or different from the threshold in S303.

Specifically, the engine 121 may transfer the content in the unmodifiable memory table 1242 to the top layer _L0 of the ordered string table area 1341 . Then, when the amount of data in the L ₀ layer exceeds the threshold, the metadata in the L ₀ layer is merged into the L ₁ layer.

S305. Merge the metadata into the metadata files in the L2 layer according to the hotness and _coldness _of the user data corresponding to the metadata in the L1 layer.

Wherein, the L2 layer includes _two metadata files, which are respectively used to store metadata corresponding to cold data and metadata corresponding to hot data. For ease of description, the metadata file used to store metadata corresponding to cold data is called a cold metadata file, and the metadata file used to store metadata corresponding to hot data is called a hot metadata file. _In the actual application process, the _two metadata files in the L2 layer may respectively include different ordered character string tables in the L2 layer. Wherein, each metadata file can be stored in a tree structure to facilitate data search.

In this embodiment, the hot or cold degree of user data can be understood as the possibility of the user data being modified (the possibility of being modified can be specifically reflected as the historical modification frequency or historical modification times of the user data and other parameters) high and low. The colder the user data is, the lower the possibility that the user data will be accessed; the hotter the user data is, the higher the possibility that the user data will be accessed. In addition, in this embodiment, cold data may be understood as user data whose possibility of being modified is lower than a certain threshold, and hot data may be understood as user data whose possibility of being modified is higher than a certain threshold.

Exemplarily, as shown in FIG. 4A, five ordered string tables may be included in the L1 layer, and each ordered string table stores metadata of different keyword ranges respectively, wherein the ordered string table ₁ stores key The metadata whose word range is key1-key20, for example, the metadata corresponding to key1, key3, key4, key7, key12, key15, key16, key18, key19, and key20 are currently stored in the ordered string table 1 in Figure 4A. String table 2 stores metadata whose key range is key21-key40, and ordered string table 3 stores metadata whose key range is key41-key60, and ordered string table 4 The metadata with the key range of key61-key80 is stored in , and the metadata with the key range of key81-key100 is stored in the ordered string table 5.

When merging the metadata in the L1 layer to the L2 layer, take the ordered string table ₁ as an example, and judge the degree of hotness and coldness of the corresponding user data for the metadata in the ordered string table 1 in turn _; When the user data is cold data, merge the metadata of the user data into the cold metadata file, for example, in Figure 4A, merge the metadata corresponding to key1, key3, key12 and key15 (that is, the key corresponding to the cold data) into the cold metadata file; when it is determined that the user data is hot data, the metadata of the user data is merged into the hot metadata file, for example, key4, key7, key16, key18, key19 and key20 (that is, the key corresponding to the hot data) are corresponding in Figure 4A The metadata of the file is merged into the hot metadata file. By analogy, the metadata in each ordered string table in layer _L1 can be _merged into layer L2.

It should be noted that, in FIG. 4A, the L1 layer includes ₅ ordered string tables and each ordered string table is used to store metadata of 20 keyword ranges as an example for illustration. In the actual application process, this example compares the number of ordered string tables included in each storage layer in the LSM tree, the keyword range corresponding to each ordered string table, and the number of metadata included in each ordered string table. The number is not limited.

Based on the example in FIG. 4A, as shown in FIG. 4B, S305 may specifically include:

_S3051 . When the amount _of data in the L1 layer exceeds the threshold size or the threshold number, trigger the _L1 layer to merge with the L2 layer.

S3052. For each ordered string table _of layer L1, traverse and read each keyword in the ordered string table in turn.

For example, when the ordered string table adopts the data structure of binary tree, the keywords of the leftmost path of the binary tree corresponding to each ordered string table can be traversed sequentially by post-order traversal and merge sorting, so as to realize traversal and read Each key in the sequence string table.

Wherein, S3053 is executed for each keyword.

S3053. Query the hotness and coldness of the user data corresponding to the keyword.

If the user data is cold data, execute S3054; if the user data is hot data, execute S3055.

In an implementation manner, the hot or cold degree of the user data may be judged according to the IO type corresponding to the user data. Specifically, considering that under normal circumstances, user data using sequential write IO is colder and hotter, and user data using random write IO is hotter. Therefore, user data using sequential write IO can be used as cold Data, the user data of random write IO will be used as hot data.

S3054. Merge the metadata of the cold data into the cold metadata file.

Wherein, in order to find metadata conveniently, the cold metadata file can be divided into multiple sub-files, wherein each sub-file can be an ordered string table, and each ordered string table stores metadata of different key ranges respectively. For example, as shown in Figure 4A, the cold metadata file in the L2 layer includes ₁₀ SSTables: ordered string table 6-ordered string table 15, these 10 SSTables are used to store key1-key10, Metadata for the key range of key11-key20...key99-key100.

S3055. Merge the metadata of the hot data into the hot metadata file.

Similar to the cold metadata file, the hot metadata file can also be divided into multiple subfiles, where each subfile can be an SSTable, and each SSTable stores metadata of different keyword ranges. Exemplarily, as shown in Figure 4A, the hot metadata file in the L2 layer includes ₁₀ SSTables: ordered string table 16-ordered string table 25, and these 10 SSTables are used to store hot metadata files respectively Metadata for key ranges of key1-key10, key11-key20...key99-key100.

Through the above process of S3051 _- _S3055 , the metadata in the L1 layer can finally be merged into different metadata files in the L2 layer respectively.

As mentioned above, since the actual address of user data is usually not regular and not easy to be compressed, it needs to occupy a relatively large storage space for storing metadata of user data. In order to reduce the storage space occupied by the metadata, in this embodiment, for the metadata in the cold metadata file, the user data corresponding to the metadata is processed, such as migrating the user data to change the actual storage of the user data. address, so that the actual addresses of multiple user data present a certain regularity, that is, the value in the metadata conforms to the set rule. In this way, the value in the metadata can be compressed according to the set rule, so as to achieve the effect of reducing the storage space occupied by the metadata. Specifically, in order to make the value in the metadata in the cold metadata file conform to the set rule, the method provided in this embodiment also includes:

S306. Traverse each sub-file in the cold metadata file in turn, and read values in the metadata included in the sub-file.

All or part of the steps in S307-S311 are respectively executed for the values in the metadata included in each sub-file.

It should be noted that, when only some subfiles in the cold metadata file need to be compressed, only this part of subfiles may be traversed, and subsequent steps may be performed on this part of subfiles.

Specifically, as shown in Figure 4A, the L2 layer may include ₁₀ sub-files: ordered string table 6-ordered string table 15, these 10 sub-files are used to store key1-key10, key11-key20...key99 respectively Keyword-scoped metadata for -key100. Furthermore, subsequent steps may be performed on the 10 subfiles to compress the 10 subfiles; or, subsequent steps may be performed on some of the 10 subfiles to compress the subfiles.

S307. Determine the address dispersion corresponding to the sub-file according to the value in the metadata included in the sub-file.

When the dispersion of addresses corresponding to the sub-file exceeds the dispersion threshold, execute S308; when the dispersion of addresses corresponding to the sub-file does not exceed the dispersion threshold, return to S306 to traverse the next sub-file.

Wherein, the address discrete degree corresponding to the sub-file can be understood as the discrete degree of the actual address of the user data corresponding to the metadata included in the sub-file.

For example, the sub-file includes metadata of n pieces of user data, and the n pieces of user data are respectively stored in n actual addresses, where n is a positive integer. Then, the higher the dispersion degree of the n actual addresses, the higher the dispersion degree of the address corresponding to the sub-file; the lower the dispersion degree of the n actual addresses, the lower the dispersion degree of the address corresponding to the sub-file.

Wherein, the degree of discreteness of the n actual addresses can be specifically reflected in that the n actual addresses can be reflected according to several laws. For example, in the first case, among the n actual addresses, m actual addresses are continuous, where m is a positive integer less than n, and the other n-m actual addresses are continuous, that is to say, it can follow two rules to reflect the n actual addresses; in the second case, among the n actual addresses, there are p actual addresses that are continuous, where p is a positive integer less than n, and there are q actual addresses that are continuous , where q is a positive integer less than n, and there are (n-p-q-1) actual addresses that are continuous, and there is another actual address that is not continuous with other actual addresses, that is to say, in the second case, it can The n actual addresses are reflected according to four rules. Then the address dispersion of the sub-file in the first case is smaller than the address dispersion of the sub-file in the second case.

It should be noted that here only the law of continuous addresses is used as an example, and there may be other rules in practical applications, such as the storage space of the same size between every two adjacent real addresses among n real addresses , and for another example, the size of the storage space between every two adjacent actual addresses among the n actual addresses changes regularly and so on.

S308. Migrate the user data corresponding to the metadata in the sub-file, so that the user data corresponding to the metadata in the sub-file is stored in a continuous storage space.

Specifically, in one implementation, the user data corresponding to the metadata of the sub-file can be stored in a continuous storage space by migrating the user data; in another implementation, the sub-file can also be The user data corresponding to the metadata of the file is segmented and stored in multiple blocks of continuous storage space. The following two implementation methods are introduced respectively:

In a first implementation manner, the above S308 may specifically include: migrating the user data corresponding to the metadata in the sub-file, so that the user data corresponding to the metadata in the sub-file is stored in a continuous storage space.

Wherein, in a possible design, the user data corresponding to the sub-file may be migrated to a continuous free storage space.

Exemplarily, if the sub-file includes metadata of 5 user data, as shown in FIG. 5A , the 5 user data are respectively stored in address 1, address 3, address 4, address 8 and address 10. Then, by reading the data in these 5 addresses respectively and migrating the data in these 5 addresses to the unused address 11-address 15 respectively, the storage spaces of these 5 user data are continuous.

In another possible design, part of the user data corresponding to the sub-file can be migrated to the continuous storage space of other user data corresponding to the sub-file, so that the user data corresponding to the metadata in the sub-file is stored in the in a contiguous storage space.

Exemplarily, if the sub-file includes metadata of 5 user data, as shown in FIG. 5B , the 5 user data are stored in address 1, address 3, address 4, address 8 and address 10 respectively. Then, by reading the data at address 8 and address 10 respectively, and migrating the data at address 8 and address 10 to address 2 and address 5 respectively, the storage spaces of these five user data are continuous.

In the second implementation manner, the above S308 may specifically include: migrating the user data corresponding to the metadata in the sub-file, so that the user data corresponding to the metadata of the sub-file is stored in segments into multiple consecutive storage spaces .

Similar to the first implementation, in the second implementation, two possible designs may also be included:

In the first possible design, the user data corresponding to the sub-files may be migrated to multiple consecutive segments of storage space.

Exemplarily, if the sub-file includes metadata of 10 user data, as shown in FIG. 16. Address 17 and Address 20. Then read the data in these 10 addresses respectively, and migrate the data in these 10 addresses to the unused address 21-address 25, address 32-address 36, so that the 10 user data The storage space of user data 1-user data 5 is continuous, and the storage space of user data 6-user data 10 is continuous.

In the second possible design, the user data corresponding to the subfile can be migrated to the continuous storage space of other user data corresponding to the subfile, so that the user data corresponding to the metadata of the subfile can be stored in segments at most in a contiguous storage space.

Exemplarily, if the sub-file includes metadata of 10 user data, as shown in FIG. 16. Address 17 and Address 20. By migrating user data 2-user data 5 to the storage space continuous with user data 1, and user data 7-user data 10 to the storage space continuous with user data 7, so that the users in these 10 user data The storage space of data 1-user data 5 is continuous, and the storage space of user data 6-user data 10 is continuous.

S309. Update the value in the metadata of the migrated user data in the subfile according to the actual address of the migrated user data.

For example, the migrated actual address may be used as a value in the metadata of the migrated user data, and updated in the metadata of the user data.

S310. Compress the values in the metadata included in the sub-file.

Compared with the related art, because the actual address of the user data is irregular, the actual address of the user data is usually stored as a value in the metadata. In this embodiment, because the user data is stored in consecutive actual addresses, it can be By summarizing the change law of continuous actual addresses, a compression algorithm reflecting this change law is generated, for example, the compression algorithm may be a binary first-order function. Exemplarily, the law of the actual address of the user data can be summarized by using machine learning to generate the binary first-order function.

In this way, in this embodiment, only the compression algorithm used for compression and the compression value corresponding to the actual address of the user data need to be persistently stored, and the actual address of the user data does not need to be stored. When the actual address of the user data needs to be read, by using the compression value corresponding to the actual address of the user data as an input of the compression algorithm, the actual address of the user data can be output by the compression algorithm.

Exemplarily, taking five user data as an example, after the user data is migrated so that the actual addresses of the five user data are continuous, the actual addresses of the five user data can be expressed as 0x0000000100000000, 0x0000000100000001, 0x0000000100000002, 0x0000000100000003, and 0x0000000001. Among them, "0x" in the actual address represents a hexadecimal number, "00000001" in the middle represents the disk ID, and the last 8 digits represent the physical block number in the disk. Through compression, the following information needs to be recorded in the metadata: start key (start key): 0000000100000000, number of data: 5, compression algorithm: 1 (indicating that the compression algorithm is a first-order function with a slope of 1), and each element Compressed values for values in data: 0, 1, 2, 3, 4.

In addition, for the values in the metadata included in the sub-files, a prefix compression method may also be used to compress the values in the metadata by extracting a common prefix. The embodiment of the present application may not limit the compression method adopted for the value in the metadata.

In an implementation manner, the method further includes:

S311. Compress keywords in the metadata included in the sub-file.

Specifically, a prefix compression method can be used to extract a common prefix (such as a volume number (LUN ID), a subnet access point identifier (snap ID), etc.) for keywords in metadata, and then compress the keywords. For another example, when the keywords in the multiple metadata in the subfile include offsets that change linearly or approximately linearly, a first-order function can also be used to represent the relationship between the offset and the keyword subscript, and the keyword subscript and the function can directly Calculate the offset part of the keyword, that is, only need to record the key coefficient and order in the function at this time, so as to realize the compression of the keyword.

In addition, in an implementation manner, the keywords and values in the metadata in this embodiment can be stored in two different logically or physically divided data entries, which can be referred to as keyword data entries and Value data entry. Wherein, the two different data entries may be two data entries that can respectively write data through independent write operations.

For example, taking the physical chunk in the hard disk in the storage system as an example, the above key data entry and value data entry can be understood as two different physical chunks. In this way, when the value data entry is modified (for example, after the value in the metadata is compressed, the compressed value of the value in the metadata is updated to the value data entry, that is, when the value data entry is modified) , has no effect on keyword data entries. Similarly, when the value data entry is written, the key data entry will not be affected.

It should be noted that, in the above example, the keyword data entry and the value data entry may be different physical chunks as an example to illustrate the two data entries to which the keyword data entry and the value data entry belong. In practical applications, the two data entries to which the keyword data entry and the value data entry belong may also be storage space units of other granularity, which may not be limited in this embodiment.

Exemplarily, before the keywords and values in the metadata included in the subfile are respectively compressed by S310 and S311, as shown in FIG. 7 , the keywords in the metadata included in the subfile (Key1 -Key10) and values (Value1-Value10 in the figure), are stored in the key data entry and the value data entry respectively.

After the keywords and values in the metadata included in the subfile are compressed by S310 and S311, the keyword data entry is used to store the compressed value of the keyword in the metadata in the subfile, and the value data entry is used to store The compressed value of the value in the metadata in the subfile.

Specifically, as shown in FIG. 8 , the keyword data entry and the value data entry respectively include a header part (header) and a content part (vlaue). Among them, the content part of the keyword data entry is used to record the compressed value of the keyword; the header part of the keyword data entry is used to record the n kinds of compression algorithms used to compress the keyword, and which keywords in the content part each compression algorithm applies to . Wherein, the content part in the keyword data entry can be organized according to a tree structure, such as balanced+tree (balanced+tree, B+tree), adaptive radix tree (the adaptive radix tree, ARtree), etc.

A value data entry includes a header part and a content part. Among them, the content part is used to record the compressed value of the actual address in the storage space with different serial numbers; the header part is used to record the m compression algorithms used to compress the actual address, and which storage spaces in the content part each compression algorithm is applicable to.

For example, in FIG. 8 , the content of the key data entry includes: key_1' to key_10', key_1' to key_10' are compressed values of keys key1-key10 of 10 user data. The header part of the keyword data entry records two compression algorithms: compression algorithm 1 and compression algorithm 2, and the range of keywords applicable to compression algorithm 1 and compression algorithm 2 respectively (that is, compression algorithm 1 corresponds to key1-key5 and compression algorithm 2 corresponds to key6 -key10).

The content part of the value data entry includes 10 storage spaces with serial numbers V1-V10, which respectively store the compressed values of the actual addresses of 10 user data. The header part of the value data entry records three compression algorithms: compression algorithm 13, compression algorithm 14, and compression algorithm 15, and the storage spaces (ie, V1-V3, V4-V7, and V8-V10) respectively applicable to the three compression algorithms.

Wherein, the compressed values of the keywords in the content part of the keyword data item respectively point to different storage spaces of the content part of the value data item. For example, in FIG. 8 , key_1' in the key data entry points to storage space V1 in the value data entry, key_2' points to storage space V2 in the value data entry, and so on.

Exemplarily, when accessing user data according to the identifier key1 of the user data: first look up the compression algorithm corresponding to key1 in the header part of the keyword data entry, as shown in Figure 8, the compression algorithm corresponding to key1 is compression algorithm 11; after that, According to the compression algorithm 11 and key1, the compressed value key_1' corresponding to key1 is obtained; then the storage space V1 in the value data entry is determined according to key_1'; then the value1_offset in the storage space V1 is read from the value data entry; and then by reading In the title part of the value data entry, it can be known that the compression algorithm corresponding to the storage space V1 is the compression algorithm 13; after that, according to the compression algorithm 13 and value1_offset, the actual address of the user data can be obtained, and then the data in the actual address can be read to complete the access User data.

In the above implementation manner, keywords and values in multiple metadata (for example, multiple metadata in a subfile in the above implementation manner) are stored separately. In this way, when the metadata value needs to be compressed, it is only necessary to update the compressed value of the metadata value obtained after compression to the original storage space in the corresponding value data entry, and there is no need to modify the key Word data entries are modified, which improves compression efficiency.

In another example, in the case where the keywords of the metadata do not need to be compressed, only the keywords in the metadata (that is, the uncompressed keywords) may be stored in the keyword data entry as shown in FIG. 9 ; On the other hand, the value data entry is still stored in a manner similar to the above design. In this way, when the metadata value needs to be recompressed, it is only necessary to update the compressed value of the metadata value obtained after recompression to the original storage space in the corresponding value data entry. Modifications are made to key data entries, which improves the effect of compression efficiency.

In addition, in an implementation manner, the method further includes:

S312. Periodically detect whether the data change amount of the cold metadata file exceeds a change threshold.

If the data change amount of the cold metadata file exceeds the change threshold, execute S306; if it does not exceed the change threshold, wait for the next cycle and re-execute S312.

Wherein, the data change amount of the cold metadata file may specifically refer to the change amount of user data corresponding to the metadata in the cold metadata file within a preset time period. In this embodiment, multiple pieces of metadata in a cold metadata file are referred to as a metadata set.

Exemplarily, the preset time period may be the period from the last processing of the user data corresponding to the metadata in the cold metadata file to the current moment; further exemplary, the preset time period may also be preset Set a fixed duration. The method for setting the length of the preset time period may be set according to actual needs, and this application may not limit it. In addition, the change amount of the user data in the cold metadata file may be the number of changes of the user data in the cold metadata file, or the data volume of the changed user data in the cold metadata file. In practical applications, technicians may use appropriate parameters to reflect the variation of user data in the cold metadata file according to actual requirements, and this application may not limit this.

In the above implementation, considering that during the operation of the storage system, the storage system will continuously receive new data write requests for writing new user data or modifying previous user data, so even cold metadata files Metadata in may also change gradually. Therefore, in the above-mentioned implementation, by periodically detecting whether the change of the cold metadata file exceeds the change threshold, when it is determined that the change threshold is exceeded, S306 is executed, so that the corresponding technical means in S306-S311 are used to update the metadata file again. The value is compressed.

In addition, in an implementation manner, the method further includes: compressing the metadata in the hot metadata file.

For example, prefix compression or slope compression is used for the metadata keywords in the hot metadata file; in addition, when the metadata value in the hot metadata file has no regularity, the metadata value may not be compressed.

In the above implementation, considering that the hot data in the storage system is likely to be modified, the possibility of changing the value of the metadata of the hot data is high. Compressing the metadata of hot data in one way will lead to low compression efficiency. Especially in the scenario where user data is written by means of additional write or redirect-on-write (ROW), each time the user data is modified, the modified data content of the user data will be Deposit a new physical address, so the above problem will be more obvious.

For example, in the scenario where user data is written in the append or ROW mode, it may occur that: the value in the metadata of a certain user data (the user data is hot data) has just been compressed by the method of S308-310 above , the actual address of the user data changes due to the modification of the user data, which is equivalent to the process of compressing the value in the metadata of the user data, which is meaningless.

Therefore, in this example, the metadata is first divided into cold metadata files and hot metadata files (that is, the above S304), and then on the one hand, according to the process of S308-310, the value of the metadata in the cold metadata files is compressed , on the other hand, the metadata value in the hot metadata file may not be compressed. In this way, the efficiency of metadata compression can be improved.

Of course, in some other scenarios, cold data and hot data may not be distinguished, that is, the content of S304 above is not executed, but the metadata in the _L2 layer is taken as a whole, and all or part of the metadata in this whole , adopting processes such as migration of user data, so that the value of the metadata conforms to the set rule, and then compresses all or part of the value of the metadata. In this regard, there is no limitation in this example.

_In addition, in the above _- mentioned embodiment, the metadata compression process is performed at the L2 layer mainly in the scenario where the metadata _of the L1 layer in the LSM tree is merged to the L2 layer, and the metadata compression method provided by the present application is carried out. introduce. In the actual application process, this method can also be applied to compress metadata of other data structures, for example, this method can also be used to perform metadata compression on other storage layers in the LSM tree, or this method can also be applied to other than Metadata for data structures other than the LSM tree are compressed.

In addition, in the above embodiment, the user data corresponding to the metadata is mainly migrated to store the user data corresponding to the metadata in a continuous storage space, so that the actual addresses of the multiple user data appear as regularity to compress the values in the metadata of these user data.

In some other embodiments, if the actual address of the user data is the address in the logical block group (chunk group) corresponding to the user data in the LUN, or the address in the physical block (chunk) of the user data in the hard disk , the user data may not be migrated, so that the actual addresses of multiple user data show regularity. For example, the actual address of the storage space where the user data is stored can be modified so that the actual addresses of multiple user data show regularity, that is, the values in the metadata of multiple user data conform to the set rule, The values in the metadata of these user data are thus compressed. Then, the mapping relationship between the actual address of the user data before modification and the underlying physical address is updated to the mapping relationship between the actual address of the user data after modification and the underlying physical address.

To give a specific example, when the value in the metadata refers to the chunk group ID of the storage space where the user data resides and the offset in the chunk group. In a possible design, by modifying the mapping rules of chunk group IDs and offsets in chunk groups, the chunk group IDs and offsets in chunk groups in the storage space of multiple user data can be regularized, for example, making multiple user data The chunk group ID of the storage space and the offset in the chunk group are continuous, so that the values in the metadata of these user data can be compressed.

In addition, this embodiment also provides a metadata compression device, which can be used to perform some or all of the steps in the above-mentioned metadata compression method of this embodiment.

It can be understood that, in order to realize the functions in the above metadata compression method, the metadata compression device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and method steps described in each example in this embodiment, the technical solutions provided in this embodiment can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives the hardware depends on the specific application scenario and design constraints of the technical solution.

In this embodiment, the metadata compression apparatus may be located in a hardware device used to manage metadata in the storage system. For example, metadata compressors are located in engines in centralized storage systems. For another example, the metadata compression device is located in a storage server with a metadata management function in the distributed storage system.

FIG. 10 is a schematic structural diagram of a metadata compression device provided by the present application. The metadata compression device 40 includes an acquisition unit 401 , a processing unit 402 and a compression unit 403 . The metadata compression device is used to realize the functions of some or all steps in the method described above in FIG. 3 .

For example, the acquiring unit 401 is configured to execute one or more items of S301 and S306 in FIG. 3 . The processing unit 402 is configured to execute one or more items of S302-S305, S307-S309, and S312 in FIG. 3 . The compression unit 403 is configured to execute one or more items of S310 and S311 in FIG. 3 .

For a more detailed description of the acquisition unit 401 , the processing unit 402 and the compression unit 403 , you can directly refer to the relevant description in the method shown in FIG. 3 , which will not be repeated here.

FIG. 11 is a schematic structural diagram of a chip provided by the present application. The chip 50 is used to implement the metadata compression method provided in this application. Specifically, the chip may be a chip used to realize the functions of the controller in the engine 121 . Wherein, the chip 50 includes:

The processor 501 is configured to execute the metadata compression method provided in this application.

Specifically, the processor 501 may include a general-purpose central processing unit (central processing unit, CPU) and a memory, and the processor 501 may also be a microprocessor, a field programmable gate array (Field Programmable Gate Array, FPGA) or a specific application integration Circuit (application-specific integrated circuit, ASIC), etc. In the scenario where the processor 501 includes a CPU and a memory, the CPU executes computer instructions stored in the memory to execute the metadata compression method provided in this application.

In addition, the chip 50 may further include: a memory 502 . Computer instructions are stored in the memory 502, and the processor 501 executes the computer instructions stored in the memory to execute the metadata compression method provided in this application.

Specifically, the memory 502 may be a read-only memory (read-only memory, ROM) or other types of static storage devices that can store static information and instructions, or a random access memory (random access memory, RAM) that can store information and instructions Other types of dynamic storage devices can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage , optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store program codes in the form of instructions or data structures and can be used by Any other medium accessed by a computer, but not limited to.

In addition, the chip 50 may further include: an interface 503 . Interface 503 can be used to receive and send data. The interface 502 may be a communication interface or a transceiver or the like.

In addition, the chip 50 may further include a communication line 504 . For example, communication line 504 may be a data bus for transferring information between the aforementioned components.

For a more detailed description of the above-mentioned metadata compression apparatus 40 and chip 50 , reference may be made directly to relevant descriptions in the above-mentioned metadata compression method, which will not be repeated here.

The method steps in the embodiments of the present application may be implemented by means of hardware, or may be implemented by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, PROM, EPROM, EEPROM, registers, hard disk, mobile hard disk, CD-ROM or any other form of storage medium known in the art . An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC. In addition, the ASIC can be located in a network device or a terminal device. Certainly, the processor and the storage medium may also exist in the network device or the terminal device as discrete components.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer programs or instructions. When the computer program or instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, network equipment, user equipment, or other programmable devices. The computer program or instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be downloaded from a website, computer, A server or data center transmits to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrating one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disc (digital video disc, DVD); it may also be a semiconductor medium, such as an SSD.

In each embodiment of the present application, if there is no special explanation and logical conflict, the terms and/or descriptions between different embodiments are consistent and can be referred to each other, and the technical features in different embodiments are based on their inherent Logical relationships can be combined to form new embodiments.

In the present application, "at least one" means one or more, "multiple" means two or more, and other quantifiers are similar. "And/or" describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate: A exists alone, A and B exist simultaneously, and B exists independently. Furthermore, the singular forms "a", "an" and "the" do not mean "one or only one" but "one or more" unless the context clearly dictates otherwise. in one". For example, "a device" means reference to one or more such devices. Furthermore, at least one (at least one of)......." means one or any combination of subsequent associated objects, such as "at least one of A, B and C" includes A, B, C, AB, AC, BC, or ABC. In the text description of the application, the character "/" generally indicates that the front and rear related objects are a kind of "or" relationship; in the formula of the application, the character "/" indicates that the front and rear Associated objects are a "division" relationship.

It can be understood that the various numbers involved in the embodiments of the present application are only for convenience of description, and are not used to limit the scope of the embodiments of the present application. The size of the serial numbers of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic.

Claims

A method for compressing metadata, comprising:

Acquiring n pieces of metadata, one piece of metadata includes a key-value pair, the key-value pair includes a keyword and a value, the keyword is used to indicate the identity of the data corresponding to the metadata, and the value is used to indicate the The actual address of the data storage, the n is a positive integer greater than 1;

Processing m pieces of data corresponding to at least part of the metadata in the n pieces of metadata to obtain n target values corresponding to the n pieces of metadata conforming to the set rule, where m is a positive integer less than or equal to n;

Compress the n target values.
The method according to claim 1, characterized in that the n actual addresses indicated by the n target values conforming to the set rules are continuous.
The method according to claim 2, characterized in that the m data corresponding to the at least part of the metadata are processed to obtain n target values corresponding to the n metadata conforming to the set rules, include:

Migrating the m pieces of data, so as to store the data corresponding to the n pieces of metadata in a storage space with continuous actual addresses;

saving the actual addresses of the n data stored in the continuous storage space as the n target values.
The method according to any one of claims 1-3, wherein the method further comprises:

The n pieces of metadata are selected from the multiple pieces of metadata according to the hotness and coldness of the data corresponding to the pieces of metadata included in the metadata set, and the data corresponding to the n pieces of metadata is cold data.
The method according to any one of claims 1-3, wherein the n pieces of metadata are metadata in the first storage layer in a structured synthetic LSM tree; the LSM tree is used to store metadata; The LSM tree includes a plurality of storage tiers, and the plurality of storage tiers includes the first storage tier.
The method according to any one of claims 1-3, wherein the key and the value are respectively stored in two data entries.
The method according to any one of claims 1-3, wherein the method further comprises:

Detecting the amount of data change in the metadata set; the metadata set is used to record metadata of multiple data;

The acquiring n pieces of metadata includes: acquiring the n pieces of metadata included in the metadata set after determining that the amount of data change in the metadata set exceeds a change threshold.
The method according to any one of claims 1-3, wherein the method further comprises:

Obtain the degree of dispersion of the actual address of the data corresponding to the n pieces of metadata;

The processing of m data corresponding to at least some of the n metadata includes: after determining that the degree of dispersion is greater than a discrete threshold, processing the m data corresponding to at least some of the n metadata The m data for processing.
The method according to any one of claims 1-8, wherein the method is applied to a centralized storage system, and the method is executed by an engine in the centralized storage system.
The method according to any one of claims 1-8, wherein the method is applied to a distributed storage system, the distributed storage system includes multiple storage servers, and the method consists of the multiple storage servers One or more storage servers in the server execute.
A metadata compression device is characterized in that it comprises:

An acquisition unit, configured to acquire n pieces of metadata, one piece of metadata includes a key-value pair, the key-value pair includes a keyword and a value, the keyword is used to indicate the identity of the data corresponding to the metadata, the The value is used to indicate the actual address of the data storage, and the n is a positive integer greater than 1;

A processing unit, configured to process m data corresponding to at least some of the n metadata, to obtain n target values corresponding to the n metadata conforming to a set rule, where m is less than or equal to n a positive integer;

A compression unit, configured to compress the n target values.
The device according to claim 11, characterized in that the n actual addresses indicated by the n target values conforming to the set rule are continuous.
The device according to claim 12, wherein the processing unit is configured to process the m pieces of data corresponding to the at least part of the metadata, and obtain the data corresponding to the n pieces of metadata conforming to the set rule. n target values, including:

The processing unit is specifically configured to migrate the m pieces of data, so as to store the data corresponding to the n pieces of metadata in a storage space with continuous actual addresses;

The processing unit is specifically configured to save the actual addresses of the n data stored in the continuous storage space as the n target values.
The device according to any one of claims 11-13, wherein the processing unit is further configured to select from the multiple The n pieces of metadata are selected from the pieces of metadata, and the data corresponding to the n pieces of metadata is cold data.
The device according to any one of claims 11-13, wherein the n pieces of metadata are metadata in the first storage layer in the structured synthetic LSM tree; the LSM tree is used to store metadata; The LSM tree includes a plurality of storage tiers, and the plurality of storage tiers includes the first storage tier.
The device according to any one of claims 11-13, wherein the key and the value are stored in two data entries respectively.
The device according to any one of claims 11-13, wherein the processing unit is further configured to detect a data change amount of a metadata set; the metadata set is used to record metadata of multiple data;

The acquisition unit is configured to acquire n pieces of metadata, including: the acquisition unit is specifically configured to acquire the metadata included in the metadata set after determining that the amount of data change in the metadata set exceeds a change threshold. n metadata.
The device according to any one of claims 11-13, wherein the acquiring unit is further configured to acquire the degree of discreteness of the actual address of the data corresponding to the n pieces of metadata;

The processing unit is configured to process m pieces of data corresponding to at least part of the metadata in the n pieces of metadata, including: the processing unit is specifically configured to, after determining that the degree of dispersion is greater than a dispersion threshold, process m data corresponding to at least part of the n metadata are processed.
The device according to any one of claims 11-13, wherein the metadata compression device is located in an engine in a centralized storage system.
The device according to any one of claims 11-13, wherein the metadata compression device is located in a storage server in a distributed storage system.
A storage device, characterized by comprising a memory and a processor, the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory, so as to realize claims 1-10 any one of the methods described.
A storage system, characterized by comprising an engine and a plurality of hard disks, the plurality of hard disks are used to store data, and the engine is used to execute the method according to any one of claims 1-10.
A storage system, characterized by comprising multiple storage servers, the multiple storage servers are used to store data, and the first server among the multiple storage servers is used to perform the described method.
A chip, characterized in that it includes a memory and a processor, the memory is used to store computer instructions, and the processor is used to call and run the computer instructions from the memory, so as to implement claims 1-10 any one of the methods described.
A computer-readable storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1-10 is realized.
A computer program product, characterized in that the computer program product includes instructions, and when the instructions are run on a processor, the method according to any one of claims 1-10 is implemented.