CN113805789A

CN113805789A - Metadata processing method in storage device and related device

Info

Publication number: CN113805789A
Application number: CN202010739813.8A
Authority: CN
Inventors: 王晨; 高蒙; 崔文林; 罗四维; 任仁
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-06-11
Filing date: 2020-07-28
Publication date: 2021-12-17

Abstract

The application discloses a metadata processing method in storage equipment and related equipment, and belongs to the technical field of computers. The method comprises the following steps: a network card in the storage device receives an IO request, wherein the IO request comprises a data reading request or a data writing request; the network card executes a metadata processing task corresponding to the IO request; and under the condition that the network card determines that the metadata processing task fails to be executed, the network card requests the CPU in the storage device to execute the metadata processing task. Therefore, the application provides a method for dynamically executing metadata processing tasks in a self-adaptive manner by using a network card and a CPU. Therefore, partial metadata processing tasks can be adaptively unloaded from the CPU to the network card, the data processing pressure of the CPU can be reduced, and overlong time delay caused by the fact that the network card processes the metadata can be avoided, so that the data reading and writing efficiency of the storage device is improved.

Description

Metadata processing method in storage device and related device

The present application claims priority from chinese patent application No. 202010526832.2 entitled "a storage system, storage node, and data storage method" filed 11.06/11/2020, which is incorporated herein by reference in its entirety.

Technical Field

The present application relates to the field of storage technologies, and in particular, to a metadata processing method in a storage device and a related device.

Background

To guarantee the query of the data, metadata of the data is also configured for the data stored in the storage device. The metadata is used to indicate the storage location of the data. When writing data, it is often necessary to write the metadata of the data at the same time. When reading data, the metadata of the data needs to be acquired first, so as to read the data based on the metadata. Therefore, how to process the metadata in the process of reading and writing the data affects the reading and writing efficiency of the storage device to a certain extent.

Disclosure of Invention

The application provides a metadata processing method in a storage device and a related device, which can improve the efficiency of reading and writing data of the storage device. The technical scheme is as follows:

in a first aspect, a method for processing metadata in a storage device is provided. In the method, a network card in a storage device receives an input/output (IO) request, where the IO request includes a read data request or a write data request. And the network card executes the metadata processing task corresponding to the IO request. When the network card determines that the metadata processing task fails to be executed, the network card requests a Central Processing Unit (CPU) in the storage device to execute the metadata processing task.

In the application, in order to reduce the pressure of the CPU on processing data, when the network card receives an IO request, the network card does not directly transfer the IO request to the CPU, but the network card executes the metadata processing task first. If the metadata processing task fails to execute, the CPU is requested to execute the metadata processing task. Therefore, the metadata processing task corresponding to the IO request can be prevented from being executed by the network card or the CPU. Therefore, the storage device can adaptively unload part of metadata processing tasks from the CPU to the network card, so that the data processing pressure of the CPU can be reduced, and the overlong time delay caused by the fact that the network card processes the metadata can be avoided, thereby improving the efficiency of reading and writing data of the storage device.

Based on the method provided in the first aspect, in a possible implementation manner, the network card includes a memory, and the case where the execution of the metadata processing task fails includes: the memory of the network card is not sufficient to provide the memory resources required for the metadata processing task.

In the process of executing the metadata processing task by the network card, if the available memory of the network card is not enough to provide the memory required by the metadata processing task, the network card may determine that the metadata processing task fails to be executed. Therefore, the network card can quickly judge whether the metadata processing task fails in the process of executing the metadata processing task, and the flexibility of executing the metadata processing task is improved.

Based on the method provided by the first aspect, in a possible implementation manner, when the IO request is a read data request, the case that the execution of the metadata processing task fails includes: the address length of the logical address carried by the read data request exceeds a length threshold.

In a scenario of a data reading request, if a network card directly reads data with a logical address length exceeding a length threshold, the network card needs to interact for multiple times to read the data, so that the time required by the network card to read the data is long. Therefore, in the application, if the address length of the logical address carried by the read data request exceeds the length threshold, the CPU is requested to execute the metadata processing task, so as to improve the efficiency of reading the data of which the address length of the logical address exceeds the length threshold.

Based on the method provided by the first aspect, in a possible implementation manner, the storage device includes a first index repository and a second index repository, and a data amount of metadata stored in the first index repository is smaller than a data amount of metadata stored in the second index repository. In this scenario, when the IO request is a read data request, the case that the metadata processing task fails to execute includes: the network card does not acquire the metadata corresponding to the read data request in the first index database. At this time, the network card requesting the CPU in the storage device to execute the metadata processing task includes: and the network card instructs the CPU to acquire the metadata corresponding to the read data request from the second index database.

In addition, because the data volume required to be processed for acquiring the metadata from the first index base is smaller than the data volume required to be processed for acquiring the metadata from the second index base, in the embodiment of the present application, the network card may acquire the metadata from the first index base first, and the CPU may acquire the metadata from the second index base when the network card does not acquire the metadata from the first index base, thereby improving the efficiency of acquiring the metadata.

Based on the method provided by the first aspect, in a possible implementation manner, the first index library includes metadata of hotspot data, and the second index library includes metadata of non-hotspot data.

The first index database comprises the metadata of the hot spot data, and the hot spot data is data with high access frequency, so that the efficiency of reading the metadata of the hot spot data can be improved by separately storing the metadata of the hot spot data in one index database.

Based on the method provided by the first aspect, in a possible implementation manner, when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

Based on the method provided by the first aspect, in a possible implementation manner, when the IO request is a read data request, the metadata processing task is to acquire metadata corresponding to the read data request.

In the present application, operations that need to be executed by the metadata processing task are different in different IO requests.

Based on the method provided by the first aspect, in a possible implementation manner, the metadata corresponding to the IO request is used to indicate a storage location of data corresponding to the IO request.

In the embodiment of the application, the metadata corresponding to the IO request includes information related to the storage location of the data, so that in a data reading scenario, the network card can acquire the information related to the storage location of the data based on the index library, and thus the network card can bypass the CPU to read the data, and the efficiency of the storage device in reading the data is improved.

Based on the method provided in the first aspect, in one possible implementation manner, the metadata includes a key-value pair, a key in the key-value pair indicates a logical address of data corresponding to the IO request, and a value in the key-value pair indicates a physical address of the data corresponding to the IO request.

Through the key-value pairs, the corresponding relation between the logical address and the physical address of the data can be stored in the metadata, so that the technical effect that the network card can acquire the information related to the storage position of the data based on the index database is achieved, the network card can bypass a CPU to read the data, and the efficiency of the storage device for reading the data is improved.

Based on the method provided in the first aspect, in a possible implementation manner, the metadata includes a key-value pair, a key in the key-value pair indicates a logical address of data corresponding to the IO request, and a value in the key-value pair indicates fingerprint information of the data corresponding to the IO request.

The physical address of the data can be continuously inquired through the fingerprint information of the data, so that the technical effect that the network card can acquire the information related to the storage position of the data based on the index database can be realized by the mode, the network card can bypass the CPU to read the data, and the efficiency of the storage device for reading the data is improved.

Based on the method provided by the first aspect, in a possible implementation manner, the storage device is a storage array or a storage node in a distributed storage system. The method provided by the embodiment of the application can be applied to a single storage device and can also be applied to a certain storage node of a distributed storage system.

In a second aspect, a method for processing metadata in a storage device is provided. In the method, a network card in a storage device receives a plurality of IO requests, the IO requests comprise data reading requests or data writing requests, and when the network card determines that the number of the IO requests exceeds a set number threshold, the network card requests a CPU in the storage device to process a metadata processing task corresponding to at least one IO request in the plurality of IO requests.

In the application, the execution result of the metadata execution task to be currently processed can be pre-judged based on the number of the IO requests, when the number of the IO requests exceeds a certain number, the pre-judging network card cannot successfully execute the metadata processing task, and at this time, the CPU can be requested to execute the metadata processing task. Not only the flexibility of metadata processing is improved. And the network card and the CPU can adaptively and dynamically execute the metadata processing task. The at least one IO request may be a part of IO requests in the plurality of IO requests, or may be all of the IO requests. Therefore, partial metadata processing tasks can be adaptively unloaded from the CPU to the network card, the data processing pressure of the CPU can be reduced, and overlong time delay caused by the fact that the network card processes the metadata can be avoided, so that the data reading and writing efficiency of the storage device is improved.

When the number of the IO requests exceeds a certain number, the CPU can process all the IO requests, or the CPU can process one part of the IO requests, and the network card can process the other part of the IO requests, so that the flexibility of processing the metadata is improved.

Based on the method provided by the second aspect, in a possible implementation manner, when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

Based on the method provided by the second aspect, in a possible implementation manner, when the IO request is a read data request, the metadata processing task is to acquire metadata corresponding to the read data request.

In a possible implementation manner, based on the method provided by the second aspect, the metadata corresponding to the IO request is used to indicate a storage location of data corresponding to the IO request.

Based on the method provided in the second aspect, in one possible implementation, the metadata includes key-value pairs, a key of a key-value pair indicating a logical address of data corresponding to the IO request, and a value of a key-value pair indicating a physical address of data corresponding to the IO request.

Based on the method provided in the second aspect, in one possible implementation manner, the metadata includes a key-value pair, a key of the key-value pair indicates a logical address of data corresponding to the IO request, and a value of the key-value pair indicates fingerprint information of the data corresponding to the IO request.

Based on the method provided by the second aspect, in one possible implementation manner, the storage device is a storage array or a storage node in a distributed storage system.

The beneficial effects of the above related explanations of the metadata can refer to the technical effects of the related implementation manners in the metadata processing method provided in the first aspect, and are not described herein again.

In a third aspect, there is provided a metadata processing apparatus having a function of implementing the metadata processing method in the first aspect described above. The metadata processing device comprises at least one module, and the at least one module is used for realizing the metadata processing method provided by the first aspect.

In a fourth aspect, there is provided a metadata processing apparatus having a function of implementing the metadata processing method in the second aspect described above. The metadata processing device comprises at least one module, and the at least one module is used for realizing the metadata processing method provided by the second aspect.

In a fifth aspect, a network card is provided, where the network card includes a communication interface and a processor, the communication interface is used for communicating with a CPU in a storage device, and the processor is used for implementing the functions of the method in any one of the first aspect and the second aspect.

In a sixth aspect, a storage device is provided, which includes a network card and a CPU;

the network card is used for receiving an IO request, executing a metadata processing task corresponding to the IO request, and requesting a CPU to execute the metadata processing task under the condition that the network card determines that the metadata processing task fails to be executed, wherein the IO request comprises a data reading request or a data writing request;

the CPU is used for executing metadata processing tasks based on the request of the network card.

For a specific implementation, reference may be made to the implementation of the metadata processing method in the storage device provided in the first aspect, which is not described herein again.

The network card comprises a memory, and the condition that the execution of the metadata processing task fails comprises the following steps: the memory of the network card is not enough to provide the memory resource required by the metadata processing task.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, when the IO request is a read data request, the case that the execution of the metadata processing task fails includes: the address length of the logical address carried by the read data request exceeds a length threshold.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, the storage device includes a first index repository and a second index repository, where a data amount of metadata stored in the first index repository is smaller than a data amount of metadata stored in the second index repository, and when the IO request is a read data request, a case that execution of the metadata processing task fails includes: the network card does not acquire metadata corresponding to the read data request in the first index database; the network card requesting a CPU in the storage device to execute the metadata processing task comprises the following steps: and the network card instructs the CPU to acquire the metadata corresponding to the read data request from the second index database.

In a possible implementation manner, based on the storage device provided by the sixth aspect, the first index library includes metadata of hotspot data, and the second index library includes metadata of non-hotspot data.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, when the IO request is a read data request, the metadata processing task is to acquire metadata corresponding to the read data request.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, the metadata corresponding to the IO request is used to indicate a storage location of data corresponding to the IO request.

Based on the storage device provided in the sixth aspect, in one possible implementation manner, the metadata includes a key-value pair, a key in the key-value pair is used to indicate a logical address of data corresponding to the IO request, and a value in the key-value pair is used to indicate a physical address of the data corresponding to the IO request.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, the metadata includes a key-value pair, a key in the key-value pair is used to indicate a logical address of data corresponding to the IO request, and a value in the key-value pair is used to indicate fingerprint information of the data corresponding to the IO request.

Based on the storage device provided in the sixth aspect, in a possible implementation manner, the storage device is a storage array or a storage node in a distributed storage system.

A seventh aspect provides a storage device, including a network card and a CPU;

the network card is used for receiving a plurality of input/output (IO) requests, and requesting a Central Processing Unit (CPU) to process a metadata processing task corresponding to at least one IO request in the plurality of IO requests when the number of the IO requests exceeds a set number threshold;

the CPU is used for executing a metadata processing task corresponding to at least one IO request based on a request of the network card.

For a specific implementation, reference may be made to an implementation of the metadata processing method in the storage device provided in the second aspect, which is not described herein again.

Based on the storage device provided in the seventh aspect, in one possible implementation manner, the at least one IO request is all IO requests in the multiple IO requests.

Based on the storage device provided in the seventh aspect, in one possible implementation manner, the at least one IO request is a subset of the plurality of IO requests; the network card is also used for executing metadata processing tasks corresponding to the rest IO requests except at least one IO request in the plurality of IO requests.

Based on the storage device provided in the seventh aspect, in a possible implementation manner, when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

Based on the storage device provided in the seventh aspect, in a possible implementation manner, when the IO request is a read data request, the metadata processing task is to obtain metadata corresponding to the read data request.

Based on the storage device provided in the seventh aspect, in a possible implementation manner, the metadata corresponding to the IO request is used to indicate a storage location of data corresponding to the IO request.

Based on the storage device provided in the seventh aspect, in one possible implementation manner, the metadata includes a key-value pair, a key in the key-value pair indicates a logical address of data corresponding to the IO request, and a value in the key-value pair is used to indicate a physical address of the data corresponding to the IO request.

Based on the storage device provided in the seventh aspect, in a possible implementation manner, the metadata includes a key-value pair, a key in the key-value pair is used to indicate a logical address of data corresponding to the IO request, and a value in the key-value pair is used to indicate fingerprint information of the data corresponding to the IO request.

In an eighth aspect, a computer-readable storage medium is provided, in which instructions are stored, which, when executed on a computer, cause the computer to perform the metadata processing method of the first or second aspect.

In a ninth aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the metadata processing method of the first or second aspect.

The technical effects obtained by the third aspect to the ninth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect or the second aspect, and are not described herein again.

Drawings

FIG. 1 is a schematic diagram of a memory system according to an embodiment of the present disclosure;

FIG. 2 is a schematic block diagram of another embodiment of a storage system;

FIG. 3 is a flowchart of a metadata processing method in a storage device according to an embodiment of the present disclosure;

FIG. 4 is a flowchart of another metadata processing method in a storage device according to an embodiment of the present disclosure;

FIG. 5 is a flowchart of another metadata processing method in a storage device according to an embodiment of the present disclosure;

FIG. 6 is a flowchart of another metadata processing method in a storage device according to an embodiment of the present disclosure;

fig. 7 is a schematic diagram of a metadata processing apparatus according to an embodiment of the present application;

fig. 8 is a schematic diagram of another metadata processing apparatus according to an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Before explaining the embodiments of the present application in detail, an application scenario of the embodiments of the present application is explained.

The performance of the index library in the storage system has a great influence on the overall performance of the storage system, and the index library is used for storing metadata and is usually stored by adopting a certain data structure. Metadata is data that describes various attributes of data, including the mapping relationship between logical addresses and physical addresses of the data, or other storage locations that describe the data, as well as information describing the relationship between various data. Where the logical address is an address presented to the upper layer application and the physical address is used to indicate where the data is stored on the storage medium. When reading data, often only the logical address of the data to be read can be obtained from the data reading request, which needs to obtain the physical address corresponding to the logical address from the index database, and obtain the data from the space indicated by the physical address. When writing data, the physical address of the data and the mapping relationship between the physical address and the logical address need to be stored in the index database so as to be used when reading the data next time.

As described above, the storage system accesses the index library at the same time when data access (data access means reading or writing data) is performed. The metadata processing method provided by the embodiment of the application is applied to the scene of accessing the index database.

The following explains an architecture of a storage system according to an embodiment of the present application. As shown in fig. 1, the storage system provided in this embodiment includes a host cluster and a storage node cluster.

Wherein a host cluster includes one or more hosts 100 (two hosts 100 are shown in fig. 1, but not limited to two hosts 100). Host 100 is a computing device on the user side, such as a server, desktop computer, or the like. An application program (application)101 (application for short) and a client program (client) 102 (client for short) are run on the host 100. Application 101 is a generic term for various application programs presented to a user. The client 102 is configured to receive an input/output (IO) request triggered by the application 101, interact with the storage node 200, and send the IO request to the storage node 200. Client 102 is also used to receive data from storage nodes and forward the data to application 101. The client 102 may also be implemented by a hardware component located inside the host 100. It will be appreciated that when the client 102 is a software program, the functionality of the client 102 is implemented by a processor included in the host 100 running the program. Any one of the clients 102 in the host cluster may access any one of the storage nodes 200 in the storage node cluster over a network.

The storage node cluster includes one or more storage nodes 200 (three storage nodes 200 are shown in fig. 1, but not limited to three storage nodes 200), and the storage nodes 200 may be interconnected through an Internet Protocol (IP) network or other networks. The storage node 200 is a storage device, such as a server or a controller of a storage array. In hardware, as shown in fig. 1, the storage node 200 at least includes a processor 202, a memory 203, a network card 201, and a hard disk 204.

The processor 202 is a Central Processing Unit (CPU) and is configured to process an IO request from outside the storage node 200 or a request generated inside the storage node 200.

The memory 203 is used to temporarily store data received from the host 100 or data internally read. When the storage node 200 receives a plurality of write data requests transmitted from the host 100, the data in the plurality of write data requests may be temporarily stored in the memory 203. When the total amount of data in the memory 203 reaches a certain threshold, the data stored in the memory 203 is downloaded to the hard disk 204 for storage. Memory 203 includes volatile memory, such as random-access memory (RAM). The memory 203 may have a power-saving function, which means that when the system is powered down and powered up again, the data stored in the memory 203 will not be lost. In general, a memory having a power retention function is called a nonvolatile memory.

It should be noted that the memory 203 may be a memory of the storage node itself. Or the memory used by the storage node may be partitioned from a shared memory pool (global memory pool). The shared memory pool is created in advance on each storage node, and the storage space of the shared memory pool is provided by the memory or other storage on each storage node. When in use, each storage node can apply a part of storage space to the shared memory pool as its own memory.

The network card 201 is used to communicate with the host 100. Specifically, the storage node 200 may receive a request from the host 100 through the network card 201, or may send a request to the host 100 through the network card 201.

The hard disk 204 is used for storing data, and may be a magnetic disk or other types of storage media, such as a solid state disk or a shingled magnetic recording hard disk. It should be noted that the hard disk in the storage system shown in fig. 1 may be a hard disk inside the storage node. Optionally, the hard disk is also a hard disk on another storage device externally connected to the storage node. In this case, the storage node 200 is mainly used for calculating or storing data, and the calculation of data mainly includes metadata management, data deduplication, virtualization of storage space, address translation, and the like.

The metadata management may refer to reading and writing or updating metadata from the index repository. For example, storing the metadata to the index repository or reading the metadata from the index repository, etc. The deduplication is that when data is stored, if the same data as the data to be stored is already stored in the storage node, the data is not stored. Whether the same data exist or not is judged through fingerprint information of the data, and the fingerprint information is used for uniquely identifying the data. The fingerprint information may be stored in some database of the storage node. For example, when an index library is deployed on a storage node, the database may be the index library on the storage node. Alternatively, the database may be another database independent of the index database.

The virtualized storage space means that the storage space provided by the hard disk is virtualized into a logical unit, so that a logical address which can be perceived by a host is provided. Address translation refers to the translation of a logical address to a physical address or the translation of a physical address to a logical address.

In addition, the storage node 200 may also include a bus (not shown in FIG. 1) for communication between various components within the storage node 200.

Referring to fig. 2, as shown in fig. 2, in another network architecture of a storage system provided in this embodiment, an application 101 and a client 102 are deployed inside a storage node 200, so that the application 101 may directly trigger an IO request (the IO request in this embodiment includes a write data request or a read data request) through the client 102 in the storage node 200, be processed by the storage node 200, or be sent to another storage node 200 for processing. At this time, the IO request sent by the client 102 to the storage node 200 specifically refers to the client 102 sending the IO request to the processor 202. For the network architecture shown in fig. 2, if the hard disk in fig. 2 is a hard disk inside the storage node 200, the IO request does not need to pass through other networks when arriving at the hard disk 204 from the client 102. If the hard disk in fig. 2 is a hard disk on another storage device externally connected to the storage node 200, the IO request needs to pass through a one-hop network (between the storage node and the other storage device where the hard disk is located) when arriving at the hard disk 204 from the client 102. In the network architecture shown in fig. 1, the IO request from the client 102 to the hard disk 204 needs to pass through at least one hop network (the network between the host and the storage node). Otherwise, the components included in the storage node 200 and the functions thereof are similar to those of the storage node 200 in fig. 1, and are not described herein again.

For the storage system shown in fig. 1 or fig. 2, an index library may be deployed on each storage node, where the index library is used to store metadata for data stored on the corresponding storage node. This scenario of deploying an index repository may be referred to as a distributed deployment index repository. In a scenario of distributed deployment of index libraries, the index library on each storage node may be deployed in a memory of the storage node, or may be deployed on a hard disk of the storage node.

Optionally, an index library may be centrally deployed on a storage node in the storage system, and the index library is used for storing metadata of data stored in the entire storage system. This scenario of deploying an index repository may be referred to as a centralized deployment index repository. In a scenario of centralized deployment of the index library, the centrally deployed index library may be deployed in a memory of the storage node, or may be deployed in a hard disk of the storage node.

In addition, considering that the hot spot data in the storage system is frequently used data, the metadata of the hot spot data can be separately stored in one index repository. Data other than the hotspot data or metadata including the hotspot data is stored in another index repository. The access frequency of the embodiment of the application can be determined by the access times of the data in unit time. Thus, the index library provided by the embodiment of the application can be divided into the first index library and the second index library. The first index base stores metadata of the hotspot data. The second index database stores metadata including hotspot data and non-hotspot data, or stores metadata of non-hotspot data except hotspot data. Illustratively, the metadata in the first index repository may be stored by way of a hash (hash) table. The metadata in the second index repository may be stored by a tree structure. At this time, the first index repository may also be referred to as a hash table index repository, and the second index repository may also be referred to as a tree structure index repository.

In the case that the index libraries comprise a first index library and a second index library, if the metadata needs to be queried from the index libraries, the metadata can be queried from the first index library. If the metadata is not inquired in the first index database, the metadata is inquired from the second index database. Therefore, the metadata is prevented from being directly inquired from the full-scale index library, and the efficiency of inquiring the metadata is improved.

In addition, the index library can store information in metadata of certain data in a manner of collecting key-value pairs. For example, when metadata corresponding to an IO request is used to indicate a storage location of data corresponding to the IO request, a key in a key-value pair indicates a logical address of the data, and a value in the key-value pair indicates a physical address of the data. At this time, the key-value pair is used to store a mapping relationship between logical addresses and physical addresses of data. Optionally, a key in a key-value pair indicates a logical address of the data and a value in a key-value pair indicates fingerprint information of the data. The fingerprint information has been described above and will not be described herein. The physical address of the data can be continuously inquired through the fingerprint information.

The storage manner of the metadata in the index library is described above by taking key-value pairs as an example. Alternatively, the metadata in the index library may be stored in other manners, which are not illustrated herein.

In addition, if the index libraries deployed on the storage node include a first index library and a second index library, the first index library may be stored in the memory of the storage node, and the second index library is stored in the hard disk of the storage node. Optionally, the first index repository and the second index repository may both be stored in the memory of the storage node, or both may be stored in the hard disk of the storage node. The embodiment of the present application does not limit the specific storage locations of the first index repository and the second index repository.

In addition, the storage system shown in fig. 1 or fig. 2 further includes a storage pool for providing storage space originating from the hard disk 204 in the storage node 200.

The actual address of the storage space provided by the hard disk 204 is not directly exposed to the storage node 200 or the host 100. In practical applications, part or all of the hard disks 204 included in each storage node 200 form the storage pool, each hard disk 204 is divided into a plurality of blocks (chunks), a plurality of blocks from different hard disks 204 or different storage nodes 200 form a storage object, and the storage object is the minimum allocation unit of the storage pool. When a storage node applies for storage space from a storage pool, the storage pool may provide one or more storage objects to the storage node. The storage node further virtualizes the storage space provided by the storage object into Logical Units (LU) for use by the host 100. Each logical unit has a unique Logical Unit Number (LUN). Since the host 100 is directly aware of the logical unit number, those skilled in the art typically directly refer to the logical unit with a LUN. Each LUN has a LUN ID for identifying the LUN. The specific location of data within a LUN may be determined by the starting address and the length (length) of the data. For the start address, those skilled in the art generally refer to a Logical Block Address (LBA). It will be appreciated that the three factors, LUN ID, LBA and length, identify a certain address segment. The host 10 generates a read data request, which typically carries LUN ID, LBA, and length, and these three factors may be referred to as logical address of data.

In addition, the storage system shown in fig. 1 or fig. 2 described above belongs to a scenario of distributed deployment, and in the scenario of distributed deployment of the storage system, the storage system includes a plurality of storage nodes. Optionally, the present application may also be applied in a centralized storage scenario, for example, the metadata processing method of the present application is implemented inside a single storage array or inside a single storage node. The storage array is a combination of at least one controller and a plurality of hard disks, and the method may be performed by the controller. The single storage node may be any of the storage nodes 200 shown in fig. 1 or 2.

The foregoing is used to explain the network architecture of the storage system according to the embodiment of the present application. The metadata processing method provided in the embodiments of the present application is explained in detail below.

Fig. 3 is a flowchart of a metadata processing method in a storage device according to an embodiment of the present application. The storage device may be any of the storage nodes of fig. 1 or fig. 2. As shown in fig. 3, the method includes the following steps.

Step 301: a network card in the storage device receives an IO request, where the IO request includes a read data request or a write data request.

The IO request is a data access request sent by a client in the host shown in fig. 1 or fig. 2. For the storage system shown in fig. 1 or fig. 2, the client calculates and obtains the corresponding storage node according to the address of the data to be accessed by the IO request and a preset algorithm, and sends the IO request to the corresponding storage node. For the scenario shown in fig. 1, the client in the host sends the IO request to the network card in the storage node, and when the network card receives the IO request, the IO request is processed by the method provided by the present application. For the scenario shown in fig. 2, the client in the host still sends the IO request to the network card in the storage node, and the network card processes the IO request uniformly by the method provided in the present application.

In addition, when the IO request is a write data request, the write data request includes data to be stored and a logical address of the data to be stored. The logical address of the data to be stored includes LUN ID, LBA, and length. The operations that the storage device needs to perform include: and storing the data corresponding to the data writing request to the storage device, and storing the corresponding relation between the logical address and the physical address of the data to the index library. And when the IO request is a read data request, the read data request comprises a logic address of data to be read. The operations that the storage device needs to perform include: and acquiring a physical address corresponding to the logical address carried by the read data request from the index database, and reading data according to the acquired physical address.

When the correspondence between the logical address and the physical address of the data is in a key-value pair manner, and the IO request is a data writing request, the storage device needs to store a key-value pair in the index repository. When the IO request is a data read request, the storage device needs to obtain a key-value pair from the index library.

In addition, because the network card may receive a large number of IO requests in a short time, and the network card needs to use a certain time length for processing each IO request, the network card may also add the IO request to the task queue after receiving the IO request. And processing the IO requests sequentially through the following steps according to the sequence in the task queue.

Step 302: and the network card executes the metadata processing task corresponding to the IO request.

In the embodiment of the present application, the metadata processing task refers to a query or modification task for processing metadata of data corresponding to an IO request. For example, when the IO request is a data writing request, the metadata processing task corresponding to the IO request stores metadata corresponding to the data writing request in the index library, and completes updating of the index library. The metadata corresponding to the write data request includes a physical address of the data to be stored, or a mapping relationship between the physical address and a logical address, or other address index information. When the IO request is a data reading request, the metadata processing task corresponding to the IO request is to query the index database to obtain metadata corresponding to the data reading request. For example, a physical address corresponding to a logical address carried by the read data request is obtained.

Because the network card needs to use certain memory resources to execute the metadata processing task, the memory resources mainly come from the memory contained in the network card itself. Therefore, in one possible implementation, the network card may determine the execution of the metadata processing task based on the available memory resources. Specifically, the network card determines the memory resources required for executing the metadata processing task, and if the available memory of the network card is not enough to provide the required memory resources, the network card determines that the metadata processing task is failed to be executed. And if the memory available to the network card is enough to provide the required memory resource, the network card executes the metadata processing task. That is, in the embodiment of the present application, the case that the execution of the metadata processing task fails includes: the memory of the network card is not sufficient to provide the memory resources required for the metadata processing task.

The memory resource available for the network card refers to a remaining space in the memory of the network card. In addition, the memory resource of the network card may be a part of the memory divided from the shared memory pool to be used by the network card, except for the memory of the network card itself, which is not specifically limited in this embodiment of the present application. The shared memory pool has already been introduced in the system architectures shown in fig. 1 and 2, and will not be described herein again.

In addition, because the IO request usually has a time delay requirement, in another possible implementation manner, after the network card receives the IO request, the network card executes the metadata processing task, and determines in real time the time that has been taken to execute the metadata processing task in the execution process. And if the time length exceeds the time delay threshold value, the network card judges that the execution of the metadata processing task fails.

The time length threshold is a set time length, and the time length threshold can be set by an administrator based on the performance of the network card.

In the implementation manner described above, after receiving the IO request, the network card directly executes the metadata processing task, counts the time length spent for executing the metadata processing task, and after the time length spent exceeds the time length threshold, does not continue to execute the metadata processing task, but completes the execution of the metadata processing task through the following step 303.

It should be noted that the implementation manner for determining whether the metadata processing task fails to be executed based on the memory resource or the time delay may be applied to a read data request or a write data request. In addition, the network card may also determine whether the metadata processing task fails to be executed in other manners, which is not illustrated herein.

Step 303: and under the condition that the network card determines that the metadata processing task fails to be executed, the network card requests the CPU in the storage device to execute the metadata processing task.

Based on step 302, the execution result of the metadata processing task acquired by the network card is obtained by the network card during the metadata processing task. Therefore, requesting the CPU to execute the metadata processing task may refer to requesting the CPU to re-execute the metadata processing task, or may refer to requesting the CPU to execute the remaining metadata processing task whose network card is not completed.

The implementation manner of the network card requesting the CPU to execute the metadata processing task may be: the network card sends a task execution request to the CPU. The task execution request instructs the CPU to execute the metadata processing task corresponding to the IO request. When the CPU receives the task execution request, the metadata processing task can be executed. The task execution request may carry an identifier of the IO request, so that the CPU directly executes the metadata processing task corresponding to the IO request based on the identifier of the IO. In addition, the task execution request may also carry information indicating an execution progress of the network card executing the metadata processing task, so that the CPU continues to execute the remaining metadata processing tasks that the network card has not completed directly based on the identifier of the IO and the execution progress of the metadata processing task.

It should be noted that, when executing the metadata processing task, if the operations such as updating and querying the index library are completed through the CPU, the advantage is that a relatively complex index library can be processed, but at the same time, the overhead of scheduling the processing operation on the CPU is also brought, and the access delay is increased. The other method is to Access the index library through the processing capability of a high-performance network card (e.g., one-sided Remote Direct Memory Access (one-sided RDMA)), but because the network card cannot process an excessively complex index library and has a certain limitation on processing efficiency, the network card often needs to complete the processing of the index library through multiple network interactions, which also has a negative impact on performance.

Therefore, in the embodiment of the present application, in order to reduce the pressure of the CPU for processing data, when the network card receives an IO request, the network card does not directly transfer the IO request to the CPU, but the network card first obtains the execution result of the metadata processing task. And if the execution result is failure, requesting the CPU to execute the metadata processing task corresponding to the IO request. Therefore, the metadata processing task corresponding to the IO request can be prevented from being executed by the network card or the CPU. Therefore, the storage device can adaptively unload part of metadata processing tasks from the CPU to the network card, thereby not only reducing the data processing pressure of the CPU, but also avoiding overlong time delay caused by the network card processing metadata.

The embodiment shown in fig. 3 is that the network card determines whether the metadata processing task fails to be executed in the process of executing the metadata processing task. Optionally, in this embodiment of the application, the network card may also pre-judge an execution result of the metadata processing task before the metadata processing task is executed. And if the execution result of the metadata processing task is judged to be failure, the CPU is requested to process the metadata processing task.

The embodiment shown in fig. 4 is used to explain this anticipation. As shown in fig. 4, the metadata processing method in the storage device includes the following steps.

Step 401: a network card in a storage device receives a plurality of IO requests, where the IO requests include a read data request or a write data request.

The explanation of step 401 may refer to the explanation of step 301 in the embodiment of fig. 3, and is not described herein again.

Step 402: when the network card determines that the number of the plurality of IO requests exceeds a set number threshold, the network card requests a CPU in the storage device to process a metadata processing task corresponding to at least one IO request in the plurality of IO requests.

When the number of the received multiple IO requests exceeds the number threshold, the amount of data to be processed for executing the metadata processing tasks corresponding to the IO requests is large, so that the time required for executing the metadata processing tasks corresponding to the IO requests is also long, and therefore, in order to avoid the problem that the time delay for directly executing the metadata processing tasks corresponding to the IO requests by the network card is large, the network card can request the CPU to process the IO requests. That is, the network card directly judges whether the metadata processing task is executed by itself or executed by the CPU by the number of received IO requests.

The implementation manner of the network card requesting the CPU to execute the metadata processing task corresponding to the at least one IO request may be: the network card sends a task execution request to the CPU. The task execution request instructs the CPU to execute the metadata processing task corresponding to the at least one IO request. When the CPU receives the task execution request, the metadata processing task corresponding to the at least one IO request can be executed.

In one possible implementation, the network card may request the CPU to process all IO requests of the plurality of IO requests in step 402. At this time, at least one IO request in step 402 is all IO requests in the plurality of IO requests in step 401.

In a possible implementation manner, in step 402, the network card may request the CPU to process one part of the IO requests, and the network card itself processes another part of the IO requests. At this point, at least one IO request in step 402 is a subset of the plurality of IO requests. In this implementation manner, the network card executes the metadata processing tasks corresponding to the remaining IO requests except for at least one IO request in the plurality of IO requests in step 401.

When the network card requests the CPU to process a part of the IO requests in the plurality of IO requests, the network card may determine the number of IO requests that can be processed by the network card based on the current available memory of the network card, and then use the other IO requests beyond the number as the at least one IO request. For the available memory of the network card, reference may be made to the related explanation in the embodiment shown in fig. 3, and details are not described here.

It should be noted that the embodiment shown in fig. 4 may be applied to a read data request, and may also be applied to a write data request. In addition, the explanation of the metadata processing task in the embodiment shown in fig. 4 may refer to the explanation of relevant contents in the embodiment of fig. 3, and is not described herein again.

Therefore, in the embodiment of the present application, the execution result of the metadata execution task to be currently processed may be pre-judged based on the number of the IO requests, when the number of the IO requests exceeds a certain number, the pre-judging network card cannot successfully execute the metadata processing task, and at this time, the CPU is requested to execute the metadata processing task. The pre-judging mode not only improves the flexibility of metadata processing, but also can realize the self-adaptive dynamic execution of the metadata processing task by the network card and the CPU. Therefore, partial metadata processing tasks can be adaptively unloaded from the CPU to the network card, the data processing pressure of the CPU can be reduced, and overlong time delay caused by the fact that the network card processes the metadata can be avoided, so that the data reading and writing efficiency of the storage device is improved.

In addition, the embodiment shown in fig. 4 is to prejudge the execution result of the metadata execution task to be currently processed based on the number of IO requests. Optionally, for any IO request, the network card may also estimate a time length for executing a metadata processing task corresponding to the IO request; and if the time length exceeds the time delay threshold value, the network card judges that the execution result of the metadata processing task is failure. And in turn requests the CPU to perform the metadata processing task.

The time length for executing the metadata processing task is the time length which is estimated in advance by the network card and is required for executing the metadata processing task. That is, the network card estimates the time length before actually executing the metadata processing task, and determines whether to execute the metadata processing task by itself or to execute the metadata processing task by the CPU based on the estimated time length. The network card can acquire the length from the head of the task queue to the IO request in the task queue to which the IO request is added and the average processing time of each IO request, and estimate the time required for processing the IO request according to the two pieces of information.

Alternatively, the network card may also pre-determine whether to execute the metadata processing task by itself or execute the metadata processing task by the CPU in other manners, which are not illustrated herein.

The embodiments shown in fig. 3 and fig. 4 can be applied to a scenario in which the IO request is a data write request, or a scenario in which the IO request is a data read request. The following explains and explains the metadata processing method provided in the embodiment again by taking the IO request as a write data request and a read data request as examples respectively.

Fig. 5 is a schematic flowchart of a metadata processing method in a data writing scenario according to an embodiment of the present application. As shown in fig. 5, the method includes the following steps.

Step 501: and the client sends a data writing request to the network card of the storage node through the network card, wherein the data writing request indicates to store data and updates the metadata corresponding to the data in the index database.

Step 502: and the network card of the storage node puts the data writing request into a task queue controlled by the network card, and the network card executes the data writing request in the task queue according to the sequence in the task queue.

When the IO request is a write data request, the network card stores the data to be stored in the write data request through step 503, and executes the metadata processing task based on the physical address of the data to be stored through step 504. The physical address of the data to be stored may be obtained from the storage node by the client in advance, and the embodiment of the present application does not limit how the client determines the physical address of the data to be stored.

Step 503: and the network card writes data to be stored.

In this embodiment of the application, the network card may write data to be stored in a write-ahead loading (WAL) manner.

The log before writing is a technology for ensuring data integrity, and in brief, before a data writing request is actually executed, the fact that a data writing operation is executed is recorded, the record is the log, and data is cached in a memory. All subsequent modification records, if any, are also saved in the log. The log is stored in the hard disk in a persistent mode, and after the log is stored successfully, the operation of storing the data in the memory into the hard disk is executed. In the process of storage, no matter which operation step has errors, the error can be played back once according to the log stored in the hard disk, and a correct result is obtained. In practical application, because the data volume of the data to be stored carried by the data writing request is usually large, the operation is complicated, the data writing is not necessarily sequential, and if each operation needs to wait for the result to be written into the hard disk to execute the next operation, the efficiency is very low. And the method of the log before writing can improve the writing efficiency because the data volume of the log is small and the log is written in sequence. Typically, logs include, but are not limited to: the data to be stored (the format of the data to be stored contained in the log is different from that of the data to be stored described above), the time for receiving the data to be stored, the corresponding operation type (for example, a write instruction or a read instruction), the access address of the data to be stored, and the like.

And executing a data writing request according to a log before writing mode, writing the log firstly, then writing data, and accumulating the data stored in the memory in the storage device to a certain degree and then brushing the data to the hard disk once, so that the times of writing the data into the hard disk of the storage node can be reduced, and the network resource is saved. In addition, since the log is already persistently stored in the hard disk before the data is stored in the hard disk, even if a failure occurs during the storage of the data, the data can be recovered by playing back the log.

It should be noted that the WAL method is only one possible implementation manner for the network card to write in the data to be stored, and the embodiment of the present application does not limit the implementation manner for the network card to write in the data to be stored.

In addition, the data to be stored may be stored in the aforementioned global memory pool and persisted through a non-volatile medium. The mapping relation between the physical address and the logical address included in the metadata of the data to be stored may be stored in a key-value pair manner. At this time, the content encoded in the value in the key-value pair may be a physical address of the global memory pool, so that the subsequent network card can bypass the CPU to directly read the physical address of the data when processing the read data request, thereby improving the efficiency of reading the data.

Step 504: the network card judges the memory resources required by updating the index database. And if the available memory of the network card can meet the memory resource required by the updating process, the network card executes the metadata processing task.

The above-mentioned network card executing metadata processing task means: the network card stores the metadata of the data to the index database to complete the updating of the index database. At this time, the write data request sent by the client is successfully processed.

Under the condition that the index libraries comprise a first index library and a second index library, the network card executes the metadata processing task specifically comprising the following three implementation modes:

in a first possible implementation manner, if the metadata of the hot data stored in the first index database and the metadata of the non-hot data other than the hot data stored in the second index database are stored, the network card executing the metadata processing task means that the network card only updates the first index database. And after the CPU determines that the data is non-hotspot data based on a data elimination algorithm, the metadata of the data can be migrated to a second index library, and the metadata in the first index library is deleted.

In a second possible implementation manner, if the metadata of the hotspot data stored in the first index repository and the metadata of the hotspot data and the non-hotspot data stored in the second index repository are, the network card executing the metadata processing task may mean that only the first index repository is updated, the CPU background completes the storage of the metadata in the second index repository at the same time to complete the update of the second index repository, and the metadata in the first index repository is not deleted after the metadata is stored in the second index repository.

In a third possible implementation manner, if the metadata of the hotspot data stored in the first index database and the metadata of the hotspot data and the non-hotspot data stored in the second index database are stored, the network card executing the metadata processing task may also mean that the network card updates the first index database and the second index database at the same time.

In addition, the implementation manner of the network card determining whether the available memory is sufficient to provide the memory resource required for executing the metadata processing task may be: the network card determines the metadata which needs to be stored in the index database, and then determines whether the memory needs to be reallocated to the index database to store the metadata. And if the memory needs to be reallocated to the index base to store the metadata, determining that the memory of the network card is insufficient to provide the required memory resources. If the network card can store the metadata in the index database without reallocating the memory to the index database, the memory of the network card is determined to be sufficient to provide the required memory resources.

Step 505: if the process of updating the index library is complex, so that the available memory of the network card cannot meet the memory resource required by the updating process, the network card gives the metadata processing task to the CPU for processing, and the CPU finishes the updating of the metadata in the index library. And simultaneously, the network card marks the task related to the data in the task queue as being processed by the CPU, and then the arrived request related to the same data is queued in the task queue and is not processed for the moment until the CPU finishes processing, and then the arrived update request related to the same data does not start processing. However, the network card still processes the IO request through the above steps 501 to 504 without waiting for the IO request of other data that is not processed by the CPU.

In step 505, the completion of the updating of the metadata in the index repository by the CPU may refer to: the CPU executes the entire metadata processing task, that is, the CPU completes storing the metadata of the data to be stored in the index repository to complete updating of the index repository.

At this time, in the case that the index library includes the first index library and the second index library, the implementation manner of the CPU executing the metadata processing task may refer to the implementation manner of the network card executing the metadata processing task in step 505, and is not described herein again.

Optionally, the completion of the updating of the metadata in the index repository by the CPU in step 505 may refer to: and the CPU completes the tasks left in the process of executing the metadata processing task by the network card. And will not be described in detail herein.

Step 506: the CPU processes the complex index base updating operation to complete the updating of the index base. At this time, the write data request sent by the client is successfully processed. In addition, the network card also marks the tasks related to the data in the task queue as being completed by the CPU processing.

In summary, in the embodiment of the present application, in order to relieve the pressure of the CPU for processing data, when the network card of the storage node receives a write data request, the network card does not directly transfer the write data request to the CPU, but the network card pre-judges the execution result of the metadata processing task corresponding to the write data request. And if the execution result is failure, requesting the CPU to execute the metadata processing task corresponding to the write request. Therefore, the metadata processing task corresponding to the write request can be prevented from being executed by the network card or the CPU. Therefore, the embodiment of the application provides a method for dynamically executing metadata processing tasks in a self-adaptive manner by using a network card and a CPU. Therefore, partial metadata processing tasks can be adaptively unloaded from the CPU to the network card, the data processing pressure of the CPU can be relieved, and overlong time delay caused by the fact that the network card processes the metadata can be avoided.

It should be noted that, in the embodiment shown in fig. 5, the network card determines the execution result of the metadata processing task corresponding to the write data request based on the memory resource as an example. Optionally, the network card may also determine the execution result of the metadata processing task based on other implementation manners, and the specific implementation manner refers to the embodiment shown in fig. 3, which is not described in detail herein.

Fig. 6 is a schematic flowchart of a metadata processing method in a read data scenario according to an embodiment of the present application. As shown in fig. 6, the method includes the following steps.

Step 601: the client sends a data reading request to the network card of the storage node through the network card, wherein the data reading request indicates to read data on a certain logical address or indicates to read data on a large-range logical address.

When the IO request is a read data request, in addition to determining whether the execution result of the metadata processing task fails through memory resources or time delay in step 302 in the embodiment shown in fig. 3, the execution result of the metadata processing task may be determined based on the address length of the logical address carried in the read data request. For example, when the address length of the logical address carried by the read data request exceeds the length threshold, the read data request may be processed by the CPU in order to avoid that the network card needs to interact for multiple times to acquire the data to be read. That is, when the address length of the logical address carried by the read data request exceeds the length threshold, the network card determines that the execution of the data processing task corresponding to the read data request fails. If the address length of the logical address carried by the read data request is lower than the length threshold, the network card can execute the metadata processing task, and then the network card continues to judge whether to request the CPU to execute the metadata processing task according to the actual metadata processing execution condition.

The following steps 602 and 603 are explained for the above.

Step 602: and if the address length of the logical address carried by the read data request is lower than the length threshold, executing the metadata processing task by the network card.

In the case where the index library includes the aforementioned first index library and second index library, based on the aforementioned explanation of the first index library and second index library, it can be seen that the data amount of the metadata stored in the second index library is larger than the data amount of the metadata in the first index library, and therefore the amount of resources consumed to acquire the metadata from the second index library is larger than the amount of resources consumed to acquire the metadata from the first index library. Therefore, the implementation manner of executing the metadata processing task by the network card may be as follows: the network card acquires a physical address corresponding to the logical address carried by the read data request from the first index library; and if the physical address corresponding to the logical address carried by the read data request is not inquired in the first index database, determining that the execution of the metadata processing task fails. At this point the CPU is requested to perform a metadata processing task, via step 603.

The length threshold may be a difference between two adjacent logical addresses, or may be a specified length, which is not limited in this embodiment of the present application.

For example, when the read data request indicates to read data in a certain logical address area, the network card may first execute the metadata processing task corresponding to the read data request in the first index repository. If the metadata processing task fails to execute, the CPU is requested to execute the metadata processing task in the second index repository, via step 603 described below.

Step 603: when the network card receives a data reading request, if the address length of a logical address carried by the data reading request exceeds a length threshold, the execution of the metadata processing task is determined to fail. In this case, the network card requests the CPU to execute the metadata processing task. Alternatively, if the network card does not successfully execute the metadata processing task in the first index repository in step 602, the network card also requests the CPU to execute the metadata processing task.

That is, in step 603, there are two scenarios that require the CPU to perform the metadata processing task. The first scenario is: the address length of the logical address carried by the read data request is lower than the length threshold, but the network card does not inquire the metadata of the data to be read in the first index database. The second scenario is: the address length of the logical address carried by the read data request exceeds a length threshold.

In the first scenario, since the network card has already queried the metadata in the first index repository, and the query result is that the metadata is not queried, the CPU executes the metadata processing task by: and the CPU acquires the metadata of the data to be read from the second index database.

In the second scenario, if the metadata of the hotspot data stored in the first index repository and the metadata of other data than the hotspot data stored in the second index repository are stored, the execution of the metadata processing task by the CPU may refer to: and the CPU acquires the metadata of the data to be read from the first index database and the second index database.

Furthermore, in the second scenario, if the metadata of the hotspot data stored in the first index repository and the metadata including the hotspot data and the non-hotspot data are stored in the second index repository, the CPU may refer to: and the CPU acquires the metadata of the data to be read from the second index database. The technical effect of doing so is: the CPU can be prevented from repeatedly querying the metadata from the first index repository.

Step 604: and the CPU executes the metadata processing task to finish the query work on the index database, and the read data request sent by the client is successfully processed.

In step 604, the CPU may perform the query work of the index library through different scenarios shown in step 603, respectively.

Step 605: if the CPU can not inquire the physical address corresponding to the logical address carried by the data request on the index database, continuously inquiring the filter, if the filter indicates that the logical address is not in the storage layer with lower performance, returning a data reading failure message, otherwise, continuously inquiring the data corresponding to the logical address in the storage layer with low performance.

To further improve the efficiency of reading data, the storage system may store portions of data that are rarely accessed to a lower performance storage layer (tier). The storage layer with lower performance may be part or all of the storage space in the hard disk shown in fig. 1 or fig. 2. The index store does not have metadata for the data in the lower performing storage tier. Instead, a filter (filter) is configured for the lower-performance storage tier, and the filter functions to query whether certain data is stored in the lower-performance storage tier.

Therefore, in step 605, if the CPU does not query the physical address corresponding to the logical address carried in the read data request in the index database, the query is continued through the filter. And if the data to be read is inquired in the storage layer with lower performance through the filter, reading the data from the storage layer with lower performance. And if the data to be read cannot be inquired through the filter, the CPU returns a data reading failure message to the client.

In addition, in the embodiment shown in fig. 6, if the address length of the logical address carried in the read data request is lower than the length threshold, the network card may execute the metadata processing task based on the first index library, and then the network card may determine whether to request the CPU to execute the metadata processing task in the second index library based on the execution result in the first index library. Optionally, the network card may also directly execute the metadata processing task based on the first index library by the network card without considering the address length of the logical address carried in the read data request, and then the network card determines whether to request the CPU to execute the metadata processing task in the second index library according to the execution result of the metadata processing task. And will not be described in detail herein.

Fig. 7 is a metadata processing apparatus, which is disposed in a network card of a storage system according to an embodiment of the present application. As shown in fig. 7, the apparatus 700 includes:

the receiving module 701 is configured to receive an IO request, where the IO request includes a read data request or a write data request. The specific implementation process refers to step 301 in the embodiment of fig. 3.

And the execution module 702 is configured to execute a metadata processing task corresponding to the IO request. The specific implementation process refers to step 302 in the embodiment of fig. 3.

A requesting module 703, configured to request a central processing unit CPU in the storage device to execute the metadata processing task if it is determined that the metadata processing task fails to be executed. The specific implementation process refers to step 303 in the embodiment of fig. 3.

It should be noted that the network card shown in fig. 1 or fig. 2 may include a communication interface for communication and a component having a processing function, such as a processor (not shown in fig. 1 or fig. 2). At this time, the receiving module 701 and the requesting module 703 in fig. 7 may be implemented by a communication interface on the network card in fig. 1 or fig. 2. The execution module 702 in fig. 7 may be implemented by a component having a processing function on the network card in fig. 1 or fig. 2. Optionally, the modules shown in fig. 7 may also be implemented by components having a processing function on a network card, which is not specifically limited in this embodiment of the application.

Optionally, the network card includes a memory, and the condition that the metadata processing task fails to be executed includes: the memory of the network card is not enough to provide the memory resource required by the metadata processing task.

Optionally, when the IO request is a read data request, the case that the metadata processing task fails to execute includes: the address length of the logical address carried by the read data request exceeds a length threshold.

Optionally, when the IO request is a read data request, the storage device includes a first index repository and a second index repository, a data amount of metadata stored in the first index repository is smaller than a data amount of metadata stored in the second index repository, and the case that the execution of the metadata processing task fails includes: the network card does not acquire metadata corresponding to the read data request in the first index database; the network card requesting a CPU in the storage device to execute the metadata processing task comprises the following steps: and the network card instructs the CPU to acquire the metadata corresponding to the read data request from the second index database.

Optionally, the first index repository includes metadata of hotspot data, and the second index repository includes metadata of non-hotspot data.

Optionally, when the IO request is a write data request, the metadata processing task stores metadata corresponding to the write data request.

Optionally, when the IO request is a read data request, the metadata processing task is to obtain metadata corresponding to the read data request.

Optionally, the metadata corresponding to the IO request is used to indicate a storage location of data corresponding to the IO request.

Optionally, the metadata comprises key-value pairs, a key of a key-value pair indicating a logical address of the data corresponding to the IO request, a value of a key-value pair indicating a physical address of the data corresponding to the IO request.

Optionally, the metadata includes key-value pairs, a key of a key-value pair indicating a logical address of data corresponding to the IO request, a value of a key-value pair indicating fingerprint information of the data corresponding to the IO request.

Optionally, the storage device is a storage array, or a storage node in a distributed storage system.

In the embodiment of the present application, in order to reduce the pressure of the CPU for processing data, when the network card receives an IO request, the network card does not directly transfer the IO request to the CPU, but the network card executes the metadata processing task first. And if the execution fails, requesting the CPU to execute the metadata processing task corresponding to the IO request. Therefore, the metadata processing task corresponding to the IO request can be prevented from being executed by the network card or the CPU. Therefore, the storage device can adaptively unload part of metadata processing tasks from the CPU to the network card, thereby not only reducing the data processing pressure of the CPU, but also avoiding overlong time delay caused by the network card processing metadata.

It should be noted that: in the metadata processing apparatus provided in the above embodiment, when processing metadata, only the division of the above functional modules is exemplified, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the metadata processing apparatus provided in the foregoing embodiment and the metadata processing method embodiment belong to the same concept, and specific implementation processes thereof are described in the method embodiment and are not described herein again.

Fig. 8 is a metadata processing apparatus, which is disposed in a network card of a storage system according to an embodiment of the present application. As shown in fig. 8, the apparatus 800 includes:

the receiving module 801 is configured to receive a plurality of input/output IO requests, where the IO requests include a read data request or a write data request. The specific implementation manner may refer to step 401 in the embodiment of fig. 4.

The request module 802 is configured to request a central processing unit in the storage device to process a metadata processing task corresponding to at least one IO request of the multiple IO requests when it is determined that the number of IO requests exceeds a set number threshold. The specific implementation manner may refer to step 402 in the embodiment of fig. 4.

It should be noted that the network card shown in fig. 1 or fig. 2 may include a component such as a processor (not shown in fig. 1 or fig. 2) for a communication interface and having a processing function. At this time, the receiving module 801 and the requesting module 80 in fig. 8 may be implemented by a communication interface on the network card in fig. 1 or fig. 2. Alternatively, the processing function may be implemented by other components with processing functions on the network card. This is not particularly limited in the embodiments of the present application.

Optionally, the at least one IO request is all IO requests of the plurality of IO requests.

Optionally, the at least one IO request is a subset of the plurality of IO requests, the apparatus further comprising:

and the execution module is used for executing the metadata processing tasks corresponding to the rest IO requests except at least one IO request in the plurality of IO requests.

In the embodiment of the present application, the execution result of the metadata execution task to be currently processed may be pre-judged based on the number of IO requests, and when the number of IO requests exceeds a certain number, the pre-judging network card cannot successfully execute the metadata processing task, and at this time, the CPU may be requested to execute the metadata processing task. Not only the flexibility of metadata processing is improved. And the network card and the CPU can adaptively and dynamically execute the metadata processing task. Therefore, partial metadata processing tasks can be adaptively unloaded from the CPU to the network card, the data processing pressure of the CPU can be reduced, and overlong time delay caused by the fact that the network card processes the metadata can be avoided, so that the data reading and writing efficiency of the storage device is improved.

In the above embodiments, the implementation may be wholly or partly realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Versatile Disk (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above-mentioned embodiments are provided not to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of metadata processing in a storage device, the method comprising:

a network card in the storage device receives an input/output (IO) request, wherein the IO request comprises a data reading request or a data writing request;

the network card executes a metadata processing task corresponding to the IO request;

and under the condition that the network card determines that the metadata processing task fails to be executed, the network card requests a Central Processing Unit (CPU) in the storage device to execute the metadata processing task.

2. The method of claim 1, wherein the network card includes a memory, and wherein the failure to execute the metadata processing task comprises: the memory of the network card is not enough to provide the memory resource required by the metadata processing task.

3. The method of claim 1 or 2, wherein when the IO request is a read data request, the case of the metadata processing task failing to execute comprises: and the address length of the logic address carried by the read data request exceeds a length threshold value.

4. The method of any of claims 1-3, wherein the storage device includes a first index repository and a second index repository, wherein an amount of metadata stored by the first index repository is less than an amount of metadata stored by the second index repository, and wherein when the IO request is a read data request, the failure of the metadata processing task to execute comprises: the network card does not acquire metadata corresponding to the read data request in the first index database; the network card requesting the CPU in the storage device to execute the metadata processing task includes: and the network card instructs the CPU to acquire the metadata corresponding to the read data request from the second index database.

5. The method of claim 4, wherein the first indexed repository includes metadata for hotspot data and the second indexed repository includes metadata for non-hotspot data.

6. The method of claim 1 or 2, wherein when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

7. The method of any of claims 1-5, wherein when the IO request is a read data request, the metadata processing task is to obtain metadata corresponding to the read data request.

8. The method of any of claims 1-7, wherein the metadata corresponding to the IO request is to indicate a storage location of data corresponding to the IO request.

9. The method of claim 8, wherein the metadata comprises a key-value pair, a key of the key-value pair to indicate a logical address of data corresponding to the IO request, a value of the key-value pair to indicate a physical address of data corresponding to the IO request.

10. The method of claim 8, wherein the metadata comprises key-value pairs, a key of the key-value pair to indicate a logical address of data corresponding to the IO request, a value of the key-value pair to indicate fingerprint information of the data corresponding to the IO request.

11. The method of any of claims 1-10, wherein the storage device is a storage array or a storage node in a distributed storage system.

12. A method of metadata processing in a storage device, the method comprising:

a network card in the storage device receives a plurality of input/output (IO) requests, wherein the IO requests comprise data reading requests or data writing requests;

and when the network card determines that the number of the IO requests exceeds a set number threshold, the network card requests a Central Processing Unit (CPU) in the storage device to process a metadata processing task corresponding to at least one IO request in the plurality of IO requests.

13. The method of claim 12, wherein the at least one IO request is all IO requests of the plurality of IO requests.

14. The method of claim 12, wherein the at least one IO request is a subset of the plurality of IO requests, the method further comprising:

and the network card executes metadata processing tasks corresponding to the rest IO requests except the at least one IO request in the plurality of IO requests.

15. The method of any of claims 12-14, wherein when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

16. The method of any of claims 12-14, wherein when the IO request is a read data request, the metadata processing task is to obtain metadata corresponding to the read data request.

17. The method of any of claims 12-16, wherein the metadata corresponding to the IO request is to indicate a storage location of data corresponding to the IO request.

18. The method of claim 17, wherein the metadata comprises a key-value pair, a key of the key-value pair indicating a logical address of data corresponding to the IO request, a value of the key-value pair indicating a physical address of data corresponding to the IO request.

19. The method of claim 17, wherein the metadata comprises key-value pairs, a key of the key-value pair to indicate a logical address of data corresponding to the IO request, a value of the key-value pair to indicate fingerprint information of the data corresponding to the IO request.

20. A network card located in a storage device, the network card comprising a communication interface for communicating with a central processor in the storage device and a processor for implementing the method of any one of claims 1-11 or 12-19.

21. A storage device is characterized in that the storage device comprises a network card and a Central Processing Unit (CPU);

the network card is used for receiving an input/output (IO) request, executing a metadata processing task corresponding to the IO request, and requesting the CPU to execute the metadata processing task under the condition that the network card determines that the metadata processing task fails to be executed, wherein the IO request comprises a data reading request or a data writing request;

and the CPU is used for executing the metadata processing task based on the request of the network card.

22. The storage device of claim 21, wherein the network card includes a memory, and wherein the failure to execute the metadata processing task comprises: the memory of the network card is not enough to provide the memory resource required by the metadata processing task.

23. The storage device of claim 21 or 22, wherein when the IO request is a read data request, the case where the metadata processing task fails to execute comprises: and the address length of the logic address carried by the read data request exceeds a length threshold value.

24. The storage device according to any one of claims 21 to 23, wherein the storage device comprises a first index repository and a second index repository, the first index repository stores metadata with a smaller amount of data than the second index repository, and when the IO request is a read data request, the case where the metadata processing task fails to execute comprises: the network card does not acquire metadata corresponding to the read data request in the first index database; the network card requesting the CPU in the storage device to execute the metadata processing task includes: and the network card instructs the CPU to acquire the metadata corresponding to the read data request from the second index database.

25. The storage device of claim 24, wherein the first index repository includes metadata for hotspot data and the second index repository includes metadata for non-hotspot data.

26. The storage device of claim 21 or 22, wherein when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

27. The storage device of any of claims 21-25, wherein when the IO request is a read data request, the metadata processing task is to obtain metadata corresponding to the read data request.

28. The storage device of any of claims 21-27, wherein the metadata corresponding to the IO request is to indicate a storage location of data corresponding to the IO request.

29. The storage device of claim 28, wherein the metadata comprises a key-value pair, a key of the key-value pair to indicate a logical address of data corresponding to the IO request, a value of the key-value pair to indicate a physical address of data corresponding to the IO request.

30. The storage device of claim 28, wherein the metadata comprises a key-value pair, a key of the key-value pair to indicate a logical address of data corresponding to the IO request, a value of the key-value pair to indicate fingerprint information for the data corresponding to the IO request.

31. The storage device of any of claims 21-30, wherein the storage device is a storage array or a storage node in a distributed storage system.

32. A storage device is characterized in that the storage device comprises a network card and a Central Processing Unit (CPU);

the network card is used for receiving a plurality of input/output (IO) requests, and requesting the CPU to process a metadata processing task corresponding to at least one IO request in the plurality of IO requests when the number of the IO requests is determined to exceed a set number threshold;

and the CPU is used for executing a metadata processing task corresponding to at least one IO request based on the request of the network card.

33. The storage device of claim 32, wherein the at least one IO request is all IO requests of the plurality of IO requests.

34. The storage device of claim 32, wherein the at least one IO request is a subset of the plurality of IO requests; the network card is further configured to execute metadata processing tasks corresponding to the remaining IO requests, except the at least one IO request, in the plurality of IO requests.

35. The storage device of any of claims 32-34, wherein when the IO request is a write data request, the metadata processing task is to store metadata corresponding to the write data request.

36. The storage device of any of claims 32-34, wherein when the IO request is a read data request, the metadata processing task is to obtain metadata corresponding to the read data request.

37. The storage device of any of claims 32-36, wherein the metadata corresponding to the IO request is to indicate a storage location of data corresponding to the IO request.

38. The storage device of claim 37, wherein the metadata comprises a key-value pair, a key of the key-value pair indicating a logical address of data corresponding to the IO request, a value of the key-value pair indicating a physical address of data corresponding to the IO request.

39. The storage device of claim 37, wherein the metadata comprises a key-value pair, a key of the key-value pair to indicate a logical address of data corresponding to the IO request, a value of the key-value pair to indicate fingerprint information for the data corresponding to the IO request.

40. A computer-readable storage medium having instructions stored therein, which when executed on a computer, cause the computer to perform the method of any of claims 1-11 or claims 12-19.