WO2024093958A1

WO2024093958A1 - Access method and apparatus for storage pool

Info

Publication number: WO2024093958A1
Application number: PCT/CN2023/128077
Authority: WO
Inventors: 陈雷明; 何锐; 刘宸睿
Original assignee: 华为云计算技术有限公司
Priority date: 2022-11-01
Filing date: 2023-10-31
Publication date: 2024-05-10

Abstract

Disclosed in embodiments of the present application is an access method for a storage pool, for use in improving the flexibility of accessing a storage pool by a computing device cluster. The method in the embodiments of the present application comprises: a computing device cluster provides an application program interface, wherein the application program interface is used by a computing node in the computing device cluster to access a storage pool by means of the application program interface, and a software development kit (SDK) of the application program interface is deployed on a virtual machine or a container of the computing device cluster; the computing device cluster sends a storage pool access request to a data forwarding agent module by calling the application program interface, wherein the data forwarding agent module is used for processing a data forwarding process of the computing device cluster; and the computing device cluster sends the storage pool access request to a storage node in the storage pool by means of the data forwarding agent module.

Description

A storage pool access method and device

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on November 1, 2022, with application number 202211366811.4, and invention name “A storage pool access method and device”, and the priority of the Chinese patent application filed with the State Intellectual Property Office on March 30, 2023, with application number 202310330666.2, and invention name “A storage pool access method and device”, the entire contents of which are incorporated by reference in this application.

Technical Field

Embodiments of the present application relate to the computer field, and in particular, to a method and device for accessing a storage pool.

Background technique

With the development of data access service technology, more and more computing device clusters use storage pools to access data. In the process of accessing the storage pool, the current computing device cluster needs the engine layer of the data service to realize the storage and data extraction of the computing device cluster.

Currently, some engine layers of computing device clusters are deployed based on physical machines, that is, computing devices access storage pools through physical network interfaces. When the computing device cluster needs to apply for new storage resources from the storage pool, a new physical network interface needs to be added, resulting in the computing device cluster being unable to flexibly apply for storage resources.

Although the engine layer of some computing device clusters can be deployed based on virtual machines, the computing device cluster accesses the storage resources of the storage pool through cloud disks. The expansion of the computing device cluster requires the simultaneous addition of cloud disks, and different computing nodes can only access their corresponding cloud disks, resulting in insufficient flexibility in the computing device cluster's access to the storage pool and poor expansion elasticity.

Summary of the invention

Embodiments of the present application provide a method and device for accessing a storage pool, which are used to improve the flexibility of accessing a storage pool.

The first aspect of the embodiment of the present application is a method for accessing a storage pool. The method can be executed by a computing device, or by a component of a computing device cluster, such as a processor, chip or chip system of a computing device cluster, or by a logic module or software that can realize all or part of the functions of a computing device cluster. Taking the execution of a computing device cluster as an example, the method provided in the first aspect includes the following steps: the computing device cluster provides an application program interface, the application program interface is used for computing nodes in the computing device cluster to access the storage pool through the application program interface, and the software toolkit SDK of the application program interface is deployed on a virtual machine or container of the computing device cluster. The computing device cluster sends a storage pool access request to a data forwarding proxy module by calling the application program interface, and the data forwarding proxy module is used to process the data forwarding process of the computing device cluster. The computing device cluster sends a storage pool access request to the storage node in the storage pool through the data forwarding proxy module.

In an embodiment of the present application, the virtual machine or container of the computing node in the computing device cluster can deploy a software toolkit of the application program interface, so that the virtual machine or container of the computing node can access any storage node in the storage pool based on the application program interface. Compared with the computing device cluster accessing the storage pool through a physical network interface or accessing the storage pool in a cloud disk manner, the flexibility of the computing device cluster in accessing individual storage nodes in the storage pool is improved.

In one possible implementation, the computing device cluster calls the application program interface and sends a storage pool access request to the data forwarding agent module of the data processing unit through a direct memory access DMA controller, without the participation of the central processing unit of the computing node, thereby realizing direct transmission of the storage pool access request.

In the embodiment of the present application, the computing nodes of the computing device cluster can send storage pool access requests to the data forwarding proxy module through direct memory access (DMA), thereby improving the communication efficiency between the computing nodes and the data processing unit and improving the input and output (IO) performance of the computing device cluster.

In one possible implementation, when the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling an application program interface, the computing device cluster sends a log data write request to the data forwarding proxy module by calling an application program interface. The log data write request carries a log identifier (log ID), and the log identifier is used to indicate the data write location in the storage pool of the data forwarding proxy module.

In the embodiment of the present application, the virtual machine of the computing device can send a log data write request to the data forwarding proxy module through the application program interface, thereby improving the flexibility of the computing device cluster in writing log data in the storage pool and improving the log data of the computing device cluster in the storage pool. Data writing efficiency.

In one possible implementation, in a process in which a computing device cluster sends a storage pool access request to a data forwarding proxy module by calling an application program interface, the computing device cluster sends a log data read request to the data forwarding proxy module by calling an application program interface, and the log data read request carries a log identifier, which is used to indicate the data reading position of the data forwarding proxy module in the storage pool.

The virtual machine of the computing device in the embodiment of the present application can send a log data read request to the data forwarding proxy module through the application program interface, thereby improving the flexibility of the computing device cluster in reading log data from the storage pool and improving the reading efficiency of the computing device cluster in reading log data from the storage pool.

In one possible implementation, the computing device cluster determines the partition identifier (pt ID) based on the log identifier. The computing device cluster queries the routing table according to the partition identifier to determine one or more storage node identifiers corresponding to the partition identifier. Specifically, the computing device cluster determines the partition identifier based on the log identifier and the number of partitions, queries the routing table based on the partition identifier, and determines one or more storage node identifiers (disk ID) corresponding to the partition identifier.

In the embodiment of the present application, the computing device can determine the storage node identifier that the computing device cluster wants to access based on the log identifier carried in the storage pool access request, thereby enabling the computing node to accurately determine the location of the storage node to be accessed, thereby improving the feasibility of the computing device cluster accessing the storage nodes in the storage pool.

In a possible implementation, the data forwarding proxy module is deployed in the data processing unit DPU chip. The computing node can offload the data forwarding process to the data forwarding proxy module of the data processing unit, and the data forwarding proxy module in the data processing unit implements the forwarding of the storage pool access message.

The computing node can offload the data forwarding process to the data forwarding proxy module of the data processing unit for execution, thereby improving the processing efficiency of the computing device cluster for computing tasks and also improving the input and output IO performance of the computing device cluster.

In one possible implementation, the application program interface includes a log interface, and the virtual machine or container of the computing node can call the log interface to write log data to the storage pool, or the virtual machine of the computing node can call the log interface to read log data from the storage node of the storage pool.

In the embodiment of the present application, the application program interface can be multiple types of application program interfaces, wherein the application program interface includes a log interface. The virtual machine or container of the computing node can call the log interface to read log data in the storage node of the storage pool, thereby improving the reading and writing efficiency of the computing device cluster for log data.

The second aspect of the embodiment of the present application provides a storage pool access device, the device includes a transceiver unit and a processing unit. Among them, the processing unit is used to provide an application program interface, the application program interface is used for computing nodes in a computing device cluster to access the storage pool through the application program interface, and the software toolkit SDK of the application program interface is deployed on a virtual machine or container of the computing device cluster. The transceiver unit is used to send a storage pool access request to a data forwarding proxy module by calling the application program interface, and the data forwarding proxy module is used to process the data forwarding process of the computing device cluster. The transceiver unit is also used to send a storage pool access request to a storage node in the storage pool through the data forwarding proxy module.

In a possible implementation manner, the transceiver unit is specifically configured to call an application program interface and send a storage pool access request to the forwarding agent module through a direct memory access DMA controller.

In a possible implementation, the transceiver unit is specifically used to send a data write request to the data forwarding proxy module by calling an application program interface, and the data write request carries a log identifier, which is used to indicate the data write location of the data forwarding proxy module in the storage pool.

In a possible implementation, the transceiver unit is specifically used to call an application program interface to send a data read request to the data forwarding proxy module, the data read request carries a log identifier, and the log identifier is used to indicate a data reading position of the data forwarding proxy module in the storage pool.

In a possible implementation, the processing unit is further configured to determine a partition identifier based on the log identifier, query a routing table according to the disk table identifier, and determine one or more storage node identifiers corresponding to the partition identifier.

In a possible implementation, the data forwarding proxy module is deployed on a data processing unit DPU chip.

A third aspect of an embodiment of the present application provides a computing device cluster, which includes one or more computing devices, and the computing device includes a processor, which is coupled to a memory, and the processor is used to store instructions. When the instructions are executed by the processor, the cloud server executes the method described in the first aspect or any possible implementation method of the first aspect.

A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

A fifth aspect of an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, the computer implements the method described in the first aspect or any possible implementation method of the first aspect.

It can be understood that the beneficial effects that can be achieved by any of the computing device clusters, computer-readable media, or computer program products provided above can refer to the beneficial effects in the corresponding methods, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a system architecture of a storage pool access system provided in an embodiment of the present application;

FIG2 is a schematic diagram of a flow chart of a storage pool access method provided in an embodiment of the present application;

FIG3 is a schematic diagram of a storage pool access method provided in an embodiment of the present application;

FIG4 is a schematic diagram of another storage pool access method provided in an embodiment of the present application;

FIG5 is a schematic diagram of another storage pool access method provided in an embodiment of the present application;

FIG6 is a schematic diagram of a storage pool access device provided in an embodiment of the present application;

FIG7 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application.

Detailed ways

Embodiments of the present application provide a method and apparatus for accessing a storage pool, which are used to improve the flexibility of a computing device cluster in accessing a storage pool.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

First, some terms involved in the embodiments of the present application are introduced to facilitate technical personnel in this field to understand the solution.

A virtual machine (VM) is a complete computer system that is simulated by software and has complete hardware system functions and runs in a completely isolated environment. Any work that can be done in a server can be done in a virtual machine. When creating a virtual machine in a server, part of the hard disk and memory capacity of the physical machine is used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent hard disk and operating system, and the user of the virtual machine can operate the virtual machine just like using a server.

Remote dictionary server (Redis) is an open source high-performance key-value database developed in C language. Redis can store data in memory and is often used as a distributed cache to improve the performance of reading or writing data.

A data processing unit (DPU) is a dedicated processor built around data. It supports a variety of infrastructure layer services, such as storage, security, and quality of service. DPU enhances the computing device's processing capabilities for network security and network protocols as well as distributed storage.

The following introduces the system architecture used by the storage pool access method provided in the embodiment of the present application.

Please refer to Figure 1, which is a schematic diagram of the system architecture used by a storage pool access method provided in an embodiment of the present application. In the example shown in Figure 1, the storage pool access system 100 includes a computing device cluster 101 and a storage pool 102, wherein the computing device cluster 101 includes a computing node 1011 and a data processing unit 1012, and the computing node 1011 in the computing device cluster 101 can be one or more computing nodes, each computing node corresponds to a data processing unit 1012, and the storage pool 102 includes one or more storage nodes 1021. The specific functions of each part of the storage pool access system 100 provided by the implementation of the present application are introduced below.

The computing device cluster 101 is used to process the computing tasks of the user and to transmit data to the storage nodes in the storage pool 101. The computing device cluster 101 includes one or more computing nodes 1011. The computing nodes 1011 can be computing devices that constitute the computing device cluster 101. The computing nodes 1011 are, for example, physical servers or desktop computers. At the hardware level, the computing nodes 1011 are provided with processors and memory. At the software level, the computing nodes 1011 are running applications and clients. The client is used to receive data access requests triggered by the application. The data processing unit 1012 interacts with the storage node 102 and sends a data access request to the storage node 102.

One or more virtual machines or containers may also be deployed on the computing node 1011. This application takes the virtual machine as an example. Users can process computing tasks based on the virtual machine and persist the data generated by executing the computing tasks to the storage pool 102. The virtual machine can also provide users with a variety of data services, such as remote dictionary service (remote dictionary server, Redis). The virtual machine can call the application interface to send the data generated by the Redis service to the data processing unit 1012, which is forwarded to the storage pool 102 for data persistence. The virtual machine can also call the application interface to store the data of the storage pool 102 in the memory of the computing node 1011.

The data processing unit 1012 is used to process the interaction process between the computing node 101 and the storage pool 102. The data processing unit 1012 includes a data forwarding agent module, which can forward the storage pool access request sent by the virtual machine in the computing node 101 to the storage node of the storage pool 102. The data processing unit 1012 can be integrated into the computing device as a chip, or it can be an independent device, which is not specifically limited.

The storage pool 102 is used to store data sent by the computing device cluster 101. The storage pool 102 includes one or more storage nodes 1021, such as servers, desktop computers and hard disks. The storage node 1021 includes one or more controllers and one or more hard disks. The hard disk is used to store data, and the hard disk can be a magnetic disk or other types of storage media, such as a solid-state drive. The controller is used to write data to the hard disk or read data from the hard disk according to the data access request sent by the computing node 1011. In the process of reading and writing data, the controller needs to convert the address carried in the storage pool access request into an address that the hard disk can recognize.

A method and device for accessing a storage pool provided in an embodiment of the present application are described below in conjunction with the accompanying drawings.

Please refer to Figure 2, which is a flow chart of a storage pool access method provided by an embodiment of the present application. In the example shown in Figure 2, the storage pool access method provided by an embodiment of the present application includes the following steps:

201. The computing device cluster provides an application program interface, and the application program interface is used for computing nodes in the computing device cluster to access the storage pool through the application program interface.

The computing device cluster provides an application program interface, which is used for computing nodes in the computing device cluster to access the storage pool through the application program interface. The software toolkit SDK of the application program interface is deployed on the virtual machine or container of the computing node in the computing device cluster, and the virtual machine of the computing node can call the application program interface to communicate with the data processing unit in the computing device cluster.

The data service in the embodiment of the present application includes a Redis service, which can cache data in the storage pool. Specifically, the Redis service can call an application program interface in the virtual machine to read the data in the storage pool into the memory, or the Redis service can call an application program interface in the virtual machine to write the data in the memory to the storage pool to achieve data persistence.

The application program interface in the embodiment of the present application includes a log interface, an application program interface for transmitting log data, and the virtual machine of the computing node can call the log interface to write the log data to the storage pool. Among them, the log data is record oriented and append-only data. The log data is not stored in the form of a single byte, but is recorded in the form of an indivisible data block, and the log data can only be appended and cannot be modified.

Please refer to Figure 3, which is a schematic diagram of calling an application program interface in an embodiment of the present application. In the embodiment shown in Figure 3, a software toolkit of a log interface is installed in the virtual machine of a computing node in a computing device cluster, and the software tool can provide a log interface, and the virtual machine accesses the storage pool by calling the log interface. For example, the virtual machine VM1 appends the log data generated by the Redis service to the storage pool by calling the log interface.

202. The computing device cluster sends a storage pool access request to the data forwarding proxy module by calling an application program interface.

The computing device cluster sends a storage pool access request to the data forwarding proxy module of the data processing unit by calling the application program interface, and the data forwarding proxy module is used to process the data forwarding process of the computing device cluster. For example, the virtual machine of the computing node in the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling the log interface. The storage pool access request is used to request to write or read log data in the storage pool, and the data forwarding proxy module can process the forwarding process of the log data.

In one possible implementation, the data forwarding proxy module is deployed in a data processing unit DPU chip, that is, the computing node can offload the data forwarding process to the data processing unit, and the data processing unit implements the forwarding of the storage pool access request. In the embodiment of the present application, the data processing unit DPU chip can be embedded in the network card of the computing device cluster, without specific limitation.

In the embodiment of the present application, the computing node can offload the data forwarding process to the data forwarding proxy module of the data processing unit for execution, thereby improving the processing efficiency of the computing device cluster for computing tasks, improving the data transmission efficiency of the data processing unit, and improving the IO performance of the computing device cluster.

In one possible implementation, the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling an application program interface. When a request is made, the virtual machine of the computing node in the computing device cluster calls the application program interface and sends a storage pool access request to the data forwarding agent module through the direct memory access DMA controller.

For example, the virtual machine of the computing node calls the log interface and sends the log data from the memory of the computing node directly to the data forwarding agent module of the data processing unit through the direct memory access DMA controller, without the participation of the central processing unit of the computing node, thereby realizing direct transmission of log data.

Please continue to refer to Figure 3. In the embodiment shown in Figure 3, the virtual machine of the computing node in the computing device cluster sends a storage pool access request to the forwarding proxy module of the data processing unit by calling the log interface, and the storage pool access request includes a log data write request and a log data read request. For example, the Redis service of the virtual machine VM1 sends the log data write request to the data forwarding proxy module of the data processing unit by calling the log interface, and the virtual machine VM1 can send the log data write request to the data forwarding proxy module by direct memory access.

In the embodiment of the present application, the computing node can send a storage pool access request to the data forwarding agent module by means of direct memory access DMA, thereby improving the communication efficiency between the computing node and the data processing unit and further improving the IO performance of the computing device cluster.

In one possible implementation, when the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling an application program interface, the computing device cluster sends a log data write request to the data forwarding proxy module by calling an application program interface. The log data write request carries a log identifier (log ID). The log identifier is used to indicate the data write location in the storage pool of the data forwarding proxy module, such as a storage node identifier.

Please refer to Figure 4, which is a schematic diagram of a storage pool data writing process provided by an embodiment of the present application. In the example shown in Figure 4, the virtual machine of the computing node sends the log data generated by the Redis service to the data forwarding proxy module through the log interface. For example, the virtual machine wants to send 2M of log data, where the log data contains 4 data blocks, and the size of each log data block is 512k. After the virtual machine calls the log interface to send a log data write request, the service quality QOS control module of the computing node first checks whether the current data traffic is overloaded. If the current data traffic is overloaded, the QOS control module rejects the log data write request.

In the example shown in FIG4 , if the QOS control module checks that the current data traffic is not overloaded, the verification module of the computing node continues to perform cyclic redundancy check code CRC verification on the log data write request. If the verification fails, the verification module rejects the log data write request. If the verification passes, the computing node performs erasure code (EC) encoding on the log data in the log data write request through the encoding module to generate encoded data blocks, which include log data blocks and verification blocks. For example, the computing node obtains 6 data blocks after performing EC encoding on the log data in the log data write request through the encoding module, of which 4 data blocks are log data blocks and 2 data blocks are verification blocks. The computing device sends the EC-encoded data blocks to the data forwarding agent module according to the log data write request, and the data forwarding agent module sends them to the storage nodes of the storage pool.

In one possible implementation, the computing device cluster determines the partition identifier (pt ID) based on the log identifier. Specifically, the computing device cluster determines the partition identifier based on the log identifier (log ID) and the partition number (pt num), and the partition identifier is used to query the storage node identifier (disk ID) corresponding to the partition identifier in the routing table. The computing device cluster can query the routing table based on the partition identifier to determine one or more storage node identifiers corresponding to the partition identifier.

In the example shown in FIG4 , in the process of sending the EC-encoded data block to the data forwarding proxy module according to the log data write request, it is necessary to determine the write location of the log data in the storage node according to the log identifier carried in the log data write request. Specifically, the computing node calculates the partition identifier according to the log identifier and the number of partitions, queries the routing table according to the partition identifier, and determines the storage node identifier corresponding to the partition identifier. The computing device sends the data write request to the buffer corresponding to the storage node identifier in the data forwarding proxy module according to the storage node identifier.

In the embodiment of the present application, the computing device cluster can determine the partition identifier according to the log identifier and the number of partitions. The partition identifier, log identifier and the number of partitions satisfy the following calculation relationship:
ptID＝log ID％ptNum

In the embodiment of the present application, the computing device can determine the storage node identifier that the computing device cluster wants to access based on the log identifier carried in the storage pool access request, thereby enabling the computing node to accurately determine the location of the storage node to be accessed, thereby improving the accuracy of the computing device cluster accessing the storage nodes in the storage pool.

In one possible implementation, in the process of the computing device cluster sending a storage pool access request to the data forwarding proxy module by calling an application program interface, the computing device cluster sends a data read request to the data forwarding proxy module by calling an application program interface, and the data read request carries a log identifier, which is used to indicate the data reading position of the data forwarding proxy module in the storage pool, such as the identifier of the storage node.

Please refer to Figure 5, which is a schematic diagram of a process of reading data from a storage pool provided by an embodiment of the present application. In the example shown in Figure 5, the Redis service of the virtual machine in the computing node needs to read log data from the storage pool. After the virtual machine calls the log interface to send a log data read request, the service quality QOS control module of the computing node first checks whether the current data traffic is overloaded. If the current data traffic is overloaded, the QOS control module rejects the log data read request of the Redis service.

In the example shown in Figure 5, if the QOS control module checks that the current data traffic is not overloaded, the splitter of the computing node determines the log data block corresponding to the log data to be read by the log data read request, determines the reading position of each log data in the storage pool according to the log identifier of the log data read request, and sends the log data read request corresponding to each log data block to the data forwarding agent module of the data processing unit.

In the example shown in FIG5 , the log data read request sent by the computing node carries an offset (offset) and a length (len), which are used to indicate the specific location of the log data in the storage node, such as the storage address of the log data in the storage node. The computing node sends the log data read request to the data forwarding agent module through the memory direct access DMA controller.

203. The computing device cluster sends a storage pool access request to a storage node in the storage pool through a data forwarding proxy module.

The computing device cluster sends a storage pool access request to the storage node in the storage pool through the data forwarding proxy module. Specifically, after the data forwarding proxy module in the data processing unit receives the storage pool access request sent by the virtual machine of the computing node, it sends the storage pool access request to the controller corresponding to the storage node according to the storage pool access request.

In one possible implementation, the data forwarding agent module in the data processing unit can transmit data with the storage nodes in the storage pool through the transmission control protocol (TCP).

After receiving the storage pool access request, the storage node controller writes data to the hard disk or reads data from the hard disk according to the storage pool access request. Specifically, after receiving the log data write request, the storage node controller writes the log data to the hard disk according to the log data write request, and after receiving the log data read request, the storage node controller reads the log data in the hard disk according to the log data read request.

Please continue to refer to Figure 3. In the embodiment shown in Figure 3, after the data forwarding agent module of the data processing unit receives the storage pool access request sent by the virtual machine of the computing node through the log interface, the data forwarding processing module forwards the storage pool access request to the IO controller corresponding to the storage node in the storage pool. The storage pool access request includes a log data write request and a log data read request.

For example, in the example shown in Figure 3, the Redis service in the virtual machine VM1 sends log data to the data forwarding agent module of the data processing unit by calling the log interface. The data forwarding agent module forwards the log data to the corresponding controller of the storage node in the storage pool, and the controller of the storage node writes the log data to the hard disk of the storage node.

Please continue to refer to Figure 4. In the example shown in Figure 4, after the data forwarding proxy module of the data processing unit receives the data write request sent by the computing node, the data forwarding proxy module sends the log data write request to the storage pool. Specifically, the data forwarding proxy module sends the log data write request to the storage node corresponding to the storage node identifier in the storage pool according to the storage node identifier, thereby writing the log data to the corresponding storage node and completing the persistence of the log data.

Please continue to refer to Figure 5. In the example shown in Figure 5, after the data forwarding proxy module of the data processing unit receives the data read request sent by the computing node, the data forwarding proxy module sends the log data read request to the storage pool. Specifically, the data forwarding proxy module sends the log data read request to the storage node corresponding to the storage node identifier in the storage pool. After the computing node reads the log data block from different storage nodes, the computing device performs a cyclic redundancy check code CRC check on the read log data block, and merges the log data blocks through a merge module to obtain the log data to be read.

It can be seen from the above embodiments that the virtual machine or container of the computing node in the embodiments of the present application can deploy a software toolkit of the application program interface, so that the virtual machine or container of the computing node can access any storage node in the storage pool based on the application program interface, thereby improving the flexibility of the computing device cluster to access individual storage nodes in the storage pool.

The above describes the storage pool access method provided by the embodiment of the present application. The following describes the storage pool access device provided by the embodiment of the present application.

Please refer to Figure 6, which is a schematic diagram of the structure of a storage pool access device provided in an embodiment of the present application. In the example shown in Figure 6, the storage pool access device is used to implement the various steps performed by the computing node in the above embodiments, and the storage pool access device 600 includes a transceiver unit 601 and a processing unit 602.

The processing unit 602 is used to provide an application program interface, which is used by computing nodes in the computing device cluster to access the storage pool through the application program interface, and the software tool kit SDK of the application program interface is deployed on the virtual machine or container of the computing device cluster. Unit 601 is used to send a storage pool access request to a data forwarding proxy module by calling an application program interface, and the data forwarding proxy module is used to process the data forwarding process of the computing device cluster. The transceiver unit 601 is also used to send a storage pool access request to a storage node in the storage pool through the data forwarding proxy module.

In a possible implementation manner, the transceiver unit 601 is specifically configured to call an application program interface and send a storage pool access request to the forwarding agent module through a direct memory access DMA controller.

In a possible implementation, the transceiver unit 601 is specifically used to send a data write request to the data forwarding proxy module by calling an application program interface. The data write request carries a log identifier, which is used to indicate the data write location of the data forwarding proxy module in the storage pool.

In a possible implementation, the transceiver unit 601 is specifically used to call an application program interface to send a data read request to the data forwarding proxy module, and the data read request carries a log identifier, which is used to indicate the data reading position of the data forwarding proxy module in the storage pool.

In a possible implementation, the processing unit 602 is further configured to determine a partition identifier based on the log identifier, query a routing table according to the disk table identifier, and determine one or more storage node identifiers corresponding to the partition identifier.

It should be understood that the division of the units in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware. For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can also be stored in the memory in the form of a program, and called and executed by a certain processing element of the device. The function of the unit. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can also be a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software calling through a processing element.

It is worth noting that, for the above method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required for the present invention.

Other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of the present invention. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

Please refer to Figure 7, which is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 7, the computing device 700 includes: a processor 701, a memory 702, a communication interface 703 and a bus 704, and the processor 701, the memory 702 and the communication interface 703 are coupled through a bus (not marked in the figure). The memory 702 stores instructions, and when the execution instructions in the memory 702 are executed, the computing device 700 executes the method executed by the computing node in the above method embodiment.

The computing device 700 may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuit forms. For another example, when a unit in the device can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

Processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

The memory 702 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EPROM). The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

The memory 702 stores executable program codes, and the processor 701 executes the executable program codes to respectively implement the functions of the transceiver unit and the processing unit, thereby implementing the storage pool access method. That is, the memory 702 stores instructions for executing the storage pool access method.

The communication interface 703 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

In addition to the data bus, the bus 704 may also include a power bus, a control bus, and a status signal bus. The bus may be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus may be divided into an address bus, a data bus, a control bus, etc.

Please refer to FIG8 , which is a schematic diagram of a computing device cluster provided in an embodiment of the present application. As shown in FIG8 , the computing device cluster 800 includes at least one computing device 700 .

As shown in Fig. 8, the computing device cluster 800 includes at least one computing device 700. The memory 702 in one or more computing devices 700 in the computing device cluster 800 may store the same instructions for executing the above storage pool access method.

In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster 800 may also store partial instructions for executing the above storage pool access method. In other words, the combination of one or more computing devices 700 may jointly execute instructions for executing the above storage pool access method.

It should be noted that the memory 702 in different computing devices 700 in the computing device cluster 800 may store different instructions, which are respectively used to execute part of the functions of the above storage pool access device. That is, the instructions stored in the memory 702 in different computing devices 700 may implement the functions of one or more modules in the transceiver unit and the processing unit.

In some possible implementations, one or more computing devices 700 in the computing device cluster 800 may be connected via a network, which may be a wide area network or a local area network.

Please refer to Figure 9, which is a schematic diagram of a computer device in a computer cluster provided by an embodiment of the present application being connected through a network. As shown in Figure 9, two computing devices 700A and 700B are connected through a network. Specifically, the network is connected through a communication interface in each computing device.

In a possible implementation, the memory in the computing device 700A stores instructions for executing the functions of the transceiver module, and the memory in the computing device 700B stores instructions for executing the functions of the processing module.

It should be understood that the functions of the computing device 700A shown in Figure 9 can also be completed by multiple computing devices. Similarly, the functions of the computing device 700B can also be completed by multiple computing devices.

In another embodiment of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the processor of the device executes the computer-executable instructions, the device executes the method executed by the computing device in the above method embodiment.

In another embodiment of the present application, a computer program product is provided, the computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device executes the method executed by the computing device in the above method embodiment.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system. Or some features may be ignored or not performed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

A method for accessing a storage pool, comprising:

The computing device cluster provides an application program interface, the application program interface is used for computing nodes in the computing device cluster to access the storage pool through the application program interface, and a software tool kit SDK of the application program interface is deployed on a virtual machine or a container of the computing device cluster;

The computing device cluster sends a storage pool access request to the data forwarding proxy module by calling the application program interface, and the data forwarding proxy module is used to process the data forwarding process of the computing device cluster;

The computing device cluster sends the storage pool access request to the storage nodes in the storage pool through the data forwarding proxy module.
The method according to claim 1, characterized in that the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling the application program interface, comprising:

The computing device cluster calls the application program interface and sends the storage pool access request to the data forwarding agent module through a direct memory access DMA controller.
The method according to claim 1 or 2, characterized in that the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling the application program interface, comprising:

The computing device cluster sends a log data write request to the data forwarding proxy module by calling the application program interface, and the log data write request carries a log identifier, and the log identifier is used to indicate the data write position of the data forwarding proxy module in the storage pool.
The method according to claim 1 or 2, characterized in that the computing device cluster sends a storage pool access request to the data forwarding proxy module by calling the application program interface, comprising:

The computing device cluster sends a log data read request to the data forwarding proxy module by calling the application program interface, and the log data read request carries a log identifier, and the log identifier is used to indicate a data reading position of the data forwarding proxy module in the storage pool.
The method according to claim 3 or 4, characterized in that the method further comprises:

The computing device cluster determines a partition identifier based on the log identifier;

The computing device cluster queries a routing table according to the partition identifier to determine one or more storage node identifiers corresponding to the partition identifier.
The method according to any one of claims 1 to 5 is characterized in that the data forwarding agent module is deployed on a data processing unit DPU chip.
A storage pool access device, characterized by comprising:

A processing unit, configured to provide an application program interface, wherein the application program interface is used for computing nodes in the computing device cluster to access a storage pool through the application program interface, and a software tool kit SDK of the application program interface is deployed on a virtual machine or a container of the computing device cluster;

A transceiver unit, configured to send a storage pool access request to a data forwarding proxy module by calling the application program interface, wherein the data forwarding proxy module is configured to process a data forwarding process of the computing device cluster;

The transceiver unit is further configured to send the storage pool access request to the storage nodes in the storage pool through the data forwarding agent module.
The device according to claim 7, characterized in that the transceiver unit is specifically used for:

The application program interface is called, and the storage pool access request is sent to the data forwarding agent module through a direct memory access DMA controller.
The device according to claim 7 or 8, characterized in that the transceiver unit is specifically used for:

By calling the application program interface, a data write request is sent to the data forwarding proxy module, the data write request carries a log identifier, and the log identifier is used to indicate the data write position of the data forwarding proxy module in the storage pool.
The device according to claim 7 or 8, characterized in that the transceiver unit is specifically used for:

The application program interface is called to send a data read request to the data forwarding proxy module, wherein the data read request carries a log identifier, and the log identifier is used to indicate a data reading position of the data forwarding proxy module in the storage pool.
The device according to claim 9 or 10, characterized in that the processing unit is further used for:

Determine a partition identifier based on the log identifier;

The routing table is queried according to the disk table identifier to determine one or more storage node identifiers corresponding to the partition identifier.
The device according to any one of claims 7 to 11 is characterized in that the data forwarding agent module is deployed on a data processing unit DPU chip.
A computing device cluster, characterized in that it includes a processor, the processor is coupled to a memory, and the processor is used to store instructions. When the instructions are executed by the processor, the computing device cluster executes the method according to any one of claims 1 to 6.
A computer-readable storage medium having instructions stored thereon, characterized in that when the instructions are executed, a computer is caused to execute the method according to any one of claims 1 to 6.
A computer program product, comprising instructions, wherein when the instructions are executed, a computer implements the method according to any one of claims 1 to 6.