WO2022160308A1

WO2022160308A1 - Data access method and apparatus, and storage medium

Info

Publication number: WO2022160308A1
Application number: PCT/CN2021/074545
Authority: WO
Inventors: 伍波
Original assignee: 华为技术有限公司
Priority date: 2021-01-30
Filing date: 2021-01-30
Publication date: 2022-08-04
Also published as: CN116670636A

Abstract

Provided in embodiments of the present application are a data access method and apparatus, and a storage medium, the method comprising: a storage node receiving a remote direct memory access (RDMA) connection request sent by a computing node, wherein the RDMA connection request carries a target solid state disk identifier, and the RDMA connection request is used to instruct the storage node to establish an access connection between the computing node and the target solid state disk; the storage node acquiring and storing link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue (CQ); and the storage node sending, to the target solid state disk, a first address corresponding to the stored link establishment information, so as to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ. By means of the embodiments of the present application, the efficiency of data access can be effectively improved. Moreover, the CPU/memory overhead of a storage node is reduced, and a system bottleneck can be effectively eliminated.

Description

Data access method, device and storage medium

technical field

The present application relates to the technical field of data processing, and in particular, to a data access method, device and storage medium.

Background technique

At present, in the existing distributed storage system composed of three-level models of computing nodes, storage nodes, and storage media, the storage nodes are in the middle position and undertake the functions of protocol conversion and data caching, and their work efficiency will directly affect the entire system. performance. A single storage node can generally provide a bandwidth of about 10 to 20 GB, and can provide an Input/Output Per Second (IOPS) capability of about 100 to 1000K. However, a solid state drive (Solid State Disk, SSD) with a single Non-Volatile Memory Express (NVMe) interface can generally provide 5GB of bandwidth and 1200K IOPS. If one storage node manages 24 SSDs, the bandwidth and IOPS that 24 SSDs can provide are about 6 times and 28 times that of storage nodes, respectively. It can be seen that the performance of storage nodes is far from fully exploiting the high-performance advantages of SSDs.

This is because in the existing solution, there is no connection between the computing node and the SSD, and all commands sent by the computing node need to be parsed and processed by the software on the storage node, which is extremely inefficient. In addition, after the software of the storage node is parsed, the NVMe command is sent to the SSD. This operation consumes a certain amount of CPU resources. As the performance of the SSD improves and the IOPS is further improved, the subsequent CPU may become a performance bottleneck. On the other hand, the data flow between the SSD and the computing node needs to be transferred through the memory on the storage node. Due to the limited memory bandwidth, with the further improvement of the SSD read and write bandwidth, the subsequent memory may also become a performance bottleneck.

SUMMARY OF THE INVENTION

The present application discloses a data access method, device and storage medium, which can improve data access efficiency, reduce CPU/memory overhead of storage nodes, and eliminate system bottlenecks.

In a first aspect, an embodiment of the present application provides a data access method, which is applied to a storage node in a distributed storage system. The method includes: the storage node receives a remote data sent by a computing node in the distributed storage system. A direct data access RDMA connection request, where the RDMA connection request carries the target solid-state drive identifier in the distributed storage system, and the RDMA connection request is used to instruct the storage node to establish a connection between the computing node and the target solid-state drive. An access connection between hard disks; the storage node acquires and stores the link establishment information corresponding to the access connection, wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state hard disk has data storage. The storage node sends the stored first address corresponding to the link establishment information to the target SSD, so as to instruct the target SSD to obtain the CQ in the link establishment information, and monitor all described CQ.

According to the embodiment of the present application, the storage node sends the address of the chain establishment information corresponding to the access connection between the stored computing node and the target solid state drive to the target solid state drive, so that the target solid state drive obtains the CQ in the chain establishment information, And monitor the CQ, so as to realize the direct access connection between the computing node and the target solid-state hard disk. Compared with the prior art, the command sent by the computing node needs to be parsed by the storage node and then sent to the SSD, and the data flow between the SSD and the computing node in the prior art needs to pass through the storage node. In the operation mode of memory transfer, using the direct access connection between the computing node and the target solid-state hard disk provided by the embodiment of the present application can effectively improve the efficiency of data access; It reduces the CPU/memory overhead of storage nodes and effectively eliminates system bottlenecks.

Further, the link establishment information further includes a memory area MR and a queue pair QP, where the MR and QP are used to instruct the target solid-state disk to perform data access. Specifically, the MR is used to indicate an area of the target solid-state disk for data access, and the QP is used to instruct the target solid-state disk to operate the QP to send and receive data.

Among them, QP includes send queue (SendQueue, SQ) and receive queue (ReceiveQueue, RQ), and these queues manage various types of messages. When the target solid-state hard disk obtains from the CQ that there is a data access task, the target solid-state hard disk transmits and receives data by operating the QP. Among them, the SSD places the task of sending data in the sending queue SQ for data sending. On the other hand, the SSD obtains the task of receiving data from the receiving queue, and then performs data receiving processing.

Optionally, the storage node stores the link-building information corresponding to the access connection in a shared memory space, where the shared memory space is a Controller Memory Buffer (CMB) of the target solid-state hard disk. ), or the shared memory space is a host memory buffer (Host Memory Buffer, HMB) of the storage node.

As an optional implementation manner, the method further includes: the storage node detects the access connection; when a network abnormality is detected, or when receiving a heartbeat abnormality message sent by the target solid-state hard disk message, the storage node deletes the link establishment information on the shared memory space.

When an abnormality is detected, the access connection between the SSD and the computing node is released in time to ensure the security of data access.

As another optional implementation manner, the method further includes: the storage node sends the received instruction of disconnecting the access connection sent by the computing node to the target solid-state disk; When the message of successful link disconnection sent by the target solid state disk is sent, the storage node deletes the link establishment information on the shared memory space.

When the computing node no longer needs to establish an access connection with the SSD, the connection between the computing node and the SSD is disconnected, so as to be re-established when data access is required. By adopting this method, system resources can be saved.

In a second aspect, an embodiment of the present application provides a data access method, which is applied to a target solid-state disk in a distributed storage system, the method includes: the target solid-state disk receives a data sent by a storage node in the distributed storage system. The message carries a first address, where the first address is the link establishment corresponding to the access connection between the computing node in the distributed storage system and the target SSD stored by the storage node The address of the information, the link establishment information includes the completion queue CQ, and the CQ is used to indicate that the target SSD has a data access task; the target SSD obtains the link establishment information according to the first address. the CQ, and listen to the CQ.

Through the embodiment of the present application, the target solid-state drive obtains the CQ in the link establishment information corresponding to the access connection between the computing node and the target solid-state drive based on the address sent by the storage node, and monitors the CQ, thereby realizing the connection between the computing node and the target solid-state drive. Direct access connection between this target SSD. Compared with the prior art, the command sent by the computing node needs to be parsed by the storage node and then sent to the SSD, and the data flow between the SSD and the computing node in the prior art needs to pass through the storage node. In the operation mode of memory transfer, using the direct access connection between the computing node and the target solid-state hard disk provided by the embodiment of the present application can effectively improve the efficiency of data access; It reduces the CPU/memory overhead of storage nodes and effectively eliminates system bottlenecks.

Further, the link establishment information further includes a memory area MR and a queue pair QP, and the method further includes: the target solid-state disk acquires the MR and QP in the link establishment information according to the first address, and performs data analysis. access.

Optionally, the storage node stores the link establishment information corresponding to the access connection in a shared memory space, where the shared memory space is a controller memory buffer of the target solid-state hard disk, or the shared memory The memory space is the host memory buffer of the storage node.

As an optional implementation manner, the method further includes: the target solid-state drive detects the access connection; when detecting that there is an abnormal heartbeat, or when receiving a network abnormality message sent by the storage node message, the target solid state drive confirms whether the commands in the access connection are all completed; if the commands in the access connection are all completed, the target solid state drive stops monitoring the CQ.

By releasing the access connection between the SSD and the computing node in time when an abnormality is detected, the security of data access can be guaranteed.

As another optional implementation manner, the method further includes: the target solid-state drive receives an instruction to disconnect the access connection sent by the storage node; the target solid-state drive confirms the command in the access connection Whether all the commands are completed; if all the commands in the access connection are completed, the target solid-state disk stops monitoring the CQ.

By disconnecting the connection between the SSD and the compute node so that it can be re-established when communication is required. By adopting this method, system resources can be saved.

In a third aspect, an embodiment of the present application provides a data access device, including: a receiving module configured to receive a remote direct data access RDMA connection request sent by a computing node in a distributed storage system, where the RDMA connection request carries a target Solid-state disk identifier, the RDMA connection request is used to instruct the storage node to establish an access connection between the computing node and the target solid-state disk; an acquisition module is used to acquire and store the link establishment information corresponding to the access connection , wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state hard disk has a data access task; a sending module is used to send the stored first address corresponding to the link establishment information to a the target solid-state disk to instruct the target solid-state disk to obtain the CQ in the link establishment information, and monitor the CQ.

Wherein, the link establishment information further includes a memory area MR and a queue pair QP, the MR is used to indicate a data access area of the target solid state drive, and the QP is used to instruct the target solid state drive to operate the QP to Send and receive data.

Wherein, the device stores the link establishment information corresponding to the access connection in the shared memory space. The shared memory space is a controller memory buffer of the target solid state disk, or the shared memory space is a host memory buffer of the device.

Optionally, the device further includes a detection module configured to detect the access connection; when a network abnormality is detected, or when a message with abnormal heartbeat sent by the target solid-state drive is received, delete all the information. the link establishment information on the shared memory space.

Optionally, the device further includes a link disconnection module, configured to send the received instruction of disconnecting the access connection sent by the computing node to the target solid-state drive; When the message of successful link disconnection is received, the link establishment information on the shared memory space is deleted.

In a fourth aspect, an embodiment of the present application provides a data access device, including: a receiving module configured to receive a message sent by a storage node in a distributed storage system, where the message carries a first address, wherein the first The address is the address of the link establishment information corresponding to the access connection between the computing node and the device in the distributed storage system stored by the storage node, and the link establishment information includes the completion queue CQ; the processing module is used for Acquire the CQ in the link establishment information according to the first address, and monitor the CQ.

Optionally, the link establishment information further includes a memory area MR and a queue pair QP, and the processing module is further configured to acquire the MR and QP in the link establishment information according to the first address, and perform data access.

The storage node stores the link establishment information corresponding to the access connection in a shared memory space, where the shared memory space is a controller memory buffer of the device, or the shared memory space is a The host memory buffer of the storage node.

Optionally, the device further includes a detection module for detecting the access connection; when it is detected that there is an abnormal heartbeat, or when receiving a message sent by the storage node that there is a network abnormality, confirming that the connection is abnormal. Whether the commands in the access connection are all completed; if the commands in the access connection are all completed, stop monitoring the CQ.

Optionally, the device further includes a link disconnection module, configured to receive an instruction to disconnect the access connection sent by the storage node; confirm whether the commands in the access connection are all completed; Commands are completed, stop monitoring the CQ.

In a fifth aspect, an embodiment of the present application provides a data access device, including a processor, where the processor is configured to: receive a remote direct data access RDMA connection request sent by a computing node in a distributed storage system, and the processor The RDMA connection request carries the target solid-state drive identifier, and the RDMA connection request is used to instruct the storage node to establish an access connection between the computing node and the target solid-state drive; obtain and store the link establishment information corresponding to the access connection , wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target SSD has a data access task; send the first address corresponding to the stored link establishment information to the target SSD , to instruct the target SSD to obtain the CQ in the link establishment information, and monitor the CQ.

Further, the link establishment information also includes a memory area MR and a queue pair QP, the MR is used to indicate the area where the target solid-state disk data is accessed, and the QP is used to indicate that the target solid-state disk operates the QP. to send and receive data.

The processor stores the link establishment information corresponding to the access connection in a shared memory space, where the shared memory space is a controller memory buffer of the target solid-state hard disk, or the shared memory space is the host memory buffer of the device.

As an optional implementation manner, the processor is further configured to: detect the access connection; when a network abnormality is detected, or when receiving a message that the target solid-state hard disk has abnormal heartbeat , delete the link establishment information on the shared memory space.

As another optional implementation manner, the processor is further configured to: send the received instruction of disconnecting the access connection sent by the computing node to the target solid-state hard disk; When the message of successful link disconnection is sent by the solid state drive, the link establishment information on the shared memory space is deleted.

In a sixth aspect, an embodiment of the present application provides a data access device, including a processor, where the processor is configured to: receive a message sent by a storage node in a distributed storage system, where the message carries a first address, wherein , the first address is the address of the link establishment information corresponding to the access connection between the computing node in the distributed storage system and the device stored by the storage node, and the link establishment information includes the completion queue CQ; Acquire the CQ in the link establishment information according to the first address, and monitor the CQ.

Further, the link establishment information further includes a memory area MR and a queue pair QP, and the processor is further configured to: acquire the MR and QP in the link establishment information according to the first address, and perform data access.

The storage node stores the link establishment information corresponding to the access connection in a shared memory space, where the shared memory space is the controller memory buffer of the device, or the shared memory space is the storage The node's host memory buffer.

As an optional implementation manner, the processor is further configured to: detect the access connection; when detecting that there is an abnormal heartbeat, or when receiving a message sent by the storage node that there is a network abnormality , confirm whether the commands in the access connection are all completed; if the commands in the access connection are all completed, stop monitoring the CQ.

As another optional implementation manner, the processor is further configured to: receive an instruction sent by the storage node to disconnect the access connection; confirm whether the commands in the access connection are all completed; The commands in the connection are all completed, stop monitoring the CQ.

In a seventh aspect, the present application provides a computer storage medium, comprising computer instructions, when the computer instructions are executed on an electronic device, the electronic device is made to perform any one of the possible implementations of the first aspect to the second A method provided by any possible embodiment of the aspect.

In an eighth aspect, an embodiment of the present application provides a computer program product, which, when the computer program product runs on a computer, enables the computer to execute any possible implementation manner of the first aspect and any possible implementation manner of the second aspect provided method.

In a ninth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the The processors are interconnected by lines; the interface circuit is used for receiving signals from the memory of the electronic device and sending the signals to the processor, the signals including computer instructions stored in the memory; when the processing The electronic device executes the method when the computer executes the computer instructions.

In a tenth aspect, an embodiment of the present application provides a distributed storage system, including a computing node, and the above-mentioned data access device.

It can be understood that the device according to the third aspect, the device according to the fourth aspect, the device according to the fifth aspect, the device according to the sixth aspect, the computer storage medium according to the seventh aspect, the The computer program product described in the eighth aspect or the chip system described in the ninth aspect is used to execute any one of the methods provided in the first aspect and any possible implementation manner of the second aspect. Therefore, for the beneficial effects that can be achieved, reference may be made to the beneficial effects in the corresponding method, which will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the accompanying drawings involved in the embodiments or the background technology of the present application will be briefly introduced below.

1a is a schematic diagram of the networking of a distributed storage system in the prior art;

Fig. 1b is a business schematic diagram of a distributed storage system in the prior art;

2 is a schematic flowchart of a data access method provided by an embodiment of the present application;

3a is a schematic diagram of a service of a distributed storage system provided by an embodiment of the application;

3b is a schematic flowchart of a data access method provided by an embodiment of the present application;

4 is a schematic flowchart of a data access method provided by an embodiment of the present application;

5 is a schematic flowchart of another data access method provided by an embodiment of the present application;

6 is a schematic diagram of a data access method provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of another data access method provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a data access device according to an embodiment of the present application;

FIG. 9 is a schematic structural diagram of another data access device provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of still another data access apparatus provided by an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

Referring to FIG. 1a, it is a schematic diagram of the networking of a distributed storage system in the prior art. The system includes a computing cluster, a storage cluster, and multiple solid-state drives (SSDs). The computing cluster is used to calculate and generate data; the storage cluster is used to save data, provide data access, and ensure data reliability and security. The computing cluster includes multiple computing nodes; the storage cluster includes multiple storage nodes; and the storage nodes can be connected to multiple SSDs. The computing node communicates with the storage node through the network for data storage and access. As shown in FIG. 1b, it is a schematic diagram of the business of the distributed storage system in the prior art. Among them, the remote direct data access (Remote Direct Memory Access, RDMA) connection mode is used for communication between the computing node and the storage node. The storage node and the SSD are connected through the Peripheral Component Interconnect Express (PCIE), and the NVME protocol is used for communication. There is no connection between the computing node and the SSD; all input/output (IO) operation commands of the computing node use the network-based non-volatile storage standard (Non-Volatile Memory Express Over Fabrics, NVMe-OF), Send commands to the storage node through RDMA, and the software on the storage node converts specific IO commands into NVME commands and sends them to the SSD. The read and write data streams on the SSD need to pass through the dynamic random access memory (Dynamic Random Access Memory, DRAM) on the storage node for transit, and interact with the computing node through RDMA.

Since all commands sent by the computing nodes need to be parsed and processed by the software on the storage nodes, the efficiency is extremely low. Moreover, after the storage node software parses, it sends the command to the SSD, which consumes a certain amount of CPU resources. At the same time, the data flow between the SSD and the computing node needs to be transferred through the memory on the storage node. As the read and write bandwidth of the SSD is further improved, the subsequent memory may also become a performance bottleneck.

To this end, an embodiment of the present application provides a data access method, wherein the storage node sends the address of the link establishment information corresponding to the access connection between the stored computing node and the target solid-state hard disk to the target solid-state hard disk, so that the target solid-state hard disk is The CQ in the link establishment information is obtained, and the CQ is monitored, so as to realize the direct access connection between the computing node and the target solid-state hard disk. Compared with the prior art, the command sent by the computing node needs to be parsed by the storage node and then sent to the SSD, and the data flow between the SSD and the computing node in the prior art needs to pass through the storage node. In the operation mode of memory transfer, using the direct access connection between the computing node and the target solid-state hard disk provided by the embodiment of the present application can effectively improve the efficiency of data access; It reduces the CPU/memory overhead of storage nodes and effectively eliminates system bottlenecks.

The data access method provided by the embodiment of the present application is described in detail below.

Referring to FIG. 2 , a schematic flowchart of a data access method provided by an embodiment of the present application is shown. Wherein, the data access method is applied to storage nodes in a distributed storage system. The method includes steps 201-203, as follows:

201. A storage node receives a remote direct data access RDMA connection request sent by a computing node, where the RDMA connection request carries a target solid-state disk identifier, and the RDMA connection request is used to instruct the storage node to establish a connection between the computing node and the target. Access connections between SSDs;

Wherein, the above-mentioned target solid-state disk identifier may be an ID of the target solid-state disk SSD, or the like.

202. The storage node acquires and stores the link establishment information corresponding to the access connection, wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state disk has a data access task;

The above-mentioned complete queue (Complete Queue, CQ) is used to indicate that the target solid-state hard disk has a data access task, that is, it is used to inform the SSD that there is a command that needs to be processed.

Further, the above chain establishment information also includes a memory region (Memory Region, MR) and a queue pair (Queue Pairs, QP).

Among them, QP includes send queue (SendQueue, SQ) and receive queue (ReceiveQueue, RQ), and these queues manage various types of messages. When the target solid-state hard disk obtains from the CQ that there is a data access task, the target solid-state hard disk transmits and receives data by operating the QP. Among them, the SSD places the task of sending data in the sending queue SQ for data sending. On the other hand, the SSD obtains the task of receiving data from the receiving queue, and then performs data receiving processing. Specifically, the QP will be mapped to the virtual address space of the application, so that the application communicates with the peer directly through the QP, such as operating the link, controlling the sending and receiving of data, and so on.

The above-mentioned MR is a memory area for accessing data.

The above-mentioned MR, QP, and CQ may be referred to as link establishment information (MR, QP, CQ) for short.

Among them, multiple RDMA connections can be established between the storage node and the SSD. That is to say, there is link establishment information corresponding to each RDMA connection in the multiple RDMA connections.

The storage node stores the link establishment information related to the RDMA connection based on the received RDMA connection request. Wherein, the storage node stores the link establishment information corresponding to the access connection in the shared memory space.

As an implementation manner, the storage node establishes the link establishment information corresponding to the access connection in the controller memory buffer (Controller Memory Buffer, CMB) of the target solid-state hard disk. That is to say, the above-mentioned first address is the address at which the storage node establishes the link establishment information corresponding to the access connection on the controller memory buffer CMB of the above-mentioned target solid-state hard disk.

As another implementation manner, the storage node establishes the link establishment information corresponding to the access connection in the host memory buffer (Host Memory Buffer, HMB) of the storage node. That is to say, the above-mentioned first address is the address at which the storage node establishes the link establishment information corresponding to the access connection on the host memory buffer HMB of the storage node.

That is to say, the above-mentioned shared memory space may be the controller memory buffer CMB of the target solid state disk, or may be the host memory buffer HMB of the storage node.

As shown in FIG. 3a, a schematic diagram of a service of a distributed storage system provided by an embodiment of the present application is shown. Wherein, the storage node establishes the link establishment information (MR, QP, CQ) on the target solid state drive, so that the target solid state drive can directly access data with the computing node through the link establishment information.

203. The storage node sends the stored first address corresponding to the link establishment information to the target SSD, to instruct the target SSD to obtain the CQ in the link establishment information, and monitor the CQ.

As an implementation manner, the storage node notifies the SSD of the address of the link establishment information corresponding to the access connection on the CMB space of the SSD, so that the SSD obtains the CQ in the link establishment information according to the address, and resolves the CQ address by parsing the CQ address. , then listen for commands on the CQ.

As another implementation manner, the storage node notifies the SSD of the address of the link establishment information corresponding to the access connection on the host memory buffer HMB of the storage node, so that the SSD can obtain the CQ in the link establishment information according to the address, and pass The CQ address is parsed out, and then the command on the CQ is monitored.

Further, the storage node forwards the connection successful message sent by the target solid state disk to the computing node.

The storage node receives the successful connection response message sent by the target solid-state drive, and forwards the response message to the above-mentioned computing node, so that heartbeat detection starts between the computing node and the SSD.

As an implementation manner, after step 203, the method further includes:

The storage node detects the access connection;

When a network abnormality is detected, or when a message of abnormal heartbeat is received from the target solid-state disk, the storage node deletes the link establishment information on the shared memory space.

The above-mentioned network abnormality may specifically be a network interruption or the like. For example, network card failure, network card loose, etc., or storage node/computing node crash, etc. It should be noted that, in the event of an abnormal network, the storage node can delete all the chain-building information established on the target solid-state hard disk.

Among them, heartbeat information is sent regularly between the computing node and the SSD. Once a component on the entire path is abnormal, the heartbeat will be abnormal. The abnormal heartbeat refers to: for example, the sent heartbeat message does not receive a corresponding response. At this time, the SSD can send a message to the storage node to inform the storage node that there is an abnormal heartbeat.

Optionally, after step 203, an embodiment of the present application further provides a data access method, which is used to disconnect the above-mentioned access connection. As shown in Figure 3b, it includes steps 301-302, as follows:

301. The storage node sends the received instruction of disconnecting the access connection sent by the computing node to the target solid-state drive;

Specifically, when the computing node no longer needs to access data on the target solid-state disk, the computing node sends a link disconnection instruction to the storage node.

302. The storage node deletes the link establishment information on the shared memory space when receiving the message that the link disconnection is successful sent by the target solid-state hard disk.

Wherein, after the target solid-state hard disk receives the above-mentioned instruction to disconnect the access connection, it stops monitoring the above-mentioned CQ after all the commands on its current connection are completed (ie, drained). Then send a message that the chain is disconnected successfully to the storage node.

Specifically, the storage node receives the message of successful link disconnection sent by the target solid-state drive, and then releases the link establishment information (MR, QP, CQ) related to the access connection above from the shared memory space. For example, from the CMB space of an SSD, or from the storage node's host memory buffer HMB.

Further, the storage node sends the message that the link disconnection is successful to the computing node.

Specifically, the storage node returns the link disconnection result to the computing node. The compute node stops sending heartbeat information.

According to the embodiment of the present application, the storage node sends the address of the chain establishment information corresponding to the access connection between the stored computing node and the target solid state drive to the target solid state drive, so that the target solid state drive obtains the CQ in the chain establishment information, And monitor the CQ, so as to realize the direct access connection between the computing node and the target solid-state hard disk. Compared with the prior art, the command sent by the computing node needs to be parsed by the storage node before sending the command to the SSD, and the data flow between the SSD and the computing node in the prior art needs to pass through the storage node. In the operation mode of memory transfer, using the direct access connection between the computing node and the target solid-state drive provided by the embodiment of the present application can effectively improve the efficiency of data access; It reduces the CPU/memory overhead of storage nodes and effectively eliminates system bottlenecks.

On the other hand, when using the CMB space of the SSD for data transfer, the memory of the storage node is not used, which is beneficial to avoid the bandwidth bottleneck of the memory of the storage node and perform parallel expansion of the SSD. In the case of limited SSD connection resources, the storage node cooperates with the SSD for connection management, which can adapt to scenarios with massive connections.

Referring to FIG. 4 , it is a schematic flowchart of another data access method provided by an embodiment of the present application. Wherein, the data access method is applied to a target solid-state hard disk in a distributed storage system. The method includes steps 401-402, as follows:

401. The target solid-state disk receives a message sent by a storage node in the distributed storage system, where the message carries a first address, where the first address is the address in the distributed storage system stored by the storage node. the address of the link establishment information corresponding to the access connection between the computing node and the target solid state disk, the link establishment information includes the completion queue CQ;

Further, the above chain establishment information also includes a memory region (Memory Region, MR) and a queue pair (Queue Pairs, QP). Among them, QP includes send queue (SendQueue, SQ) and receive queue (ReceiveQueue, RQ), and these queues manage various types of messages. When the target solid-state hard disk obtains from the CQ that there is a data access task, the target solid-state hard disk transmits and receives data by operating the QP. Among them, the SSD places the task of sending data in the sending queue SQ for data sending. On the other hand, the SSD obtains the task of receiving data from the receiving queue, and then performs data receiving processing. The QP will be mapped to the virtual address space of the application, so that the application communicates with the peer directly through the QP, such as operating the link, controlling the sending and receiving of data, etc. The above-mentioned MR is a memory area for accessing data.

The method further includes: acquiring, by the target solid-state disk, the MR and QP in the link establishment information according to the first address, and performing data access.

Wherein, the above-mentioned storage node stores the link establishment information corresponding to the access connection in the shared memory space. Specifically, the shared memory space is a controller memory buffer of the target solid-state hard disk, or the shared memory space is a host memory buffer of the storage node.

As an implementation manner, the first address is an address at which the storage node establishes the link establishment information corresponding to the access connection on the controller memory buffer CMB of the target solid-state hard disk.

As another implementation manner, the above-mentioned first address is an address at which the storage node establishes the link establishment information corresponding to the access connection on the host memory buffer HMB of the storage node.

402. The target solid-state disk acquires the CQ in the link establishment information according to the first address, and monitors the CQ.

After receiving the first address sent by the storage node, the SSD obtains the CQ in the link establishment information according to the first address, and then monitors the command on the CQ.

At the same time, the SSD sends a response message of successful connection to the computing node through the storage node, so that heartbeat detection starts between the computing node and the SSD.

Optionally, the target solid-state hard disk SSD monitors the command on the CQ, for example, after receiving the command to write data, the target solid-state hard disk SSD operates the storage node to perform the data writing operation through the address of the network card register. The storage node maps the address of the network card register to the SSD in advance. This solution only takes writing data as an example for description, which may also be other commands, which are not specifically limited in this solution.

As an implementation manner, after step 402, the method further includes:

The target solid-state disk detects the access connection;

When detecting that there is an abnormal heartbeat, or when receiving a message that there is a network abnormality sent by the storage node, the target solid-state drive confirms whether the commands in the access connection are all completed;

If all the commands in the access connection are completed, the target solid state drive stops monitoring the CQ.

Among them, heartbeat information is sent regularly between the computing node and the SSD. Once a component on the entire path is abnormal, the heartbeat will be abnormal. When an abnormal heartbeat is detected, the SSD stops monitoring CQ after emptying the task.

Alternatively, when the network is abnormal, the storage node sends a network abnormality message to the SSD, and after the SSD empties the tasks, it stops monitoring the CQ.

Optionally, after step 402, an embodiment of the present application further provides a data access method, which is used to disconnect the above-mentioned access connection. As shown in Figure 5, it includes steps 501-503, which are as follows:

501. The target solid-state disk receives an instruction sent by the storage node to disconnect the access connection;

Specifically, when the computing node no longer needs to access data on the target solid-state disk, the computing node sends a disconnection instruction to the storage node.

502. The target solid-state drive confirms whether the commands in the access connection are all completed;

503. If all the commands in the access connection are completed, the target solid-state disk stops monitoring the CQ.

Specifically, after the SSD empties the commands on the current access connection, it stops monitoring the CQ.

As an implementation manner, if the commands in the current access connection are not completed, the SSD continues to complete each command until all commands are completed, and then stops monitoring the CQ. The SSD also sends a link disconnection success message to the storage node.

Referring to FIG. 6 , a schematic flowchart of a data access method provided by an embodiment of the present application is shown. The data access method includes steps 601-606, which are as follows:

601. The computing node initiates an RDMA connection request to the storage node, where the connection request carries the target SSD identifier;

As an implementation manner, before step 601, the method further includes: the SSD reports the number of connections it supports to the storage node, so that the storage node establishes at least one access between the SSD and the computing node according to the number of connections supported by the SSD connect.

Further, the SSD also reports the CMB address to the storage node, so that the storage node performs subsequent operations of storing the link establishment information according to the CMB address of the SSD.

Alternatively, the storage node notifies the SSD of the available HMB, so that the SSD can subsequently operate the link establishment information stored on the HMB.

Before step 601, the method further includes: the storage node maps the address of the network card register to the SSD, so that the storage node operates the storage node to access data through the address of the network card register during subsequent data access operations.

602. The storage node establishes the chain establishment information (MR, QP, CQ) corresponding to the access connection initiated by the computing node on the shared memory space;

As an implementation manner, the storage node establishes the link establishment information on the CMB space of the target SSD based on the address of the CMB reported by the above-mentioned SSD.

As another implementation manner, the storage node establishes the link establishment information on the host memory buffer HMB of the storage node.

603. The storage node notifies the SSD of the address of the above-mentioned chain establishment information (MR, QP, CQ) in the shared memory space;

604. After parsing the CQ address in the link building information according to the above address, the SSD monitors the command on the CQ;

The storage node notifies the SSD of the address on the shared memory space of the link-building information (MR, QP, CQ) corresponding to the access connection, so that the SSD can obtain the link-building information (MR, QP, CQ) according to the address. The CQ address in the above chain building information (MR, QP, CQ) is parsed out, and then the command on the CQ is monitored.

605. The SSD returns the connection result;

The SSD sends a connection success message to the storage node, and the storage node sends the connection success message to the computing node.

606. Start heartbeat detection between the computing node and the SSD.

For the specific implementation of the above steps, reference may be made to the foregoing embodiments, which will not be repeated here.

Through the embodiment of the present application, the direct access connection between the computing node and the target solid-state hard disk is realized, and the subsequent computing node and the SSD directly interact without software processing of the storage node. The CPU on the storage node side only participates in the process of building the chain of the computing node, and the workload is greatly reduced, which is beneficial to the overall performance improvement. Compared with the prior art, the command sent by the computing node needs to be parsed by the storage node and then sent to the SSD, and the data flow between the SSD and the computing node in the prior art needs to pass through the storage node. In the operation mode of memory transfer, using the direct access connection between the computing node and the target solid-state hard disk provided by the embodiment of the present application can effectively improve the efficiency of data access; It reduces the CPU/memory overhead of storage nodes and effectively eliminates system bottlenecks.

On the basis of the foregoing embodiment, referring to FIG. 7 , a schematic flowchart of a data access method provided by an embodiment of the present application is shown. The data access method includes steps 701-705, which are as follows:

701. The computing node sends a disconnection instruction to the target SSD to the storage node;

702. The storage node sends a link disconnection instruction to the target SSD;

703. After the SSD empties the command on the current access connection, stop monitoring the CQ;

If the command in the current access connection is not completed, the SSD continues to complete each command until all the commands are completed, and then stops monitoring the access connection; and sends a link disconnection success message to the storage node.

704. The storage node releases the chain establishment information (MR, QP, CQ) corresponding to the access connection from the shared memory space;

The storage node receives the message of successful link disconnection sent by the target SSD, and then releases the link establishment information corresponding to the access connection from the shared memory space. For example, from the CMB space of an SSD, or from the storage node's host memory buffer HMB.

705. The storage node returns a link disconnection result to the computing node, and the computing node stops sending heartbeats.

Through the embodiments of the present application, when the computing node no longer needs to establish an access connection with the SSD, the connection between the computing node and the SSD is disconnected, so as to re-establish when data access is required. By adopting this method, system resources can be saved. Among them, the CPU on the storage node side only participates in the work when the computing node is disconnected, and the workload is greatly reduced, which is beneficial to the overall performance improvement.

Referring to FIG. 8 , a data access apparatus provided by an embodiment of the present application is shown. As shown in FIG. 8 , the data access device includes a receiving module 801, an obtaining module 802 and a sending module 803, and the details are as follows:

A receiving module 801, configured to receive a remote direct data access RDMA connection request sent by a computing node in a distributed storage system, where the RDMA connection request carries a target solid-state drive identifier, and the RDMA connection request is used to instruct the storage node to establish an access connection between the computing node and the target SSD;

An obtaining module 802, configured to obtain and store link establishment information corresponding to the access connection, wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state disk has a data access task;

A sending module 803, configured to send the stored first address corresponding to the link establishment information to the target solid-state drive, so as to instruct the target solid-state drive to obtain the CQ in the link establishment information, and monitor the CQ.

For the specific implementation means of the above modules, reference may be made to the foregoing embodiments, which will not be repeated here.

Referring to FIG. 9 , another data access device provided by an embodiment of the present application is shown. As shown in FIG. 9, the data access device includes a receiving module 901 and a processing module 902, as follows:

A receiving module 901, configured to receive a message sent by a storage node in a distributed storage system, where the message carries a first address, where the first address is a computation in the distributed storage system stored by the storage node The address of the link establishment information corresponding to the access connection between the node and the device, the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the device has a data access task;

The processing module 902 is configured to acquire the CQ in the link establishment information according to the first address, and monitor the CQ.

An embodiment of the present application provides a data access device, including a processor, where the processor is configured to: receive a remote direct data access RDMA connection request sent by a computing node in a distributed storage system, where the RDMA connection request carries A target solid-state disk identifier, where the RDMA connection request is used to instruct the device to establish an access connection between the computing node and the target solid-state disk; acquire and store link establishment information corresponding to the access connection, wherein the The link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state drive has a data access task; the first address corresponding to the stored link establishment information is sent to the target solid-state drive to indicate the target solid-state drive. The target solid-state disk acquires the CQ in the link establishment information, and monitors the CQ.

On the other hand, an embodiment of the present application further provides a data access device, including a processor, wherein the processor is configured to: receive a message sent by a storage node in a distributed storage system, where the message carries a first address, The first address is the address of the link establishment information corresponding to the access connection between the computing node in the distributed storage system and the device stored by the storage node, and the link establishment information includes the completion queue CQ ; Acquire the CQ in the link establishment information according to the first address, and monitor the CQ.

The present application provides a data access device. As shown in FIG. 10 , the data access device 1000 includes at least one processor 1001 , at least one memory 1002 and at least one communication interface 1003 . The processor 1001, the memory 1002 and the communication interface 1003 are connected through the communication bus and complete the communication with each other.

The processor 1001 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the above programs.

The communication interface 1003 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (Wireless Local Area Networks, WLAN).

Memory 1002 may be read-only memory (ROM) or other type of static storage device that can store static information and instructions, random access memory (RAM), or other type of static storage device that can store information and instructions It can also be an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of being executed by a computer Access any other medium without limitation. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

Wherein, the memory 1002 is used for storing the application code for executing the above solution, and the execution is controlled by the processor 1001 . The processor 1001 is configured to execute the application code stored in the memory 1002 .

The code stored in memory 1002 may perform any of the data access methods provided above.

An embodiment of the present application further provides a chip system, the chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor pass through line interconnection; the interface circuit is used to receive signals from the memory of the electronic device and send the signals to the processor, the signals include computer instructions stored in the memory; when the processor executes the When executing the computer instructions, the electronic device performs the method.

Embodiments of the present application also provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer or processor is run on a computer or a processor, the computer or the processor is made to execute any one of the above methods. or multiple steps.

Embodiments of the present application also provide a computer program product including instructions. The computer program product, when run on a computer or processor, causes the computer or processor to perform one or more steps of any of the above methods.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted over a computer-readable storage medium. The computer instructions can be sent from one website site, computer, server or data center to another website site, computer, server or data center for transmission. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.

In this application, "at least one" means one or more, and "plurality" means two or more. "And/or", which describes the association relationship of the associated objects, indicates that there can be three kinds of relationships, for example, A and/or B, which can indicate: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A, B can be singular or plural.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented. The process can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed , which may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random storage memory RAM, magnetic disk or optical disk and other mediums that can store program codes.

The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application shall be covered by this within the protection scope of the application examples. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

A data access method, characterized in that it is applied to a storage node in a distributed storage system, the method comprising:

The storage node receives a remote direct data access RDMA connection request sent by a computing node in the distributed storage system, where the RDMA connection request carries the target solid-state disk identifier in the distributed storage system, and the RDMA connection request The request is used to instruct the storage node to establish an access connection between the computing node and the target solid-state disk;

The storage node acquires and stores the link establishment information corresponding to the access connection, wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state disk has a data access task;

The storage node sends the stored first address corresponding to the link establishment information to the target solid-state drive, so as to instruct the target solid-state drive to acquire the CQ in the link establishment information, and monitor the CQ.
The method according to claim 1, wherein the link establishment information further comprises a memory area MR and a queue pair QP, the MR is used to indicate a data access area of the target solid-state disk, and the QP is used to Instructing the target solid state drive to operate the QP for data transmission and reception.
The method according to claim 2, wherein the storage node stores the link establishment information corresponding to the access connection in a shared memory space, wherein the shared memory space is a controller of the target solid state disk The memory buffer, or the shared memory space is the host memory buffer of the storage node.
The method according to claim 3, wherein the method further comprises:

The storage node detects the access connection;

When a network abnormality is detected, or when a message of abnormal heartbeat is received from the target solid-state disk, the storage node deletes the link establishment information on the shared memory space.
The method according to claim 3, wherein the method further comprises:

The storage node sends the received instruction of disconnecting the access connection sent by the computing node to the target solid-state drive;

The storage node deletes the link establishment information on the shared memory space when receiving the message that the link disconnection is successful sent by the target solid-state hard disk.
A data access method, characterized in that it is applied to a target solid-state hard disk in a distributed storage system, the method comprising:

The target solid-state disk receives a message sent by a storage node in the distributed storage system, and the message carries a first address, where the first address is an address in the distributed storage system stored by the storage node. The address of the link establishment information corresponding to the access connection between the computing node and the target SSD, the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target SSD has a data access task;

The target solid state disk acquires the CQ in the link establishment information according to the first address, and monitors the CQ.
The method according to claim 6, wherein the link establishment information further comprises a memory region MR and a queue pair QP, and the method further comprises:

The target solid state disk acquires the MR and QP in the link establishment information according to the first address, and performs data access.
The method according to claim 7, wherein the storage node stores the link establishment information corresponding to the access connection in a shared memory space, wherein the shared memory space is a controller of the target solid state disk The memory buffer, or the shared memory space is the host memory buffer of the storage node.
The method according to claim 7 or 8, wherein the method further comprises:

The target solid-state disk detects the access connection;

When detecting that there is an abnormal heartbeat, or when receiving a message that there is a network abnormality sent by the storage node, the target solid-state drive confirms whether the commands in the access connection are all completed;

If all the commands in the access connection are completed, the target solid state drive stops monitoring the CQ.
The method according to claim 7 or 8, wherein the method further comprises:

receiving, by the target solid-state disk, an instruction sent by the storage node to disconnect the access connection;

The target SSD confirms whether the commands in the access connection are all completed;

If all the commands in the access connection are completed, the target solid state drive stops monitoring the CQ.
A data access device, comprising a processor, wherein the processor is used for:

Receive a remote direct data access RDMA connection request sent by a computing node in a distributed storage system, where the RDMA connection request carries a target solid-state disk identifier in the distributed storage system, and the RDMA connection request is used to indicate the The processor establishes an access connection between the computing node and the target solid-state disk;

Acquire and store the link establishment information corresponding to the access connection, wherein the link establishment information includes a completion queue CQ, and the CQ is used to indicate that the target solid-state disk has a data access task;

The first address corresponding to the stored link establishment information is sent to the target solid-state disk, so as to instruct the target solid-state disk to obtain the CQ in the chain establishment information, and monitor the CQ.
The device according to claim 11, wherein the link establishment information further comprises a memory area MR and a queue pair QP, the MR is used to indicate a data access area of the target solid-state disk, and the QP is used to Instructing the target solid state drive to operate the QP for data transmission and reception.
The device according to claim 12, wherein the processor stores the link establishment information corresponding to the access connection in a shared memory space, wherein the shared memory space is a controller of the target solid-state hard disk The memory buffer, or the shared memory space is the host memory buffer of the device.
The apparatus of claim 13, wherein the processor is further configured to:

detecting the access connection;

When a network abnormality is detected, or when a heartbeat abnormality message sent by the target solid-state disk is received, the link establishment information on the shared memory space is deleted.
The apparatus of claim 13, wherein the processor is further configured to:

sending the received instruction to disconnect the access connection sent by the computing node to the target solid-state drive;

When receiving the message of successful link disconnection sent by the target solid-state hard disk, the link establishment information on the shared memory space is deleted.
A data access device, comprising a processor, wherein the processor is used for:

Receive a message sent by a storage node in a distributed storage system, where the message carries a first address, where the first address is the relationship between the computing node in the distributed storage system and the device stored by the storage node. The address of the link establishment information corresponding to the access connection between, the link establishment information includes the completion queue CQ, and the CQ is used to indicate that the device has a data access task;

Acquire the CQ in the link establishment information according to the first address, and monitor the CQ.
The device according to claim 16, wherein the link establishment information further comprises a memory region MR and a queue pair QP, and the processor is further configured to:

Acquire the MR and QP in the link establishment information according to the first address, and perform data access.
The device according to claim 17, wherein the storage node stores the link establishment information corresponding to the access connection in a shared memory space, wherein the shared memory space is a controller memory buffer of the device or, the shared memory space is the host memory buffer of the storage node.
The apparatus according to claim 17 or 18, wherein the processor is further configured to:

detecting the access connection;

When detecting that there is an abnormal heartbeat, or when receiving a message that the storage node has a network abnormality, confirming whether the commands in the access connection are all completed;

If all the commands in the access connection are completed, stop monitoring the CQ.
The apparatus according to claim 17 or 18, wherein the processor is further configured to:

receiving an instruction to disconnect the access connection sent by the storage node;

Confirm whether the commands in the access connection are completed;

If all the commands in the access connection are completed, stop monitoring the CQ.
A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1 to 10.
A computer program product, characterized in that, when the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 10.
A distributed storage system, characterized by comprising a computing node, further comprising the data access device according to any one of claims 11 to 15, and the data access device according to any one of claims 16 to 20 device.