CN116670636A

CN116670636A - Data access method, device and storage medium

Info

Publication number: CN116670636A
Application number: CN202180085306.XA
Authority: CN
Inventors: 伍波
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-01-30
Filing date: 2021-01-30
Publication date: 2023-08-29
Also published as: WO2022160308A1

Abstract

The embodiment of the application provides a data access method, a data access device and a storage medium, comprising the following steps: a storage node receives a remote direct data access (RDMA) connection request sent by a computing node, wherein the RDMA connection request carries a target solid state disk identifier, and the RDMA connection request is used for indicating the storage node to establish access connection between the computing node and the target solid state disk; the storage node acquires and stores link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ; the storage node sends a first address corresponding to the stored link establishment information to the target solid state disk so as to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ. By adopting the embodiment of the application, the data access efficiency can be effectively improved. Meanwhile, CPU/memory overhead of the storage node is reduced, and system bottlenecks can be effectively eliminated.

Description

Data access method, device and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data access method, apparatus, and storage medium.

Background

At present, in a distributed storage system formed by three-level models of existing computing nodes, storage nodes and storage media, the storage nodes are positioned at middle positions and play roles of protocol conversion and data caching, and the working efficiency of the distributed storage system can directly influence the performance of the whole system. A single storage node may typically provide a bandwidth of around 10-20 GB, which may provide an Input/Output Per Second (IOPS) per second capability of approximately 100-1000K. However, the bandwidth that a single Non-volatile storage device connection protocol (Non-Volatile Memory Express, NVMe) interface Solid State Disk (SSD) can provide can typically reach 5gb and the iops can reach 1200K. The 24 SSDs are managed by one storage node to calculate, and the 24 SSDs can provide about 6 times and 28 times of bandwidth and IOPS respectively. It can be seen that the performance of the storage nodes is far from fully exploiting the high performance advantages of SSD disks.

The method is characterized in that in the existing scheme, no connection exists between the computing node and the SSD, all commands sent by the computing node are required to be analyzed by software on the storage node, and the efficiency is extremely low. And after the storage node software analyzes, the storage node software sends the NVMe command to the SSD, which consumes certain CPU resources, and as the performance of the SSD increases, the IOPS further increases, and the subsequent CPU may become a performance bottleneck. On the other hand, the data flow between the SSD and the computing node needs to be transferred through the memory on the storage node, and the memory bandwidth is limited, so that the subsequent memory can become a performance bottleneck as the SSD read-write bandwidth is further improved.

Disclosure of Invention

The application discloses a data access method, a data access device and a storage medium, which can improve the data access efficiency, reduce the CPU/memory overhead of a storage node and eliminate the system bottleneck.

In a first aspect, an embodiment of the present application provides a data access method, applied to a storage node in a distributed storage system, where the method includes: the storage node receives a remote direct data access (RDMA) connection request sent by a computing node in the distributed storage system, wherein the RDMA connection request carries a target solid state disk identifier in the distributed storage system, and the RDMA connection request is used for indicating the storage node to establish access connection between the computing node and the target solid state disk; the storage node acquires and stores link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk; the storage node sends a first address corresponding to the stored link establishment information to the target solid state disk so as to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.

According to the embodiment of the application, the storage node sends the address of the link establishment information corresponding to the access connection between the stored computing node and the target solid state disk to the target solid state disk, so that the target solid state disk obtains the CQ in the link establishment information and monitors the CQ, and further the direct access connection between the computing node and the target solid state disk is realized. Compared with the operation mode that the command sent by the computing node in the prior art needs to be analyzed and processed by the storage node and then sent to the SSD, and the operation mode that the data flow between the SSD and the computing node in the prior art needs to be transferred through the memory on the storage node, the direct access connection between the computing node and the target solid state disk provided by the embodiment of the application can effectively improve the efficiency of data access; meanwhile, as the computing node and the target solid state disk are directly accessed and connected, CPU/memory overhead of the storage node is reduced, and system bottlenecks can be effectively eliminated.

Further, the link establishment information further includes a memory region MR and a queue pair QP, where the MR and QP are used to instruct the target solid state disk to perform data access. Specifically, the MR is configured to instruct the target solid state disk to access the data, and the QP is configured to instruct the target solid state disk to operate the QP to send and receive data.

Among them, the QP includes a Send Queue (SQ) and a receive queue (ReceiveQueue, RQ) in which various types of messages are managed. When the target solid state disk obtains the data access task from the CQ, the target solid state disk receives and transmits data by operating the QP. The SSD places a task of sending data in a sending queue SQ so as to send the data. On the other hand, the SSD acquires a task of receiving data from the reception queue, and further performs data reception processing.

Optionally, the storage node stores the link establishment information corresponding to the access connection in a shared memory space, where the shared memory space is a controller memory buffer (Controller Memory Buffer, CMB) of the target solid state disk, or the shared memory space is a host memory buffer (Host Memory Buffer, HMB) of the storage node.

As an alternative implementation, the method further includes: the storage node detects the access connection; when network abnormality is detected, or when a message with heartbeat abnormality sent by the target solid state disk is received, the storage node deletes the link establishment information in the shared memory space.

By timely releasing access connection between SSD and the computing node when abnormality is detected, safety of data access can be guaranteed.

As yet another alternative implementation, the method further includes: the storage node sends the received instruction for disconnecting the access connection, which is sent by the computing node, to the target solid state disk; and when receiving a successful link breaking message sent by the target solid state disk, the storage node deletes the link building information in the shared memory space.

When the computing node no longer needs to establish an access connection with the SSD, the connection between the computing node and the SSD is disconnected so as to be established again when data access is required. By adopting the means, system resources can be saved.

In a second aspect, an embodiment of the present application provides a data access method, which is applied to a target solid state disk in a distributed storage system, where the method includes: the target solid state disk receives a message sent by a storage node in the distributed storage system, wherein the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node in the distributed storage system and the target solid state disk, which is stored by the storage node, the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk; and the target solid state disk acquires the CQ in the link establishment information according to the first address and monitors the CQ.

According to the embodiment of the application, the target solid state disk is based on the address sent by the storage node, so that the CQ in the link establishment information corresponding to the access connection between the computing node and the target solid state disk is obtained, the CQ is monitored, and the direct access connection between the computing node and the target solid state disk is realized. Compared with the operation mode that the command sent by the computing node in the prior art needs to be analyzed and processed by the storage node and then sent to the SSD, and the operation mode that the data flow between the SSD and the computing node in the prior art needs to be transferred through the memory on the storage node, the direct access connection between the computing node and the target solid state disk provided by the embodiment of the application can effectively improve the efficiency of data access; meanwhile, as the computing node and the target solid state disk are directly accessed and connected, CPU/memory overhead of the storage node is reduced, and system bottlenecks can be effectively eliminated.

Further, the link establishment information further includes a memory region MR and a queue pair QP, and the method further includes: and the target solid state disk acquires MR and QP in the link establishment information according to the first address, and performs data access.

Optionally, the storage node stores the link establishment information corresponding to the access connection in a shared memory space, where the shared memory space is a controller memory buffer of the target solid state disk, or the shared memory space is a host memory buffer of the storage node.

As an alternative implementation, the method further includes: the target solid state disk detects the access connection; when detecting that the heartbeat is abnormal, or when receiving a message of network abnormality sent by the storage node, the target solid state disk confirms whether the commands in the access connection are all completed; and if the commands in the access connection are all completed, stopping monitoring the CQ by the target solid state disk.

As another alternative implementation, the method further includes: the target solid state disk receives an instruction for disconnecting the access connection sent by the storage node; the target solid state disk confirms whether the commands in the access connection are all completed or not; and if the commands in the access connection are all completed, stopping monitoring the CQ by the target solid state disk.

By disconnecting the SSD and the compute node for re-establishment when communication is required. By adopting the means, system resources can be saved.

In a third aspect, an embodiment of the present application provides a data access apparatus, including: the remote direct data access (RDMA) connection request is sent by a computing node in the distributed storage system, the RDMA connection request carries a target solid state disk identifier, and the RDMA connection request is used for indicating the storage node to establish access connection between the computing node and the target solid state disk; the acquisition module is used for acquiring and storing the link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that the target solid state disk has a data access task; the sending module is used for sending the stored first address corresponding to the link establishment information to the target solid state disk so as to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.

The link establishment information further comprises a memory region MR and a queue pair QP, wherein the MR is used for indicating the region of the target solid state disk for data access, and the QP is used for indicating the target solid state disk to operate the QP for data transceiving.

The device stores the link establishment information corresponding to the access connection in a shared memory space. The shared memory space is a controller memory buffer area of the target solid state disk, or the shared memory space is a host memory buffer area of the device.

Optionally, the device further comprises a detection module, configured to detect the access connection; and deleting the link establishment information in the shared memory space when network abnormality is detected or when a message with heartbeat abnormality sent by the target solid state disk is received.

Optionally, the device further includes a chain breaking module, configured to send the received instruction for disconnecting the access connection sent by the computing node to the target solid state disk; and deleting the link establishment information in the shared memory space when receiving a successful link breakage message sent by the target solid state disk.

In a fourth aspect, an embodiment of the present application provides a data access apparatus, including: the device comprises a receiving module, a receiving module and a processing module, wherein the receiving module is used for receiving a message sent by a storage node in a distributed storage system, the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node and the device in the distributed storage system, which is stored by the storage node, and the link establishment information comprises a completion queue CQ; and the processing module is used for acquiring the CQ in the link establishment information according to the first address and monitoring the CQ.

Optionally, the link establishment information further includes a memory region MR and a queue pair QP, and the processing module is further configured to obtain the MR and the QP in the link establishment information according to the first address, and perform data access.

The storage node stores the link establishment information corresponding to the access connection in a shared memory space, wherein the shared memory space is a controller memory buffer zone of the device, or the shared memory space is a host memory buffer zone of the storage node.

Optionally, the device further comprises a detection module, configured to detect the access connection; when detecting that the heartbeat is abnormal, or when receiving a message of existence of network abnormality sent by the storage node, confirming whether commands in the access connection are all completed; and stopping monitoring the CQ if all the commands in the access connection are completed.

Optionally, the device further includes a chain breaking module, configured to receive an instruction sent by the storage node to break the access connection; confirming whether commands in the access connection are all completed; and stopping monitoring the CQ if all the commands in the access connection are completed.

In a fifth aspect, an embodiment of the present application provides a data access apparatus, including a processor, where the processor is configured to: receiving a remote direct data access (RDMA) connection request sent by a computing node in a distributed storage system, wherein the RDMA connection request carries a target solid state disk identifier and is used for indicating the storage node to establish access connection between the computing node and the target solid state disk; acquiring and storing link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk; and sending the stored first address corresponding to the link establishment information to the target solid state disk to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.

Further, the link establishment information further includes a memory region MR and a queue pair QP, where the MR is configured to indicate a region where the target solid state disk data is accessed, and the QP is configured to indicate the target solid state disk to operate the QP to send and receive data.

The processor stores the link establishment information corresponding to the access connection in a shared memory space, wherein the shared memory space is a controller memory buffer zone of the target solid state disk, or the shared memory space is a host memory buffer zone of the device.

As an alternative implementation, the processor is further configured to: detecting the access connection; and deleting the link establishment information in the shared memory space when network abnormality is detected or when a message with heartbeat abnormality sent by the target solid state disk is received.

As another alternative implementation, the processor is further configured to: transmitting the received instruction for disconnecting the access connection, which is transmitted by the computing node, to the target solid state disk; and deleting the link establishment information in the shared memory space when receiving a successful link breakage message sent by the target solid state disk.

In a sixth aspect, an embodiment of the present application provides a data access apparatus, including a processor, where the processor is configured to: receiving a message sent by a storage node in a distributed storage system, wherein the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node and the device in the distributed storage system stored by the storage node, and the link establishment information comprises a completion queue CQ; and acquiring the CQ in the link establishment information according to the first address, and monitoring the CQ.

Further, the link establishment information further includes a memory region MR and a queue pair QP, and the processor is further configured to: and acquiring MR and QP in the link establishment information according to the first address, and performing data access.

As an alternative implementation, the processor is further configured to: detecting the access connection; when detecting that the heartbeat is abnormal, or when receiving a message of existence of network abnormality sent by the storage node, confirming whether commands in the access connection are all completed; and stopping monitoring the CQ if all the commands in the access connection are completed.

As another alternative implementation, the processor is further configured to: receiving an instruction for disconnecting the access connection sent by the storage node; confirming whether commands in the access connection are all completed; and stopping monitoring the CQ if all the commands in the access connection are completed.

In a seventh aspect, the present application provides a computer storage medium comprising computer instructions which, when run on an electronic device, cause the electronic device to perform a method as provided by any one of the possible implementations of the first aspect to any one of the possible implementations of the second aspect.

In an eighth aspect, the present embodiments provide a computer program product which, when run on a computer, causes the computer to perform the method as provided by any one of the possible implementations of the first aspect and any one of the possible implementations of the second aspect.

In a ninth aspect, an embodiment of the present application provides a chip system, where the chip system is applied to an electronic device; the system-on-chip includes one or more interface circuits, and one or more processors; the interface circuit and the processor are interconnected through a circuit; the interface circuit is configured to receive a signal from a memory of the electronic device and to send the signal to the processor, the signal including computer instructions stored in the memory; the electronic device performs the method when the processor executes the computer instructions.

In a tenth aspect, an embodiment of the present application provides a distributed storage system, including a computing node, and further including the data access device described above.

It will be appreciated that the apparatus according to the third aspect, the apparatus according to the fourth aspect, the apparatus according to the fifth aspect, the apparatus according to the sixth aspect, the computer storage medium according to the seventh aspect, the computer program product according to the eighth aspect, or the chip system according to the ninth aspect provided above are each adapted to perform the method provided in any one of the first aspects and any one of the possible implementations of the second aspect. Therefore, the advantages achieved by the method can be referred to as the advantages of the corresponding method, and will not be described herein.

Drawings

In order to more clearly describe the embodiments of the present application or the technical solutions in the background art, the following will briefly describe the drawings related to the embodiments of the present application or the background art.

FIG. 1a is a schematic diagram of a prior art networking of a distributed storage system;

FIG. 1b is a business schematic diagram of a prior art distributed storage system;

FIG. 2 is a flow chart of a data access method according to an embodiment of the present application;

FIG. 3a is a schematic business diagram of a distributed storage system according to an embodiment of the present application;

FIG. 3b is a flowchart illustrating a data access method according to an embodiment of the present application;

FIG. 4 is a flow chart of a data access method according to an embodiment of the present application;

FIG. 5 is a flowchart illustrating another data access method according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a data access method according to an embodiment of the present application;

FIG. 7 is a schematic diagram of another data access method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a data access device according to an embodiment of the present application;

FIG. 9 is a schematic diagram of another data access device according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of another data access device according to an embodiment of the present application.

Detailed Description

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

Referring to FIG. 1a, a schematic diagram of a distributed storage system in the prior art is shown. The system comprises a computing cluster, a storage cluster and a plurality of solid state disks SSDs. The computing cluster is used for computing and generating data; the storage cluster is used for storing data and providing access to the data, and guaranteeing the reliability and the safety of the data. Wherein the computing cluster comprises a plurality of computing nodes; the storage cluster comprises a plurality of storage nodes; the storage node may be connected with multiple SSDs. The computing nodes communicate with the storage nodes via a network for storage, access, etc. of data. Fig. 1b is a schematic service diagram of a distributed storage system in the prior art. The computing nodes and the storage nodes communicate in a remote direct data access (Remote Direct Memory Access, RDMA) connection mode. The storage nodes are connected with the SSD through a peripheral equipment high-speed connection protocol (Peripheral Component Interconnect Express, PCIE), and NVME protocol communication is adopted. The computing node is not connected with the SSD; all Input/Output (IO) operation commands OF the computing node are sent to the storage node in an RDMA mode by adopting a Non-volatile storage standard (Non-Volatile Memory Express Over Fabrics, NVMe-OF) based on a network, and specific IO commands are converted into NVME commands by software on the storage node and sent to the SSD. The read-write data flow on the SSD needs to be transferred through a dynamic random access memory (Dynamic Random Access Memory, DRAM) on the storage node, and interacts with the computing node in an RDMA mode.

All commands sent by the computing node need to be analyzed by software on the storage node, so that the efficiency is extremely low. After the storage node software analyzes, a command is sent to the SSD, and certain CPU resources are consumed in the operation. Meanwhile, the data flow between the SSD and the computing node needs to be transferred through the memory on the storage node, and the subsequent memory can become a performance bottleneck as the SSD read-write bandwidth is further improved.

Therefore, the embodiment of the application provides a data access method, and a storage node sends an address of link establishment information corresponding to access connection between a stored computing node and a target solid state disk to the target solid state disk, so that the target solid state disk acquires CQ in the link establishment information and monitors the CQ, and further direct access connection between the computing node and the target solid state disk is realized. Compared with the operation mode that the command sent by the computing node in the prior art needs to be analyzed and processed by the storage node and then sent to the SSD, and the operation mode that the data flow between the SSD and the computing node in the prior art needs to be transferred through the memory on the storage node, the direct access connection between the computing node and the target solid state disk provided by the embodiment of the application can effectively improve the efficiency of data access; meanwhile, as the computing node and the target solid state disk are directly accessed and connected, CPU/memory overhead of the storage node is reduced, and system bottlenecks can be effectively eliminated.

The data access method provided by the embodiment of the application is described in detail below.

Referring to fig. 2, a flow chart of a data access method according to an embodiment of the application is shown. The data access method is applied to storage nodes in the distributed storage system. The method comprises the steps 201-203, specifically as follows:

201. a storage node receives a remote direct data access (RDMA) connection request sent by a computing node, wherein the RDMA connection request carries a target solid state disk identifier, and the RDMA connection request is used for indicating the storage node to establish access connection between the computing node and the target solid state disk;

the target solid state disk identifier may be an ID of the target solid state disk SSD, etc.

202. The storage node acquires and stores link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk;

the Completion Queue (CQ) is used for indicating that the target solid state disk has a data access task, that is, for informing the SSD that there is a command to be processed.

Further, the link establishment information further includes a Memory Region (MR) and a Queue Pair (QP).

Among them, the QP includes a Send Queue (SQ) and a receive queue (ReceiveQueue, RQ) in which various types of messages are managed. When the target solid state disk obtains the data access task from the CQ, the target solid state disk receives and transmits data by operating the QP. The SSD places a task of sending data in a sending queue SQ so as to send the data. On the other hand, the SSD acquires a task of receiving data from the reception queue, and further performs data reception processing. Specifically, the QP will be mapped to the virtual address space of the application so that the application communicates directly with the peer through the QP, such as to operate links, control transceiving data, etc.

The MR is a memory area for accessing data.

The MR, QP, CQ described above may be simply referred to as link establishment information (MR, QP, CQ).

Wherein multiple RDMA connections may be established between the storage node and the SSD. That is, there is linking information corresponding to each RDMA connection in the plurality of RDMA connections.

The storage node stores the link establishment information related to the RDMA connection based on the received RDMA connection request. And the storage node stores the link establishment information corresponding to the access connection in a shared memory space.

As an implementation manner, the storage node establishes the link establishment information corresponding to the access connection in the controller memory buffer (Controller Memory Buffer, CMB) of the target solid state disk. That is, the first address is an address of the storage node that establishes the link establishment information corresponding to the access connection in the controller memory buffer CMB of the target solid state disk.

As another implementation, the storage node establishes the link establishment information corresponding to the access connection in a host memory buffer (Host Memory Buffer, HMB) of the storage node. That is, the first address is an address where the storage node establishes the link establishment information corresponding to the access connection on the host memory buffer HMB of the storage node.

That is, the shared memory space may be the controller memory buffer CMB of the target solid state disk, or may be the host memory buffer HMB of the storage node.

Fig. 3a is a schematic business diagram of a distributed storage system according to an embodiment of the present application. The storage node establishes the link establishment information (MR, QP, CQ) on the target solid state disk so that the target solid state disk can directly access data with the computing node through the link establishment information.

203. The storage node sends a first address corresponding to the stored link establishment information to the target solid state disk so as to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.

As one implementation, the storage node notifies the SSD of the address of the link establishment information corresponding to the access connection on the CMB space of the SSD, so that the SSD obtains the CQ in the link establishment information according to the address, parses out the CQ address, and then listens for a command on the CQ.

As another implementation manner, the storage node notifies the SSD of the address on the host memory buffer HMB of the storage node of the link establishment information corresponding to the access connection, so that the SSD obtains the CQ in the link establishment information according to the address, parses out the CQ address, and then listens for the command on the CQ.

Further, the storage node forwards the message of successful connection sent by the target solid state disk to the computing node.

And the storage node receives a response message which is sent by the target solid state disk and is successfully connected, and forwards the response message to the computing node so as to start heartbeat detection between the computing node and the SSD.

As an implementation, after step 203, the method further includes:

the storage node detects the access connection;

when network abnormality is detected, or when a message with heartbeat abnormality sent by the target solid state disk is received, the storage node deletes the link establishment information in the shared memory space.

The network anomaly may specifically be a network outage or the like. Such as network card failure, network card loosening, etc., or storage node/compute node crashes, etc. It should be noted that, under the condition of network abnormality, the storage node may delete all the link establishment information established on the target solid state disk.

The heartbeat information is sent between the computing node and the SSD at regular time, and once an abnormality occurs in a certain part on the whole path, the heartbeat abnormality can be caused. The abnormal heartbeat refers to: for example, the transmitted heartbeat message does not receive a corresponding response. At this point, the SSD may inform the storage node that there is a heartbeat anomaly by sending a message to the storage node.

Optionally, after step 203, the embodiment of the present application further provides a data access method, which is used for disconnecting the access connection. As shown in fig. 3b, it comprises steps 301-302, which are specifically as follows:

301. The storage node sends the received instruction for disconnecting the access connection, which is sent by the computing node, to the target solid state disk;

specifically, when a computing node no longer needs to access data on the target solid state disk, the computing node sends a broken link instruction to a storage node.

302. And when receiving a successful link breaking message sent by the target solid state disk, the storage node deletes the link building information in the shared memory space.

After the target solid state disk receives the instruction for disconnecting the access connection, the monitoring of the CQ is stopped after the command on the current connection is completed (namely, the command is emptied). And then sending a message of successful link breakage to the storage node.

Specifically, the storage node receives a message that the link failure sent by the target solid state disk succeeds, and then releases the link establishment information (MR, QP, CQ) related to the access connection from the shared memory space. For example, from the CMB space of the SSD or from the host memory buffer HMB of the storage node.

Further, the storage node sends the message of successful disconnection to the computing node.

Specifically, the storage node returns a broken link result to the computing node. The computing node stops sending heartbeat information.

On the other hand, when the data transfer is carried out by using the CMB space of the SSD, the memory of the storage node is not used, so that the bandwidth bottleneck of the memory of the storage node is avoided, and the parallel expansion of the SSD is carried out. Under the condition that SSD connection resources are limited, the storage node and the SSD cooperate to carry out connection management, and a scene of mass connection can be adapted.

Referring to fig. 4, a flow chart of another data access method according to an embodiment of the application is shown. The data access method is applied to the target solid state disk in the distributed storage system. The method comprises the steps 401-402, and specifically comprises the following steps:

401. the method comprises the steps that a target solid state disk receives a message sent by a storage node in a distributed storage system, wherein the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node in the distributed storage system and the target solid state disk, which is stored by the storage node, and the link establishment information comprises a completion queue CQ;

Further, the link establishment information further includes a Memory Region (MR) and a Queue Pair (QP). Among them, the QP includes a Send Queue (SQ) and a receive queue (ReceiveQueue, RQ) in which various types of messages are managed. When the target solid state disk obtains the data access task from the CQ, the target solid state disk receives and transmits data by operating the QP. The SSD places a task of sending data in a sending queue SQ so as to send the data. On the other hand, the SSD acquires a task of receiving data from the reception queue, and further performs data reception processing. The QP will be mapped to the virtual address space of the application so that the application communicates directly with the peer through the QP, such as to operate links, control transceiving data, etc. The MR is a memory area for accessing data.

The method further comprises the steps of: and the target solid state disk acquires MR and QP in the link establishment information according to the first address, and performs data access.

The storage node stores the link establishment information corresponding to the access connection in a shared memory space. Specifically, the shared memory space is a controller memory buffer of the target solid state disk, or the shared memory space is a host memory buffer of the storage node.

As an implementation manner, the first address is an address where the storage node establishes the link establishment information corresponding to the access connection on the controller memory buffer CMB of the target solid state disk.

As another implementation manner, the first address is an address where the storage node establishes the link establishment information corresponding to the access connection on the host memory buffer HMB of the storage node.

402. And the target solid state disk acquires the CQ in the link establishment information according to the first address and monitors the CQ.

After receiving the first address sent by the storage node, the SSD acquires the CQ in the link establishment information according to the first address, and then monitors commands on the CQ.

And simultaneously, the SSD transmits a response message of successful connection to the computing node through the storage node, so that heartbeat detection is started between the computing node and the SSD.

Optionally, the target solid state disk SSD listens for a command on the CQ, for example, after receiving a command to write data, the target solid state disk SSD operates the storage node to write data through an address of a network card register. The storage node maps the address of the network card register to the SSD in advance. The present embodiment is described by taking writing data as an example, and may be other commands, which is not particularly limited.

As an implementation, after step 402, the method further includes:

the target solid state disk detects the access connection;

when detecting that the heartbeat is abnormal, or when receiving a message of network abnormality sent by the storage node, the target solid state disk confirms whether the commands in the access connection are all completed;

and if the commands in the access connection are all completed, stopping monitoring the CQ by the target solid state disk.

The heartbeat information is sent between the computing node and the SSD at regular time, and once an abnormality occurs in a certain part on the whole path, the heartbeat abnormality can be caused. When the presence of a heartbeat anomaly is detected, the SSD stops monitoring CQ after draining the task.

Or when the network is abnormal, the storage node sends a message that the network is abnormal to the SSD, and the SSD stops monitoring the CQ after the task is emptied.

Optionally, after step 402, an embodiment of the present application further provides a data access method, which is used to disconnect the access connection. As shown in fig. 5, it includes steps 501-503, which are specifically as follows:

501. the target solid state disk receives an instruction for disconnecting the access connection sent by the storage node;

specifically, when a computing node no longer needs to access data on the target solid state disk, the computing node sends a chain break instruction to a storage node.

502. The target solid state disk confirms whether the commands in the access connection are all completed or not;

503. and if the commands in the access connection are all completed, stopping monitoring the CQ by the target solid state disk.

Specifically, after the SSD empties the command on the current access connection, the monitoring CQ is stopped.

As an implementation, if the commands in the current access connection are not completed, the SSD continues to complete each command until all commands are completed, and then stops monitoring CQ. The SSD also sends a message to the storage node that the disconnection was successful.

Referring to fig. 6, a flow chart of a data access method according to an embodiment of the application is shown. The data access method comprises steps 601-606, specifically as follows:

601. initiating an RDMA connection request to a storage node by a computing node, wherein the connection request carries a destination SSD (solid State disk) identifier;

as an implementation, before step 601, the method further includes: the SSD reports the number of connections it supports to the storage node, so that the storage node establishes at least one access connection between the SSD and the computing node according to the number of connections supported by the SSD.

Further, the SSD also reports the CMB address to the storage node, so that the storage node can perform subsequent operations for storing the link establishment information according to the CMB address of the SSD.

Alternatively, the storage node notifies the SSD of the usable HMB so that the SSD subsequently operates on the link establishment information stored on the HMB.

Wherein, before step 601, the method further comprises: the storage node maps the address of the network card register to the SSD, so that the subsequent SSD can access data through the address of the network card register when performing data access operation.

602. The storage node establishes link establishment information (MR, QP, CQ) corresponding to the access connection initiated by the computing node on the shared memory space;

As an implementation manner, the storage node further establishes the link establishment information on the CMB space of the destination SSD based on the address of the CMB reported by the SSD.

As another implementation, the storage node builds the linking information on the host memory buffer HMB of the storage node.

603. The storage node informs the SSD of the address of the link establishment information (MR, QP, CQ) on the shared memory space;

604. after resolving CQ addresses in the link establishment information according to the addresses, the SSD monitors commands on the CQ;

the storage node notifies the SSD of the address of the link establishment information (MR, QP, CQ) corresponding to the access connection on the shared memory space, so that the SSD acquires the link establishment information (MR, QP, CQ) according to the address, and then monitors commands on the CQ by resolving the CQ address in the link establishment information (MR, QP, CQ).

605. SSD returns a connection result;

the SSD sends the message of successful connection to the storage node, and the storage node sends the message of successful connection to the computing node.

606. Heartbeat detection is started between the computing node and the SSD.

The specific implementation of the above steps may refer to the foregoing embodiments, and are not repeated herein.

According to the embodiment of the application, the direct access connection between the computing node and the target solid state disk is realized, the subsequent computing node and the SSD are directly interacted, and the processing of storage node software is not needed. The CPU at the storage node side only participates in the work in the process of calculating the node link establishment, so that the workload is greatly reduced, and the overall performance is improved. Compared with the operation mode that the command sent by the computing node in the prior art needs to be analyzed and processed by the storage node and then sent to the SSD, and the operation mode that the data flow between the SSD and the computing node in the prior art needs to be transferred through the memory on the storage node, the direct access connection between the computing node and the target solid state disk provided by the embodiment of the application can effectively improve the efficiency of data access; meanwhile, as the computing node and the target solid state disk are directly accessed and connected, CPU/memory overhead of the storage node is reduced, and system bottlenecks can be effectively eliminated.

On the basis of the above embodiments, referring to fig. 7, a flow chart of a data access method according to an embodiment of the present application is shown. The data access method comprises steps 701-705, which are specifically as follows:

701. The computing node sends a broken link instruction to the destination SSD to the storage node;

702. the storage node sends a broken link instruction to a destination SSD;

703. after the SSD empties the command on the current access connection, stopping monitoring the CQ;

if the command in the current access connection is not finished, the SSD continuously completes each command until all commands are completed, and then the monitoring of the access connection is stopped; and sending a message of successful link breakage to the storage node.

704. The storage node releases the link establishment information (MR, QP, CQ) corresponding to the access connection from the shared memory space;

and the storage node receives a message of successful link breakage sent by the target solid state disk, and then releases link establishment information corresponding to the access connection from the shared memory space. For example, from the CMB space of the SSD or from the host memory buffer HMB of the storage node.

705. The storage node returns a broken link result to the computing node, and the computing node stops sending heartbeats.

According to the embodiment of the application, when the computing node no longer needs to establish access connection with the SSD, the connection between the computing node and the SSD is disconnected, so that the access to the data is established when the data is required. By adopting the means, system resources can be saved. The CPU at the storage node side only participates in the work in the process of calculating node broken links, so that the workload is greatly reduced, and the overall performance is improved.

Referring to fig. 8, a data access device according to an embodiment of the present application is shown. As shown in fig. 8, the data access device includes a receiving module 801, an acquiring module 802, and a transmitting module 803, which are specifically as follows:

a receiving module 801, configured to receive a remote direct data access RDMA connection request sent by a computing node in a distributed storage system, where the RDMA connection request carries a target solid state disk identifier, and the RDMA connection request is used to instruct the storage node to establish an access connection between the computing node and the target solid state disk;

an obtaining module 802, configured to obtain and store link establishment information corresponding to the access connection, where the link establishment information includes a completion queue CQ, where the CQ is used to indicate that a data access task exists in the target solid state disk;

the sending module 803 is configured to send a first address corresponding to the stored link establishment information to the target solid state disk, so as to instruct the target solid state disk to obtain the CQ in the link establishment information, and monitor the CQ.

The specific implementation manner of each module may refer to the foregoing embodiments, and will not be described herein.

Referring to fig. 9, another data access device according to an embodiment of the present application is shown. As shown in fig. 9, the data access device includes a receiving module 901 and a processing module 902, which are specifically as follows:

a receiving module 901, configured to receive a message sent by a storage node in a distributed storage system, where the message carries a first address, where the first address is an address of link establishment information corresponding to an access connection between a computing node in the distributed storage system and the device stored by the storage node, where the link establishment information includes a completion queue CQ, and the CQ is configured to indicate that a data access task exists in the device;

a processing module 902, configured to obtain the CQ in the chaining information according to the first address, and monitor the CQ.

The embodiment of the application provides a data access device, which comprises a processor, wherein the processor is used for: receiving a remote direct data access (RDMA) connection request sent by a computing node in a distributed storage system, wherein the RDMA connection request carries a target solid state disk identifier and is used for indicating the device to establish access connection between the computing node and the target solid state disk; acquiring and storing link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk; and sending the stored first address corresponding to the link establishment information to the target solid state disk to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.

In another aspect, an embodiment of the present application further provides a data access apparatus, including a processor, where the processor is configured to: receiving a message sent by a storage node in a distributed storage system, wherein the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node and the device in the distributed storage system stored by the storage node, and the link establishment information comprises a completion queue CQ; and acquiring the CQ in the link establishment information according to the first address, and monitoring the CQ.

The present application provides a data access device, as shown in fig. 10, the data access device 1000 comprises at least one processor 1001, at least one memory 1002 and at least one communication interface 1003. The processor 1001, the memory 1002, and the communication interface 1003 are connected by the communication bus and perform communication with each other.

The processor 1001 may be a general purpose Central Processing Unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above programs.

Communication interface 1003 is used to communicate with other devices or communication networks, such as ethernet, radio Access Network (RAN), wireless local area network (Wireless Local Area Networks, WLAN), etc.

The Memory 1002 may be, but is not limited to, a read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a random access Memory (random access Memory, RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a compact disc (Compact Disc Read-Only Memory) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory may be stand alone and coupled to the processor via a bus. The memory may also be integral with the processor.

Wherein the memory 1002 is used for storing application program codes for executing the above schemes, and the execution is controlled by the processor 1001. The processor 1001 is configured to execute application code stored in the memory 1002.

The code stored by memory 1002 may perform any of the data access methods provided above.

The embodiment of the application also provides a chip system, which is applied to the electronic equipment; the system-on-chip includes one or more interface circuits, and one or more processors; the interface circuit and the processor are interconnected through a circuit; the interface circuit is configured to receive a signal from a memory of the electronic device and to send the signal to the processor, the signal including computer instructions stored in the memory; the electronic device performs the method when the processor executes the computer instructions.

Embodiments of the present application also provide a computer-readable storage medium having instructions stored therein, which when run on a computer or processor, cause the computer or processor to perform one or more steps of any of the methods described above.

Embodiments of the present application also provide a computer program product comprising instructions. The computer program product, when run on a computer or processor, causes the computer or processor to perform one or more steps of any of the methods described above.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

In the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a alone, a and B together, and B alone, wherein a, B may be singular or plural.

Those of ordinary skill in the art will appreciate that implementing all or part of the above-described method embodiments may be accomplished by a computer program to instruct related hardware, the program may be stored in a computer readable storage medium, and the program may include the above-described method embodiments when executed. And the aforementioned storage medium includes: ROM or random access memory RAM, magnetic or optical disk, etc.

The foregoing is merely a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this, and any changes or substitutions within the technical scope disclosed in the embodiment of the present application should be covered in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

A data access method for use with a storage node in a distributed storage system, the method comprising:

the storage node receives a remote direct data access (RDMA) connection request sent by a computing node in the distributed storage system, wherein the RDMA connection request carries a target solid state disk identifier in the distributed storage system, and the RDMA connection request is used for indicating the storage node to establish access connection between the computing node and the target solid state disk;

the storage node acquires and stores link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk;

the storage node sends a first address corresponding to the stored link establishment information to the target solid state disk so as to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.
The method of claim 1, wherein the linking information further includes a memory region MR and a queue pair QP, the MR being configured to indicate a region of the target solid state disk for data access, the QP being configured to instruct the target solid state disk to operate the QP for data transceiving.
The method of claim 2, wherein the storage node stores the link establishment information corresponding to the access connection on a shared memory space, wherein the shared memory space is a controller memory buffer of the target solid state disk, or the shared memory space is a host memory buffer of the storage node.
A method according to claim 3, characterized in that the method further comprises:

the storage node detects the access connection;

when network abnormality is detected, or when a message with heartbeat abnormality sent by the target solid state disk is received, the storage node deletes the link establishment information in the shared memory space.
A method according to claim 3, characterized in that the method further comprises:

the storage node sends the received instruction for disconnecting the access connection, which is sent by the computing node, to the target solid state disk;

and when receiving a successful link breaking message sent by the target solid state disk, the storage node deletes the link building information in the shared memory space.
The data access method is characterized by being applied to a target solid state disk in a distributed storage system, and comprises the following steps:

The target solid state disk receives a message sent by a storage node in the distributed storage system, wherein the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node in the distributed storage system and the target solid state disk, which is stored by the storage node, the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk;

and the target solid state disk acquires the CQ in the link establishment information according to the first address and monitors the CQ.
The method of claim 6, wherein the chaining information further includes a memory region MR and a queue pair QP, the method further comprising:

and the target solid state disk acquires MR and QP in the link establishment information according to the first address, and performs data access.
The method of claim 7, wherein the storage node stores the link establishment information corresponding to the access connection on a shared memory space, wherein the shared memory space is a controller memory buffer of the target solid state disk or the shared memory space is a host memory buffer of the storage node.
The method according to claim 7 or 8, characterized in that the method further comprises:

the target solid state disk detects the access connection;

when detecting that the heartbeat is abnormal, or when receiving a message of network abnormality sent by the storage node, the target solid state disk confirms whether the commands in the access connection are all completed;

and if the commands in the access connection are all completed, stopping monitoring the CQ by the target solid state disk.
The method according to claim 7 or 8, characterized in that the method further comprises:

the target solid state disk receives an instruction for disconnecting the access connection sent by the storage node;

the target solid state disk confirms whether the commands in the access connection are all completed or not;

and if the commands in the access connection are all completed, stopping monitoring the CQ by the target solid state disk.
A data access device comprising a processor, wherein the processor is configured to:

receiving a remote direct data access (RDMA) connection request sent by a computing node in a distributed storage system, wherein the RDMA connection request carries a target solid state disk identifier in the distributed storage system, and the RDMA connection request is used for indicating the processor to establish access connection between the computing node and the target solid state disk;

Acquiring and storing link establishment information corresponding to the access connection, wherein the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that a data access task exists in the target solid state disk;

and sending the stored first address corresponding to the link establishment information to the target solid state disk to instruct the target solid state disk to acquire the CQ in the link establishment information and monitor the CQ.
The apparatus of claim 11, wherein the linking information further comprises a memory region MR and a queue pair QP, the MR being configured to indicate a region of the target solid state disk for data access, the QP being configured to instruct the target solid state disk to operate the QP for data transceiving.
The apparatus of claim 12, wherein the processor stores the linking information corresponding to the access connection on a shared memory space, wherein the shared memory space is a controller memory buffer of the target solid state disk or the shared memory space is a host memory buffer of the apparatus.
The apparatus of claim 13, wherein the processor is further configured to:

Detecting the access connection;

and deleting the link establishment information in the shared memory space when network abnormality is detected or when a message with heartbeat abnormality sent by the target solid state disk is received.
The apparatus of claim 13, wherein the processor is further configured to:

transmitting the received instruction for disconnecting the access connection, which is transmitted by the computing node, to the target solid state disk;

and deleting the link establishment information in the shared memory space when receiving a successful link breakage message sent by the target solid state disk.
A data access device comprising a processor, wherein the processor is configured to:

receiving a message sent by a storage node in a distributed storage system, wherein the message carries a first address, the first address is an address of link establishment information corresponding to access connection between a computing node in the distributed storage system and the device, which is stored by the storage node, the link establishment information comprises a completion queue CQ, and the CQ is used for indicating that the device has a data access task;

and acquiring the CQ in the link establishment information according to the first address, and monitoring the CQ.
The apparatus of claim 16, wherein the chaining information further comprises a memory region MR and a queue pair QP, the processor further configured to:

and acquiring MR and QP in the link establishment information according to the first address, and performing data access.
The apparatus of claim 17, wherein the storage node stores the linking information corresponding to the access connection on a shared memory space, wherein the shared memory space is a controller memory buffer of the apparatus or the shared memory space is a host memory buffer of the storage node.
The apparatus of claim 17 or 18, wherein the processor is further configured to:

detecting the access connection;

when detecting that the heartbeat is abnormal, or when receiving a message of existence of network abnormality sent by the storage node, confirming whether commands in the access connection are all completed;

and stopping monitoring the CQ if all the commands in the access connection are completed.
The apparatus of claim 17 or 18, wherein the processor is further configured to:

receiving an instruction for disconnecting the access connection sent by the storage node;

Confirming whether commands in the access connection are all completed;

and stopping monitoring the CQ if all the commands in the access connection are completed.
A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, which is executed by a processor to implement the method of any one of claims 1 to 10.
A computer program product, characterized in that the computer program product, when run on a computer, causes the computer to perform the method according to any one of claims 1 to 10.
A distributed storage system comprising a compute node, further comprising a data access device according to any of claims 11 to 15, and a data access device according to any of claims 16 to 20.