WO2023174341A1

WO2023174341A1 - Data read-write method, and device, storage node and storage medium

Info

Publication number: WO2023174341A1
Application number: PCT/CN2023/081675
Authority: WO
Inventors: 金浩; 屠要峰; 韩银俊; 许军宁; 陈正华
Original assignee: 中兴通讯股份有限公司
Priority date: 2022-03-16
Filing date: 2023-03-15
Publication date: 2023-09-21
Also published as: CN116804908A

Abstract

The present disclosure relates to a data read-write method, and a device, a storage node and a storage medium. The method comprises: by means of a first network link which is established with a first storage node, sending a first read-write request to the first storage node; and by means of a second network link which is established with a second storage node, receiving a read-write response from the second storage node, and using the read-write response as a response to the first read-write request, wherein the first network link and the second network link belong to the same storage protocol channel.

Description

Data reading and writing methods, equipment, storage nodes and storage media

Cross-references to related applications

This disclosure claims the priority of Chinese patent application CN202210258761.1 titled "Data reading and writing method, device, storage node and storage medium" submitted on March 16, 2022, the entire content of which is incorporated into this disclosure by reference.

Technical field

The present disclosure relates to the field of communications, and in particular, to a data reading and writing method, device, storage node and storage medium.

Background technique

A distributed storage system usually contains multiple storage nodes. Each storage node contains one or more storage devices that support the NVMe (non-volatile memory express, non-volatile memory host controller interface specification) storage layer protocol. Each storage node provides a logical address space. The target space of IO (read and write) operations may be located on any one or more storage nodes. In addition, the cluster topology may be updated at any time. These all bring technical problems to standard client device access. , For example, the standard client cannot obtain the routing information of the storage cluster in time, the proxy mode network transmission path is long, and the network delay is large.

Contents of the invention

The present disclosure provides a data reading and writing method, device, storage node and storage medium to solve the problem of long network transmission paths and the introduction of new network delays when the second server forwards requests and response results when accessing a distributed storage cluster. question. Wherein, the second server includes a proxy server.

In a first aspect, a data reading and writing method is provided, which is applied to a client device. The method includes: sending a first reading and writing request to the first storage node through a first network link established with the first storage node. The first storage node Be any storage node in the storage system; receive the read and write response from the second storage node through the second network link established with the second storage node, and use the read and write response as a response to the first read and write request, The second storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel.

In a second aspect, a data reading and writing method is provided, which is applied to a second storage node. The method includes: receiving a first reading and writing request from a first storage node, where the first storage node is any storage node in the storage system; In response to the first read and write request, a read and write response is returned to the client device through the second network link established by the client device and the second storage node. The second network link and the first network link belong to the same storage protocol channel. A network link is a network link established between the client device and the first storage node.

In a third aspect, a data reading and writing system is provided, including: a client device, a first storage node and a second storage node. The client device is configured to send a first read and write request to the first storage node through the first network link established with the first storage node, and the first storage node is any storage node in the storage system; through the first network link established with the second storage node The second network link established receives the read and write response from the second storage node, and uses the read and write response as a response to the first read and write request. The second storage node is determined by the first storage node. A node capable of executing the first read and write request; the first network link and the second network link belong to the same storage protocol channel.

In a fourth aspect, a client device is provided, including: a first sending unit and a first receiving unit. The first sending unit is configured to send the first read and write request to the first storage node through the first network link established with the first storage node, and the first storage node is any storage node in the storage system; the first receiving unit , configured to receive a read and write response from the second storage node through the second network link established with the second storage node and use the read and write response as a response to the first read and write request, the second storage node being configured by the first The node determined by the storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel.

In a fifth aspect, a second storage node is provided, including: a second receiving unit and a second sending unit. The second receiving unit is configured to receive the first read and write request from the first storage node, which is any storage node in the storage system; the second sending unit is configured to respond to the first read and write request, Return a read and write response to the client device through the second network link established by the client device and the second storage node. The second network link and the first network link belong to the same storage protocol channel. The first network link is the connection between the client device and the second storage node. The network link established by the first storage node.

In a sixth aspect, an electronic device is provided, including: a processor, a memory and a communication bus, wherein the processor and the memory complete communication with each other through the communication bus; the memory is used to store computer programs; the processor is used to execute the memory The program stored in implements the data reading and writing method described in the first aspect or the data reading and writing method described in the second aspect.

In a seventh aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the data reading and writing method described in the first aspect or the data reading and writing method described in the second aspect is implemented.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the drawings needed to be used in the embodiments or description of the prior art will be briefly introduced below. It is obvious that for those of ordinary skill in the art, Other drawings can also be obtained based on these drawings without incurring any creative effort.

Figure 1 is a schematic flow chart of the data reading and writing method in the present disclosure;

Figure 2 is another schematic flow chart of the data reading and writing method in the present disclosure;

Figure 3 is a schematic structural diagram of the data reading and writing system in the present disclosure;

Figure 4 is a schematic diagram of the link layering principle between nodes and clients in the distributed storage system in the present disclosure;

Figure 5 shows the node X, node Y and client device in the distributed storage system in this disclosure. Interaction flow chart between;

Figure 6 is a schematic structural diagram of a client device in the present disclosure;

Figure 7 is a schematic structural diagram of the second storage node in the present disclosure;

FIG. 8 is a schematic structural diagram of an electronic device in the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions in the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the present disclosure. Obviously, the described embodiments are part of the implementation of the present disclosure. examples, not all examples. Based on the embodiments in this disclosure, all other embodiments obtained by those of ordinary skill in the art without any creative efforts fall within the scope of protection of this disclosure.

It should be noted that the terms "first", "second", etc. in the description and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

A distributed storage system usually contains multiple storage nodes. Each storage node contains one or more storage devices that support the NVMe (non-volatile memory express, non-volatile memory host controller interface specification) interface specification. Multiple Storage nodes provide a logical address space. The target space of IO (read and write) operations may be located on any one or more storage nodes. In addition, the topology of the cluster may be updated at any time. These all introduce new issues to standard client device access. Technical issues such as network delays. Currently, there are two main technologies for NVMe client devices to access distributed storage systems.

The first method is that the client device copies a cluster partition table and can calculate the target storage node to be accessed. The client device directly establishes an NVMe link with the target storage device to implement IO operations. This method requires customized NVMe client devices to be updated synchronously in real time. The cluster's partition information, routing calculation rules, client devices and cluster services are highly coupled, and the implementation cost is very high.

In the second type, the client device sends an IO request to the first server. The first server determines the second server where the request target address is located and forwards the IO request to the second server. After the second server completes the IO request, it needs to notify the second server. One server, and then the first server sends the response result. The response message of this method needs to be sent from the second server to the first server, and then the latter sends it to the client device. The network transmission path is long and new network delays are introduced.

In related technology, the NVMe client device and each storage node in the distributed storage system (hereinafter referred to as the NVMe Target server) have matching network links. An NVMe Target server can only use the network link that matches the NVMe client device. Send read and write requests or receive read and write responses. if If the link used by the NVMe client device to send read and write requests and the link used to receive read and write responses do not belong to the same path, then even if the read and write requests sent match the read and write responses received, the NVMe client device cannot identify them.

The NVMe Target server and the NVMe client device establish network links through queue mapping.

The NVMe client device includes the NVMe layer and the network layer. The NVMe layer includes the submission queue and the completion queue, and the network layer includes the submission queue and the completion queue. Each NVMe Target server in the distributed storage system also includes an NVMe layer and a network layer. Similarly, the NVMe layer includes a submission queue and a completion queue, and the network layer includes a submission queue and a completion queue.

From the perspective of sending read and write requests, the establishment process of the network link between the NVMe client device and the NVMe Target server is: on the NVMe client device side, the submission queue of the NVMe layer is mapped to the submission queue of the network layer, and the NVMe client The submission queue of the network layer on the device side is connected to the completion queue of the network layer on the NVMe Target server through network transmission; the completion queue of the network layer on the NVMe Target server is mapped to the completion queue of the NVMe layer.

From the perspective of receiving read and write responses, the establishment process of the network link between the NVMe client device and the NVMe Target server is: on the NVMe client device side, the completion queue of the NVMe layer is mapped to the completion queue of the network layer, and the completion queue of the network layer The completion queue is connected to the submission queue of the network layer of the NVMe Target server through network transmission; on the NVMe Target server side, the submission queue of the network layer is mapped to the submission queue of the NVMe layer.

When the NVMe client device sends a read and write request to the NVMe Target server, the NVMe layer submission queue of the NVMe client device sends a read and write request to the network layer submission queue, and the network layer submission queue sends the read and write request to the NVMe Target server's network The completion queue of the layer sends the read and write request; after the completion queue of the network layer of the NVMe Target server receives the read and write request, it sends the read and write request to the completion queue of the NVMe layer.

When the NVMe Target server sends a read and write response to the NVMe client device, the NVMe layer submission queue of the NVMe Target server sends a read and write response to the network layer submission queue, and the network layer submission queue sends a read and write response to the network of the NVMe client device. The completion queue of the NVMe client device sends the read and write response; after receiving the read and write request, the completion queue of the network layer of the NVMe client device sends the read and write response to the completion queue of the NVMe layer.

In related technology, the NVMe layer of each NVMe client device is only set up with one submission queue and one completion queue. The network layer is also set with only one submission queue and completion queue. Therefore, one NVMe client device can only be bound to one NVMe Target. server, and can only interact with the NVMe Target server. If you need to interact with other NVMe Target servers, you need to forward them through the bound NVMe Target server.

Specifically, the read and write requests sent by the NVMe client device to the first NVMe Target server can only be sent to the NVMe client device by the first NVMe Target server through its own NVMe layer submission queue and network layer submission queue. Send read and write responses. And if the read-write response is sent by the second The NVMe Target server sends it to the NVMe client device through its NVMe layer submission queue and network layer submission queue, so the NVMe client device will not be able to recognize the read and write response. Therefore, when the read and write response corresponding to the read and write request can only be obtained through the second NVMe Target server, the read and write response must be forwarded to the NVMe client device through the first NVMe Target server, so that the NVMe client device can recognize the read and write response. Write response.

The present disclosure provides a data reading and writing method, which can be applied to client devices.

As shown in Figure 1, the method may include the following steps 101 to 102.

Step 101: Send a first read and write request to the first storage node through the first network link established with the first storage node. The first storage node is any storage node in the storage system.

Step 102: Receive a read and write response from the second storage node through the second network link established with the second storage node, and use the read and write response as a response to the first read and write request. The second storage node is The node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel.

In an exemplary embodiment, a read-write response from the second storage node is received, and the read-write response is used as a response to the first read-write request according to the storage protocol layer identifier. In an exemplary embodiment, the first network link and the second network link are both bottom network links of the same storage protocol layer link channel.

It should be understood that the read and write requests in this embodiment include read requests and/or write requests, and accordingly, the read and write responses include read responses and/or write responses. When the read and write request sent by the client device is specifically a read request, then the storage node returns a read response; when the read and write request sent by the client device is specifically a write request, then the storage node returns a write response. response.

In this embodiment, the network link established between the client device and the storage node can be used to send requests and receive responses at the same time. For example, for the client device and the first storage node, the client device can send a read and write request to the first storage node through the first network link, and the first storage node can also return a read and write response to the client device through the first network link. .

It should be understood that when the network links between the client device and different storage nodes are aggregated to implement the same storage protocol channel, when the client device sends a read and write request to a storage node, no matter which storage node the read and write response comes from, the client The device can recognize the read and write ring. Specifically in this embodiment, since the first network link and the second network link belong to the same storage protocol channel, although the read and write responses come from the second storage node, the client device can still recognize the read and write responses.

In an exemplary embodiment, the data reading and writing method includes: the client and the storage cluster establish a storage protocol layer link, a first network link, a second network link and other network layer links, and the storage protocol layer selects any network link Send an IO read and write request and receive an IO read and write response from any network link. The storage protocol layer identifier of the read and write response matches the storage protocol layer identifier of the read and write request.

This embodiment provides the following two methods to aggregate the first network link and the second network link to implement the same storage protocol channel.

First, add a storage protocol management layer on top of the NVMe layer of the client device to manage the NVMe layer. During specific implementation, different storage nodes still have uniquely matching submission queues and completion queues at the NVMe layer of the client device, but the submission queue and completion queue can be marked by the added storage protocol management layer to indicate that they can be processed through the The submission queue sends read and write requests to different storage nodes and indicates that any read or write responses received by the completion queue can be identified.

Through this storage protocol management layer marking method, although the first network link and the second network link can be aggregated into one storage protocol channel, the storage protocol management layer needs to be added, resulting in a relatively large workload.

Second, in order to improve processing efficiency, this embodiment sets a storage node that has a network link relationship with the client device. The completion queue mapped to the NVMe layer of the client device is the same completion queue; the storage node that has a network link relationship with the client device Node, the submission queue mapped to the NVMe layer of the client device is the same submission queue.

It should be understood that in this embodiment, the focus is on enabling the client device to identify the read and write response returned by the second storage node. Therefore, when the first network link and the second network link belong to the same storage protocol channel, the first network link is The completion queues of the storage node and the second storage node mapped to the NVMe layer of the client device are the same completion queue. In this way, no matter which storage node sends the read and write response, the NVMe layer of the client device receives the read and write response from the same completion queue. Read and write responses and identify them.

During specific implementation, in one embodiment, the first completion queue and the second completion queue in the network layer are obtained. The first completion queue is used to receive read and write responses from the first storage node, and the second completion queue is used to receive the read and write responses from the first storage node. a read-write response from the second storage node; and obtain the completion queue of the storage protocol layer. The completion queue of the storage protocol layer is used to receive a read-write response from the first completion queue or a read-write response from the second completion queue; The first completion queue and the second completion queue are mapped to the completion queues of the storage protocol layer.

In this embodiment, considering that the client device may send multiple read and write requests in a short period of time, in order to distinguish whether the received read and write response is a response to the first read and write request, in this embodiment, the third read and write request is Before the read-write response returned by the second storage node is used as a response to the first read-write request, the read-write response may also be verified.

In one embodiment, before using the read-write response as a response to the first read-write request, parse the storage protocol layer session identifier in the read-write response and the storage protocol layer session identifier in the first read-write request; determine the read-write response The storage protocol layer session ID in is the same as the storage protocol layer session ID in the first read and write request.

It should be understood that when the session identifier in the read-write response and the session identifier in the first read-write request are the same, it is confirmed that the read-write response and the first read-write request are requests and responses for the same session, and receipt can be confirmed at this time. The read-write response matches the first read-write request, that is, the read-write response is the response to the first read-write request.

In order to facilitate the subsequent read-write response to the first read-write request, the second storage node can be directly accessed to reduce the probability of request forwarding. This embodiment can also establish a second storage node after determining the response to the first read-write request. Mapping relationship between storage nodes and client devices. In an exemplary embodiment, this embodiment may also After determining the response to the first read-write request, a mapping relationship between the second storage node and the read-write request target data is established.

In one embodiment, the identifier of the second storage node is extracted from the read and write response; based on the identifier of the second storage node, a mapping relationship between the second storage node and the client device is established, and the mapping relationship indicates that the read and write response data is stored in the second storage node. storage node. In one embodiment, the identifier of the second storage node is extracted from the read and write response; based on the identifier of the second storage node, a mapping relationship between the second storage node and the read and write request target data is established, and the mapping relationship indicates that the read and write response data is stored in Second storage node.

In the application, the data stored in the second storage node may be hotspot data. That is to say, the mapping relationship indicates that the hotspot data is stored in the second storage node. Therefore, when the client device needs to access the hotspot data, in order to reduce the probability of request forwarding, , you can directly send an access request to the second storage node.

In one embodiment, the specific implementation of the client device accessing the read-write response in the second storage node may be: based on the mapping relationship, a second read-write request is generated, and the second read-write request is used to access the read-write response; through the second network link, send a second read-write request to the second storage node, and receive a read-write response returned by the second storage node in response to the second read-write request.

In this embodiment, in order to realize that the client device can send read and write requests to the first storage node and send requests to the second storage node, the first network link and the second network link belong to the same storage protocol channel. When, the first storage node and the second storage node are mapped to the same submission queue of the NVMe layer of the client device.

During specific implementation, in one embodiment, the first submission queue and the second submission queue in the network layer are obtained. The first submission queue is used to send read and write requests to the first storage node, and the second submission queue is used to send read and write requests to the second storage node. The node sends read and write requests; and obtains the submission queue in the storage protocol layer. The submission queue of the storage protocol layer is used to send read and write requests to the first submission queue and the second submission queue; and maps the first submission queue and the second submission queue. Submission queue to the storage protocol layer.

In the technical solution provided by this embodiment, the first read and write request is sent to the first storage node through the first network link established with the first storage node, and the first storage node is any storage node in the storage system; through The second network link established by the second storage node receives the read and write response from the second storage node, and uses the read and write response as a response to the first read and write request. The second storage node is determined by the first storage node. A node capable of executing the first read and write request; the first network link and the second network link belong to the same storage protocol channel. Since the first network link and the second network link belong to the same storage protocol channel, the storage protocol layer identification information of the read-write request and the read-write response is the same, so although the client device sends the first read-write request to the first storage node, However, it is still possible to send the read-write response to the client device after the second storage node executes the first read-write request, so that the client device can recognize that the read-write response is for the first read-write request.

The present disclosure provides a data reading and writing method, which can be applied to the second storage node; as shown in Figure 2, the method can include the following steps 201 to 202.

Step 201: Receive a first read and write request from a first storage node, which is any storage node in the storage system; Step 202: Respond to the first read and write request through the client device and the second storage node. The second network link established by the node returns a read and write response to the client device. The second network link and the first network link belong to the same storage protocol channel. The first network link is the network link established between the client device and the first storage node. . In an exemplary embodiment, the second network link and the first network link are both underlying network links of the same storage protocol channel.

In an exemplary embodiment, the method is further configured to: receive a second read-write request from the client device through the second network link, the second read-write request being used to access the read-write response; wherein the read-write response Includes second storage node data. In response to the second read and write request, a read and write response is returned to the client device over the second network link.

Based on the same concept, the present disclosure provides a data reading and writing system. For the specific implementation of the system, please refer to the description of the method embodiment section. The repeated points will not be repeated. As shown in Figure 3, the system mainly includes: client Device 301, storage cluster, the storage cluster includes: a first storage node 302 and a second storage node 303 and other storage nodes; the client device 301 establishes a storage protocol layer link channel with the storage cluster to communicate with the storage cluster. All storage nodes establish network layer links respectively. The client device 301 is configured to send a first read and write request to the first storage node 302 through the first network link established with the first storage node 302, which is any storage node in the storage system; through The second network link established by the second storage node 303 receives the read and write response from the second storage node 303, and uses the read and write response as a response to the first read and write request. The second storage node 303 is configured by the first storage node 303. The node determined by node 302 is capable of executing the first read and write request; the first network link and the second network link belong to the same storage protocol channel. Similarly, the client device 301 can send read and write requests to the second storage node 302 through the second network link established with the second storage node 303, and receive data from the second storage node 302 through the second network link established with the second storage node 303. The read and write response of the second storage node 303. In an exemplary embodiment, the first network link and the second network link are both underlying network links of the same storage protocol channel.

The following takes the client device as an NVMe client device and the storage node as an NVMe Target server as an example to describe the application environment in this disclosure.

The existing NVM Express over Fabrics Revision 1.1a specification requires that the NVMe layer IO queue and the underlying network link have a one-to-one correspondence. This disclosure proposes to decouple the storage layer and network layer, and the NVMe protocol and network link support 1:N(N Not less than 1) Mapping relationship. NVMe request messages and response messages support transmission on different network links. The client establishes network links with all servers in the distributed cluster. These network connections are mapped to the same storage layer NVMe protocol channel. Realize that one NVMe Path (path) is mapped to multiple network links, thereby completely solving the problem of standard NVMe clients accessing distributed storage clusters to obtain higher performance and better user experience. This disclosure is different from the existing MultiPath (multipath) function. Multipath means that there are multiple path channels at the NVMe protocol level. According to the protocol specification requirements, an NVMe link is selected to implement IO interaction.

Based on the above description of the inventive concept, a schematic diagram of the link principle between nodes and clients in the distributed storage system as shown in Figure 4 is given. In Figure 4, the client and all server nodes in the distributed storage system are respectively Create a network link, aggregate multiple network links to implement an NVMe access path, and the client reads and writes the entire cluster through an NVMe path.

Please refer to Figure 5, which is an interaction flow chart between node X, node Y and client device in the distributed storage system.

Step 501: The client sends an IO request to node X, requiring node X to have a data partition table and be able to determine the location information of the target space; Step 502: Node Storage node Y forwards the request message to node Y. The NVMe session life cycle of node X ends; Step 503: Storage node Y receives the NVMe request, parses and processes the NVMe message, and executes the corresponding IO request; Step 504: Node Y sends NVMe responds with messages to the client.

In order to realize the above inventive concept, this disclosure makes the following extensions to the NVMe client device and the NVMe Target server respectively.

NVMe client device side: According to the NVMe protocol definition, the client establishes a network link with the server through the connect command. The submission queue and completion queue of the NVMe protocol are mapped to the submission queue and completion queue of the network link respectively. . According to this disclosure, the connect command specifies the server address list, and the client establishes network links with all service nodes. Taking RDMA as an example, all links share the same RDMA Protection Domain (PD), and the server shares the client through the RDMA network. Terminal memory region (Memory Region).

When the client sends a message, it uses the reserved field of the NVMe message header to save the link identifier FID (Fabric ID). The network layer sends a standard NVMe request message to the correct server based on the FID carried in the message header.

When the client receives a message, multiple network links share a completion queue. The completion queue notifies the NVMe layer protocol stack of messages received by all links, which are then received and processed by the client.

The enhancement of the client in this disclosure also includes the address mapping table caching function. The response messages of all read and write requests are returned by the target node. When the client receives the response, it immediately updates the target address of the request and the mapping relationship of the target node. Subsequently, the address Access can directly initiate requests to the target node, reducing the probability of request forwarding and greatly improving read and write efficiency.

NVMe Target server side: The distributed storage cluster implements a logical storage volume and provides external storage volume services through NVMe Target. This disclosure enhances the NVMe Target server side.

First, a fabric link dedicated to the NVMe Target service is established between NVMe Target servers. This link supports forwarding NVMe requests to other nodes as they are, so the NVMe Target server can receive NVMe requests from the client and other NVMe Target servers at the same time.

Second, each NVMe Target server can calculate the server node where the target is located based on the request.

Third, the NVMe Target server only sends NVMe response messages to the client, regardless of whether the request is forwarded by other nodes.

This disclosure extends the client and server of the NVMe-oF protocol. The protocol layer messages transmitted by the network layer are consistent with the existing standard protocols. Therefore, the enhanced NVMe client of this disclosure can be compatible with the standard NVMe-oF Target server, a standard NVMe client can access the Target server enhanced by this disclosure.

This disclosure improves the NVMe protocol client and server implementation solutions. The network layer is not limited to RDMA and is compatible with network types supported by the standard NVMe protocol.

Based on the same concept, this disclosure provides a client device. For the specific implementation of the client device, please refer to the description of the method embodiment section, and the repeated points will not be repeated. As shown in Figure 6, the client device mainly includes : first sending unit 601, first receiving unit 602.

The first sending unit 601 is configured to send a first read and write request to the first storage node through the first network link established with the first storage node, and the first storage node is any storage node in the storage system; the first receiving Unit 602 is configured to receive a read and write response from the second storage node through the second network link established with the second storage node, and use the read and write response as a response to the first read and write request. The second storage node is A node capable of executing the first read and write request determined by the first storage node; the first network link and the second network link belong to the same storage protocol channel. In an exemplary embodiment, the client device includes an IO client device.

In an exemplary embodiment, the client device includes: a storage protocol layer sending unit, a storage protocol layer receiving unit, a first network layer sending unit, a first network layer receiving unit, a second network layer sending unit, a second network layer receiving unit.

The storage protocol layer sending unit sends the first read and write request, the network layer first sending unit sends the first read and write request to the first storage node, the network layer second receiving unit receives the read and write response from the second storage node, the storage protocol layer The receiving unit determines the read-write response as a response to the first read-write request. The second storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel.

In an exemplary embodiment, the client device includes: a storage protocol layer sending unit, a storage protocol layer receiving unit, a first sending unit of the network layer, a first receiving unit of the network layer, a second sending unit of the network layer, a third sending unit of the network layer. Two receiving units, etc. The storage protocol layer sending unit selects the first sending unit of the network layer to send the IO read and write request. The storage protocol layer receiving unit supports the second receiving unit of the network layer to receive the storage layer read and write response. The first sending unit of the network layer is any sending unit of the network layer. , the second receiving unit of the network layer is any receiving unit of the network layer. Among them, read and write requests include storage protocol layer information and network layer link information.

In an exemplary embodiment, the second receiving unit of the network layer transfers the read and write response from the second storage node to the storage protocol layer receiving unit, and the storage protocol layer receiving unit parses the storage protocol layer identifier and network layer identifier of the read and write response. , compare the storage protocol layer identifier of the read-write response with the storage protocol layer identifier of the first read-write request, and the read-write response received by the second receiving unit of the network is a response to the first read-write request sent by the second sending unit of the network. The storage protocol layer records the second network layer identifier and establishes a mapping relationship between the read and write request target data and the second network link identifier. The first network link and the second network link are any network link channels of the storage protocol layer.

In an exemplary embodiment, the client device is further configured to: after using the read and write response as a response to the first read and write request, extract the identification of the second storage node from the read and write response; based on the second storage node identification, establishing a mapping relationship between the second storage node and the client device, and the mapping relationship indicates that the read and write responses are stored in the second storage node.

In an exemplary embodiment, the client device is further configured to: after establishing a mapping relationship between the second storage node and the client device based on the identity of the second storage node, generate a second read-write request based on the mapping relationship. The second read-write request is used to access the read-write response; send the second read-write request to the second storage node through the second network link, and receive the read-write response returned by the second storage node in response to the second read-write request.

In an exemplary embodiment, the client device is further configured to: obtain the first submission queue and the second submission queue in the network layer before sending the second read and write request to the second storage node through the second network link, The first submission queue is used to send read and write requests to the first storage node, the second submission queue is used to send read and write requests to the second storage node; and the submission queue in the storage protocol layer is obtained, and the submission queue in the storage protocol layer is used to Send read and write requests to the first submission queue and the second submission queue; map the first submission queue of the network layer and the second submission queue of the network layer to the submission queue of the storage protocol layer.

In an exemplary embodiment, the client device is further configured to: obtain the first completion queue in the network layer before receiving the read and write response from the second storage node through the second network link established with the second storage node. and a second completion queue, the first completion queue is used to receive read and write responses from the first storage node, the second completion queue is used to receive read and write responses from the second storage node; and obtains the completion queue of the storage protocol layer , the completion queue of the storage protocol layer is used to receive read and write responses from the first completion queue of the network layer or the read and write responses of the second completion queue of the network layer; map the first completion queue of the network layer and the second completion queue of the network layer to Stores the completion queue for the protocol layer.

In an exemplary embodiment, the client device is further configured to: parse the session identifier in the read-write response and the session identifier in the first read-write request before using the read-write response as a response to the first read-write request. ; Make sure the session ID in the read-write response is the same as the session ID in the first read-write request.

Based on the same concept, the present disclosure provides a second storage node. For the specific implementation of the second storage node, please refer to the description in the method embodiment section. Repeated parts will not be repeated. As shown in Figure 7, the second storage node The node mainly includes: a second receiving unit 701 and a second sending unit 702.

The second receiving unit 701 is configured to receive the first read and write request from the first storage node, which is any storage node in the storage system; wherein the second receiving unit includes an internal network layer receiving unit. The second sending unit 702 is configured to respond to the first read and write request, return a read and write response to the client device through the second network link established by the client device and the second storage node, the second network link and the first network link. Belonging to the same storage protocol channel, the first network link is the network link established between the client device and the first storage node. Wherein, the second sending unit includes a client network layer sending unit.

In an exemplary embodiment, a second storage node includes: a cluster internal network layer sending unit, a cluster internal network layer receiving unit, a client network layer sending unit, and a client network layer receiving unit. The cluster internal network layer receiving unit receives the first read and write request from the first storage node, which is any storage node in the storage system; the client network layer sending unit communicates with the second storage node through the client device The established second network link returns a read and write response to the client device. The second network link belongs to The storage protocol layer link channel established between the client and the storage cluster. The second storage node is any storage node in the cluster.

In an exemplary embodiment, the second storage node is further configured to: in response to the first read and write request, after returning a read and write response to the client device through the second network link established by the client device and the second storage node, Receive a second read and write request from the client device through the second network link, the second read and write request is used to access the read and write response; respond to the second read and write request, return to the client device through the second network link Read and write responses.

Based on the same concept, the present disclosure also provides an electronic device. As shown in Figure 8, the electronic device mainly includes: a processor 801, a memory 802 and a communication bus 803. The processor 801 and the memory 802 communicate through the communication bus 803. complete mutual communication. Among them, the memory 802 stores a program that can be executed by the processor 801. The processor 801 executes the program stored in the memory 802 to implement the following steps: sending a message to the first storage node through the first network link established with the first storage node. For the first read and write request, the first storage node is any storage node in the storage system; through the second network link established with the second storage node, the read and write response from the second storage node is received, and the read and write response is In response to the first read and write request, the second storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel;

or,

Receive a first read and write request from a first storage node, which is any storage node in the storage system; in response to the first read and write request, through the second network established by the client device and the second storage node link to return a read and write response to the client device. The second network link and the first network link belong to the same storage protocol channel. The first network link is the network link established between the client device and the first storage node.

The communication bus 803 mentioned in the above electronic equipment may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 803 can be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, only one thick line is used in Figure 8, but it does not mean that there is only one bus or one type of bus.

The memory 802 may include random access memory (RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. In an exemplary embodiment, the memory may also be at least one storage device located remotely from the aforementioned processor 801.

The above-mentioned processor 801 can be a general-purpose processor, including a Central Processing Unit (CPU for short), a Network Processor (NP for short), etc., or it can also be a Digital Signal Processing (DSP for short). ), Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

In yet another embodiment of the present disclosure, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program. When the computer program is run on a computer, it causes the computer to execute the above embodiments. The described data reading and writing methods.

The solution provided by this disclosure establishes multiple network links with multiple storage nodes in the storage cluster and simultaneously establishes a storage protocol layer link; multiple network links belong to the same storage protocol link.

The above technical solution provided by the present disclosure has the following advantages compared with the existing technology: the method provided by the present disclosure sends a first read and write request to the first storage node through the first network link established with the first storage node. One storage node is any storage node in the storage system; through the second network link established with the second storage node, the read and write response from the second storage node is received, and the read and write response is used as the first read and write request. In response, the second storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel. Since the first network link and the second network link belong to the same storage protocol channel, although the client device sends the first read and write request to the first storage node, the first read and write request can still be executed by the second storage node. Finally, the read-write response is sent to the client device, so that the client device can recognize that the read-write response is for the first read-write request.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions, when loaded and executed on a computer, produce processes or functions in accordance with the present disclosure, in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, e.g., from a website, computer, server, or data center via a wireline (e.g., Coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, microwave, etc.) means to transmit to another website, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media. The available media may be magnetic media (such as floppy disks, hard disks, magnetic tapes, etc.), optical media (such as DVDs), or semiconductor media (such as solid state drives).

It should be noted that in this article, relational terms such as “first” and “second” are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Furthermore, the terms "comprises," "comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. public. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be practiced in other embodiments without departing from the spirit or scope of the disclosure. Therefore, the present disclosure is not to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

A data reading and writing method, applied to client devices, the method includes:

Send a first read and write request to the first storage node through the first network link established with the first storage node, where the first storage node is any storage node in the storage system;

Through the second network link established with the second storage node, a read and write response from the second storage node is received, and the read and write response is used as a response to the first read and write request, and the second The storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel.
The method according to claim 1, wherein after using the read and write response as a response to the first read and write request, it further includes:

Extract the identification of the second storage node from the read and write response;

Based on the identification of the second storage node, a mapping relationship between the second storage node and the client device is established, where the mapping relationship indicates that the read and write response is stored in the second storage node.
The method according to claim 2, wherein, after establishing the mapping relationship between the second storage node and the client device based on the identity of the second storage node, it further includes:

Based on the mapping relationship, generate a second read and write request, the second read and write request is used to access the read and write response;

Send the second read and write request to the second storage node through the second network link, and receive the read and write response returned by the second storage node.
The method according to claim 3, wherein before sending the second read and write request to the second storage node through the second network link, it further includes:

Obtain the first submission queue and the second submission queue in the network layer. The first submission queue is used to send read and write requests to the first storage node. The second submission queue is used to send read and write requests to the second storage node. write request; and obtain the submission queue in the storage protocol layer, the submission queue of the storage protocol layer is used to send read and write requests to the first submission queue and the second submission queue;

The first submission queue and the second submission queue are mapped to a submission queue of the storage protocol layer.
The method according to claim 1, wherein before receiving the read and write response from the second storage node through the second network link established with the second storage node, it further includes:

Obtain the first completion queue and the second completion queue in the network layer. The first completion queue is used to receive read and write responses from the first storage node. The second completion queue is used to receive the read and write responses from the first storage node. The read and write response of the second storage node; and obtaining the completion queue of the storage protocol layer, which is used to receive the read and write response from the first completion queue or the read and write response of the second completion queue. write response;

The first completion queue and the second completion queue are mapped to a completion queue of the storage protocol layer.
The method according to claim 1, wherein before using the read and write response as a response to the first read and write request, it further includes:

Parse the storage protocol layer session identifier in the read and write response and the storage protocol layer session identifier in the first read and write request;

It is determined that the storage protocol layer session identifier in the read and write response is the same as the storage protocol layer session identifier in the first read and write request.
A data reading and writing method, applied to the second storage node, the method includes:

Receive a first read and write request from a first storage node, where the first storage node is any storage node in the storage system;

In response to the first read and write request, return a read and write response to the client device through the second network link established by the client device and the second storage node, the second network link and the first network link Belonging to the same storage protocol channel, the first network link is a network link established between the client device and the first storage node.
The method of claim 7, wherein in response to the first read and write request, after returning a read and write response to the client device through a second network link established by the client device and the second storage node ,Also includes:

Receive a second read-write request from the client device through the second network link, where the second read-write request is used to access the read-write response;

In response to the second read and write request, the read and write response is returned to the client device through the second network link.
A data reading and writing system, including:

A client device, a first storage node and a second storage node;

The client device is configured to send a first read and write request to the first storage node through the first network link established with the first storage node. The first storage node is any storage in the storage system. Node; receiving a read and write response from the second storage node through the second network link established with the second storage node, and using the read and write response as a response to the first read and write request, The second storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel.
A client device that includes:

The first sending unit is configured to send the first read and write request to the first storage node through the first network link established with the first storage node, and the first storage node is any storage node in the storage system;

A first receiving unit configured to receive a read and write response from the second storage node through the second network link established with the second storage node, and use the read and write response as a response to the first read and write request. In response, the second storage node is a node determined by the first storage node that can execute the first read and write request; the first network link and the second network link belong to the same storage protocol channel .
A second storage node including:

The second receiving unit is configured to receive the first read and write request from the first storage node, where the first storage node is any storage node in the storage system;

The second sending unit is configured to respond to the first read and write request and return a read and write response to the client device through the second network link established by the client device and the second storage node, the second The network link and the first network link belong to the same storage protocol channel, and the first network link is a network link established between the client device and the first storage node.
An electronic device includes: a processor, a memory and a communication bus, wherein the processor and the memory complete communication with each other through the communication bus;

The memory is used to store computer programs;

The processor is configured to execute the program stored in the memory to implement the data reading and writing method described in any one of claims 1-6 or the data reading and writing method described in claims 7-8.
A computer-readable storage medium storing a computer program, which when executed by a processor implements the data reading and writing method described in any one of claims 1-6 or the data reading and writing described in claims 7-8 method.