WO2017162175A1

WO2017162175A1 - Data transmission method and device

Info

Publication number: WO2017162175A1
Application number: PCT/CN2017/077752
Authority: WO
Inventors: 王东临; 金友兵
Original assignee: 北京书生国际信息技术有限公司; 书生云公司
Priority date: 2016-03-25
Filing date: 2017-03-22
Publication date: 2017-09-28
Also published as: CN105867843A

Abstract

The embodiments of the present invention provide a data transmission method and device, which improve data transmission efficiency. The data transmission method is used for a storage system having a shared storage pool and comprises: receiving a data transmission request sent by an application on a source physical server; storing, in a shared storage pool of a storage system in which said application is located, data to be transmitted by the application; packaging the storage address of the to-be-transmitted data in network-protocol form and transmitting same.

Description

Data transmission method and device

Technical field

The present invention relates to the field of computer communication technologies, and in particular, to a data transmission method and apparatus.

Background technique

In the prior art, when data sharing is performed between applications running on different servers, it is usually necessary to transfer all data content from one server to another by using a network protocol such as TCP/IP protocol. Under the circumstance, the efficiency of data transmission is low.

Summary of the invention

In view of this, the embodiments of the present invention provide a data transmission method and apparatus, which improve data transmission efficiency.

An embodiment of the present invention provides a data transmission method for a storage system having a shared storage pool, including:

Receiving a data transmission request sent by an application on the physical server;

Storing the data to be transmitted by the application in a shared storage pool of a storage system in which it is located;

The storage address of the data to be transmitted is packaged in the form of a network protocol.

Another embodiment of the present invention further provides a data transmission apparatus for a storage system having a shared storage pool, including:

a receiving module, configured to receive a data transmission request sent by an application on a physical server;

a storage module, configured to store the data to be transmitted by the application in a shared storage pool of a storage system where the application is located;

The sending module is configured to package the storage address of the data to be transmitted in a network protocol manner.

According to the data transmission method and apparatus provided by the embodiment of the present invention, when the source server and the target server are located in a storage system sharing the same shared storage pool, the data itself is no longer transmitted between the two servers, but the hardware providing the shared storage system is provided. In the solution, by providing the source server with the mapping of the data file to the data address, and providing the mapping of the data address to the data file for the target server at the subsequent receiving end, so that the application of the source server and the target server are not changed. It can realize the only transfer of the address of the data file between the source server and the target server, without transferring all the data itself, which greatly improves the data transmission efficiency.

DRAWINGS

FIG. 1 is a schematic structural diagram of a storage system according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart diagram of a data transmission method according to an embodiment of the present invention.

FIG. 3 is a structural block diagram of a data transmission apparatus according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

FIG. 1 is a schematic structural diagram of a storage system according to an embodiment of the present invention. As can be seen from the figure, the storage system includes a storage network; a storage node is connected to the storage network, wherein the storage node is a software module that provides a storage service, instead of a hardware server including a storage medium in a general sense. And a storage device that is also connected to the storage network. Each storage device includes at least one storage medium. For example, a storage device commonly used by the inventors can place 45 storage media. Wherein, the storage network is configured such that each storage node can access all storage media without resorting to other storage nodes.

With the storage system provided by the embodiment of the present invention, each storage node can access all the storage media without using other storage nodes, so that all the storage media of the present invention are actually It is shared by the physical servers where all the storage nodes are located, thus achieving the effect of the global shared storage pool.

At the same time, it can be seen from the above description that in the embodiment of the present invention, the physical server where the storage node is located is independent of the storage device, and the storage device is more used as a channel connecting the storage medium and the storage network.

In another embodiment of the present invention, the storage node side further includes a computing node, and the computing node and the storage node are disposed in a physical server, and the physical server is connected to the storage device through the storage network. The aggregated storage system in which the computing node and the storage node are located in the same physical machine constructed by using the embodiment of the present invention can reduce the number of physical devices required, thereby reducing the cost. At the same time, the compute node can also access the storage resources it wishes to access locally. In addition, since the compute nodes and storage nodes are aggregated on the same physical server, the data exchange between the two can be as simple as shared memory, and the performance is particularly excellent.

In the storage system provided by the embodiment of the present invention, the length of the I/O data path between the computing node and the storage medium includes: (1) the storage medium to the storage node; and (2) the storage node to the computing node aggregated in the same physical server. (CPU bus path). In contrast, in the prior art storage system shown in FIG. 1, the I/O data path length between the compute node and the storage medium includes: (1) storage medium to storage node; (2) storage node to storage Network access network switch; (3) storage network access network switch to core network switch; (4) core network switch to computing network access network switch; and (5) computing network access network switch to computing node. Obviously, the total data path of the storage system of the embodiment of the present invention is only close to item (1) of the conventional storage system. That is, the storage system provided by the embodiment of the present invention can greatly improve the I/O channel performance of the storage system by extremely compressing the I/O data path length, and the actual running effect is very close to the I/O of the local hard disk. O channel, which provides higher performance than traditional network transmissions such as Ethernet.

In an embodiment of the present invention, the storage node may be a virtual machine of a physical server, a container, or a module directly running on a physical operating system of the server, and the computing node may also be a virtual machine of the same physical machine server, a container or something that runs directly on the server A module on the operating system. In one embodiment, each storage node may correspond to one or more compute nodes.

Specifically, one physical server may be divided into multiple virtual machines, one of which is used as a storage node, and the other virtual machine is used as a computing node; or a module on the physical OS is used as a storage node, so as to implement Better performance.

In an embodiment of the invention, the virtualization technology forming the virtual machine may be KVM or Zen or VMware or Hyper-V virtualization technology, and the container technology forming the container may be Docker or Rocket or Odin or Chef or LXC or Vagrant. Or Ansible or Zone or Jail or Hyper-V container technology.

In an embodiment of the present invention, all the storage media in the system may be divided according to storage logic. Specifically, the storage pool of the entire system may be divided into a logical storage hierarchy structure such as a storage area, a storage group, and a storage block. Among them, the storage block is the smallest storage unit. In an embodiment of the invention, the storage pool may be divided into at least two storage areas.

In an embodiment of the invention, each storage area may be divided into at least one storage group. In a preferred embodiment, each storage area is divided into at least two storage groups.

In some embodiments, the storage area and the storage group can be merged such that one level can be omitted in the storage hierarchy.

In an embodiment of the invention, each storage area (or storage group) may be composed of at least one storage block, wherein the storage block may be a complete storage medium or a part of a storage medium. In order to construct redundant storage inside the storage area, each storage area (or storage group) may be composed of at least two storage blocks, and when any one of the storage blocks fails, the complete storage block may be calculated from the remaining storage blocks in the group. The data is stored. The redundant storage mode can be multi-copy mode, independent redundant disk array (RAID) mode, and erasure code mode. In an embodiment of the invention, the redundant storage mode can be established by the ZFS file system. In an embodiment of the present invention, in order to combat the hardware failure of the storage device/storage medium, the plurality of storage blocks included in each storage area (or storage group) are not located in the same storage medium, or even in the same Storage devices. In an embodiment of the invention, any two storage blocks included in each storage area (or storage group) are not located in the same storage medium/storage device. In another embodiment of the present invention, the number of storage blocks located in the same storage medium/storage device in the same storage area (or storage group) is preferably less than or equal to the redundancy of the redundant storage. For example, when the RAID 5 mode of redundancy is used, the redundant storage redundancy is 1, and the number of storage blocks in the same storage group of the same storage device is at most 1; for RAID 6, the redundant storage is With a redundancy of 2, the number of memory blocks in the same storage group on the same storage device is up to 2.

In one embodiment, the storage medium may include, but is not limited to, a hard disk, a flash memory, an SRAM, a DRAM, an NVME, or the like. The access interface of the storage medium may include, but is not limited to, a SAS interface, a SATA interface, a PCI/e interface, a DIMM interface, NVMe interface, SCSI interface, and AHCI interface.

In an embodiment of the invention, the storage network may include at least one storage switching device, and the storage node accesses the storage medium through data exchange between the storage switching devices included therein. Specifically, the storage node and the storage medium are respectively connected to the storage switching device through the storage channel.

In an embodiment of the invention, the storage switching device may be a SAS switch or a PCI/e switch. Correspondingly, the storage channel may be a SAS (Serial Attached SCSI) channel or a PCI/e channel.

Taking the SAS channel as an example, compared with the traditional IP-based storage solution, the SAS-based switching solution has the advantages of high performance, large bandwidth, and a large number of disks per device. When used in conjunction with a host adapter (HBA) or a SAS interface on a server board, the storage provided by the SAS system can be easily accessed by multiple servers connected simultaneously.

Specifically, the SAS switch is connected to the storage device through a SAS line, and the storage device and the storage medium are also connected by a SAS interface. For example, the storage device internally connects the SAS channel to each storage medium (may be in the storage device) Internally set a SAS switch chip). Since the bandwidth of a SAS network can reach 24Gb or 48Gb, which is several times that of Gigabit Ethernet, and several times that of an expensive 10 Gigabit Ethernet; at the same time, the link layer SAS has an order of magnitude improvement over the IP network, in transmission. Layer, due to TCP protocol three-way handshake four times off, high overhead and TCP delay The acknowledgment mechanism and slow start sometimes cause a delay of 100 milliseconds. The delay of the SAS protocol is only a few tenths of that of TCP, and the performance is greatly improved. In summary, SAS networks offer significant advantages in terms of bandwidth and latency over Ethernet-based TCP/IP. Those skilled in the art will appreciate that the performance of the PCI/e channel can also be adapted to the needs of the system.

In an embodiment of the invention, the storage network may include at least two storage switching devices, each of which may be connected to any one of the storage devices through any one of the storage switching devices, thereby being connected to the storage medium. When any storage switching device or storage channel connected to a storage switching device fails, the storage node reads and writes data on the storage device through other storage switching devices.

Based on the storage system with shared storage pool function shown in FIG. 1, when an application located on one physical server needs to transfer data to an application on another physical server, in an embodiment of the present invention, Two physical plug-ins are installed in the two physical servers. For the convenience of description, two physical servers can be recorded as the source server and the target server, and the two plug-ins are respectively recorded as the source server plug-in and the target server plug-in. These two plugins work together, and the workflow is shown in Figure 2.

On the source server side, the source server plugin performs the following process:

Step 201: Receive a data transmission request sent by an application on the source server.

Step 202: Store the data to be transmitted by the application in a shared storage pool of a storage system where the application is located. The data to be transmitted may be stored in one or more storage media of the shared storage pool.

Step 203: The storage address of the data to be transmitted is packaged and sent in a network protocol manner.

The address information is transmitted to the corresponding target server plug-in on the target server through the network by using its existing communication protocol, such as TCP/IP, FTP, and the like. Those skilled in the art can understand that the existing communication mode can be adopted between the source server and the target server, which is not within the scope of the present invention.

The target server plug-in located on the target server performs the following process:

Step 204: Receive a data packet packaged in a network protocol manner, and obtain address information stored in the data packet. After receiving the data packet by the existing communication method, the plug-in of the target server parses the data packet to obtain the address information in the data packet. This step may be an existing communication unpacking mode, which is not limited by the present invention.

Step 205: Access, according to the received storage address information, a data file stored in the shared storage pool and stored in the address information, and return the data file to a specified target server on the target server. application.

Here, when the application of the source server sends a data transmission request, the request includes the information of the target server and the corresponding application (for example, an IP address plus a port number) in addition to the data to be transmitted.

In an embodiment of the invention, when the source server plug-in sends a data packet, the data packet includes an identifier indicating whether the current data packet contains the address of the data file or the data file itself. After receiving the data packet, the target server plug-in performs the above-mentioned process and method according to the embodiment of the present invention once it is determined that the data packet contains the address of the data file. Otherwise, the process is performed according to the prior art.

In this way, the application on the two servers of the storage system having the same shared storage pool can greatly reduce the number of data transmissions in the system when the data is transmitted in the shared storage system without any change. Greatly saves the network resources of the shared storage system. Of course, those skilled in the art can understand that in practical applications, the application on each server will be both the sender of the information and the receiver of the information, so the plug-ins installed on each physical server will be simultaneously The functions of the source server plugin and the target server plugin mentioned in the above embodiments are provided.

In an embodiment of the present invention, software code is stored in a storage system of each physical server in the shared storage system. When the software code is executed, the foregoing embodiment may be implemented by, for example, a virtual machine. And the steps of the source server plugin and the target server plugin. The source server's application needs to go through the gateway when making network communication to the target server's application. The conversion can be done at the gateway and is transparent to the application.

In an embodiment of the present invention, software codes are stored in a gateway corresponding to each physical server in the shared storage system, and when the software codes are executed, the source server plug-ins and targets mentioned in the foregoing embodiments may be executed. The steps of the server plugin.

FIG. 3 is a structural block diagram of a data transmission apparatus according to an embodiment of the present invention. As shown in Figure 3, the device comprises:

The receiving module 301 is configured to receive a data transmission request sent by an application on the physical server where the physical server is located.

The storage module 302 is configured to store the data to be transmitted by the application in a shared storage pool of a storage system where the application is located.

The sending module 303 is configured to package the storage address of the data to be transmitted in a network protocol manner.

In another embodiment of the present invention, the receiving module 301 is further configured to receive a data packet that is packaged in a network protocol manner.

The device further includes:

The obtaining module 304 is configured to obtain storage address information in the data packet.

The data providing module 305 is configured to access, by using the storage address information of the storage system, the data file stored in the shared storage pool in the address information according to the received storage address information, and return the data file to the physical entity The specified application on the server.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions, etc., which are within the spirit and principles of the present invention, should be included in the scope of the present invention. within.

Claims

A data transmission method, comprising:

Receiving a data transmission request sent by an application on the physical server;

Storing the data to be transmitted by the application in a shared storage pool of a storage system in which it is located;

The storage address of the data to be transmitted is packaged in the form of a network protocol.
The method of claim 1 wherein said storage system comprises:

Storage network

At least two storage nodes connected to the storage network;

At least one storage device connected to the storage network, each storage device including at least one storage medium,

The storage network is configured such that each physical server where the storage node is located can access all storage media without using a physical server where other storage nodes are located;

The storage medium constitutes the shared storage pool.
The method according to claim 1 or 2, wherein the information packaged in a network protocol further includes identification information of the target application.
The method of claim 1 or 2, further comprising:

Receiving a data packet packaged in a network protocol manner, and acquiring address information stored in the data packet;

Accessing, by the storage network of the storage system, a data file stored in the shared storage pool in the address information according to the received storage address information, and returning the data file to a specified application on the physical server .
The method of claim 1 or 2, wherein the network protocol is a TCP and/or IP protocol.
The method according to claim 1 or 2, wherein the storage network is a SAS network, and the storage node and the storage device are connected by a SAS switch.
A data transmission device, configured for use in a storage system having a shared storage pool, comprising:

a receiving module, configured to receive a data transmission request sent by an application on a physical server;

a storage module, configured to store the data to be transmitted by the application in a shared storage pool of a storage system where the application is located;

The sending module is configured to package the storage address of the data to be transmitted in a network protocol manner.
The device of claim 7 wherein:

The receiving module is further configured to receive a data packet that is packaged in a network protocol manner;

The device further includes:

An obtaining module, configured to obtain storage address information in the data packet;

a data providing module, configured to access, by the storage network of the storage system, a data file stored in the address storage information in the shared storage pool according to the received storage address information, and return the data file to the physical server where the data is located The specified application on.
The apparatus according to claim 7 or 8, wherein the apparatus is a plug-in installed on the physical server or on a virtual machine of the physical server.
The apparatus according to claim 7 or 8, wherein said apparatus is located on a gateway setting corresponding to said physical server.