JP2011180933A - I/o switch and computer system equipped with the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that transfer performance is deteriorated because when data on the memory of one physical server have to be copied in the memory of two or more physical servers, using a general-purpose I/O interface which does not respond to multicast data transfer requires to issue a data write instruction to each of the plurality of physical servers, which results in increase in traffic between a server-to-server communication mechanism and an I/O switch. <P>SOLUTION: The I/O switch is provided with a mechanism which comprises a register capable of holding a plurality of pieces of destination information in the control register of the server-to-server communication mechanism, and the register, when sending data to a plurality of destinations, capable of adding the plurality of pieces of destination information to the data to transfer data toward a plurality of destinations by the data write instruction of one time. The I/O switch shapes the data write instruction issued by the server-to-server communication mechanism to data write instructions to each of the destinations and simultaneously issues the data write instructions to the plurality of destinations. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an inter-server communication mechanism for performing communication between physical servers in a computer system in which three or more physical servers are connected via an I / O switch.

  In recent computer systems, the need for server integration to operate multiple virtual servers on one physical server has increased along with the improvement in processing performance such as high performance of a single CPU and multi-core CPU. ing. By performing server integration, the number of OSs and applications running on one physical server increases. As a result, an increase in I / O devices that must be connected to one physical server is expected. In order to mount more I / O devices, there is an increasing need to connect a server and an I / O device via an I / O switch such as a PCI-Express (R) switch.

  While using an I / O switch to increase the number of physical I / O devices connected to a server, I / O virtualization is shared by sharing I / O devices between physical servers or virtual servers Is also expected. I / O virtualization refers to I / O devices by providing multiple virtual I / O devices on top of physical I / O devices and assigning virtual I / O devices to each physical server or virtual server. Is shared between physical servers or virtual servers. When sharing an I / O device with multiple physical servers, prepare an I / O switch with multiple upstream ports, connect a physical server to each upstream port of this I / O switch, and The stream port is configured to connect an I / O device, and the I / O device is shared between the physical servers. By using such I / O virtualization, a large number of I / O devices are used from each of OSs and applications running on an integrated server without increasing the physical number of I / O devices. It becomes possible.

  Also, in an in-memory DB that can perform data communication at a higher speed than a database built on a conventional disk by building a database on the memory, two or more waits for data on the memory of the active physical server There is a request to synchronize the memory data of multiplexed physical servers that are copied to the memory of the physical server at high speed. In this case, it is necessary to transfer a large amount of data on the memory at high speed, and high-speed communication means between physical servers is necessary.

  For example, in Patent Document 1, in a computer system in which two or more servers and an I / O device are connected by a switch, a server high-speed communication mechanism built in the I / O switch is used for server-to-server communication. A high-speed communication means is illustrated in which the inter-communication mechanism interprets a transfer command for data transfer, controls the general-purpose I / O interface, and transfers data between servers.

JP2009-292917 JP2000-4922 JP-A-9-83522

  In the memory data synchronization process applicable to the in-memory DB, data on the memory of one active physical server must be copied to the memory of two or more standby physical servers. Must write. When a general-purpose I / O interface that does not support data multicast transmission is used, a data write command must be issued for each of a plurality of physical servers. In this case, as the number of destination physical servers increases, the traffic between the server-to-server communication mechanism and the I / O switch increases, and the data transfer time becomes longer. .

  In order to solve the above-described problems, the inter-server communication mechanism having the multicast transmission function of the present invention has a register capable of holding a plurality of destination information in the control register of the inter-server communication mechanism, and transmits data to the plurality of destinations. In this case, a mechanism for adding a plurality of destination information of the register to the data is provided, and data transfer to the plurality of destinations can be performed by one data write command. Further, the I / O switch is characterized in that the data write command issued by the inter-server communication mechanism is shaped into a data write command for each of a plurality of destinations, and the data write command is issued simultaneously to the plurality of destinations.

  In the inter-server communication mechanism and the I / O switch according to the present invention, data multicast transmission processing is performed within the I / O switch with the built-in inter-server communication mechanism, and conforms to the I / O link protocol outside the switch. Since it is transmitted to each physical server as a command, it is effective when it is desired to perform multicast data transmission at high speed even with a general-purpose I / O interface that does not support multicast transmission. Furthermore, in the inter-server communication mechanism and the I / O switch of the present invention, it is not necessary to use an I / O device corresponding to a communication protocol that supports the multicast transmission function for data transfer between physical servers, and general-purpose I / O Since this is performed inside the server-to-server communication mechanism using only the O interface, the overhead of protocol conversion does not occur. Therefore, it is advantageous in terms of communication throughput and latency. For example, when synchronizing the memory data of the active physical server to the memory of multiple standby physical servers using a general-purpose I / O interface that does not support multicast transmission, the memory data is transferred to multiple standby servers. Since the memory data transfer time is shortened, the synchronous processing performance can be improved.

It is a block diagram showing the structure of the computer system which has an I / O switch with a built-in communication mechanism between servers concerning a present Example. It is a block diagram for demonstrating the internal structure of the I / O switch with a built-in server communication mechanism based on a present Example. It is a time chart for demonstrating operation | movement of the communication mechanism between servers which concerns on a present Example. It is a figure which shows the data structure of the command processed within the I / O switch with a built-in server communication mechanism based on a present Example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Description of configuration)
FIG. 1 shows a configuration diagram of a computer system according to the present invention. The physical server 111 has a CPU 131, a memory 121, and a CPU-I / O interface 141, and the OS 101 is running on the physical server 111. The same applies to the physical server 112, which includes a CPU 132, a memory 122, and a CPU-I / O interface 142, and the OS 102 is operating on the physical server 112. The same applies to the physical server 113, which includes a CPU 133, a memory 123, and a CPU-I / O interface 143, and the OS 103 is running on the physical server 113.

  The physical servers 111, 112, and 113 are connected to the upstream port of the communication mechanism built-in I / O switch 151 via the CPU-I / O interfaces 141, 142, and 143. I / O devices 181, 182, 183, and 184 are connected to downstream ports of the communication mechanism built-in I / O switch 151, and the physical servers 111, 112, 113 are connected via the communication mechanism built-in I / O switch 151. And I / O devices 181, 182, 183, and 184 are connected to each other. Here, each of the I / O devices 181, 182, 183, and 184 is an I / O device that is shared and used by the physical servers 111, 112, and 113, either the physical server 111, 112, or 113. Alternatively, it may be an I / O device that is used exclusively by one unit. Further, an inter-server communication mechanism 171 is connected to the downstream port of the communication mechanism built-in I / O switch 151, and is connected to the physical servers 111, 112, and 113 via the communication mechanism built-in I / O switch 151. Yes. The inter-server communication mechanism 171 is built in the communication mechanism built-in I / O switch 151.

  FIG. 2 is a diagram showing the internal structure of the switch unit 161 and the inter-server communication mechanism 171 in the communication mechanism built-in I / O switch 151. The inter-server communication mechanism 171 is connected to the switch unit 161 via the I / O switch link 201. The intra-I / O switch link 201 and the inter-server communication mechanism 171 are connected by an inter-server communication mechanism switch unit interface 202. The inter-server communication mechanism 171 includes a memory read command generation unit 203 for fetching memory data, a memory write command generation unit 204 for sending out memory data, and an interrupt command generation unit 205 for generating an interrupt. . A destination information adding unit 206 for sending a command to a correct destination server is provided ahead of the memory read command generation unit 203, the memory write command generation unit 204, and the interrupt command generation unit 205. An instruction issue sequencer 208 is provided to control the operations of the memory read instruction generator 203, the memory write instruction generator 204, and the interrupt instruction generator 205. The inter-server communication mechanism 171 includes a memory data return command receiving unit 209 for capturing data read by a memory read command, and a memory data buffer 210 for storing the received memory data. is there. Finally, an inter-server communication mechanism register 211 is provided as a mechanism for controlling the inter-server communication mechanism 171 from software. This inter-server communication mechanism register 211 includes destination information 212, a transmission memory address 213, a reception 1 memory address 214, a reception 2 memory address 215, a transmission memory area length 216, and a start register 217. Note that the reception memory address register needs to have a register for a physical server as a destination, and does not need to be fixed as in the embodiment.

  The switch unit 161 is connected to the upstream port and the downstream port of the communication mechanism built-in I / O switch 151 by the I / O link 221. The I / O link 221 and the inside of the switch unit 161 are connected by an I / O interface 222. The intra-I / O switch link 201 and the switch unit 161 are connected by an inter-server communication mechanism inter-switch interface 224. The switch unit 161 determines whether the command transmitted from the server communication mechanism is unicast or multicast. If the command is multicast, the switch unit 161 is a multicast command determination / command shaping unit 226 as a mechanism for shaping the unicast command for each destination. It has. A unicast command transmitted from the inter-server communication mechanism 171 or a unicast command to a plurality of destinations shaped by the multicast command determination / command shaping unit 226, and an I / O device received via the I / O link 221 181, 182, 183, 184 or all destinations of all unicast commands from the physical servers 101, 102, 103 are determined, and a destination determination / command arbitration unit 225 is used as a mechanism for determining a command that can be transmitted for each destination. I have. The crossbar switch 223 is used to transmit a unicast command to each destination determined by the destination determination / command arbitration unit 225 to the I / O link 221 or the I / O switch link 201.

(Description of operation)
FIG. 3 is a time chart showing the operation of the inter-server communication mechanism. This time chart operates as an example of multicast transfer of data existing in the memory 121 in the physical server 111 to the memory 122 in the physical server 112 and the memory 123 in the physical server 113 in the system having the configuration shown in FIG. Indicates.

  First, the OS 101 running on the physical server 111 on the data transmission side sets the start address of the memory area of the physical server 111 to be transmitted. This setting is performed by transmitting a write command from the physical server 111 on the data transmission side to the transmission memory address register 214 of the inter-server communication mechanism 171 (301). At this time, the destination information of the physical server 111 and the information indicating the transmission side are registered in the destination information register 212. Data is written to the reception memory address 1 register 214 from the physical server 112 on the data receiving side, and the head address of the memory area of the physical server 112 to receive is set (302). At this time, the destination information of the physical server 112 and information indicating the receiving side are registered in the destination information register 212. Similarly, data is written from the physical server 113 on the data receiving side to the received memory address 2 register 215, and the head address of the memory area of the receiving physical server 113 is set (303). At this time, the destination information of the physical server 113 and information indicating the receiving side are registered in the destination information register 212. Writing to the transmission memory area length register 216 is performed, and the size of the memory area to be transmitted is set (304). Thus, the initial setting process for performing communication between servers is completed.

  Next, the inter-server communication mechanism 171 is activated by writing to the activation register 217 of the inter-server communication mechanism 171 (305). When activated, the instruction issuing sequencer 208 starts operation, and the destination information of the physical server 111 on the data transmission side is added from the memory read command generation unit 203 via the destination information addition unit 206, and the memory read command is issued. It is issued (306a). In this memory read command, the destination determination / command arbitration unit 225 of the switch unit 161 determines the destination using the transmission-side server destination information, and is correctly transmitted to the physical server 111 on the data transmission side. The physical server 111 on the data transmission side that has received the memory read command sends a data return command to the inter-server communication mechanism 171 to return the memory data to the memory read command 306a (307a). In the inter-server communication mechanism 171, the data return command is received by the memory data return command receiving unit 209 and the memory data part is stored in the memory data buffer 210. When the memory data is received, the memory write command generation unit 204 retrieves the memory data from the memory data buffer 210, adds destination information of the physical server 112 and the physical server 113 on the data receiving side via the destination information addition unit 206, A multicast memory write command is issued (308a). FIG. 4 shows the data structure of the multicast instruction and the unicast instruction. The data structure of the multicast memory write command 308a is a multicast command data structure 401. The header 411 is destination information of the physical server 112 on the reception side, and the header 412 is destination information of the physical server 113 on the reception side. Data 413 is data written to the destination memory. The multicast memory write command 308a is determined as a multicast command by the multicast command determination / command shaping unit 226 of the switch unit 161, and is shaped into a memory write command addressed to the physical server 112 and a memory write command addressed to the physical server 113. . The data structures of the shaped memory write instructions are unicast instruction data structures 421 and 422. The unicast instruction data structure 421 is a data structure of a memory write instruction that is shaped by the header 411 and the data 413 of the multicast instruction data structure 401. Similarly, the unicast instruction data structure 422 is a data structure of a memory write instruction that is shaped by the header 412 and the data 413 of the multicast instruction data structure 401. The destinations of the two formatted memory write commands are determined by the destination determination / command arbitration unit 225 using the header 431 and the header 432 which are reception side server destination information, and the physical server 112 and the physical server 113 on the data reception side. (309a, 310a). The instruction issuing sequencer 208 designates a series of operations of sending the above memory read command (306a), receiving the memory return command (307a), and sending the multicast memory write command (308a) using the send memory area length register 216. By repeating until the transferred data length is transferred, data is multicast transferred from the physical server 111 on the data transmission side to the physical server 112 and the physical server 113 on the data reception side.

  Finally, when the data transfer is completed, the instruction issuing sequencer 208 activates the interrupt instruction generating unit 205. The interrupt command generation unit issues an interrupt command to the physical server 111 on the data transmission side, the physical server 112 on the data reception side, and the physical server 113, respectively. The interrupt command generated by the interrupt command generation unit 205 is added with the destination information of the physical server 111 on the data transmission side and the physical servers 112 and 113 on the data reception side via the destination information addition unit 206, and the multicast transmission is completed. An interrupt notification is issued (311). The issued multicast transmission completion interrupt notification 311 is shaped into a transmission completion interrupt notification of a unicast command data structure for the physical servers 111, 112, and 113 in the switch unit 161 and transmitted, similarly to the multicast memory write command ( 312, 313, 314). Thus, all data transfer processing between servers is completed.

  In this embodiment, the operating entity that instructs the server-to-server communication mechanism to set or activates the server-to-server communication mechanism has been described as an OS. However, the operating entity is a device driver for the server-to-server communication mechanism. Alternatively, it may be an application program or the like, or a hypervisor or the like that manages a virtual server.

111, 112, 113 Physical server, 151 Communication mechanism built-in I / O switch, 161 switch unit, 171 Inter-server communication mechanism

Claims (3)

  An I / O switch having a plurality of upstream ports and one or more downstream ports has a control register capable of holding a plurality of destination information, and when transmitting data to a plurality of destinations, a plurality of destinations of the registers An I / O switch comprising a mechanism for adding information to the data, and a server-to-server communication mechanism capable of performing multicast transmission to a plurality of destinations.   The I / O switch according to claim 1, wherein the switch unit in the I / O switch includes a mechanism for determining whether the command issued from the inter-server communication mechanism according to claim 1 is multicast or unicast, A mechanism for shaping a multicast command into unicast commands for the number of destinations using each of a plurality of pieces of destination information in the multicast command and one data in the multicast command when the determination mechanism determines that the command is a multicast write command; An I / O switch characterized by simultaneously issuing a plurality of unicast instructions to a port that can be issued.   3. An I / O switch according to claim 1, comprising a plurality of physical servers each having one or more CPUs and a memory, a plurality of upstream ports, and one or more downstream ports. In a computer system in which a physical server is connected via the upstream port of the I / O switch, the memory of the physical server that is the data transmission source is requested by a request from the physical server to the inter-server communication mechanism in the I / O switch. A computer system characterized by multicasting data on the memory of a plurality of destination physical servers.

Images