JP2014116645A - Computer system, communication path management method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a communication load between network switches and also reduce costs of a control system of switches in a multi-path network system using the multiple network switches.SOLUTION: A computer system comprises: first and second network switches communicatively connected with each other; and a plurality of computers including a first network adapter for communicating with a network adapter of another computer via the first network switch and a second network adapter for communicating with a network adapter of another computer via the second network switch. Each computer holds path information on paths on which communication can be performed between the first and second network adapters of other computers and the first and second network adapters of its own computer respectively, and preferentially determines a communication path on which there is no communication between the first and second network switches as a communication path with another computer. Further, it manages the path information by broadcast transmission and its responses.

Description

  The present invention relates to a computer system, a communication path management method, and a program, and more particularly, to a computer system, a communication path management method, and a program for reducing communication load.

  In recent years, it has become common to connect a large number of servers to a network to construct a large-scale system and perform a large amount of data processing. In particular, development of a parallel distributed framework for mass data processing as disclosed in Non-Patent Document 1 also contributes to the realization of such a large-scale system.

Non-Patent Document 1 discloses a technique related to so-called Map-Reduce. Map-reduce is generally used in a system configuration in which a plurality of execution servers, a plurality of data servers that provide a distributed file system, and storage devices connected to the data servers are connected to be communicable. In particular,
[1] A Map phase in which data on the distributed file system is read to each execution server, processing is performed in units of records (rows), and another record in the key-value format is output as a result for each record;
[2] A Shuffle phase that collects records output by the Map phase that have the same key in the same server;
[3] A framework composed of a Reduce phase in which processing is performed on a record group collected for each key, another record is output as a result for each key or record, and the result is stored in a distributed file system. .

As described above, Map-Reduce has an advantage that the user only needs to describe the process executed in units of records in the Map phase and the Reduce phase. Since the processing in the Map phase and the Reduce phase is independent for each record, the processing is executed in parallel for each record. However, since the parallel execution is controlled dynamically by the framework, The user does not need to be aware of parallel processing.
The data stored in the distributed file system is duplicated on different servers constituting the distributed file system so that fault tolerance is ensured.

  In a system that executes such a Map-Reduce process, large-scale communication occurs between the servers and devices in each of the phases [1] to [3] described above. The total throughput of this communication is proportional to the number of servers and the like. Therefore, by performing the processing performed by each execution server and each data server on the same server, the data transfer amount on the network in the [1] Map phase can be reduced. However, the [2] Suffle phase essentially generates communication, and the [3] Reduce phase generates communication to create a copy of the data. Control is difficult.

  Therefore, in order to appropriately control the amount of communication proportional to the number of servers when processing a large amount of data, it is indispensable to expand the system scale and improve the network bandwidth. Therefore, as a method of connecting many servers via a network, a tree-type network configuration in which network switches (hereinafter referred to as “NT switches”) are connected in many stages is often used. However, this method has a problem that the communication load is concentrated on the upper network switch and becomes a bottleneck.

  Further, for example, Non-Patent Document 2 discloses another technique for improving the network bandwidth. This is a technique related to so-called SMLT (Split Multi-link Trunking). For the sake of explanation, FIG. 15 illustrates the network configuration. In the network configuration of FIG. 15, each of the plurality of servers SV10 to SVn includes a plurality of network adapters (hereinafter referred to as “NT adapters”) E100 to En110, and each NT adapter is connected to a different NT switch SW10 or SW20. Furthermore, the NT switches have a configuration in which the NT switches are communicably connected (hereinafter, a path connecting the NT switches is referred to as an “inter-switch path”). The NT switches SW10 and SW20 are “Layer-2” switches.

  When each server SV10 to SVn transmits a packet, an NT adapter to be used for packet transmission is determined by the Link-Aggregation algorithm. For example, assume that a packet is transmitted from the server SV20 to the server SV10 via the NT adapter E200 during normal operation (solid arrow). As shown in FIG. 15, the transmitted packet is transmitted to the server SV1 via the NT switch SW10 and the NT adapter E100 (solid arrow).

  When a failure occurs between the NT adapter E100 and the NT adapter SW10, the communication path is changed to transfer data from the server SV20 to the server SV10 via the E200, SW10, SW20, and E110 as shown ( Dashed arrows). That is, since the server SV20 cannot detect a failure occurring between the server SV10 and the NT switch SW10, the packet is transmitted from the NT adapter E200 in the same manner as before the failure, so that the packet is transferred through the inter-switch path. Become.

  As another further technique for improving the network bandwidth, there is a technique disclosed in Patent Document 1. A configuration example of a network to which this technology is applied is shown in FIG. The difference from the network configuration example of NPL 2 (FIG. 15) is that the NT switches SW10 and SW20 are not directly connected, and the NT switch SW30 (Layer-3) is connected above the NT switches SW10 and SW20. This is the point. A large-scale network can be constructed by preparing a plurality of configurations below the NT switch SW30 and connecting them in a torus shape.

  The technique of Patent Document 1 is different from the technique of Non-Patent Document 2 in that when a failure occurs in the connection between the server and the NT switch, the route is changed so as not to use the inter-switch path. Specifically, during normal operation, a packet from the server SV20 to the server SV10 is transmitted via E200, SW10, and E100 (similar to the solid arrow shown in FIG. 15). When a failure occurs in the connection between E100 and SW10, the route is changed so that the packet from server SV20 to server SV10 is transferred via E210, SW20, and E110 (broken line) Arrow).

  Non-Patent Document 3 discloses a technique for controlling a plurality of NT adapters mounted on a server in software. In this document, a plurality of control methods are described. Among these, balance-tlb and balance-alb can use a plurality of NT switches to improve the bandwidth. For example, as shown in FIG. It can be used in various network configurations.

  The balance-tlb selects the transmission NT adapter so that the load of the NT adapter is leveled when the server transmits a packet. By balance-alb, load leveling of the NT adapter is performed not only at the time of packet transmission but also at the time of packet reception. Specifically, the reception data amount of each NT adapter is monitored, and the reception NT adapter is controlled so that the reception load of each NT adapter is equalized. When the server receives an ARP request from another server, the NT adapter MAC address used in the reception process from that server is returned as the source MAC address in the ARP response. It has come to be realized.

JP 2008-11082 A

Jeffry Deans, Sanjay Ghemawat “MapReduce: Simplified Data Processing on Large Clusters, December 5, 2004, USENIX Association OSDI'04: 6th 6th Symmetric. /tech/full_papers/dean/dean.pdf) Roger Lapuh, “Split Multi-link Trunking (SMLT) draft-lapuh-network-smlt-08”, [online], January 7, 2009, [Search September 20, 2012], Internet (http: / /tools.ietf.org/html/draft-lapuh-network-smlt-08) Thomas Davis, “Luxux Ethernet Bonding Driver HOWTO” April 27, 2011 Update, [online], [September 20, 2012 search], (http://www.kernel.org/doc/Documentation/networking /bonding.txt)

  As a problem of Non-Patent Document 2, when a failure occurs on a network route, the route is always changed to a route that passes through an inter-switch path. If a failure occurs on multiple routes, the inter-switch path may become a communication bottleneck. There is. As a method for increasing the scale of this system, as shown in FIG. 17, it is conceivable to interconnect subnets configured as shown in FIG. 15 in a torus shape. You will use the path. Therefore, when a failure occurs on the network path, there is a problem that the bandwidth that can be used for inter-subnet communication is reduced when a network load is applied to the inter-switch path.

  A problem of the technique of Non-Patent Document 2 is that since the network topology is not considered, the frequency of occurrence of communication via the inter-switch path is increased. Specifically, since a communication route is selected at random, almost half of the communication goes through the inter-switch path.

  The problem of the technique of Patent Document 1 is that the fault tolerance is low. For example, when a failure occurs between E210 and SW20 in addition to the failure between E100 and SW10 as shown in FIG. 15, communication between server SV10 and server SV20 becomes impossible.

  In addition, as a problem common to the technologies of Non-Patent Document 2 and Patent Document 1, since the change of the packet transfer path when a failure occurs is controlled by the NT switch, an NT switch having a dedicated function is required. Become. Generality and cost issues arise.

  In order to solve the above-described problem, for example, the configuration described in claim 1 is applied. That is, at least first and second network switches that are communicably connected, and other computers via the first network adapter and the second network switch that communicate with the network adapters of other computers via the first network switch. A computer system comprising a plurality of computers having at least a second network adapter that communicates with the other network adapter, wherein the plurality of computers includes first and second network adapters of the other computers and a first computer of the own computer. And a storage unit that holds path information that can be communicated with the second network adapter, and a communication path that does not communicate between the first and second network switches based on the path information, with the other computer. A computer system having a control unit that preferentially determines the communication path of That.

According to one aspect of the present invention, it is possible to reduce the traffic on the inter-switch path and to communicate with a shorter path in communication between computers constituting the system. Furthermore, since each computer executes control of communication path selection, the network switch control system can be simplified.
Other problems and effects of the present invention will become apparent from the contents of the following embodiments.

It is a schematic diagram which shows the example of the route selection of the computer system which is one Embodiment to which this invention is applied. It is a schematic diagram which shows the other example of the route selection of this embodiment. It is a schematic diagram which shows the receiving path | route from the external subnet in this embodiment. It is a block diagram which shows typically the structure of the computer system in this embodiment. It is a schematic diagram which shows an example of the active server information in this embodiment. It is a schematic diagram which shows an example of NT adapter information in this embodiment. It is a schematic diagram which shows an example of the router information in this embodiment. It is a schematic diagram which shows an example of the single path | route information in this embodiment. It is a schematic diagram which shows an example of ARP information in this embodiment. It is a schematic diagram which shows an example of the routing information in this embodiment. It is a schematic diagram which shows an example of the setting file in this embodiment. It is a flowchart which shows an example of the whole process of the virtual NIC driver in this embodiment. It is a flowchart which shows an example of a route detection process with the router in this embodiment. It is a flowchart which shows an example of the server detection process in this embodiment. It is a flowchart of the packet reception process in this embodiment. It is a schematic diagram which shows the example of the route selection in a prior art. It is a schematic diagram which shows the other example of the route selection in a prior art. It is a schematic diagram which shows the example of a system which connects a subnet like a torus.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1A, FIG. 1B, and FIG. 2 show an outline of route selection of the computer system 1 that is an embodiment to which the present invention is applied. In the computer system 1, servers SV1 to SVn having a plurality of NT adapters E10 to En1 are connected to different L3 switches (NT switches operating in Layer-3) SW1 or SW2 for each NT adapter. One subnet having an inter-switch path enabling communication is configured.

When the system scale is small and only one subnet is required, a configuration in which the L3 switch is changed to the L2 switch (NT switch operating in Layer-2) may be applied. In order to expand the scale of the system, the present invention can be applied to a network configuration in which NT switches are L3 switches and subnets are connected by the L3 switches. For example, as shown in FIG. 17, the subnet NT switches are connected in a torus shape.
For example, when configuring a parallel distributed system to implement Map-Reduce as described above, it is only necessary to arrange as many execution servers and data servers as necessary in the subnet.

  Since inter-subnet communication uses the conventional Layer-3 function, the network topology connecting the subnets is not limited to a torus, and various types such as FatTree and mesh can be applied. A routing control protocol such as OSPF (Open Shortest Path First) is applicable as a routing control method between subnets.

  In FIG. 1A, during normal operation, when each server transfers a packet to a partner server, an NT adapter to be used for packet transmission is selected by a Link-Aggregation algorithm. That is, the NT adapter to be used for packet transmission is determined from the combination of its own IP address and the IP address of the destination server. Specifically, when a packet is transmitted from the server SV2 to the server SV1, the packet is transmitted from the NT adapter E20 and is transmitted to the server SV1 through a route of E20, SW1, and E10.

  On the other hand, when a failure occurs between E10 and SW1, the occurrence of this failure is detected by the server SV2. The communication path is changed to E21, SW2, and E11, and the packet is transmitted to the server SV1. In the above-mentioned Non-Patent Document 2, the server SV20 cannot detect the occurrence of the failure, and therefore transmits a packet from the E200 and transmits a packet using the routes E200, SW10, SW20, and E110 (see FIG. 15). In the calculation system 1, a shorter route than this can be selected.

  As shown in FIG. 1B, when a failure further occurs between E21 and SW2, the route from server SV2 to server SV1 is only the route E20, SW1, SW2, E11, so this route is used. To send packets.

FIG. 2 shows a normal operation and a route failure when a packet is transmitted from another subnet to the server SV1.
When a packet is received from another subnet, the L3 switch SW1 or SW2 performs packet routing. At this time, the L3 switch selects the shortest path possible. For example, as shown in FIG. 2, when the L3 switch SW1 receives a packet transmitted from another subnet to the server SV1, the packet is transferred using the routes SW1 and E10.

  When a failure occurs between the L3 switches SW1 and E10, packets transmitted from other subnets are transferred to the server SV1 via SW1, SW2, and E11. At this time, the L3 switch SW1 performs Layer-3 routing, rewrites the MAC address of the packet, and the NT switch SW2 operates as a Layer-2 switch.

On the other hand, when transmitting a packet to another subnet, the packet is transmitted via the L3 switch, and the computer system 1 controls the packet transmission / reception to / from the L3 switch in the same subnet. It has become.
When a failure occurs between the L3 switches SW1 and E10, routes E10, SW2, and SW1 are used.

  The L3 switches SW1 and SW2 in the same subnet have different addresses in the subnet, and it is determined by the routing function of each server SV1 or SV2 which address the packet is transferred to. This process is realized by using the function of the existing OS installed in the servers SV1 and SV2.

  Further, for example, when the server SV1 cannot reach the L3 switch SW1, a packet may be transmitted via the L3 switch SW2. In this case, it is realized by changing the routing setting of the server SV1 and changing the routing setting between the L3 switches.

  Various efficient route selections as described above are performed by route detection processing possessed by each of the servers SV1 to SVn. More specifically, each server transmits a probe packet including destination information related to communication, and receives a probe packet returned from another server, thereby managing a communicable route. It has come to be realized.

In the present embodiment, the computer system 1 will explain an example in which the processing operation as described above is realized by a general NT switch and software running on a server. Alternatively, all can be realized as hardware.
The above is the outline of route selection in the computer system 1.

Hereinafter, the server SV constituting the computer system 1 will be described in detail.
FIG. 3 shows the configuration of the server SV1. Other servers SVn have the same configuration. The server SV1 is a general-purpose server device, and includes a CPU 1, a memory 2, a communication I / F 3, a NIC 4 and 5, an STG-I / F 6, and an input / output device 7. The server SV1 is connected to the external storage device 30 so as to be communicable.

  The communication I / F 3 is connected to the NIC 4 connected to the L3 switch SW1 and the NIC 5 connected to the L3 switch SW2. The STG-I / F 6 is an I / F for communicating with the external storage apparatus of the server SV1. The external storage device stores a setting file 31 to be described later, and can be read from and updated with the server SV1 as appropriate.

  In the memory 2 of the server SV1, a virtual NIC driver 10 and a network processing unit 20 are realized by the cooperation of the CPU 1 and the program.

  In the virtual NIC driver 10, a plurality of NT adapters 4 and 5 included in the server SV1 are virtually viewed as one NT adapter. In the following description, NIC4 may be written together with eth0, and NIC5 may be written together with eth1. The virtual NIC driver 10 includes operating server information 10 for managing communication status with other servers operating in the same subnet, NT adapter information 12 for managing information on NT adapters owned by the own server, and L3 operating in the same subnet. It has router information 13 for managing the communication state with the switches SW1 and SW2, and single path information 14 for managing a range of addresses that can be taken by a server that does not support packet control by the server SV1.

  The network processing unit 20 is a Layer-3 packet control function unit provided by the OS, and includes ARP information 21 and routing information 22. The network processing unit 20 operates as a higher-level functional unit of the virtual NIC driver 10.

  FIG. 4 schematically shows a specific example of the operating server information 102. The operating server information 11 includes the IP address 111 of the server, the MAC address 112 of the NT adapter, the SW 113 indicating the adjacent L3 switch, and Time (the last time when the operation of the NT adapter (eth0) of the communication partner server was last confirmed. eth0) 114 and the last time Time (eth1) 115 at which the operation of the NT adapter (eth1) of the communication partner server is confirmed are included and managed in association with each other. One entry is registered for each NT adapter of a server in the same subnet, and the communication state with each server is managed.

  Note that the identifier of the L3 switch is unified throughout the system. In this embodiment, it is assumed that the identifier of the L3 switch SW1 is “0” and the identifier of SW2 is “1”. In addition, although the MAC address of Ethernet (registered trademark) that is generally used is 6 bytes, in this embodiment, it is described as 3 bytes for simplicity.

  FIG. 5 schematically shows a specific example of the NT adapter information 12. The NT adapter information 12 includes an NIC 121 that is an identifier of the NT adapter, an MAC address 122 of the NT adapter, and SW information 123 that is an identifier of an adjacent L3 switch, and these are managed in association with each other. A virtual NT adapter is generated from the NT adapter registered in the NT adapter information 12. Therefore, NT adapters that do not want to participate in the virtual NT adapter are not registered in the NT adapter information 12.

  In the NT adapter information 12, one entry is registered for each NT adapter of the server. FIG. 5 shows an example in which two NT adapters possessed by the server SV1 are registered.

  FIG. 6 schematically shows a specific example of the router information 13. A list of L3 switches in the subnet is registered in the router information 13, and the communication state with each L3 switch is managed. The router information 13 includes the SW address 131 that is the address of the L3 switch, the SW-MAC address that is the MAC address of the L3 switch, the SW identifier 133 that identifies the L3 switch, and between the NIC 4 (eth0) of the local server and the L3 switch SW1. Status (eth0) 134 indicating the communication state, Time (eth0) 135 indicating the last communication time between the local server NIC5 (eth0) and the L3 switch, Status indicating the communication state between the local server NIC5 (eth1) and the L3 switch (eth1) 136 and the last communication time Time (eth1) 137 between the NIC 5 (eth1) of the local server and the L3 switch are included.

  FIG. 7 schematically shows a specific example of the single path information 105. A server that has only one NIC and does not support packet control by the servers SV1 to SVn (hereinafter referred to as “unsupported server”) is connected to the L3 switch SW1 or SW2 and passes through the servers SV1 to SV1. will communicate with n. Therefore, in this embodiment, when an unsupported server is connected to the L3 switch, an address in a predetermined range is set for the unsupported server, thereby distinguishing it from the server corresponding to the packet control of the servers SV1 to SVn. become able to. For example, the first entry in FIG. 7 indicates that an unsupported server connected to the L3 switch SW1 (identifier = “0”) sets an address in the range of “192.168.1.200 to 192.168.1.219”.

  FIG. 8 shows a specific example of the ARP information 15. The ARP information 15 is information used by the OS to determine the MAC address of the communication partner from the NIC used for communication and the address of the communication partner. The ARP information 15 includes a NIC 151 indicating the NT adapter of its own server used for communication, the IP address 152 of the communication partner, and the MAC address 153 of the communication partner. In the example of FIG. 8, since 3 entries are registered and the virtual NT adapter is used from the OS, the NIC 531 of the ARP information 15 is registered with the identifier “veth0” of the virtual NT adapter. Indicates.

  FIG. 9 schematically shows a specific example of the routing information 16. The routing information 16 is used by the OS to determine the NT adapter (NIC) to be used based on the transmission destination address when transmitting the packet. The routing information 16 includes a destination address 161 indicating a packet transmission destination address or address range, a gateway address 162 that is an address of an L3 switch through which a transfer packet passes when the destination is another subnet, and a NIC used for packet transmission. NIC 163 indicating the identifier of the ID is included.

  Note that packet transmission from the OS layer is performed as follows. First, the OS refers to the routing information 16 and determines the NT adapter (NIC) to be used based on the address described in the packet. Then, based on the NT adapter (NIC) and the transmission destination address, the ARP information 15 is referred to and the transmission destination MAC address is determined. Subsequently, when the OS requests the virtual NT adapter “veth0” to transmit a packet, the virtual NIC driver 10 determines the NT adapter (NIC) that actually transfers the packet.

  The setting file 31 held in the storage device 30 is used to set initial values for the various information described above. When the system is activated, no entry is registered in the NT adapter information 12, the router information 13, and the single path information 14, and the initial value is registered by reading the setting file 31.

FIG. 10 schematically shows a specific example of the setting file 31.
The “virtual NIC and Switch” section of the setting file 31 describes the correspondence between the identifiers of the virtual NT adapters and the identifiers of the NT adapters to be associated. In the example shown in the figure, the NICs 4 (eth0) and (eth1) indicate that they correspond to the virtual NT adapter (veth0). Further, as shown in the figure, in the configuration file 31, the identifier of the proximity L3 switch is described for each NIC identifier (the portions (0) and (1) of eth0 (0) and eth (1)). ).
Note that obtaining the MAC address of the corresponding NIC from the NIC identifier is usually implemented as an OS function. Based on these pieces of information, an entry is added to the NT adapter information 12.

  In the “router” section of the configuration file 31, the address of the L3 switch, the MAC address of the L3 switch, and the identifier of the L3 switch are described. These pieces of information are set in the router information 13. Further, since the communication state with the L3 switch is unknown, “-” is set in the Status (eth0) 134 and the Status (eth1) 136 of the router information 13.

In the “single path” section of the setting file 31, a range of addresses that can be taken by a server that does not support packet control by the servers SV1 to SVn is described. Each entry includes an address range and an identifier of an L3 switch that connects an unsupported server. Based on this information, an entry is added to the single path information 14 (FIG. 7).
As will be described later, since the entry is added to the active server information 11 by receiving a probe packet from another server, the number of registered entries is “0” even when the initialization of each information is completed. Is set.

Further, based on the information described in the “virtual NIC and Switch” section and the “router” section, router information is set in advance in the ARP information 15 (FIG. 8). The information described in the “and Switch” section is used to acquire the identifier of the virtual NT adapter. As a result, the ARP request is not transmitted from the OS to the L3 switch.
The above is the configuration of the computer system 1.

Next, the operation of the virtual NIC driver 10 of the computer system 1 will be described.
First, FIGS. 11, 12, and 13 show the flow of “packet transmission processing” by the virtual NIC driver 10.
In S101, upon receiving a packet transmission request from the OS, the virtual NIC driver 10 extracts a transmission destination address from the packet, and determines whether or not the destination server is a multipath incompatible server.
That is, when the packet is an ARP request, the target IP address (Target IP Address) is the destination address, but the virtual NIC driver 10 refers to the single path 14 and the extracted destination address is the start address and end address. It is determined whether there is an entry included in the range.

  If it is determined that there is an entry, the read destination server is a server that does not support multipath control in the present embodiment, and thus the process proceeds to S125. On the other hand, if it is determined that there is no entry, the read destination server is a server that supports multipath control, and thus the process proceeds to S103.

In S103, the virtual NIC driver 10 refers to the router information 13 and confirms whether the packet transmission destination is an L3 switch. Specifically, it is determined whether or not there is an entry whose destination address included in the packet matches the address of the router information 13. If such an entry exists, the L3 switch corresponding to the entry is determined as the packet transmission destination, and the process proceeds to S105. When the packet is an ARP request, the target IP address is the transmission destination address.
Conversely, if it is determined that there is no corresponding L3 switch, the process proceeds to S109.

  In S105, the virtual NIC driver 10 determines an NT adapter to be used for packet transmission to the L3 switch ("route detection process with router"). Details of this processing will be described later with reference to FIG.

  In S107, the virtual NIC driver 10 rewrites the transmission source MAC address of the packet with the MAC address of the NT adapter determined in S105, transmits the packet, and exits this flow. When an ARP request is received from the L3 switch, it is necessary to send back an ARP response. In particular, the MAC address rewriting process plays an important role in sending back an ARP response. That is, when an ARP response including information on a route having a long route length and a low priority is returned, the L3 switch records the information in its own ARP information, and then transmits a packet to the server using the route. Because.

On the other hand, a process when the entry whose packet transmission destination address matches the address of the routing information 16 is not found in S103 will be described.
In S105, the virtual NIC driver 10 searches for an entry whose packet transmission destination address matches the address of the active server information 11, and determines in S111 whether such an entry exists. More specifically, the virtual NIC driver 10 further determines whether or not the corresponding Time (eth0) 114 or Time (eth1) 115 is within a predetermined time for the entry determined to exist.

  If the condition is satisfied, the server corresponding to the entry is determined as a packet transmission destination candidate, and the process proceeds to S113. On the other hand, if there is no entry whose packet destination address matches the address of the active server information 11, or the corresponding Time (eth0) 114 or Time (eth1) 115 is out of a certain time even if it exists, the process of S121 Proceed to

  In step S113, the virtual NIC driver 10 determines whether the packet is an ARP request. If it is an ARP request, the process proceeds to S115, and if it is not an ARP request, the process proceeds to S117.

  In S115, the virtual NIC driver 10 does not transmit the APR request packet, but generates a pseudo ARP response and returns it to the OS, and exits this flow. That is, one of the features of this embodiment is server detection using a probe packet. Therefore, there is no need to send an ARP request. However, in order for the OS to operate normally, the OS needs to behave as if it has received an ARP response. Therefore, the virtual NIC driver 10 generates a pseudo ARP response and returns it to the OS. . The ARP response includes the MAC address of the NT adapter of the communication partner server as the source MAC address.

  Since each entry of the active server information 11 is generated for each NT adapter of the communication partner server, a plurality of MAC addresses of the communication partner server exist if no failure occurs. In this case, the source MAC address of the ARP response may be any of the plurality of MAC addresses, but in order to reduce rewriting of the ARP information 15, the smallest one of the plurality of MAC addresses is adopted, and the MAC is preferably used. The address may not be changed.

On the other hand, in step S117, if the packet is not an ARP request in the determination in step S113, the virtual NIC driver 10 acquires the information registered in the entry of the operating server information 11 found in step S111 and the information described in the NT adapter information 12. Based on the above, the NT adapter to be used for packet transmission is determined.
In this case, the conditions of the route to be preferentially used include (1) a short route length, and (2) an optimal route in consideration of continuing to use the same route as much as possible when there are multiple routes with the same route length. select.

  In (1), the short path length is one of the criteria because it means that the line between the L3 switches is not used, and the network load is reduced by using this path. In (2), the reason why the same route is used as much as possible is to prevent the packet arrival order from being reversed and the communication efficiency from being lowered if the transmission route is changed each time a packet is transmitted.

  For example, when the address of the packet transmission destination is 192.168.1.11, the active server information 11 has two entries that match the address. Each entry corresponds to the NT adapter of the communication partner server. Time (eth0) and Time (eth1) in the active server information 11 are times when communication with the NT adapters eth0 and eth1 of the local server is confirmed. If this is within a certain time, it can be considered that communication is possible. In this case, the combination of the NT adapter of the communication partner server's MAC address “ab: cd: 10” and its own server eth0, eth1, and the NT adapter of the communication partner server's MAC address “ab: cd: 11” and its own server All combinations of eth0 and eth1 can communicate, and there are a total of four communication paths.

  There are two types of communication path lengths. One is the shortest route that passes through the L3 switch only once, and the other is a bypass route that passes through the L3 route twice. With reference to the SW 113 of the entry registered in the operating server information 11, an entry on the NT adapter information 12 is searched such that this value matches the SW 123 of the NT adapter information 12. The first entry of the operating server information 11 corresponds to the first entry of the NT adapter information 12. The second entry of the NT adapter information 12 corresponds to the second entry of the operating server information 11.

The NT adapter having the NIC 121 of the entry of the NT adapter information 12 thus found as an identifier constitutes the shortest path. Specifically, for the NT adapter with the MAC address “ab: cd: 10” of the communication partner server, it is eth0 of the own server, and for the NT adapter with the MAC address “ab: cd: 11” of the communication partner server. In this case, eth1 of the local server constitutes the shortest path.
The route that is actually used uses the shortest route with priority, and when there is no shortest route, the detour route is used. In this example, since there are a plurality of shortest paths, any one of the shortest paths is used. Which one to use may be determined based on the IP address or MAC address of both servers that communicate.
Moreover, although it may change dynamically according to the load of NT adapter, it is desirable to control so that a route change is not performed frequently. The same applies when there is no shortest path and there are a plurality of detour paths.

In the process of S117, the virtual NIC driver 10 uses the shortest route when there is a network failure or the like, and uses the route as shown in FIG. Will do.
In S110, the virtual NIC driver 10 rewrites the transmission source MAC address of the packet with the MAC address of the NT adapter, and transmits the packet.

Next, the migration process when no entry is found in the active server information 11 in S111 will be described.
In S121, the virtual NIC driver 10 performs “(communication partner) server detection process”. In this process, it is confirmed that communication with the communication partner server is possible by broadcasting the probe packet in the subnet. Normally, ARP uses broadcast transmission, but in this embodiment, one of the features different from normal ARP is that a communication path is specified in probe packet broadcast transmission. Details of the processing will be described later with reference to FIG.
In this embodiment, an example of broadcasting at the time of packet transmission processing is used. However, various configurations may be considered for triggers of transmission such that some or all servers broadcast periodically.

  In S123, the virtual NIC driver 10 determines whether or not communication with the packet transmission destination server is possible. If communication is not possible, the virtual NIC driver 10 returns an error to the OS, and if communication is possible, the process proceeds to S113.

Finally, a process when the destination server is a non-multipath compatible server in the determination of S101 will be described.
In S125, the NIC driver 10 refers to the single path information 14, and determines the L3 switch corresponding to the SW column 143 of the corresponding entry as a path. Here, the virtual NIC driver 10 transmits a packet via this L3 switch, and the shortest path is a path that passes through the NT adapter of its own server directly connected to this L3 switch. The entry of the NT adapter information 12 in which the SW column 113 of the NT adapter information 12 matches the value of the SW is searched, and the value of the NIC column 111 of the entry is determined as the NT adapter of the local server constituting the shortest path. .

  If the NT adapter does not fail and packet transmission is possible, the virtual NIC driver 10 transmits a packet from the NT adapter. If a failure has occurred, a packet is transmitted from another NT adapter registered in the NT adapter information 12.

In S127, the virtual NIC driver 10 rewrites the packet transmission source MAC address with the MAC address of the NT adapter determined to be used for transmission processing, and transmits the packet.
The above is the overall processing when the virtual NIC driver 10 transmits a packet.

Next, the details of the “route detection process with the router” in S103 described above will be described.
FIG. 12 shows a flow of route detection processing with the router.
In S201, the virtual NIC driver 10 refers to the router information 13 when determining the NT adapter to be used for packet transmission to the L3 switch, and reads the status and time of the NT adapter having the highest priority.

For the communication with the L3 switch, the NT adapter directly connected to the L3 switch has the highest priority. The second and subsequent priorities are in the order of numbers included in the NT adapter identifiers with the highest priority NT adapter as a reference. For example, when there are three NT adapters eth0 to eth2, and the highest priority NT adapter is eth1, the order is eth1, eth2, and eth0. By using such priorities, the usage frequency of the NT adapter can be averaged.
The NT adapter directly connected to the L3 switch is found by referring to the NT adapter information 12 and by referring to the entry having the SW identifier that matches the value of the SW column 123 obtained above. The NIC of the found entry is an NT adapter that is directly connected to the L3 switch. In the case of the router information 13 shown in FIG. 6, since the SW identifier is “0”, the NIC is eth0.

  In step S203, the virtual NIC driver 10 determines whether the time of the NT adapter that is the highest priority read in step S301 has elapsed for a certain period of time. If the period of time has elapsed, the process proceeds to step S209. If not, the process proceeds to S205.

  In step S205, the virtual NIC driver 10 determines whether the status of the NT adapter having the highest priority read in step S201 is “Online”. In the case of “Online”, the NT adapter is determined as an NT adapter to be used for packet transmission, and this flow is exited. On the contrary, if TimeStatus is “Offline”, the process proceeds to S207.

  In S207, the virtual NIC driver 10 determines whether the state is other than the state determined in S203 and S205, and if it is not any of the cases determined in S203 and S205 (S207: No), the process returns to S201. Investigate the availability of the next priority NT adapter. On the contrary, when it is not in any of the determined states of S203 and S205 (S207: Yes), the communication state is unknown, and the process proceeds to S209.

In step S209, the virtual NIC driver 10 transmits an ARP request from the NT adapter to the L3 switch. The source MAC address included in the ARP request is the MAC address of the NT adapter. The target address of the ARP request is the address of the L3 switch.
In step S211, the virtual NIC driver 10 determines whether or not the ARP response can be received from the L3 switch. If the virtual NIC driver 10 can receive the ARP response, the process proceeds to step S213. If the virtual NIC driver 10 cannot receive the ARP response, the process proceeds to step S215.

In S213, the virtual NIC driver 10 updates the status of the corresponding entry in the router information 13 to “Online” and the Time to the current time, and exits the flow.
In S215, since the virtual NIC driver 10 cannot communicate with the L3 switch, the status of the corresponding entry in the router information 13 is updated to “Offline”, the Time is updated to the current time, and the process returns to S201.
The above is the “route detection process with the router”.

Next, the “server detection process” described in S111 of FIG. 11 will be described in detail.
FIG. 13 shows the flow of the “communication partner server detection process”. For the sake of convenience, this figure shows not only the virtual NIC driver 10 of the server that has transmitted the probe packet, but also the processing of the receiving virtual NIC driver of the server that has received the probe packet.

  In S301, the transmission-side virtual NIC driver 10 broadcasts probe packets from all NT adapters registered in the NT adapter information 12. The probe packet includes the address of its own server, the MAC address of the source NT adapter, the identifier of the L3 switch to which the NT adapter is connected, and the IP address of the communication partner server.

  The identifier of the L3 switch connected to the MAC address is acquired from the NT adapter information 12. Since each probe packet is transmitted by broadcast, it is transmitted to all NT adapters of all servers in the subnet through a line between the L3 switches.

In S401, the virtual NIC driver on the receiving side receives the probe packet transmitted by broadcast.
In step S <b> 403, the receiving virtual NIC driver that has received the probe packet registers the entry content of the probe packet in the active server information 11. That is, the IP address, MAC address, and L3 switch identifier included in the received probe packet are registered in the IP address column 111, the MAC address column 112, and the SW identifier column 113 of the operating server information 11.
If there is an entry of the probe packet already received on the active server information 11, the time information in the corresponding Time (eth0) column 114 and Time (eth1) column 115 is updated. When received from the NT adapter ethi (i = 0, 1) of the local server, Time (ethi) (i = 0, 1) is updated to the current time.

  In step S405, the reception-side virtual NIC driver determines whether the IP address of the transmission source included in the probe packet is the same as the IP address of its own server. If they are the same (S405: Yes), the process proceeds to S407, and if they are different (S405: No), the flow exits.

  In S407, the receiving virtual NIC driver broadcasts a return probe packet. At this time, the reply probe packet does not include the address of the communication partner server. In this way, depending on whether or not the address of the communication partner server is included in the probe packet, the virtual NIC driver 10 that originally transmitted the probe packet transmits first whether the probe packet has been transmitted first. This is because it is possible to distinguish whether the packet is returned to the probe packet. Another object of the present invention is to prevent the packet from being transmitted indefinitely by not replying to the probe packet broadcasted as a reply.

  On the other hand, in S303, the virtual NIC driver 10 waits for a certain period of time until it detects a response to the probe packet broadcast to another server (S303: No). If it is received (S303: Yes), the process proceeds to S305, and if it is not received even after a certain period of time, an error occurs and the process is skipped.

  In step S <b> 305, the virtual NIC driver 10 performs processing for registering the IP address, MAC address, and identifier of the L3 switch included in the probe packet received as a reply from another server in the operating server information 11. . Through this process, the virtual NIC driver 10 can manage a communicable path in which no failure has occurred.

Since the returned probe packet is also broadcast, the virtual NIC driver 10 can check the availability of all network paths. Since the returned probe packet is delivered to all the servers in the subnet, the probe packet returned by other servers is read and the operating server information 11 is updated.
The above is the “server detection process”. The “server detection process” is generated at the timing of S111 (FIG. 11). However, the present invention is not limited to this example, and the process may be performed periodically.

Finally, a process when the virtual NIC driver 10 receives packet data in the computer system 1 in which a communicable path is managed by the processes shown in FIGS.
FIG. 14 shows the flow of packet data reception processing.
In step S501, the virtual NIC driver 10 determines whether the received bucket data is broadcast transmission. If it is not broadcast transmission (S501: No / for example, when the own server is designated as a specific destination, such as unicast or multicast), the process proceeds to S503, and the virtual NIC driver 10 sends the received packet to the OS. hand over.

  On the other hand, if the packet is a broadcast transmission packet in S501 (S501: Yes), the virtual NIC driver 10 proceeds to S505, and determines whether the packet is a packet received by the reception NT adapter.

  Since all NT adapters of the own server receive the broadcast packet, if all NT adapters pass the received broadcast packet to the OS, the OS assumes that the same broadcast packet has been received a plurality of times.

  Therefore, in this embodiment, NT adapters used for receiving broadcast packets are set in advance for each server, and broadcast packets received by other NT adapters are discarded. For example, among the NT adapters registered in the NT adapter information 12, only packets received by the NT adapter having the smallest SW identifier 113 among the NT adapters in which no failure has occurred are passed to the OS. .

When the virtual NIC driver 10 determines that the packet is received by the receiving NT adapter (S505: Yes), the process proceeds to S504, where the packet is passed to the OS and is received by an adapter other than the receiving NT adapter. When judging (S505: No), it progresses to S507 and a packet is discarded.
The above is the flow of processing when receiving a packet.

As described above, according to the computer system 1, it is possible to reduce the communication load of the inter-switch path and reduce the control cost of the network switch.
In addition, each server dynamically detects an available route, so that each server performs management of a route failure and an alternative route, thereby improving fault tolerance. In particular, in the case of a large-scale system using Map-Reduce, there are many communications between a plurality of servers constituting this, and management of the communication path greatly affects the processing time. In the computer system 1, since each server manages the route, the shortest available route is selected at the time of data transmission to other servers. Therefore, the processing time is longer than the method of detecting the failure by the non-connection route after data transmission. It can be expected to contribute to speeding up.

  The present invention is not limited to the various examples described above, and various configurations can be applied without departing from the spirit of the present invention.

  For example, in the computer system 1, each server has network adapters for the number of L3 switches, and each server is connected to each L3 switch. However, all servers have network adapters for the number of L3 switches. There is no need to prepare. That is, a server with a small number of network adapters is always considered to have failed, but there is no problem in system operation.

  In the computer system 1, the number of L3 switches in the subnet is two, but the number of L3 switches may be more than this. At this time, the L3 switches in the subnet are preferably connected to each other in L2.

  In the computer system 1, each server is directly connected to the L3 switch. However, the L3 switch and the server may be connected via another L2 switch. That is, even in such a configuration, because the network topology of Layer 3 is the same, the server can uniquely distinguish the closest L3 switch from the other L3 switches.

  Needless to say, the software constituting each functional unit of the computer system 1 can be recorded on a non-temporary recording medium using various electric and magnetic methods. Of course, it is also possible to download and install these software on a server or the like via a network.

SV1 to SVn ... server, SW1, SW2 ... NT switch, 4/5 ... NIC, 10 ... virtual NIC driver, 11 ... operating server information, 12 ... NT adapter information, 13 ... Router information, 14 ... Single route information, 20 ... Network processing unit, 21 ... ARP information, 22 ... Routing information, 31 ... Configuration file

Claims (9)

At least first and second network switches communicatively connected;
A plurality of computers having at least a first network adapter that communicates with a network adapter of another computer via the first network switch and a second network adapter that communicates with a network adapter of another computer via the second network switch; A computer system comprising:
The plurality of computers are:
A storage unit for holding communication path information between the first and second network adapters of the other computer and the first and second network adapters of the own computer;
A computer system comprising: a control unit that preferentially determines a communication path without communication between the first and second network switches as a communication path with the other computer from the path information. The computer system according to claim 1,
The plurality of computers is a computer system that generates the route information from the content of a response of broadcast transmission to the other computers. The computer system according to claim 2,
Broadcast transmission to the other computer is transmitted from each of the first and second network adapters, and includes an address of the own computer, a MAC address of the network adapter, and identification information of a network switch to which the network adapter is connected,
The response to the broadcast transmission is transmitted from each of the first and second network adapters of the other computer, and the MAC address of the first and second network adapters of the other computer and the identification information of the connected network switch are obtained. Including computer system. The computer system according to claim 3,
The other computer records the address of the own computer, the MAC address of the own network adapter, and the identification information of the connection destination network switch of the own network adapter included in the broadcast transmission in the storage unit as route information in the other computer. Computer system to do. The computer system according to claim 3,
The broadcast transmission to the other computer further includes a destination computer address,
The other computer that has received the broadcast transmission is:
A computer system that transmits the response when the computer address of the transmission destination is the address of its own computer. The computer system according to claim 5,
The other computer is a computer system that broadcasts the response. The computer system according to claim 1,
The first and second network switches are L3 (Layer 3) switches,
The control unit further transmits an ARP request from the first network adapter to the first network switch,
By receiving a response to the ARP, it is determined whether or not the communication path of the first network adapter can be used,
When the ARP request is not available, an ARP request is transmitted from the second network adapter to the second network switch, and the second network adapter communicates via the second network switch by receiving a response to the ARP. Decide the route to use,
When the second network adapter receives an ARP request from the second network switch,
A computer system that responds to the second network switch with an ARP response in which the MAC address of the network adapter of the local server included in the determined route is described in a transmission source MAC address. At least first and second network switches communicatively connected;
A plurality of computers having at least a first network adapter that communicates with a network adapter of another computer via the first network switch and a second network adapter that communicates with a network adapter of another computer via the second network switch; A communication path management method for a computer system comprising:
The plurality of computers are
Storing communicable path information between the first and second network adapters of the other computer and the first and second network adapters of the own computer;
A communication path management method including preferentially determining, from the path information, a communication path without communication between the first and second network switches as a communication path with the other computer. At least first and second network switches communicatively connected;
A plurality of computers having at least a first network adapter that communicates with a network adapter of another computer via the first network switch and a second network adapter that communicates with a network adapter of another computer via the second network switch; In the computer system comprising:
A procedure for storing path information communicable between the first and second network adapters of the other computer and the first and second network adapters of the own computer;
A program for executing, from the path information, a procedure for preferentially determining a communication path without communication between the first and second network switches as a communication path with the other computer.

Images