JP2008165531A - Method for failover (restoration) of defective node in computer system having a plurality of nodes - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for restoration of a defective node in a torus type computer system. <P>SOLUTION: When one of calculation nodes is broken during execution of calculation in a computer system including a torus network and a tree network including a plurality of IO nodes, each of the calculation nodes forming a link with a terminal IO node of the tree network, an IO node which forms a link with the defective node is reported as an alternate node specified by a one-dimension increased address of the address of the defective node to a node adjacent to the defective node. When the adjacent node receives a packet for the defective node, the packet is routed to the alternate node, and a job designated by the packet which reached the alternate node is processed in the alternate node. The alternate node confirms an address of calculation node to which a packet including the processing result of the job is transmitted, and transmits the packet from the calculation node connected to the alternate node to a calculation node closest to the address. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

The present invention relates to a computer system having a plurality of nodes. In particular, the present invention relates to a method for repairing a fault node in a torus network (so-called failover).
Here, “failover” refers to a function of continuing processing to an alternative node when a failure occurs in a node.

  Conventionally, a torus network (hereinafter also referred to as “TORUS”) is used in a large-scale parallel computing system (computer system). The torus network connects communication nodes arranged adjacent to each other when communication nodes (hereinafter also referred to as “calculation nodes” and “IO nodes”) are arranged at each lattice point in a three-dimensional space having a certain three-dimensional shape. Communication network. Most preferably, the torus network is a square if it is a two-dimensional torus and a cube if it is a three-dimensional torus.

  FIG. 1 shows the overall configuration of the computer system 10 and the information processing apparatus 20. The computer system 10 has a plurality of communication nodes (in the example described later, the communication nodes are arranged at three-dimensional lattice points (x, y, z)). The computer system 10 executes a program for numerical calculation, for example, in each of the plurality of communication nodes. The information processing apparatus 20 transmits a program execution request to each communication node in the computer system 10. In this execution request, not only the processing contents to be executed but also the execution result received from any other communication node is used to execute the program, or the execution result is executed to any other communication node. Includes instructions on what to do. That is, the computer system 10 executes the program in parallel in response to a request from the information processing apparatus 20 and returns the execution result to the information processing apparatus 20. As a result, the program can be executed more efficiently than when the program is executed by a single communication node.

  FIG. 2 shows the components of the torus network of computer system 10. The computer system 10 includes a communication node (hereinafter referred to as “calculation node” and “torus node”) 12 and a communication link (hereinafter referred to as “link”) 13. Each of the communication nodes 12 executes a program in parallel with each of the other communication nodes. Each of the communication nodes 12 is typically a processor (CPU, MPU or central processing unit). Each of the communication nodes 12 may be a storage device such as a DRAM, or a processor may be provided at the same time.

  The information processing apparatus 20 includes a CPU and a hard disk. A conventional computer system 10 having a plurality of nodes includes a torus network 10 composed of a plurality of computing nodes 12 and one IO node. In addition to the torus network link 13 (FIG. 2), each computation node constituting the main part of the computer system is connected to the tree network link 15 (connection relation of the computation nodes 12 in FIG. 3) and the top one. The IO node 14 forms a tree network link. The computer system 10 is connected to the information processing apparatus 20 by this tree network.

Since the torus network is connected only to the adjacent computing nodes (nodes between the nearest grid points), the routing overhead at each node is small and the configuration is simple, thus realizing a hardware system. Is easy. Since the network itself is scalable, it is often used in Massive Parallel computer systems such as IBM BlueGene / L. However, in the torus network, since it is connected only to an adjacent calculation node, when one calculation node fails, it is difficult to have an alternative node for that node.

In general, in a system that considers redundancy, when a node fails, a node that replaces the node is assigned. Subsequent processing is performed at the alternative node instead of the failed node. FIG. 4 shows a case where one of the nodes arranged at the two-dimensional lattice point position (x, y) fails. Since the torus network itself is connected only to adjacent nodes, it is not possible to place an alternative node at the logically same three-dimensional lattice point position (x, y, z) of the torus. Even if an alternative node cannot be assigned or can be assigned, routing to the alternative node becomes very complicated. Therefore, the overhead is large and the performance is significantly reduced. Such a problem becomes conspicuous when a replacement node is provided when a torus node (calculation node) arranged at a three-dimensional lattice point fails.

In order to solve this problem, the IBM parallel computer system is operated as follows. For example, IBM BlueGene / L is a large-scale integrated system having a plurality of nodes. In a parallel computer system having a large number of nodes, it is hardware that can be expanded into a scalable node. However, if the number of nodes increases, the probability that a failure will occur increases. When a specific node fails, the power is turned off, the node is replaced, and then the calculation is restarted from the checkpoint written (backed up) on the hard disk (HDD). As the number of nodes increases, the failure rate increases, which causes a problem (problem) that greatly reduces the throughput of the entire system.

Patent Document 1 provides a method for reconstructing a parallel network when a hardware failure occurs in a multiprocessor parallel network. This method is intended to solve the problem of recovering a network in which a hardware failure has occurred in a multiprocessor parallel network. In a parallel computer system including a large number of nodes, a group including a failed processor is replaced with a group including a redundant processor so that a hardware failure can be recovered. For this purpose, Patent Document 1 uses a switch module to dynamically divide the torus without reconnecting. For example, when an error occurs in one node of a 4 × 4 × 4 3D torus, it is divided into 1 × 4 × 4 and 3 × 4 × 4 (so that a failed node is included in 1 × 4 × 4), This is a method of redoing the calculation with 3 × 4 × 4. This method changes the number of nodes. In this method, 5 × 4 × 4 is divided into 1 × 4 × 4 + 4 × 4 × 4 from the beginning, and if an error occurs, the failure node is subdivided into 1 × 4 × 4. . In these methods, the failure node in the middle of the calculation execution is recovered in the parallel network, and the calculation up to the middle is wasted.

Patent Publication No. 2004-532447

As described above, when a hardware failure such as a node failure occurs in a parallel system having a plurality of nodes, it does not solve the waste of computation being executed. In addition, since it is necessary to significantly change the configuration of the existing torus network, the performance of calculation execution cannot be improved. In particular, in fields such as scientific and engineering calculations and financial engineering calculated over a long period of time, if it is desired to continuously obtain a calculation history in a non-stop manner, there is a great loss for users and system operators.

Accordingly, an object of the present invention is to provide a torus network (computer system) that can solve the above-described problems.
It is another object of the present invention to provide a method of repairing (failing over) a failed node in a computer system (torus network) having a plurality of nodes that can solve the above-described problems.

For this purpose, the present invention provides a torus network composed of a plurality of calculation nodes arranged at a three-dimensional lattice point (address) and forming a link between adjacent lattice points, and a tree network composed of a plurality of IO nodes. Each of the compute nodes forms a link with the IO node at the end of the tree network and is a method of failing over if one compute node fails during computation execution in a computer system. This method includes a step of detecting a failure calculation node during execution of the calculation, and the IO node linked to the failure calculation node (failure node) is identified by an address that is one-dimensionally increased to the address of the failure node. And a step of routing the packet to the alternative node when a computation node (adjacent node) adjacent to the failed node receives a packet addressed to the failed node. To do.
Further, in this method, the plurality of calculation nodes of the computer system are a × b × c arrays connected as a three-dimensional torus, and each of the calculation nodes is connected to adjacent calculation nodes by + and −. Form 6 links in x, y, z direction,
The terminal IO node of the computer system forms a link with a predetermined number of compute nodes in an a × b array of z-planes of the three-dimensional torus, and the compute node has a total of seven links. It is characterized by having.
Further, in this method, the step of using the IO node as an alternative node includes an address (x, y, z) of the alternative node using the IO node forming a link with the failed node (x, y, z) as an alternative node. 1) including the step of notifying a calculation node adjacent to the failed node.
The method further includes the step of processing, in the alternative node, a job designated by the packet that has reached the alternative node.
In this method, the alternative node confirms an address of a calculation node to which a packet including the processing result of the job is sent, and sends the packet from a calculation node connected to the alternative node to a calculation node closest to the address. The method further comprises the step of sending.
Further, in this method, the routing step is a step of sending the packet to the alternative node when the adjacent computing node is connected to the alternative node.
Further, in this method, when the adjacent node is connected to another IO node different from the alternative node, the routing step sends the packet that has reached the adjacent node to the other IO node, and -It is the step which sends to the said alternative node via a network, It is characterized by the above-mentioned.
In this method, the calculation node and the IO node include at least one CPU and a memory.

For this purpose, the present invention provides a torus network composed of a plurality of calculation nodes arranged at a three-dimensional lattice point (address) and forming a link between adjacent lattice points, and a tree network composed of a plurality of IO nodes. And each of the compute nodes forms a link with an IO node at the end of the tree network when a failure occurs in one of the compute nodes during the computation in the computer system (a ) A program for failing over. This program is stored in the computer
(B) detecting a failure calculation node (failure node);
(C) Informing the node (adjacent node) adjacent to the failed node of the IO node forming the link with the failed node as an alternative node specified by an address that is one-dimensionally increased to the address of the failed node When,
(D) when the neighboring node receives a packet addressed to the failed node, routing the packet to the alternative node;
(G) processing the job designated by the packet reaching the alternative node at the alternative node;
(H) the alternative node confirms an address of a calculation node that sends a packet including the processing result of the job, and sends the packet from a calculation node connected to the alternative node to a calculation node closest to the address; ,
Is executed.

For this purpose, the present invention comprises a torus network composed of a plurality of calculation nodes arranged at three-dimensional lattice points and forming links between adjacent lattice points, and a tree network composed of a plurality of IO nodes, Each of the calculation nodes forms a link with the IO node at the end of the tree network, and further, when the calculation node fails during the execution of the calculation, the IO node forming the link with the failure node , A computer system comprising: means for making the alternative node specified by an address one-dimensionally increased to the address of the failed node as an alternative node.
Further, in this computer system, when the adjacent computing node receives a packet addressed to the failed node, the computer node routes to the alternative node, and the alternative node indicates the processing result of the job specified by the packet. Means for confirming an address to which a packet is sent, selecting a calculation node having an address closest to a destination from addresses of a plurality of calculation nodes connected to the alternative node, and sending the packet to the calculation node of the address;
It is further provided with the feature.

According to the present invention, it is possible to assign an alternative node at the time of failure without changing the torus configuration of a computer system including a plurality of calculation nodes.
Further, according to the present invention, since the configuration of the torus is not substantially changed, the change of the torus type computer system can be minimized.
In addition, according to the present invention, a calculation execution job (checkpoint) until a fault node is detected in a torus type computer system can be recovered in a short time, and subsequent calculations can be resumed from the checkpoint.

  Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments (examples) do not limit the invention according to the scope of claims, and the features described in the embodiments are described below. Not all combinations are essential for the solution of the invention.

The following method makes it possible to assign an alternative node at the time of failure without substantially changing the configuration of a plurality of nodes of the three-dimensional lattice point (x, y, z) of the torus.
1. At least one link (connection) is added to each node (referred to as a “computation node”) constituting the torus network.
2. One added link is connected to an IO node outside the torus network. The conversion rule described below is applied with an IO node forming a link with a failed calculation node (failed node) as an alternative node of the failed node.
3. The IO node 14 outside the torus network is connected to the computing nodes 12 on the plurality of torus networks in a star shape. In FIG. 3, IO nodes outside the torus network are connected in a tree shape.
4). Routing from a torus network compute node to an IO node outside the torus is substantially routed in the torus network as detailed below. This routing method is a characteristic content of the present invention. This routing method can minimize the change in the configuration of the torus node (compute node) of the existing parallel network system. In other words, in this method, the configuration of the existing torus node of the computer system in which the failed node occurs is maintained in a pseudo manner, and failover (replacement of the failed node) is automatically performed.

FIG. 5 shows an embodiment of the hardware configuration of a computer system including a three-dimensional torus according to the present invention. In the case of a three-dimensional torus, each node (x, y, z) has six links because it has links (link: meaning having a connection relationship) in the positive and negative directions of each axis. The three-dimensional torus is connected to the information processing apparatus 20 (FIG. 3) via a tree network composed of a plurality of IO nodes 14.

As shown in FIG. 3, in the conventional IBM computer system, the tree network is composed of torus nodes (calculation nodes) 12 other than the highest-level IO node 14. On the other hand, in the computer system according to the present invention, a plurality of torus nodes 12 (calculation nodes) are not used together, and a tree network is configured by only a plurality of IO nodes 14.

In this embodiment, since one link is added to the calculation node 12, each calculation node has seven links. Six of the seven links of each computation node 12 are connected to a torus node (adjacent computation node 12) as before. The added one connects a torus node (calculation node) 12 to an IO node 14 at the end of the tree network as shown in FIG. The terminal IO node functions as alternate nodes S1, S2, S3, and S4 (see FIG. 6). When the IO node 14 is the same hardware as the computation node 12 on the torus, seven links may be provided as shown in FIGS. 5 and 6 from the viewpoint of design efficiency. Further, like the IO node of IBM BlueGene / L, the IO node has a larger capacity memory than the calculation node on the torus. Of the seven links of the IO nodes connected to the nodes on the torus, six are connected to the nodes on the torus and the other one is connected to the terminal IO node. As shown in FIG. 5, seven IO nodes 14 that are not connected to nodes (calculation nodes) on the torus are connected to other IO nodes, and the IO nodes 14 constitute a tree network structure. Yes. Each calculation node 12 may form a link with two or more (multiple) terminal IO nodes 14.

FIG. 6 is a diagram in which a three-dimensional torus is cut out on a plane where z = 2. A method of assigning a terminal IO node (alternative node) to a failed computation node will be described with reference to FIG. In addition, a method for routing (routing) a packet destined for the failed node to a substitute node and a method for routing a job processing result from the substitute node will be described in the following sequence.

In the case of normal operation (there is no failure node), all the processes are closed in a plurality of calculation nodes arranged at the three-dimensional lattice points (x, y, z) constituting the torus network. The IO node and this tree network are not used for normal processing. However, in existing large-scale parallel network systems such as IBM BlueGene / L, the IO nodes that make up the tree network are used to send the processing results of each computation node to the information processing device 20 (FIG. CPU and HDD). The

The IO node has the same number of virtual torus addresses as the torus node (calculation node) connected to the IO node. Referring to FIG. 6, when the IO node S2 is an alternative node, the virtual torus address (calculation node) possessed by this node is one-dimensionally added to the computation node (x, y, z) that forms a link with this node. (X, y, z, 1). For example, the alternative node S2 in FIG. 6 has six addresses (6, 5, 2, 1) (6, 6, 2, 1) (6, 7, 2, 1) (6, 8, 2, 1). The (6, 9, 2, 1) (6, 10, 2, 1) computation node is substituted. In this way, the terminal IO node S2 has a plurality of virtual torus addresses, so that routing in the event of a node failure can be performed smoothly.

The computer system of the present invention backs up the information of the processing results of these six calculation nodes to the storage unit (see FIG. 7) of the alternative node S2 at regular intervals during execution of a predetermined calculation. In a conventional torus type computer system, information on the processing result of each calculation node during execution of a predetermined calculation is backed up to the HDD (see FIGS. 1 and 3) of the information processing apparatus 20 via the IO node. . The information on the processing results of each backed up computation node is used to identify a checkpoint where to start the subsequent processing of a given computation after the hardware failure is repaired if a hardware failure occurs in the system. used.

Even during normal operation of the conventional IBM BlueGene / L, each node periodically sends the minimum information necessary for resuming processing to the IO node to which it is connected in preparation for a system failure. . The terminal IO stores information transmitted from each torus node (calculation node) in its own memory. In the conventional system, the minimum information necessary for resuming the processing is written in the HDD. It was necessary to interrupt the processing until all node information was written to the HDD. Then, it is determined from the information stored in the HDD the check point at which the next processing can be started. After replacing the failed node, the execution of all the calculation nodes is started from the check point. A torus type parallel computer system has a torus network composed of a plurality of computation nodes arranged at three-dimensional lattice points, and a tree network composed of these computation nodes and one IO node. As shown in FIG. 3, this tree network is formed by a tree network link comprising a plurality of computation nodes 12 and one dedicated IO node 14 at the highest level (FIG. 3 of the tree configuration). .

In the present invention, the time until writing to the memory of the alternative node (terminal IO node) is merely interrupted, so the time can be greatly reduced. The IO nodes 14 constituting the tree network can write the information of each calculation node (torus node) to the HDD in a longer cycle. In addition, since the IO node 14 can write to the HDD asynchronously with the normal processing, it is not necessary to stop the entire processing during that time.

In FIG. 6, there are adjacent nodes (nearest nodes) above and below the failure node in the z-axis direction. Here, for simplicity of explanation, the adjacent nodes in the z-axis direction are omitted.
1. Assume that the node (6, 7, 2) has failed.
2. When the system detects the failure of the node of (6, 7, 2), the IO node S2 is determined to be an alternative node of (6, 7, 2), and the adjacent node (6, 6, 2) (6, 6) of the failed node. 8,2) (7,7,2) (5,7,2) and store it.
3. Finally, the process returns to the checkpoint written in the terminal IO node (alternative node) and the process is resumed.
4). Normal operation is the same except for the failed node and its neighboring nodes.
5. The packet sent to the failed node reaches the adjacent node by normal routing.
6). When the adjacent node receives a packet addressed to the failed node (6, 7, 2), the address is increased by one dimension, and routing is performed as the alternative node S2 (6, 7, 2, 1).
7). The packet addressed to the alternative node (6, 7, 2, 1) is sent to the seventh link from the adjacent node according to the normal torus routing, and reaches the alternative node S2. The terminal IO node S2 addressed to the alternative node (6, 7, 2, 1) is connected to the failed node by a seventh link, either directly or via another IO node. The IO node S2 forms the seventh link of the calculation node (adjacent node) adjacent to the failed node. At this time, since the alternative node has a plurality of torus addresses, it is possible to send a packet to the IO node S2 connected in a star shape as if it were connected to the torus.
8). At this time, in the case of a node adjacent to the failed node (adjacent node) ((6, 8, 2) (6, 6, 2)) that forms a direct link with the same alternate node S2 as the failed node, direct replacement A packet is sent to the node S2.
9. In the case of an adjacent node ((5,7,2) (7,7,2)) connected to the IO node S1 or S3 at the other end from the failed node, the packet cannot be sent directly to the alternative node. Do the following routing.
(9-1). A packet addressed to (6, 7, 2) reaching (5, 7, 2) or (7, 7, 2) is sent to S1 or S3 as a packet addressed to (6, 7, 2, 1).
(9-2). The packet sent to S1 or S3 is sent via the tree network (S5) to S2: (6, 7, 2, 1) which is an alternative node of the failed node.
10. A packet that reaches the alternative node is processed by the alternative node.
11. The alternative node S2 confirms the destination address of the packet including the processing result, selects the address closest to the destination address from the six addresses of the torus node (calculation node) connected to S2, and sends the packet to that address. send. When the destination address is (5, 9, 2), S2 is linked to itself (6, 10, 2), (6, 9, 2), (6, 8, 2), (6, 6 , 2), (6, 5, 2) is selected from (6, 9, 2) and a packet is sent.
12 The torus node (calculation node) that receives the packet from the alternative node S2 processes the packet by normal routing.
By using the routing method to an alternative node according to the present invention, it is possible to realize a failover (FailOver) of a node on a torus network, which has not been possible in the past or has been very difficult and practically difficult.

  FIG. 7 shows a functional configuration of the calculation node 12 and the IO node 14 (also referred to as a node). The communication node (the calculation node 12 and the IO node 14) includes a storage unit 300, a reception unit 310, a selection unit 320, and a transmission unit 330. The storage unit 300 stores information indicating the topology of communication paths from the communication node 12 via the torus network 13 and the tree network 14 to other links and communication nodes. A more detailed example is shown in FIG.

FIG. 8 shows an exemplary data structure of the storage unit 300. The storage unit 300 stores a conversion rule for converting a node when a node of a packet is a destination (lattice point) and an optimal node destination is selected from six adjacent nodes.

  The storage unit 300 of the node adjacent to the failed node (x, y, z) receives (x, y, z) when the adjacent node receives a packet destined for the failed node (x, y, z). Is applied to the address (x, y, z, 1) which is one-dimensionally increased. This address specifies the IO node that forms a link with the failed node. In the present invention, this IO node is assigned as a substitute node for the failed node. As a conversion rule of the failure node (x, y, z), in order to indicate an IO node that directly forms a link with the failure node (x, y, z), it is expressed in four dimensions (x, y, z, 1).

  As a specific example, FIG. 8 shows that when a failure node (6, 7, 2) exists, a conversion table that indicates that the IO node S2 is assigned as an alternative node (see FIG. 6) is adjacent to the failure node. To the calculation nodes (6, 8, 2) (6, 6, 2) (7, 7, 2) (5, 7, 2) to be stored, and the respective storage units 300 hold the conversion tables instructing S2.

  In addition, the conversion rule of this figure is an example, and various variations can be considered for the data structure of the memory | storage part 300. FIG. For example, the storage unit 300 may store a conversion rule only for coordinates that need to be converted. Further, if the IO node directly linked to the node (x, y, z) can be indicated, the four-dimensional expression is not limited. When the adjacent node receives a packet addressed to the failed node (x, y, z), a flag indicating whether the node of (x, y, z) is normal (0) or failed (1) is stored in the storage unit 300. It may be provided. When the adjacent node receives a packet package destined for the node (x, y, z), and the flag is 1 (on), it is determined that the node (x, y, z) is faulty. When a calculation node adjacent to the failed node receives a packet addressed to the failed node, the IO node S2 linked to the failed node transfers the packet as an alternative node.

  Returning to FIG. The receiving unit 310 receives the communication packet in association with the destination of the communication packet. The receiving unit 310 receives coordinate values (x, y, z) when the destination node is arranged in the three-dimensional lattice space as the destination of the communication packet. For example, if one adjacent node (6, 8, 2) is a communication packet destined for the node (6, 7, 2), the receiving unit 310 receives the coordinates (6, 7, 2, 1). . Based on the conversion rule stored in the storage unit 300, the selection unit 320 selects a transfer destination node that transfers the communication packet next to the node on the communication path to the received destination.

FIG. 9 is a flowchart of packet routing when a failure node occurs during the execution of various large-scale calculations such as science and technology, using the above conversion rule. Typically, a computer network having a torus network having a plurality of computing nodes and a plurality of IO nodes shown in FIG. 6 in a tree network, each computing node having a terminal IO node and at least one link. Consider the system case. An operation of routing the failure calculation node (6, 7, 2) shown in FIG. 6 as an alternative node by the IO node S2 will be described.
(A) One computer node (6, 7, 2) has a failure while the computer system is executing a calculation.
(B) The information processing apparatus 20 (FIG. 1) or the monitoring system resident in the apparatus detects the failure node (6, 7, 2). The monitoring system can detect through a tree network constructed by IO nodes.
(C) The monitoring system notifies the adjacent node that the IO node S2 is an alternative node.
The monitoring system converts the conversion node (failed node (6, 7, 2) → alternate node (6, 7, 2, 1) into four neighboring nodes (6, 8, 2), (through the IO node S2 of the tree network). 6, 6, 2) and stored in the respective storage units 300. Further, this conversion rule is transmitted to two adjacent nodes (7, 6, 2) or (5) via S 2 → S 5 → S 3 or S 1. 7, 7, 2) and stored in the respective storage units Note that routing between the IO nodes 14 (S 1, S 2, S 3,...) Constituting the tree network 15 is performed by each IO node. Pre-set in hardware.
(D) Upon receiving a packet addressed to the failed node (x, y, z), the six adjacent nodes are divided into the case of routing (e) or (d) according to the conversion measurement to the alternative node.
(E) In the case of the adjacent node (6, 8, 2) (6, 6, 2) connected to the alternative node S2 (6, 7, 2, 1): The packet is directly sent to the alternative node S2.
(F) In the case of the adjacent node (7, 7, 2) connected to the alternative node S3 different from the alternative node S2: the failed node (6, 7, 2) that has reached the adjacent node (7, 7, 2) The addressed packet is sent to S3 as a packet addressed to the alternative node S2 (6, 7, 2, 1). The packet sent to S3 is finally sent to the alternative node S2 (6, 7, 2, 1) via the tree network. (G) The alternative node S2 (6, 7, 2, 1) is reached. The packet is processed in S2.
(H) The alternative node S2 confirms the address to be sent as a new packet of the processing result of the job specified by the packet.
For example, when the packet is addressed to the calculation node (5, 9, 2), the calculation node (6, 9, 2) closest to the destination is selected from the addresses of the six torus nodes connected to S2, and the calculation is performed. A packet is sent to the torus node (5, 9, 2) via the node for the purpose.

As described above, according to the present embodiment and the modification, it is possible to assign an alternative node at the time of failure without changing the torus configuration of the computer system including a plurality of calculation nodes.
In addition, even if a node fails, processing can be continued by automatically assigning an alternative node with minimum overhead, minimizing the impact on the throughput of a parallel network system with multiple nodes. It becomes possible.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

1 shows the overall configuration of a computer system 10 and an information processing apparatus 20. 2 shows components of a torus network of computer system 10. It shows that IO nodes outside the torus network are connected in a tree shape. A case where one of the nodes arranged at the two-dimensional lattice point position (x, y) fails is shown. The three-dimensional torus of the present invention has the following hardware configuration. FIG. 3 shows a cutaway view of the three-dimensional torus network of the present invention in the z = 2 plane. A functional configuration of the calculation node and the IO node 12 is shown. An example of the data structure of the memory | storage part 300 is shown. The flowchart which fails over a failure node by the routing method of the packet addressed to the failure node which occurred in the middle of the calculation item using the conversion rule is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Computer system 20 Information processing apparatus 12 Computation node 13 Torus network link 14 IO node 15 Tree network link 300 Storage part 310 Reception part 320 Selection part 330 Transmission part

Claims (15)

A torus network composed of a plurality of calculation nodes arranged at three-dimensional lattice points (addresses) and forming links between adjacent lattice points;
In a computer system having a tree network composed of a plurality of IO nodes and each of the calculation nodes forming a link with the IO node at the end of the tree network, one calculation node fails during the execution of the calculation. It is a way to fail over when
Detecting a faulty compute node during the computation; and
The IO node linked to the failure calculation node (failure node) as an alternative node specified by an address that is one-dimensionally increased to the address of the failure node;
When a computing node (adjacent node) adjacent to the failed node receives a packet addressed to the failed node, routing the packet to the alternative node;
A fail over method comprising: The plurality of computing nodes of the computer system is an a × b × c array connected as a three-dimensional torus, each of the computing nodes to an adjacent computing node in the x, y, and z directions Form 6 links to
The terminal IO node of the computer system forms a link with a predetermined number of compute nodes in an a × b array of z planes of the three-dimensional torus;
The method of claim 2, wherein the compute node has a total of seven links. The step of using the IO node as an alternative node includes:
Informing the computation node adjacent to the failed node of the address (x, y, z, 1) of the alternative node with the IO node forming a link with the failed node (x, y, z) as the alternative node The method according to claim 1 or 2, comprising.   The method according to claim 3, further comprising: processing at the alternative node a job specified by the packet that has reached the alternative node. The alternative node further comprises a step of confirming an address of a calculation node that sends a packet including the processing result of the job, and sending the packet from a calculation node connected to the alternative node to a calculation node closest to the address. The method of claim 4.   The method according to claim 4, wherein the routing step is a step of sending the packet to the alternative node when the adjacent computing node is connected to the alternative node.   When the adjacent node is connected to another IO node other than the alternative node, the routing step sends the packet that has reached the adjacent node to the other IO node, and passes through the tree network. The method of claim 4, wherein the method is the step of sending to the alternative node.   The method of claim 1, wherein the compute node and the IO node include at least one CPU and memory. A torus network composed of a plurality of calculation nodes arranged at three-dimensional lattice points (addresses) and forming links between adjacent lattice points;
Each of the computing nodes in a computer system having a tree network comprising a plurality of IO nodes and forming a link with an IO node at the end of the tree network during execution of the computation. (A) the computer to fail over when a failure occurs
(B) detecting a failure calculation node (failure node);
(C) Informing the node (adjacent node) adjacent to the failed node of the IO node forming the link with the failed node as an alternative node specified by an address that is one-dimensionally increased to the address of the failed node When,
(D) when the neighboring node receives a packet addressed to the failed node, routing the packet to the alternative node;
(G) processing the job designated by the packet reaching the alternative node at the alternative node;
(H) the alternative node confirms an address of a calculation node that sends a packet including the processing result of the job, and sends the packet from a calculation node connected to the alternative node to a calculation node closest to the address; ,
A program that executes The program according to claim 9, wherein the routing step is a step of sending the packet to the alternative node when the adjacent computing node is connected to the alternative node. When the adjacent node is connected to another IO node other than the alternative node, the routing step sends the packet that has reached the adjacent node to the other IO node, and passes through the tree network. The program according to claim 9, which is a step of sending to the alternative node. A torus network composed of a plurality of calculation nodes arranged at three-dimensional lattice points and forming links between adjacent lattice points;
With a tree network consisting of multiple IO nodes,
Each of the compute nodes forms a link with the IO node at the end of the tree network;
Further, when the calculation node fails during the execution of the calculation, the replacement node specified by the address that is one-dimensionally added to the address of the failure node is used as the replacement node for the IO node that forms the link with the failure node. And means to
A computer system with The adjacent computing node, upon receiving a packet addressed to the failed node, routes to the alternative node;
The alternative node confirms an address to which the packet of the processing result of the job specified by the packet is sent, and selects a calculation node having an address closest to the destination from the addresses of a plurality of calculation nodes connected to the alternative node. Means for sending and sending the packet to the computing node of the address;
The computer system of claim 12, further comprising:   14. The computer system according to claim 13, wherein the routing means sends the packet to the alternative node when the adjacent computing node is connected to the alternative node.   When the adjacent node is connected to another IO node other than the alternative node, the routing means sends the packet that has reached the adjacent node to the other IO node, and passes through the tree network. The computer system of claim 13, wherein the computer system sends to the alternative node.

Images