JP5435205B2 - Multi-node system, node, memory dump processing method, and program - Google Patents

Description

  The present invention relates to a multi-node system that executes a memory dump when a failure occurs in a node, a node used therefor, a memory dump processing method using these, and a program for realizing them.

  Generally, an OS (Operating System) of a computer system executes kernel panic processing when normal processing cannot be continued due to hardware or software abnormality. Examples of the kernel panic process include a mere stop of the process in the computer system, a memory dump process in which the contents of the main memory at the time of failure occurrence are stored in an internal disk, and a restart of the computer system. Among these, from the viewpoint of investigating the cause of the failure, a memory dump process (see, for example, Patent Documents 1 to 3) for collecting a memory dump is effective.

  For example, Patent Document 1 discloses a memory dump process in a distributed memory type computer. In the memory dump process disclosed in Patent Document 1, a processor (node) having a fault requests a system management processor (node) to collect memory contents, and the collected memory contents are sent to the system management processor (node). Saved.

  Similarly to Patent Document 1, Patent Document 2 discloses a memory dump process in a distributed memory type computer. However, in the process disclosed in Patent Document 2, unlike the process disclosed in Patent Document 1, the node in which the failure has occurred secures an access path between it and the storage device. For this reason, according to Patent Document 2, the reliability of the memory dump is achieved.

  Patent Document 3 discloses memory dump processing in a client / server system. In the processing disclosed in Patent Document 3, a memory dump is executed from a host computer (operator terminal) to a faulty terminal. Specifically, the host computer transfers a memory dump program for collecting the memory contents to the terminal where the failure has occurred, and causes the memory dump to be performed by executing this program. Then, the host computer restarts the faulty terminal, and then causes the faulty terminal to transfer the memory contents collected by the faulty terminal.

Japanese Patent Laid-Open No. 11-306051 JP 2007-334668 A JP 7-146807 A

  By the way, in the memory dump process disclosed in Patent Document 1 or Patent Document 2, the node in which the failure occurs executes the memory transfer process. For this reason, if the data required for the memory transfer process is destroyed due to a failure, a kernel double panic may occur. In addition, since the memory contents are collected by a function specific to the node, the memory dump cannot be performed while the node itself cannot be used due to a failure, and the memory dump may fail.

  In addition, the memory dump process disclosed in Patent Document 3 cannot be performed unless a normal kernel process such as program execution can be normally performed in a faulty terminal. There is a problem that it cannot be applied to various scenes.

  Furthermore, in the memory dump process disclosed in Patent Document 3, it is necessary to restart the terminal in which the failure has occurred, as described above. For this reason, there is a problem that memory dump analysis cannot be performed immediately when the failed terminal is a computer that takes time to restart, or when the failed hardware needs to be replaced. . Furthermore, the problem that the analysis of the memory dump cannot be performed immediately has a possibility that the contents of the dumped memory may be lost when a failure with low reproducibility or a failure analysis is urgently required.

  An object of the present invention is to provide a multi-node system, a node, a memory dump processing method, and a program capable of solving the above-described problem and executing memory dump processing without being affected by the state of a failed node. It is in.

In order to achieve the above object, a multi-node system according to the present invention is a multi-node system having a plurality of nodes capable of communicating with each other,
When a failure occurs in any of the plurality of nodes, the node in which the failure has not occurred acquires information stored in the memory of the node in which the failure has occurred. .

In order to achieve the above object, the node in the present invention is a node that functions as one of a plurality of nodes constituting a multi-node system,
Memory,
A remote memory access mechanism capable of acquiring information stored in the memory of another node other than the node;
An information acquisition request unit that requests another node other than the node to acquire information stored in the memory of the node when a failure occurs in the node;
Information for acquiring information stored in the memory of the other node by the remote memory access mechanism when a request for acquisition of information stored in the memory is received from another node other than the node And an acquisition unit.

In order to achieve the above object, a memory dump processing method according to the present invention is a method for performing a memory dump using a multi-node system having a plurality of nodes capable of communicating with each other.
(A) When a failure occurs in any of the plurality of nodes, the information stored in the memory of the failed node by the failed node in the non-failed node Requesting the acquisition of
(B) receiving a request in the step (a) and obtaining information stored in the memory of the failed node by a node in which the failure has not occurred. It is characterized by.

Furthermore, in order to achieve the above object, a program in the present invention is a program for performing a memory dump in a multi-node system having a plurality of computers capable of communicating with each other by the computer,
In the computer,
If a failure occurs in the computer, (a) request the acquisition of information stored in the memory of the computer from the computer in which the failure does not occur,
When no failure has occurred in the computer, the step of (b) acquiring information stored in the memory of the computer in which the failure has occurred is executed.

  With the above features, according to the multi-node system, the node, the memory dump processing method, and the program of the present invention, the memory dump process can be executed without being affected by the state of the failed node.

FIG. 1 is a block diagram showing a schematic configuration of a multi-node system according to Embodiment 1 of the present invention. FIG. 2 is a flowchart showing a series of operations in the node where a failure has occurred in the first embodiment. FIG. 3 is a flowchart showing a series of operations in a node in which no failure has occurred in the first embodiment. FIG. 4 is a block diagram showing a schematic configuration of the multi-node system according to Embodiment 2 of the present invention. FIG. 5 is a flowchart showing a series of operations in a node where a failure has occurred in the second embodiment of the present invention.

(Embodiment 1)
Hereinafter, a multi-node system, a node, a memory dump processing method using these, and a program for realizing them will be described with reference to FIGS. 1 to 3 according to Embodiment 1 of the present invention. First, the configuration of the multi-node system according to the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a schematic configuration of a multi-node system according to Embodiment 1 of the present invention.

  As shown in FIG. 1, the multi-node system 104 according to the first embodiment has nodes 100 and 101 that can communicate with each other. Although only the node 100 and the node 101 are shown in FIG. 1, the multi-node system 104 may actually have more nodes.

  Further, in the multi-node system 104, when a failure occurs in either the node 100 or the node 101, the node in which the failure has not occurred acquires information stored in the memory of the failed node. In other words, the memory dump process is executed. Here, the configuration of the nodes 100 and 101 will be described below.

  As shown in FIG. 1, in the first embodiment, the nodes 100 and 101 are computers having the same configuration, and both of them are a remote memory access mechanism 110, a memory 120, a secondary storage device 130, CPU (central processing unit) 150.

  The remote memory access mechanism 110 is configured to be able to acquire information stored in the memory of another node other than the node having the remote memory access mechanism 110 (own node). In the first embodiment, the remote memory access mechanism 110 includes DMA (Direct Memory Access), and can execute input / output (I / O) of the memory of the own node without going through the CPU 150 of the own node.

  In addition, the remote memory access mechanism 110 can request the remote memory access mechanism 110 of another node other than its own node to transfer information stored in the memory of the other node. An instruction for performing memory I / O exchanged between the nodes is hereinafter referred to as an “internode main memory transfer instruction”. The remote memory access mechanism 110 that has received the inter-node main memory transfer instruction causes the memory 120 to output information, and transmits the output information to the remote memory access mechanism 110 that is the output source of the inter-node main memory transfer instruction. .

  In each node, the memory 120 stores the kernel data 121 of the OS 140 and is connected to the remote memory access mechanism 110 and the secondary storage device 130. The OS 140 is a program executed by hardware resources including the CPU 150, and includes an information acquisition request unit 141, an information acquisition unit 142, and an information output unit 143. In addition, the kernel data 121 stored in the memory 120 is a target of the memory dump process (information acquisition process) in the first embodiment.

  Among them, the information acquisition request unit 141 acquires information (kernel data 121) stored in the memory 120 of the own node in another node other than the own node when a failure occurs in the own node, that is, Request acquisition of memory dump data.

  In the first embodiment, when a failure occurs in a node, first, the OS 140 of the node in which the failure occurs executes kernel panic processing. Next, in order to acquire the kernel data 121 in the memory 120 (execute a memory dump), the information acquisition request unit 141 requests acquisition of information from any other node. In addition, the OS 140 of the node in which the failure has occurred requests the acquisition of information, and then temporarily stops the kernel panic process. Then, when the acquisition of information at another node is completed, the OS 140 resumes the kernel panic process.

  Further, when the information acquisition unit 142 receives a request for acquisition of information stored in the memory from another node other than the own node, the information acquisition unit 142 is stored in the memory of another node by the remote memory access mechanism 110. Information (kernel data 121) is obtained. Specifically, the information acquisition unit 142 transmits an inter-node main memory transfer command from the remote memory access mechanism 110 of its own node to the remote memory access mechanism 110 of another node, and executes a memory dump. Then, the information acquisition unit 142 stores the acquired memory dump data (kernel data 121) in the memory 120 of the own node.

  The information output unit 143 outputs the information stored in the memory 120 of the own node to the secondary storage device 130 provided in the own node or the shared disk device 103 connected to the nodes 100 and 101, Can be stored. That is, the information output unit 143 outputs the memory dump data acquired from other nodes and stored in the memory 120 of the own node to the secondary storage device 130 or the shared disk device 103.

  In the first embodiment, the information output unit 143 can also output the memory dump data to the secondary storage device 130 or the shared disk device 103 asynchronously with the information transfer process between the nodes. In the first embodiment, the shared disk device 103 is shared by the nodes 100 and 101, and constitutes a distributed file system such as NFS (Network File System), for example.

  Further, the node on which the output by the information output unit 143 is performed is a normally operating node in which no failure has occurred. Therefore, the secondary storage device 130 is not limited to a device built in each node, and the secondary storage device 130 is a device that is shared among a plurality of nodes and that constitutes a distributed file system such as NFS. Can be used.

  In the first embodiment, the node 100 and the node 101 are connected via an inter-node connection device 102 that connects the nodes, thereby enabling communication with each other. Specifically, the internode connection apparatus 102 is a router or the like, and is connected to the nodes 100 and 101 by a dedicated line. The inter-node connection device 102 transmits the inter-node main memory transfer command to the remote memory access mechanism 110 of the node specified by the inter-node main memory transfer command. Further, the internode connection apparatus 102 transfers the information output from the remote memory access mechanism 110 of the node for which the information transfer is requested to the requesting node.

  In the first embodiment, the multi-node system 104 may be configured such that the memories of the plurality of nodes are shared by the plurality of nodes. Specifically, the multi-node system 104 can include, for example, a system called ccNUMA (cache coherency non-uniform memory access). In this system, the I / O of the memory of another node that is remotely arranged can be performed in the same manner as the I / O to the memory that is locally arranged. In this case, the nodes 100 and 101 do not have to include the remote memory access mechanism 110.

  Next, with regard to the operation in the multi-node system 104 shown in FIG. 1, paying attention to one node, it is divided into a case where a failure has occurred and a case where no failure has occurred. Will be described. FIG. 2 is a flowchart showing a series of operations in the node where a failure has occurred in the first embodiment. FIG. 3 is a flowchart showing a series of operations in a node in which no failure has occurred in the first embodiment.

  Note that the memory dump processing method in the first embodiment is implemented by operating the multi-node system in the first embodiment shown in FIG. For this reason, the description of the memory dump processing method according to the first embodiment is replaced with the following description of the operation of the multi-node system 104.

  In the following description, FIG. 1 will be referred to as appropriate. Furthermore, it is assumed that a failure has occurred in the node 100 and no failure has occurred in the node 101. The node 100 is assumed to be the “failure node 100” and the node 101 is assumed to be the “dump acquisition node 101”. In addition, inter-node main memory transfer between the failed node 100 and the dump acquisition node 101 is performed via the inter-node connection device 102 connected by a high-speed dedicated line.

  First, a description will be given focusing on the failed node 100 with reference to FIG. In FIG. 2, similarly to the general case, when a fatal failure that prevents the operation from continuing in the node 100 occurs, the kernel panic process is executed by the OS 104 of the node 100. .

  As shown in FIG. 2, first, the information acquisition request unit 141 of the failed node 100 selects an arbitrary node as a dump acquisition node from other nodes that can communicate with each other (step A201). In the present embodiment, the node 101 is a dump acquisition node.

  Next, the information acquisition request unit 141 of the failed node 100 transmits an acquisition request (dump acquisition request) of information stored in the memory 120 of the failed node 100 to the dump acquiring node 101 (step A202). Then, after step A202 ends, the OS 104 temporarily stops the kernel panic process (step A203).

  While the process stop at step A203 continues, an inter-node main memory transfer command is transmitted from the dump acquisition node 101, which will be described later, and the I / O of the memory 120 of the failed node 100 is the remote memory of the failed node 100. Access mechanism 110 performs. For this reason, the CPU 150 of the failed node 100 does not perform processing.

  Next, the information acquisition requesting unit 141 of the failed node 100 receives a notification (dump acquisition end notification) from the dump acquisition node 101 that information acquisition has ended (step A204). Then, the OS 104 of the failed node 100 restarts the kernel panic process that has been stopped in Step A203, triggered by Step A204 (A205).

  Next, a description will be given focusing on the dump acquisition node 101 with reference to FIG. The process shown in FIG. 3 is executed from when the failed node 100 starts the kernel panic process to when it ends.

  As shown in FIG. 3, first, the information acquisition unit 142 of the dump acquisition node 101 receives information stored in the memory 120 of the failed node 100 by the remote memory access mechanism 110 in response to reception of the dump acquisition request. Obtain (step A301). The information acquired at this time is kernel data 121 (memory dump data) stored in the memory of the failed node 100. Further, the address required when requesting the inter-node main memory transfer to the failed node (100) is always constant.

  Next, the information acquisition unit 142 of the dump acquisition node 101 compares the address of the acquired memory dump data with the address of the data to be acquired, and the memory dump data to be acquired still exists. It is determined whether or not (step A302).

  As a result of the determination in step A302, when the memory dump data to be acquired still exists, the information acquisition unit 142 transmits an inter-node main memory transfer command (step A303). In step A303, the destination of the inter-node main memory transfer command is set to the failed node 100. The transfer start address is set to the memory address of the memory dump data scheduled to be received in the next step A301.

  On the other hand, as a result of the determination in step A302, when there is no memory dump data to be acquired, that is, when all the information of the memory 120 to be acquired is acquired, the information acquisition unit 142 determines that the failure node 100 In step A304, a dump acquisition end notification is transmitted. After step A304 ends, the information output unit 143 outputs the acquired dump data to the secondary storage device 130 or the shared disk device 103 asynchronously with the dump acquisition process shown in FIG. Then, the process in the dump acquisition node 101 ends.

  As described above, in the multi-node system 104 and the memory dump processing method according to the first embodiment, the memory dump process is performed by a node in which no failure has occurred. That is, according to the first embodiment, it is possible to execute the memory dump process without being affected by the state of the failed node.

  Specifically, in the first embodiment, a normally operating node (dump acquisition node 101) acquires the memory contents (dump data) of the failed node 100 by an inter-node main memory transfer instruction, In the node 100, the contents of the memory 120 can be directly transferred by DMA. Therefore, since the OS 140 of the node in which the failure has occurred (failed node 100) does not need to perform processing, the memory dump processing can be executed even when the OS 140 of the failed node 100 cannot operate due to the failure. Furthermore, even if it is necessary to restart the OS 140 in the failed node 100, the dump data can be referred to without being affected by the restart.

  In the first embodiment, since a normal node (dump acquisition node 101) in which no failure has occurred acquires memory dump data, memory dump data is automatically acquired immediately after the occurrence of a failure. Processing can begin. For this reason, quick dump analysis is possible. In particular, in the field of HPC (High Performance Computing), multi-node systems that connect and operate multiple nodes are the mainstream, and when a failure occurs in multiple nodes, the operator individually performs memory dumps of multiple nodes. Need to be collected. The operation of collecting memory dumps of a large number of nodes is complicated for the operator, and there is a possibility that the memory dump may be lost due to an erroneous operation at the time of collection. However, according to the first embodiment, such a problem can be solved.

  Furthermore, in the first embodiment, since the storage destination of the memory dump data is a normally operating node (dump acquisition node 101), the memory dump data can be referred to immediately after the memory dump processing is completed. Become. In the first embodiment, the memory dump data can be stored in a shared disk device such as a distributed file system for centralized management. Therefore, data management can be facilitated and the capacity of the secondary storage device 130 required for one node can be reduced.

  Conventionally, in a multi-node system using a UNIX (registered trademark) OS, programs such as “disk dump”, “net dump”, and “k dump” are used to obtain a memory dump.

  Among these, in “disk dump”, memory dump data is stored in the internal disk of the node where the failure has occurred, and it is also necessary to separately prepare a disk driver corresponding to dump acquisition. “Netdump” is used in a system that acquires memory dump data by a client-server method, and the memory dump data is stored in the internal disk of the server. Thus, “netdump” requires a dedicated network driver.

  On the other hand, according to the first embodiment, no special disk driver or dedicated network driver is required. In a multi-node system using a UNIX OS, if the first embodiment is used instead of “disk dump” or “net dump”, the cost of the system can be reduced.

  In “kdump”, it is necessary to make the kernel to be activated in the memory dump data acquisition process resident in the main memory in advance. For this reason, since the amount of memory consumption increases, it is difficult to use the mounted memory in an HPC field or the like that is operated with a policy that allows the user to use the mounted memory as much as possible. On the other hand, according to the first embodiment, there is no need to make the program for acquiring the memory dump data resident in the main memory, and the first embodiment is useful also in the HPC field.

  Moreover, the program in this Embodiment 1 should just be a program which makes the computer which comprises a node perform step A201-A205 shown in FIG. 2, and step A301-A304 shown in FIG. By executing the program in the first embodiment, the multi-node system and the memory dump processing method in the first embodiment can be realized. In this case, the CPU of the node functions as an information acquisition request unit, an information acquisition unit, and an information output unit, and performs processing.

(Embodiment 2)
Next, a multi-node system, a node, a memory dump processing method using these, and a program for realizing these will be described with reference to FIG. 4 and FIG. First, the configuration of the multi-node system according to the second embodiment will be described with reference to FIG. FIG. 4 is a block diagram showing a schematic configuration of the multi-node system according to Embodiment 2 of the present invention.

  As shown in FIG. 4, in the multi-node system 108 according to the second embodiment, each of the nodes 105 and 106 does not include the remote memory access mechanism 110 shown in FIG. In the second embodiment, the CPU 150 of each node functions as the remote memory access mechanism 110.

  In the second embodiment, communication between the node 105 and the node 106 is performed via a network 107 such as a LAN, unlike the first embodiment. Further, the inter-node connection device 102 shown in FIG. 1 is incorporated in the network 107, and the inter-node connection device is not shown in FIG. Except for these points, the multi-node system 108 in the second embodiment is configured in the same manner as the multi-node system 104 in the first embodiment shown in FIG.

  Next, the operation in the multi-node system 108 shown in FIG. 4 will be described taking as an example a case where a failure has occurred in a node. FIG. 5 is a flowchart showing a series of operations in a node where a failure has occurred in the second embodiment of the present invention.

  The memory dump processing method according to the second embodiment is implemented by operating the multi-node system according to the second embodiment shown in FIG. For this reason, the description of the memory dump processing method according to the second embodiment is replaced with the following description of the operation of the multi-node system 108.

  In the following description, FIG. 4 will be referred to as appropriate. Furthermore, it is assumed that a failure has occurred in the node 105 and no failure has occurred in the node 106. The node 105 is assumed to be a “failure node 105” and the node 106 is assumed to be a “dump acquisition node 106”. The operation of the dump acquisition node 106 is the same as the operation shown in FIG. 3 in the first embodiment.

  As shown in FIG. 5, first, the information acquisition request unit 145 of the failed node 105 selects an arbitrary node as a dump acquisition node from other nodes that can communicate with each other (step A501). In this embodiment, the node 106 is a dump acquisition node.

  Next, the information acquisition request unit 145 of the failure node 105 transmits an acquisition request (dump acquisition request) of information stored in the memory 120 of the failure node 105 to the dump acquisition node 106 (step A502). Then, after the end of step A502, the OS 104 temporarily stops the kernel panic process (step A503).

  In the second embodiment, while the process is stopped in step A503, I / O of the memory 120 of the failed node 105 is performed by the CPU 150 functioning as a remote memory access mechanism. Therefore, after execution of step A503, the CPU 150 shifts to a “memory transfer mode” in which only the main memory transfer command from the dump acquisition node 106 is accepted and transfer processing is performed (A504).

  Next, the information acquisition request unit 145 of the failure node 105 receives a notification (dump acquisition end notification) that the information acquisition is completed from the dump acquisition node 106 (step A505). Then, the CPU 105 of the failed node 105 returns to the “normal mode” before entering the memory transfer mode, triggered by step A505 (step A506). Thereafter, the OS 140 resumes the kernel panic process that was stopped in step A503 (A507).

  As described above, also in the multi-node system 108 and the memory dump processing method according to the second embodiment, the memory dump process is performed by a node in which no failure has occurred. Therefore, even when the second embodiment is used, similarly to the first embodiment, the memory dump process can be executed without being affected by the state of the failed node. Also according to the second embodiment, all the effects described in the first embodiment can be obtained.

  In the second embodiment, communication between the failure node 105 and the dump acquisition node 106 is performed via a network 107 such as a LAN. Therefore, it is assumed that the information acquisition request unit 145, the information acquisition unit 142, and the information output unit 143 are implemented with a protocol suitable for the network to be used, such as TCP / IP.

  INDUSTRIAL APPLICABILITY The present invention is effective for collecting memory dump data of a failed node in a multi-node system having a plurality of nodes capable of communicating with each other and performing main memory transfer between nodes. . Therefore, the present invention has industrial applicability.

100 node 101 node 102 inter-node connection device 103 shared disk device 104 multi-node system 105 node 106 node 107 network 108 multi-node system 110 remote memory mechanism 120 memory 121 kernel data 130 secondary storage device 140 OS
141, 145 Information acquisition request unit 142 Information acquisition unit 143 Information output unit

Claims (12)

A multi-node system having a plurality of nodes capable of communicating with each other,
Each of the plurality of nodes includes a remote memory access mechanism that acquires information stored in its own memory in response to a transfer command from a node other than the node, and outputs the acquired information.
When a failure occurs in any of the plurality of nodes, the failed node acquires information stored in the memory of the failed node to a node in which the failure has not occurred. When requested,
The node that has received the request and has not failed outputs the transfer command to the remote memory access mechanism of the failed node and stores it in the memory of the failed node. A multi-node system characterized in that information is obtained.   The multi-node system according to claim 1, wherein a memory of each of the plurality of nodes is shared by the plurality of nodes. A storage device connected to the plurality of nodes and shared by each of the plurality of nodes;
2. The multi-node according to claim 1, wherein when the node in which the failure has not occurred acquires information stored in the memory of the node in which the failure has occurred, the node stores the acquired information in the storage device. system. The information acquired nodes is, after completion of acquisition of the information, the generated node of the fault, notifies the end of acquisition of the information, the multi-node system according to claim 1. After the failed node requests acquisition of information stored in the memory of the failed node, the processing in the node is stopped, and the information from the node that acquired the information is The multi-node system according to claim 4 , wherein the processing is resumed after receiving an acquisition end notification. A node functioning as one of a plurality of nodes constituting a multi-node system,
Memory,
Depending on the transfer command from the node other than the node, it obtains the information stored in its memory, and outputs the acquired information, the remote memory access mechanism,
If on the node fails, the node other than the node, the information acquisition request unit for requesting the acquisition of information stored in the memory of the node,
From node other than the node, when receiving a request for acquisition of information stored in its memory, the remote memory access mechanism of the nodes other than the node, and outputs the transfer command, the node A node, comprising: an information acquisition unit that acquires information stored in a memory of a node other than the node. Have a mutually communicable plurality of nodes, and wherein each of the plurality of nodes, in accordance with the transfer command from a node other than the node, obtains the information stored in its memory, and obtains the and outputs the information, that provides a remote memory access mechanism, a method for performing a memory dump using a multi-node system,
(A) When a failure occurs in any of the plurality of nodes, the information stored in the memory of the failed node by the failed node in the non-failed node Requesting the acquisition of
(B) The transfer command is output to the remote memory access mechanism of the failed node by the node that has received the request in the step (a) and the failure has not occurred, and the occurrence of the failure Obtaining information stored in the memory of the node
A memory dump processing method. The memory dump according to claim 7 , further comprising: (c) notifying the end of the acquisition of the information to the failed node by the node that acquired the information after the end of the step (b). Processing method. (D) after completion of the step (a), stopping the processing in the failed node;
The memory dump processing method according to claim 8 , further comprising: (e) resuming the processing that was stopped in the step (d) after the step (c) is completed. Have a plurality of can communicate with each other computer, and, each of the plurality of computers, according to the transfer instruction from the computer other than the computer, acquires the information stored in its memory, and obtains the and outputs the information, that provides a remote memory access mechanism, a program for performing a memory dump in a multi-node system by the computer,
In the computer,
If a failure occurs in the computer, (a) request the acquisition of information stored in the memory of the computer from the computer in which the failure does not occur,
If no failure has occurred in the computer, (b) information stored in the memory of the failed node is output to the remote memory access mechanism of the computer in which the failure has occurred. A program characterized by having a step executed. When the step (b) is executed,
The program according to claim 10 , further comprising: causing the computer to further execute a step of notifying the computer in which the failure has occurred after completion of the step of (c) (b). When step (a) is executed,
After the completion of the step (a), the processing in the computer was stopped, and when the completion of the acquisition of the information was notified by the execution of the step (c) by a computer other than the computer, the processing was stopped The program according to claim 11 , further causing the computer to execute a step of resuming processing.

Images