JP2014096031A - Failure information management device, information processing device, distributed parallel processing system, failure information management method and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for improving processing efficiency in failure analysis work by reducing the amount of failure analysis information when failure occurs in a distributed parallel processing system.SOLUTION: A failure information management device includes: a failure information storage 11; a failure information reception part 12 that receives a piece of failure information from a node 20; a similarity determination section 13 that determines whether the failure information storage 11 stores any piece of failure information that satisfies a predetermined similar condition relative to the received failure information; an output YES/NO information transmission part 14 that transmits a piece of output YES/NO information representing YES/NO of output of failure analysis information including a content of a main storage which is used for processing a failure occurred at the node 20, according to the determination result made by the similarity determination section 13; and a failure information registration section 15 that, when it is determined that no failure information satisfying the predetermined similar condition relative to the received failure information is stored in the failure information storage 11, registers the received failure information to the failure information storage 11.

Description

  The present invention relates to a technique for managing information related to a failure that occurs during execution of distributed parallel processing.

  In general, when a failure occurs during the execution of a process, the information processing apparatus outputs a memory image (context, stack, data) being executed as a core file. Such a core file is used for analyzing the cause of the failure. Hereinafter, such a core file is referred to as failure analysis information. The failure analysis information is generally created for each process.

  In recent years, distributed parallel processing systems that execute processing by a plurality of information processing apparatuses (nodes) are well known. In such a distributed parallel processing system, the entire process is divided into a plurality of processes that can be executed in parallel, and each process is distributed and executed on a plurality of nodes as different processes. When a failure occurs during the execution of distributed parallel processing in such a distributed parallel processing system, a plurality of nodes output failure analysis information for each process being executed. For this reason, it is necessary to analyze a plurality of pieces of failure analysis information in order to analyze the cause of the failure.

  On the other hand, the number of nodes in a distributed parallel processing system is increasing year by year. For this reason, the total capacity of the failure analysis information to be output is also increasing.

  Also, in the distributed parallel processing system, for example, when executing a distributed parallel program by MPI (Message Passing Interface), in most cases, similar processing is executed at each node in synchronization. In this case, the processes on a plurality of nodes often generate the same failure at the same place in the distributed parallel program. In this case, a plurality of similar failure analysis information is generated for each process. For example, when an out-of-array reference occurs in a 10-process distributed parallel program, 9 processes often cause the same exception at the same location. Even in that case, nine similar failure analysis information is generated for nine processes. However, in practice, it is almost always sufficient to analyze one piece of failure analysis information.

  As a technique related to such a problem, for example, there is one described in Patent Document 1. In this related technology, when a failure occurs in a distributed parallel processing system, a failure occurrence node is specified and failure analysis data is collected from the node. Furthermore, this related technology identifies other nodes that are considered to be related to the occurrence of a failure at the failed node (for example, the communication partner node of the failed node), and also provides failure analysis data from the identified other nodes. Collect.

  Another technique related to such a problem is described in Patent Document 2, for example. This related technique reduces the weight of failure analysis information when a plurality of processes use a shared memory. Specifically, this related technique outputs the contents of the shared memory as a separate file among the memory images included in the failure analysis information regarding each process, and the failure analysis information includes information for specifying the separate file. At this time, this related technology does not output a separate file if another file representing the contents of the shared memory related to the target process has already been created, but includes failure identification including information identifying the separate file. Output information. As a result, this related technique reduces the weight of failure analysis information generated by each process.

Japanese Patent Laying-Open No. 2005-4513 JP 2008-197980 A

  However, the techniques described in Patent Document 1 and Patent Document 2 have the following problems.

  The related technique described in Patent Document 1 specifies failure occurrence nodes and related nodes, and acquires failure analysis data from those nodes. However, when this related technique is applied to a distributed parallel processing system that executes the distributed parallel program as described above, there is a high possibility that the same failure will occur at the same location in all nodes that perform the same processing. Therefore, this related technique collects similar failure analysis data from all such nodes. Therefore, this related technique cannot reduce the enlargement of the entire capacity of the failure analysis information and the inefficiency of the analysis work associated therewith.

  The related technique described in Patent Document 2 can reduce the weight of failure analysis information generated for a plurality of processes using a shared memory. However, in a distributed parallel processing system that executes a distributed parallel program as described above on a plurality of nodes that do not have a shared memory, this related technique cannot reduce the weight of failure analysis information. Therefore, this related technique cannot reduce the enlargement of the entire capacity of the failure analysis information and the inefficiency of the analysis work associated therewith.

  In order to avoid the compression of the disk capacity due to the enlargement of the total capacity of the failure analysis information, it may be possible to use a resource limiting function provided by a general operating system. However, in this case, when the resource limit is exceeded, subsequent failure analysis information is not output. Therefore, information necessary for analysis may not be collected.

  The present invention has been made to solve the above-described problems, and provides a technique for making analysis work more efficient by suppressing the enlargement of the entire capacity of failure analysis information when a failure occurs in a distributed parallel processing system. The purpose is to do.

  The failure information management apparatus according to the present invention includes a failure information storage unit that stores failure information related to a failure that occurred during process execution in each node that executes a distributed parallel program in a distributed manner, and is transmitted by the node in response to the occurrence of the failure. A failure information receiving unit that receives the failure information, and a failure information receiving unit that determines whether failure information satisfying a predetermined similarity condition for the failure information received from the node is stored in the failure information storage unit. According to the determination result by the sex determination unit and the similarity determination unit, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage device used by the process at the time of the failure occurrence in the node is output. The output availability information transmission unit that transmits to the node, and the failure information that satisfies a predetermined similarity condition with respect to the failure information received from the node, If it is determined not to be stored in the information storage unit, the fault information received from the node, and a fault information registration unit that registers the failure information storage unit.

  The information processing apparatus of the present invention is an information processing apparatus that operates as a node that can execute a distributed parallel program in a distributed manner, and includes a process execution unit that executes processing based on the distributed parallel program, and the process execution unit If a failure occurs during process execution by the failure information, a failure information collection unit that collects failure information related to the failure and failure information that is collected by the failure information collection unit to the failure information management device described above A failure analysis information output unit that outputs failure analysis information including the contents of the main storage device used by the process when the output permission information received from the failure information management device indicates that output is possible; .

  Moreover, the distributed parallel processing system of this invention is provided with the above-mentioned failure information management apparatus and the above-mentioned information processing apparatus.

  In the failure information management method of the present invention, when a failure occurs during process execution on a node that executes a distributed parallel processing program in a distributed manner, the failed node collects failure information related to the failure, and stores the failure information. The failure information management device determines whether failure information satisfying a predetermined similarity condition for the received failure information is stored in the failure information storage unit, and the predetermined similarity condition If failure information satisfying the condition is not stored, output enable / disable information indicating that failure analysis information including the contents of the main storage device used by the process at the time of the failure is output to the node. , When the failure information received from the node is registered in the failure information storage unit and failure information satisfying a predetermined similarity condition is stored, The failure analysis information is transmitted to the node, indicating that the failure analysis information is output, and the failure analysis information is received when the output permission information received from the failure information management device indicates that the failure has occurred. Is output.

  The computer program according to the present invention uses a failure information storage unit that stores failure information relating to a failure that occurred during process execution in each node that executes a distributed parallel program in a distributed manner, and responds to the occurrence of the failure by the node. A failure information receiving step for receiving the failure information transmitted in response to the failure information, and whether or not failure information satisfying a predetermined similarity condition for the failure information received from the node is stored in the failure information storage unit. A similarity determination step to be determined, and an output indicating whether failure analysis information including the contents of the main memory used by the process at the time of the failure occurrence in the node is output according to the determination result of the similarity determination step Output permission information transmission step for transmitting permission information to the node, and failure information received from the node. A failure information registration step of registering the failure information received from the node in the failure information storage unit when it is determined that the failure information satisfying the similar condition is not stored in the failure information storage unit; Let the device run.

  Another computer program of the present invention includes a process execution step for executing processing based on a distributed parallel program, and a failure for collecting failure information related to the failure when a failure occurs during the process execution in the process execution step. An information collection step; a failure information transmission step for transmitting the failure information collected in the failure information collection step to a device executing the above-described computer program; and the output availability information received from the device is output. If yes, the computer device is caused to execute a failure analysis information output step of outputting failure analysis information including the contents of the main memory used by the process.

  The present invention can provide a technique for suppressing the enlargement of the entire capacity of failure analysis information when a failure occurs in a distributed parallel processing system, and making analysis work more efficient.

It is a block diagram which shows the structure of the distributed parallel processing system as the 1st Embodiment of this invention. It is a functional block diagram of each apparatus which comprises the distributed parallel processing system as the 1st Embodiment of this invention. It is a flowchart explaining operation | movement of the distributed parallel processing system as the 1st Embodiment of this invention. It is a block diagram which shows the structure of the distributed parallel processing system as the 2nd Embodiment of this invention. It is a functional block diagram of each apparatus which comprises the distributed parallel processing system as the 2nd Embodiment of this invention. It is a flowchart explaining operation | movement of the distributed parallel processing system as the 2nd Embodiment of this invention. It is a schematic diagram explaining the specific example of operation | movement of the distributed parallel processing system as the 2nd Embodiment of this invention. It is another schematic diagram explaining the specific example of operation | movement of the distributed parallel processing system as the 2nd Embodiment of this invention. It is another schematic diagram explaining the specific example of operation | movement of the distributed parallel processing system as the 2nd Embodiment of this invention. It is another schematic diagram explaining the specific example of operation | movement of the distributed parallel processing system as the 2nd Embodiment of this invention. It is a schematic diagram explaining the summary information output in the specific example of operation | movement of the distributed parallel processing system as the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
A configuration of a distributed parallel processing system 1 as a first embodiment of the present invention is shown in FIG.

  In FIG. 1, the distributed parallel processing system 1 includes a failure information management device 10 and one or more information processing devices 20. The failure information management apparatus 10 and each information processing apparatus 20 are connected to be communicable with each other via a network. The information processing apparatus 20 can execute a distributed parallel program in a distributed manner. Hereinafter, the information processing apparatus 20 is also referred to as a node 20. Although FIG. 1 shows one failure information management apparatus 10 and four nodes 20, the number of apparatuses included in the distributed parallel processing system of the present invention is not limited.

  Here, the failure information management device 10 and the node 20 include a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), a storage device such as a hard disk, and a network interface. Each is constituted by a computer device. Note that the failure information management apparatus 10 may be configured by the same computer apparatus as any of the nodes 20 included in the distributed parallel processing system 1.

  Next, a functional block configuration of each device constituting the distributed parallel processing system 1 is shown in FIG.

  In FIG. 2, the failure information management apparatus 10 includes a failure information storage unit 11, a failure information reception unit 12, a similarity determination unit 13, an output availability information transmission unit 14, and a failure information registration unit 15. In addition, the node 20 includes a process execution unit 21, a failure information collection unit 22, a failure information transmission unit 23, and a failure analysis information output unit 24. Here, the failure information storage unit 11 is configured by a storage device. Further, the failure information receiving unit 12, the output availability information transmitting unit 14, the failure information transmitting unit 23, and the failure analysis information outputting unit 24 read the computer program stored in the network interface, ROM, and storage device into the RAM and execute it. CPU. Further, the similarity determination unit 13, the failure information registration unit 15, the process execution unit 21, and the failure information collection unit 22 are configured by a CPU that reads a computer program stored in the ROM and the storage device into the RAM and executes it. The hardware configuration of the failure information management apparatus 10, each node 20, and each functional block of each apparatus is not limited to the above-described configuration.

  First, each functional block of the node 20 will be described.

  The process execution unit 21 executes processing based on the distributed parallel program.

  When a failure occurs during process execution by the process execution unit 21, the failure information collection unit 22 collects failure information related to the failure. Here, the failure information may include, for example, failure cause information indicating the cause of the failure. Further, the failure information may include traceback information representing a history of instructions until the failure occurs. In addition, the failure information may include various types of information related to failures that can be collected in the node 20 when a failure occurs.

  The failure information transmission unit 23 transmits the failure information collected by the failure information collection unit 22 to the failure information management apparatus 10.

  The failure analysis information output unit 24 receives output permission information from the failure information management device 10. Then, the failure analysis information output unit 24 outputs failure analysis information when the received output availability information indicates that output is possible. Here, the failure analysis information is information including the contents (memory image) of the main memory (RAM) used by the process when the failure occurs. For example, the failure analysis information output unit 24 may output the failure analysis information as a file to the storage device of the own device.

  Next, each functional block of the failure information management apparatus 10 will be described.

  The failure information storage unit 11 stores failure information regarding a failure that occurred during the process execution of the distributed parallel program in the node 20.

  The failure information receiving unit 12 receives failure information transmitted by the node 20 in response to the occurrence of a failure.

  The similarity determination unit 13 determines whether failure information satisfying a predetermined similarity condition with respect to failure information received from the node 20 is stored in the failure information storage unit 11. For example, the similar condition may be that the failure cause information included in the failure information is the same. Further, for example, the similar condition may be that the traceback information included in the failure information is the same.

  The output propriety information transmission unit 14 transmits output propriety information indicating the output propriety of the failure analysis information to the corresponding node 20 according to the determination result by the similarity determination unit 13. Specifically, the output availability information transmission unit 14 outputs an output rejection when failure information satisfying a predetermined similarity condition for failure information received from the node 20 is stored in the failure information storage unit 11. Output enable / disable information indicating is transmitted. Further, the output permission information transmission unit 14 outputs an output indicating that the output is permitted when failure information satisfying a predetermined similarity condition with respect to the failure information received from the node 20 is not stored in the failure information storage unit 11. Send availability information.

  The failure information registration unit 15 stores the received failure information in the failure information storage unit when failure information satisfying a predetermined similarity condition with respect to the failure information received from the node 20 is not stored in the failure information storage unit 11. 11 is registered.

  The operation of the distributed parallel processing system 1 configured as described above will be described with reference to FIG. In the node 20, it is assumed that the process execution unit 21 is already executing the process of the distributed parallel program. In FIG. 3, the left diagram shows the operation of the failure information management apparatus 10, the right diagram shows the operation of the node 20, and the broken arrows connecting the left and right indicate the flow of data.

  First, the failure information collection unit 22 of the node 20 collects failure information in response to the occurrence of a failure (step S1).

  Next, the failure information transmission unit 23 transmits the failure information collected in step S1 to the failure information management apparatus 10 (step S2).

  Next, the failure information receiving unit 12 of the failure information management device 10 receives the failure information transmitted in step S2 (step S3).

  Next, the similarity determination unit 13 determines whether failure information satisfying a predetermined similarity condition with respect to the failure information received in step S3 is stored in the failure information storage unit 11 (step S4). .

  Here, when it is determined that failure information satisfying a predetermined similarity condition has not yet been stored, the output permission information transmitting unit 14 transmits output permission information indicating that output is possible to the node 20 (step S5). ).

  Next, the failure information registration unit 15 registers the failure information received in step S3 in the failure information storage unit 11 (step S6).

  On the other hand, if it is determined in step S4 that the failure information satisfying the predetermined similarity condition has already been stored, the output permission information transmitting unit 14 transmits the output permission information indicating the output rejection to the node 20 ( Step S7).

  Next, the failure analysis information output unit 24 of the node 20 receives the output propriety information transmitted in step S5 or step S7, and determines whether or not the received output propriety information indicates that output is possible (step S8). .

  Here, when it is determined that the output permission information indicates that output is possible, the failure analysis information output unit 24 outputs the failure analysis information (step S9).

  On the other hand, when it is determined that the output permission information indicates output failure, the failure analysis information output unit 24 does not output the failure analysis information.

  As described above, the distributed parallel processing system 1 ends the operation.

  Next, effects of the first exemplary embodiment of the present invention will be described.

  The distributed parallel processing system as the first exemplary embodiment of the present invention can suppress the enlargement of the entire capacity of the failure analysis information when a failure occurs in the distributed parallel processing system, and can make the analysis work more efficient. .

  The reason is that each node that executes the distributed parallel program collects fault information when a fault occurs and transmits it to the fault information management apparatus. The similarity determination unit of the fault information management apparatus uses a fault similar to the received fault information. Determine whether information is already stored in the failure information storage unit, and if similar failure information is not yet stored, register the failure information in the failure information storage unit and output failure analysis information This is because the output enable / disable information indicating enable / disable is transmitted to the node, and when similar failure information is already stored, the output enable / disable information indicating whether or not the failure analysis information is output is transmitted to the node. This is because each node outputs failure analysis information when it receives output permission information indicating that output is possible from the failure information management device.

  As a result, in the distributed parallel processing system according to the present embodiment, when a similar failure occurs in a plurality of nodes, after any node outputs the failure analysis information, the failure is similar to the other nodes. Duplicate analysis information is not output. In particular, when the failure cause information and the traceback information included in the failure information are assumed to be the same condition, the distributed parallel processing system as the present embodiment has the same cause at the same location in the same distributed parallel processing program. The failure analysis information is not output repeatedly for the failure that has occurred. Therefore, this embodiment suppresses the enlargement of the failure analysis information. By using this embodiment, the analysis operator only has to analyze one piece of failure analysis information for the same failure caused by the same cause that occurs in the same location of the distributed parallel program. . Therefore, this embodiment makes analysis work more efficient.

(Second Embodiment)
Next, a second embodiment of the present invention will be described in detail with reference to the drawings. Note that, in each drawing referred to in the description of the present embodiment, the same reference numerals are given to the same configuration and steps that operate in the same manner as in the first embodiment of the present invention, and the detailed description in the present embodiment. Description is omitted.

  The configuration of a distributed parallel processing system 2 as a second embodiment of the present invention is shown in FIG. In FIG. 4, the distributed parallel processing system 2 includes a failure information management device 30, one or more nodes (information processing devices) 40, and a shared information storage device 50. The failure information management device 30, the node 40, and the shared information storage device 50 are connected to each other by an inter-node connection device 90 so as to communicate with each other. Here, the inter-node connection device 90 is a device that realizes high-speed data transfer between devices by connecting a plurality of devices with a crossbar switch or the like. Note that the failure information management device 30 and the shared information storage device 50 may each be configured by a computer device that constitutes one of the nodes 40. That is, the failure information management device 30 and the shared information storage device 50 may be realized on any of the nodes 40 that can execute the distributed parallel program in a distributed manner. Further, each computer device constituting the failure information management device 30, the shared information storage device 50, and the node 40 has a data transfer module for performing data transfer with other devices via the inter-node connection device 90. It shall be.

  Next, the functional block configuration of each device constituting the distributed parallel processing system 2 is shown in FIG. In FIG. 5, the failure information management device 30 includes a failure information registration unit 35 instead of the failure information registration unit 15 with respect to the failure information management device 10 as the first embodiment of the present invention. The difference is that an information output unit 36 is provided. Also, the node 40 replaces the failure information collecting unit 22 with the failure information collecting unit 42 and the failure analysis information output unit 24 with respect to the node 20 as the first embodiment of the present invention. The difference is that the output unit 44 is provided.

  First, each functional block of the failure information management apparatus 30 will be described.

  As with the failure information registration unit 15 in the first exemplary embodiment of the present invention, the failure information registration unit 35 stores failure information that satisfies a predetermined similarity condition for failure information received from the node 40. If it is determined that the failure information is not stored in the unit 11, the received failure information is registered in the failure information storage unit 11.

  Further, when it is determined that the failure information satisfying a predetermined similarity condition with respect to the failure information received from the node 40 is stored in the failure information storage unit 11, the failure information registration unit 35 is already stored. The failure information is updated to include the identification information of the node 40 included in the newly received failure information. Further, when the failure information includes the identification information of the process in which the failure has occurred, the failure information registration unit 35 adds the failure information that has already been stored and satisfies the similar condition to the newly received failure information. It may be updated to include additional process identification information.

  The summary information output unit 36 outputs the failure information stored in the failure information storage unit 11 as summary information. For example, the summary information output unit 36 may output the summary information after all processes of the distributed parallel program are completed on each node 40. The summary information output unit 36 may output summary information to the shared information storage device 50.

  Next, each functional block of the node 40 will be described.

  When a failure occurs during process execution by the process execution unit 21, the failure information collection unit 42 collects failure cause information and traceback information regarding the failure. Then, the failure information collection unit 42 generates failure information including the collected failure cause information and traceback information, the identification information of the own device, and the identification information of the process.

  The failure analysis information output unit 44 is different from the failure analysis information output unit 24 according to the first embodiment of the present invention in that failure analysis information is output to the shared information storage device 50. The failure analysis information output unit 44 may output the failure analysis information on a shared file of the shared information storage device 50.

  The shared information storage device 50 stores summary information output from the failure information management device 30 and failure analysis information output from each node 40. Further, the shared information storage device 50 may store the failure analysis information output from each node 40 in a shared file.

  The operation of the distributed parallel processing system 2 configured as described above will be described with reference to FIG. 6, the distributed parallel processing system 2 executes steps S1 to S9 in substantially the same manner as the distributed parallel processing system 1 according to the first embodiment of the present invention described with reference to FIG.

  However, in the present embodiment, the output destination of the failure analysis information in step S9 is different in the node 40. The failure analysis information output unit 44 outputs failure analysis information to the shared information storage device 50 (step S9).

  In the present embodiment, the failure information management apparatus 30 operates differently when it is determined in step S4 that failure information satisfying a predetermined similarity condition has already been stored.

  In this case, first, the output propriety information transmitting unit 14 transmits output propriety information indicating the output disapproval to the node 40 (step S7).

  Next, the failure information registration unit 35 adds the identification information of the node 40 and the identification information of the process included in the newly received failure information to the failure information satisfying the similarity condition already stored in the failure information storage unit 11. Is added and included (step S11).

  Moreover, in this Embodiment, after performing step S1-S9 and S11, the failure information management apparatus 30 outputs summary information.

  Specifically, when the summary information output unit 36 determines that each process of the target distributed parallel program has ended in each node 40 (Yes in step S12), the failure stored in the failure information storage unit 11 until then. Information is output to the shared information storage device 50 as summary information (step S13).

  As described above, the distributed parallel processing system 2 ends the operation.

  Next, specific examples of operations in the distributed parallel processing system 2 will be described with reference to FIGS. 7 to 11, the distributed parallel processing system 2 includes a failure information management device 30, four nodes 40 a to 40 d, and a shared information storage device 50. Note that the four nodes 40a to 40d have 40a to 40d as identification information (node IDs), respectively.

  In this example, the failure information includes failure cause information, traceback information, a node ID, and a process ID. 7 to 11, for the sake of explanation, the failure cause information and the traceback information are shown as character strings. However, the failure cause information may be expressed as a numerical value indicating the type of the failure cause information and the traceback information. May be represented by an instruction address value or the like.

  At this time, first, the operation of the distributed parallel processing system 2 when a failure occurs in the node 40a will be described with reference to FIG.

  Here, first, the fault information collection unit 42 of the node 40a collects the fault information 110a shown in FIG. 7 (step S1) and transmits it to the fault information management apparatus 30 (step S2).

  Next, in the failure information management apparatus 30, the failure information receiving unit 12 receives the failure information 110a (step S3).

  Here, the failure information storage unit 11 has not yet stored any failure information.

  Therefore, the similarity determination unit 13 determines that failure information satisfying a predetermined similarity condition with respect to the received failure information 110a is not stored in the failure information storage unit 11 (No in step S4).

  Next, since the failure information similar to the received failure information 110a has not yet been registered, the output availability information transmitting unit 14 transmits output availability information indicating that output is possible to the node 40a (step S5). .

  Next, the failure information registration unit 35 registers the failure information 110a in the failure information storage unit 11 (step S6).

  Next, in the node 40a, the failure analysis information output unit 44 receives the output enable / disable information and determines that the output is enabled (Yes in step S8).

  Therefore, the failure analysis information output unit 44 outputs the failure analysis information 440a shown in FIG. 7 to the shared information storage device 50 (step S9).

  Subsequently, it is assumed that a failure has occurred in the node 40b. The operation of the distributed parallel processing system 2 at this time will be described with reference to FIG.

  Here, first, the fault information collection unit 42 of the node 40b collects the fault information 110b shown in FIG. 8 (step S1) and transmits it to the fault information management apparatus 30 (step S2).

  Next, in the failure information management device 30, the failure information receiving unit 12 receives the failure information 110b (step S3).

  Next, the similarity determination unit 13 determines that failure information that satisfies a predetermined similarity condition for the received failure information 110b is not stored in the failure information storage unit 11 (No in step S4). Specifically, since the failure cause information and the traceback information are different between the received failure information 110b and the already stored failure information 110a, the similarity determination unit 13 must satisfy a predetermined similarity condition. Deciding.

  Next, since the failure information similar to the received failure information 110b has not yet been registered, the output availability information transmitting unit 14 transmits output availability information indicating that output is possible to the node 40b (step S5). .

  Next, the failure information registration unit 35 registers the failure information 110b in the failure information storage unit 11 (step S6).

  Next, in the node 40b, the failure analysis information output unit 44 receives the output enable / disable information and determines that the output is enabled (Yes in step S8).

  Therefore, the failure analysis information output unit 44 outputs the failure analysis information 440b shown in FIG. 8 to the shared information storage device 50 (step S9).

  Subsequently, it is assumed that a failure has occurred in the node 40c. The operation of the distributed parallel processing system 2 at this time will be described with reference to FIG.

  Here, first, the fault information collection unit 42 of the node 40c collects the fault information 110c shown in FIG. 9 (step S1) and transmits it to the fault information management apparatus 30 (step S2).

  Next, in the failure information management apparatus 30, the failure information receiving unit 12 receives the failure information 110c (step S3).

  Next, the similarity determination unit 13 determines that failure information satisfying a predetermined similarity condition with respect to the received failure information 110c is stored in the failure information storage unit 11 (Yes in step S4). Specifically, the similarity determination unit 13 uses the same failure cause information “SIGSEEGV” and the same traceback information “function C” as the already stored failure information 110b and newly received failure information 110c. () + ZZZ, function AA () + XXX, main () ”, it is determined that a predetermined similarity condition is satisfied.

  Next, since it is determined that the fault information 110b similar to the received fault information 110c has already been registered, the output propriety information transmitting unit 14 transmits output propriety information indicating output prohibition to the node 40c. (Step S7).

  Next, the failure information registration unit 35 adds the node ID “40c” and the process ID “3000” included in the received failure information 110c to the failure information 110b to update the failure information storage unit 11 (step S11). ).

  Next, in the node 40c, the failure analysis information output unit 44 receives the output propriety information and determines that it indicates output failure (No in step S8).

  Therefore, the failure analysis information output unit 44 of the node 40c does not output failure analysis information.

  Subsequently, it is assumed that a failure has occurred in the node 40d. The operation of the distributed parallel processing system 2 at this time will be described with reference to FIG.

  Here, first, the fault information collection unit 42 of the node 40d collects the fault information 110d shown in FIG. 10 (step S1) and transmits it to the fault information management apparatus 30 (step S2).

  Here, it is assumed that the content of the failure information 110d satisfies a predetermined similarity condition with respect to the failure information 110b.

  Therefore, the failure information management device 30 and the node 40d operate in the same manner as the failure information management device 30 and the node 40c described with reference to FIG. That is, the failure analysis information output unit 44 of the node 40d does not output failure analysis information. Further, the failure information management apparatus 30 adds the node ID “40d” and the process ID “4000” included in the newly received failure information 110d to the failure information 110b.

  Next, the summary output operation of the failure information management apparatus 30 will be described with reference to FIG.

  Here, first, the summary information output unit 36 of the failure information management apparatus 30 determines that each process has ended in the nodes 40a to 40d (Yes in step S12).

  Therefore, the summary output unit 36 outputs summary information 360 including the failure information 110 a and 110 b stored in the failure information storage unit 11 to the shared information storage device 50.

  As described above, in the case where three failures are caused by the same location and the same cause among the failures occurring in the four nodes 40a to 40d of the distributed parallel processing system 2, the distributed parallel processing system 2 has two failures. The analysis information 440a and 440b and the summary information 360 are output. Therefore, the distributed parallel processing system 2 suppresses the enlargement of the entire capacity of the failure analysis information compared to the case where the failure analysis information is output for each of the failures that occurred in the four processes. The analysis operator does not need to analyze the four pieces of failure analysis information, and only needs to analyze the two pieces of analysis information 440a and 440b while referring to the summary information 360. Also, when the failure analysis information is output on the shared file in the shared information storage device 50, the distributed parallel processing system 2 outputs the failure analysis information output time compared to the case where the four failure analysis information is output on the shared file. Can also be prevented.

  Next, the effect of the second exemplary embodiment of the present invention will be described.

  The distributed parallel processing system as the second exemplary embodiment of the present invention can further improve the efficiency of failure analysis work in the distributed parallel processing system.

  The reason is that if the failure information similar to the failure information collected when a failure occurs in each node is not yet stored in the failure information storage unit of the failure information management device, the failure information management device This is because the node stores the failure analysis information to the shared information storage device. Also, the failure information management device outputs the failure information accumulated in the failure information storage unit to the shared information accumulation device as summary information after the process of the distributed parallel program is completed in all nodes.

  Thereby, the analysis worker refers to the failure analysis information output to the shared information storage device according to the summary information output to the shared information storage device when a failure occurs in the distributed parallel processing system according to the present embodiment. Therefore, analysis work may be performed. The summary information includes a summary of the failure in which the cause of the failure and the traceback information do not overlap. Also, the failure analysis information indicated by the summary is output with no cause of failure or traceback information overlapping. Therefore, this embodiment can further improve the efficiency of the analysis work.

  In addition, the distributed parallel processing system according to the second exemplary embodiment of the present invention can suppress an increase in the output time of the failure analysis information when the failure analysis information is output on the shared file of the shared information storage device. .

  The reason is that the distributed parallel processing system according to the present embodiment does not output the failure analysis information in which the cause of the failure or the traceback information overlaps on the shared file. As a result, the present embodiment reduces the output time of failure analysis information compared to the case where all failure analysis information output for each node in which a failure has occurred is output on a shared file.

  In each embodiment of the present invention described above, the failure information management apparatus may be configured by the same computer apparatus as any node that executes the distributed parallel program.

  In each embodiment of the present invention described above, the failure information management device and each node have been described mainly with respect to an example in which the failure information management device and the node are connected via an inter-node connection device. As long as it is communicably connected, it may be connected by other techniques. In this case also, it is desirable that the failure information management device and each node are connected so as to be communicable at a communication speed suitable for distributed parallel processing.

  Further, in each of the embodiments of the present invention described above, the failure information collection unit of each node has been described mainly with respect to an example of collecting failure cause information and traceback information as failure information. Other information that can be collected at the time of occurrence may be collected and included in the failure information, and transmitted to the failure information management apparatus.

  Further, in each of the embodiments of the present invention described above, the similarity condition used by the similarity determination unit of the failure information management apparatus has been mainly described as an example where the failure cause information and the traceback information are the same. The predetermined similarity condition may be other conditions.

  Further, in each of the embodiments of the present invention described above, the operation of the failure information management apparatus and each node described with reference to each flowchart is stored in the storage device (storage medium) of the computer apparatus as the computer program of the present invention. In addition, the computer program may be read and executed by the CPU. In such a case, the present invention is constituted by the code of the computer program or a storage medium.

  Moreover, each embodiment mentioned above can be implemented in combination as appropriate.

  The present invention is not limited to the above-described embodiments, and can be implemented in various modes.

A part or all of each of the above-described embodiments can be described as in the following supplementary notes, but is not limited thereto.
(Appendix 1)
A failure information storage unit that stores failure information related to failures that occurred during process execution on each node that executes distributed distributed programs in a distributed manner;
A failure information receiving unit for receiving the failure information transmitted in response to the occurrence of the failure by the node;
A similarity determination unit that determines whether failure information satisfying a predetermined similarity condition with respect to failure information received from the node is stored in the failure information storage unit;
In response to a determination result by the similarity determination unit, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage device used by the process at the time of the failure in the node is output to the node An output propriety information transmitting unit to perform,
When it is determined that failure information satisfying a predetermined similarity condition with respect to the failure information received from the node is not stored in the failure information storage unit, the failure information received from the node is changed to the failure information. A failure information registration unit to be registered in the storage unit;
Fault information management device comprising:
(Appendix 2)
When the failure information includes traceback information representing a history of instructions up to the occurrence of the failure,
The failure information management apparatus according to appendix 1, wherein the similarity determination unit includes that the traceback information is the same in the predetermined similarity condition.
(Appendix 3)
When the failure information includes failure cause information indicating the cause of the failure,
The failure information management apparatus according to appendix 1 or appendix 2, wherein the similarity determination unit includes that the failure cause information is the same in the predetermined similarity condition.
(Appendix 4)
When the failure information includes identification information of the node where the failure has occurred,
The failure information registration unit stores in the failure information storage unit when it is determined that failure information satisfying a predetermined similarity condition with respect to the failure information received from the node is stored in the failure information storage unit The failure information according to any one of appendix 1 to appendix 3, wherein the fault identification information is updated so as to include the node identification information included in the fault information received from the node. Management device.
(Appendix 5)
When the failure information includes identification information of the process in which the failure has occurred,
The failure information registration unit stores in the failure information storage unit when it is determined that failure information satisfying a predetermined similarity condition with respect to the failure information received from the node is stored in the failure information storage unit The failure information according to any one of appendix 1 to appendix 4, wherein the fault identification information is updated to include process identification information included in the failure information received from the node. Management device.
(Appendix 6)
The failure information management apparatus according to any one of appendix 1 to appendix 5, further comprising a summary information output unit that outputs the failure information stored in the failure information storage unit as summary information.
(Appendix 7)
An information processing apparatus that operates as a node that can execute a distributed parallel program in a distributed manner,
A process execution unit that executes processing based on the distributed parallel program;
When a failure occurs during process execution by the process execution unit, a failure information collection unit that collects failure information related to the failure;
A failure information transmitting unit that transmits the failure information collected by the failure information collecting unit to the failure information management device according to any one of appendix 1 to appendix 6,
A failure analysis information output unit that outputs failure analysis information including the contents of the main storage device used by the process when the output permission information received from the failure information management device indicates that output is possible;
An information processing apparatus comprising:
(Appendix 8)
The failure information management device according to any one of appendix 1 to appendix 6,
The information processing apparatus according to appendix 7,
Distributed parallel processing system.
(Appendix 9)
The distributed parallel processing system according to appendix 8, further comprising a shared information storage device that stores the failure analysis information output from the information processing device.
(Appendix 10)
When a failure occurs during process execution on a node that executes a distributed parallel processing program in a distributed manner,
The failed node is
Collect failure information about the failure,
Sending the failure information to a failure information management device;
The failure information management device includes:
Determine whether failure information satisfying a predetermined similarity condition with respect to the received failure information is stored in the failure information storage unit,
When failure information satisfying a predetermined similarity condition is not stored, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage used by the process at the time of the failure at the node is output to the node And the failure information received from the node is registered in the failure information storage unit,
When failure information satisfying a predetermined similarity condition is stored, output enable / disable information indicating output failure of the failure analysis information is transmitted to the node,
The failure information management method in which the node in which the failure has occurred outputs the failure analysis information when the output availability information received from the failure information management apparatus indicates that output is possible.
(Appendix 11)
Using a failure information storage unit that stores failure information related to failures that occur during process execution on each node that executes distributed parallel programs in a distributed manner.
A failure information receiving step of receiving the failure information transmitted in response to the occurrence of the failure by the node;
A similarity determination step of determining whether or not failure information satisfying a predetermined similarity condition with respect to failure information received from the node is stored in the failure information storage unit;
In accordance with the determination result of the similarity determination step, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage device used by the process at the time of the failure occurrence in the node is transmitted to the node. An output propriety information transmission step;
When it is determined that failure information satisfying a predetermined similarity condition with respect to failure information received from the node is not stored in the failure information storage unit, failure information received from the node is stored in the failure information storage Failure information registration step to be registered in the department,
Is a computer program that causes a computer device to execute.
(Appendix 12)
A process execution step for executing processing based on a distributed parallel program;
If a failure occurs during process execution in the process execution step, a failure information collection step for collecting failure information related to the failure;
A failure information transmission step of transmitting the failure information collected in the failure information collection step to an apparatus that executes the computer program according to attachment 11,
When the output availability information received from the device indicates that output is possible, a failure analysis information output step for outputting failure analysis information including the contents of the main storage device used by the process;
Is a computer program that causes a computer device to execute.
(Appendix 13)
Receiving from the node fault information related to a fault that has occurred during process execution in a node that executes the distributed parallel program in a distributed manner;
Determining whether failure information satisfying a predetermined similarity condition with respect to failure information received from the node is stored in the failure information storage unit;
According to the determination result, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage device used by the process at the time of the failure at the node is output to the node,
If it is determined that failure information satisfying a predetermined similarity condition with respect to the failure information received from the node is not stored in the failure information storage unit, the failure information received from the node is stored in the failure information storage unit. The failure information management method to be registered.
(Appendix 14)
If a failure occurs during process execution based on a distributed parallel program, the failure information related to the failure is collected.
The collected failure information is transmitted to a device that executes the failure information management method described in Appendix 13,
A failure information management method for outputting failure analysis information including contents of a main storage device used by the process when the output permission information received from the device indicates that output is possible.

DESCRIPTION OF SYMBOLS 1, 2 Distributed parallel processing system 10, 30 Fault information management apparatus 11 Fault information storage part 12 Fault information receiving part 13 Similarity judgment part 14 Output availability information transmission part 15, 35 Fault information registration part 36 Summary information output part 20, 40 Node 21 Process execution unit 22, 42 Fault information collection unit 23 Fault information transmission unit 24, 44 Fault analysis information output unit 50 Shared information storage device 90 Inter-node connection device

Claims (10)

A failure information storage unit that stores failure information related to failures that occurred during process execution on each node that executes distributed distributed programs in a distributed manner;
A failure information receiving unit for receiving the failure information transmitted in response to the occurrence of the failure by the node;
A similarity determination unit that determines whether failure information satisfying a predetermined similarity condition with respect to failure information received from the node is stored in the failure information storage unit;
In response to a determination result by the similarity determination unit, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage device used by the process at the time of the failure in the node is output to the node An output propriety information transmitting unit to perform,
When it is determined that failure information satisfying a predetermined similarity condition with respect to the failure information received from the node is not stored in the failure information storage unit, the failure information received from the node is changed to the failure information. A failure information registration unit to be registered in the storage unit;
Fault information management device comprising: When the failure information includes traceback information representing a history of instructions up to the occurrence of the failure,
The failure information management apparatus according to claim 1, wherein the similarity determination unit includes the fact that the traceback information is the same in the predetermined similarity condition. When the failure information includes failure cause information indicating the cause of the failure,
The failure information management apparatus according to claim 1, wherein the similarity determination unit includes that the failure cause information is the same in the predetermined similarity condition. When the failure information includes identification information of the node where the failure has occurred,
The failure information registration unit stores in the failure information storage unit when it is determined that failure information satisfying a predetermined similarity condition with respect to the failure information received from the node is stored in the failure information storage unit 4. The update according to claim 1, wherein the failure information is updated to include additional node identification information included in the failure information received from the node. 5. Fault information management device.   The failure information management apparatus according to any one of claims 1 to 4, further comprising a summary information output unit that outputs the failure information stored in the failure information storage unit as summary information. An information processing apparatus that operates as a node that can execute a distributed parallel program in a distributed manner,
A process execution unit that executes processing based on the distributed parallel program;
When a failure occurs during process execution by the process execution unit, a failure information collection unit that collects failure information related to the failure;
A failure information transmission unit that transmits the failure information collected by the failure information collection unit to the failure information management device according to any one of claims 1 to 5,
A failure analysis information output unit that outputs failure analysis information including the contents of the main storage device used by the process when the output permission information received from the failure information management device indicates that output is possible;
An information processing apparatus comprising: The failure information management device according to any one of claims 1 to 5,
An information processing apparatus according to claim 6;
Distributed parallel processing system. When a failure occurs during process execution on a node that executes a distributed parallel processing program in a distributed manner,
The failed node is
Collect failure information about the failure,
Sending the failure information to a failure information management device;
The failure information management device includes:
Determine whether failure information satisfying a predetermined similarity condition with respect to the received failure information is stored in the failure information storage unit,
When failure information satisfying a predetermined similarity condition is not stored, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage used by the process at the time of the failure at the node is output to the node And the failure information received from the node is registered in the failure information storage unit,
When failure information satisfying a predetermined similarity condition is stored, output enable / disable information indicating output failure of the failure analysis information is transmitted to the node,
The failure information management method in which the node in which the failure has occurred outputs the failure analysis information when the output availability information received from the failure information management apparatus indicates that output is possible. Using a failure information storage unit that stores failure information related to failures that occur during process execution on each node that executes distributed parallel programs in a distributed manner.
A failure information receiving step of receiving the failure information transmitted in response to the occurrence of the failure by the node;
A similarity determination step of determining whether or not failure information satisfying a predetermined similarity condition with respect to failure information received from the node is stored in the failure information storage unit;
In accordance with the determination result of the similarity determination step, output enable / disable information indicating whether or not failure analysis information including the contents of the main storage device used by the process at the time of the failure occurrence in the node is transmitted to the node. An output propriety information transmission step;
When it is determined that failure information satisfying a predetermined similarity condition with respect to failure information received from the node is not stored in the failure information storage unit, failure information received from the node is stored in the failure information storage Failure information registration step to be registered in the department,
Is a computer program that causes a computer device to execute. A process execution step for executing processing based on a distributed parallel program;
If a failure occurs during process execution in the process execution step, a failure information collection step for collecting failure information related to the failure;
A failure information transmission step of transmitting the failure information collected in the failure information collection step to an apparatus that executes the computer program according to claim 9;
When the output availability information received from the device indicates that output is possible, a failure analysis information output step for outputting failure analysis information including the contents of the main storage device used by the process;
Is a computer program that causes a computer device to execute.

Images