JP2009176146A - Multi-processor system, failure detecting method and failure detecting program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multi-processor system capable of detecting failures occurring in all CPUs of the system. <P>SOLUTION: The multi-processor system includes a plurality of CPUs, where a WD daemon 11 detects failures using a watchdog timer. The system includes a primary storage device 2 for storing CPU information storing a CPU identifier operated when a standby task acquires the right of execution for each corresponding task and CPU moving rules as rules for a watchdog daemon to sequentially move and circulate the CPU, and a WD management part 12 which updates the CPU information corresponding to the task of the watchdog daemon based on the CPU moving rules and writes the updated CPU information in the primary storage device 2. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a multiprocessor system that detects a failure using a watchdog timer (WDT) for monitoring whether or not the system is operating normally, a failure detection method, and a failure detection program.

  Conventionally, in a computer system such as a server, the following method is generally used as an effective method for specifying the cause of a failure due to a bug in a program or the like. When a failure occurs due to a bug in the program, the OS (Operating System) has a dump function that outputs the memory contents used by the program at the time of the failure to a disk or the like as a “dump file”. Then, the cause of the failure is identified by analyzing the contents of the dump file with a dedicated tool or the like. The dump file also includes register information, task information, and stack information including the address of the program that was running when the failure was detected.

  In general computer systems, a watchdog timer is mounted as one means for confirming the normality of an operating program. Normally, a task on the OS called a watchdog daemon (hereinafter referred to as a WD daemon) clears the WDT counter every time it acquires the execution right of a CPU (Central Processing Unit). FIG. 21 is a diagram illustrating the movement of the WDT and the WD daemon during normal operation in the conventional technology.

  If a fault such as runaway occurs in the program, the WD daemon cannot acquire the right to execute the CPU, and clearing to the WDT is not performed. FIG. 22 is a diagram illustrating the movement of the WDT and the WD daemon when a failure occurs in the related art. If this clearing is not carried out for a certain period of time, WDT detects a failure as a watchdog timeout (hereinafter referred to as WD timeout), notifies the CPU with an interrupt, etc., and this dump file output process is triggered by this notification. The

  On the other hand, in recent years, a multiprocessor system that executes a plurality of tasks in parallel on a plurality of CPUs is frequently used as a technique for improving the processing capability of a computer system and executing various tasks. Even in a multiprocessor system, in order to confirm the normality of an operating program, failure detection by a WDT and a WD daemon on the OS is generally performed as in the above example.

  Further, as a technique for the occurrence of a failure in a computer system that enables load distribution by connecting a plurality of computers, for example, there is a technique described in Patent Document 1 below. In the following Patent Document 1, in a distributed system composed of a plurality of computers, when a failure occurs by detecting a failure of each computer, it is possible to determine a backup computer without inquiring of a host computer Is disclosed.

JP 62-072052 A

  However, on each CPU of a multiprocessor system, tasks perform parallel operations while repeatedly acquiring and releasing CPU execution rights, and when a task releases execution rights, which task executes next. Is determined by the scheduler, which is a function of the OS, and the scheduler switches to the next task. Usually, since the task stays on the same CPU as much as possible, after the CPU execution right is released, the next execution right will be scheduled to be executed on the same CPU. Since the above-mentioned WD daemon is also a kind of task, it normally runs on the same CPU. Therefore, the WD daemon operates on the same CPU every time the execution right is acquired, and clears the WDT.

  For this reason, when the conventional technology for detecting a failure using the WDT and the WD daemon is applied to a multiprocessor system, a runaway failure such as an infinite loop occurs on a CPU different from the CPU on which the WD daemon is operating. However, the WD daemon has a problem that since the operating CPU is normal, the WDT continues to be cleared, and the failure of the entire system cannot be detected.

  In the case of an OS having a load balance function, when a large number of tasks are biased on a certain CPU, the tasks may be forcibly moved to another CPU to perform load distribution. In this case, there is a high possibility that the WD daemon is assigned to a CPU other than the CPU in which the runaway failure has occurred. In this case, as in the above example, there is a problem that the failure of the entire system cannot be detected. . FIG. 23 is a diagram illustrating a case where WDT in the related art cannot detect a failure. Thus, even if the task of CPU # 2 runs away, if the WD daemon is operating on CPU # 0 or CPU # 1, WDT clears the counter and cannot detect a failure.

  Also, if the same number of WDTs as the CPUs are installed, it is possible to monitor failures that occur across all CPUs. However, the cost is high and dedicated hardware is installed. Sexuality is impaired. In recent years, the introduction of a server system that runs a general-purpose OS on a board equipped with a general-purpose CPU and supports various services with the same hardware has spread mainly in carriers and the like. If such systems are equipped with dedicated hardware for fault monitoring and fault information collection, the versatility will be impaired, making it difficult to introduce.

  The present invention has been made to solve the above-described problems caused by the prior art, and is a multiprocessor capable of detecting a failure occurring in all CPUs of the system without installing dedicated hardware. It is an object to provide a system, a failure detection method, and a failure detection program.

  In order to solve the above-described problems and achieve the object, the present invention includes a plurality of processors, and the watchdog daemon detects a failure using a watchdog timer and notifies the occurrence of the failure when the failure is detected. In this multiprocessor system, the processor information stored for each task corresponding to the identifier of the processor that operates when the waiting task acquires the execution right and the processor on which the watchdog daemon operates are moved sequentially. A processor movement rule, which is a rule for making a circulation, and updating the processor information corresponding to the task of the watchdog daemon based on the processor movement rule, and updating the updated processor information. And a watchdog management means for writing to the storage means. To.

  Further, the present invention relates to a watchdog history recording means for associating the identifier of the processor in which the watchdog daemon is operated and the time when the watchdog daemon clears the watchdog timer in the storage means in association with each other. When it is determined that there is a processor for which the watchdog daemon has not been operated for a predetermined time based on the watchdog activation history, a watch for notifying the occurrence of a failure indicating that the processor has failed And a dog activation monitoring means.

  The present invention further comprises watchdog daemon behavior specifying means for accepting a change request from a user for the processor movement rule, wherein the watchdog management means rewrites the processor movement rule based on the change request. Features.

  Further, the present invention collects program running information including register information including task instructions, task information, and stack information for a predetermined period when a failure occurrence is notified, and the collected program running information is stored in the storage means. It further comprises program running history recording means for writing to

  Further, the present invention is characterized in that the output destination device of the program travel history can be changed.

  According to the present invention, the operation processor of the WD daemon is moved according to a predetermined processor movement rule, so that the WD daemon can operate on all the processors evenly, and the multiprocessor is installed without installing dedicated hardware. Even if a failure occurs in any processor in the system, it is possible to detect a failure due to a WD timeout.

  Further, according to the present invention, the processor number and time at which WDT is cleared is stored as a WD activation history, and there is a processor that has not been cleared for a predetermined reference time based on the WD activation history. Since it is determined that a failure has occurred on the processor and a failure notification is issued, even if the OS has a load balancing function, all processors can be detected by detecting a processor that has not been operated for a long time. There is an effect that failure detection can be performed for the processor.

  In addition, according to the present invention, since the change instruction of the processor for the processor movement rule is accepted and the stored processor movement rule is rewritten based on the change instruction, the processor movement rule can be arbitrarily changed at any time, There is an effect that the user's request can be easily reflected.

  Further, according to the present invention, when there is a failure notification, register information including task execution instructions, task information, stack information, time information, and the like are collected and stored for a predetermined arbitrary time. As a result, unlike a normal dump file that stores information only when a failure occurs, it is possible to check information for a predetermined time, and even for failures that cause an infinite loop, stored program running history By analyzing the above, it is possible to easily identify the fault location.

  In addition, according to the present invention, the output destination of the dump file can be specified, so that it is possible to select more convenient output to the medium according to the usage status of the system, and an apparatus for outputting the dump file Even if there is a failure, it is possible to record a dump file by changing the output destination.

  Exemplary embodiments of a multiprocessor system, a failure detection method, and a failure detection program according to the present invention will be described below in detail with reference to the accompanying drawings. In the description of the present embodiment, the CPU will be described as an example of the processor, but the processor may be in other forms such as an MPU (Micro Processing Unit). The watchdog daemon and the watchdog timer may also have a function of monitoring the system operating state at a predetermined timing and detecting the occurrence of a failure.

  FIG. 1 is a diagram illustrating a functional configuration example of a first embodiment of the multiprocessor system according to the present invention. The multiprocessor system of the present embodiment includes a control unit 1 and a primary storage device 2 that are constituted by a plurality of CPUs (not shown). In addition, on the control unit 1, an OS is operated. As shown in FIG. 1, the multiprocessor system of this embodiment includes a scheduler 10 that assigns a CPU to be used for a task to be executed, and a WD daemon that clears a counter for the WDT each time a CPU execution right is acquired. 11, a WD management unit 12, and a dump file output unit 13 that generates a dump file when a failure is detected by the WD daemon 11. The output destination device 3 includes a magnetic disk, a memory, a display device, and the like, and is a device that is an output destination of the dump file generated by the dump file output unit 13.

  Here, the scheduler 10, the WD daemon 11, and the dump file output unit 13 are a part of the general functions of the OS, but the present invention is not limited to this, and means other than the OS that realizes the same function is used. It may be.

  The primary storage device 2 stores task information including CPU information, which is information about which CPU is used for each task, and a CPU movement rule in which a CPU movement rule assigned to the WD daemon 11 is stored. The

  Next, the operation of this embodiment will be described. Here, description will be made assuming that the multiprocessor system of this embodiment includes a total of four CPUs (CPU # 0 to CPU # 3). First, as a precondition, FIG. 2 shows state transitions of tasks operating on these CPUs. The “execution state” refers to a state in which the execution right of the CPU is passed by the scheduler 10 and the task is executing a process. The “standby state” refers to a state in which the processing of a task is completed, the CPU execution right is given to another task, and the computer is sleeping. The “execution waiting state” refers to a state in which a task is woken up from the standby state and enters an execution waiting queue prepared for each CPU. The WD daemon 11 is one of the tasks, and clears the counter to the WDT as a task process every time the CPU executes the execution state while performing the state transition of FIG.

  FIG. 3 is a diagram for explaining a task execution waiting queue prepared for each CPU. In the waiting queue, data is stored in a list as task information for each task. The wake-up task enters the tail of the execution queue (stored as tail task information). When a scheduling opportunity comes, the scheduler 10 gives the execution right of the CPU in preference to the task corresponding to the task information stored at the head (Head) of the queue. For example, in FIG. 3, the execution right is given in order from the task corresponding to the task information stored at the head (Head). Then, the execution right is given to the tasks corresponding to the order of the task information, and finally the execution right is given to the task information of the tail. The task that has finished processing on the CPU shifts to a standby state, is removed from the queue, and waits until the next execution trigger.

  Based on the above, the processing procedure of the present embodiment will be described. FIG. 4 is a flowchart illustrating an example of a processing procedure according to the present exemplary embodiment. FIGS. 5-1, 5-2, 5-3, and 5-4 are first, second, third, and fourth conceptual diagrams for explaining the operation of the present embodiment, respectively. First, it is assumed that CPU movement rules are determined in advance and stored in the primary storage device 2. Here, as an example, it is assumed that a rule of “move to round robin format and descending order of CPU numbers (CPU # 3 → CPU # 2 → CPU # 1 → CPU # 0)” is set.

  It is assumed that the task A executed on the CPU # 3 finishes the process at a certain point and the scheduler 10 assigns the execution right of the CPU # 3 to the WD daemon 11 (step S11). Specifically, as shown in the first conceptual diagram shown in FIG. 5A, after the task processing of the task A is completed, the task A is removed from the execution waiting queue of the CPU # 3 and enters the standby state. The execution right is given by the scheduler 10 to the task of the WD daemon 11 that is the head of the execution waiting queue # 3. Then, the WD daemon 11 requests clearing of the WDT counter (step S12).

  When the counter clearing is completed by executing the WDT counter clear request, the WD daemon notifies the WD management unit 12 that the WDT clear process has been performed on the CPU # 3 (step S13). Receiving the notification, the WD management unit 12 reads the CPU migration rule of the primary storage device, and determines the CPU to execute the task of the next WD daemon 11 based on the CPU migration rule (step S14). In this case, since the CPU movement rule is “round robin format and descending order of CPU numbers”, the CPU to be executed when the WD daemon 11 is started next time is determined to be CPU # 2.

  Next, the WD management unit 12 retrieves and reads out the task information of the WD daemon 11 from the task information group corresponding to each task, and refers to the CPU information area included in the read task information. At this time, since the task of the WD daemon 11 is executed by the CPU # 3, a numerical value (here, “3”) indicating the CPU # 3 is recorded in the referred CPU information area. The WD management unit 12 rewrites the CPU information area to a numerical value (“2”) indicating the CPU # 2 determined in step S14 (step S15). When the rewriting in step S15 is completed, the WD management unit 12 notifies the WD daemon 11 of the completion of the change of the CPU information (step S16), and the WD daemon 11 changes to a standby state. Then, as shown in the second conceptual diagram of FIG. 5B, the WD daemon 11 is removed from the execution waiting queue of the CPU # 3, and the next task B acquires the execution right of the CPU by the scheduler 10, and task processing To start.

  When the waiting WD daemon 11 is woken up after a certain period of time, the task of the WD daemon 11 enters the execution waiting queue again as described above. At this time, the task information in the task information of the WD daemon 11 is included. Since the CPU information of “# 2” has been rewritten to “2”, the WD daemon 11 automatically enters the execution waiting queue of CPU # 2 as shown in the third conceptual diagram of FIG. Stored as task information). As a result, as shown in the fourth conceptual diagram of FIG. 5-4, when the task of the WD daemon 11 moves to the head of the queue after the elapse of time, the execution right of the CPU # 2 is passed by the scheduler, and the above-mentioned As in step S12, the clearing of the WDT counter is requested. Thereafter, the processing after step S13 is performed, but the CPU determined in step S14 is CPU # 1 in accordance with the CPU movement rule.

  The CPU in which the WD daemon 11 operates (repeating the WDT counter) while repeating such operations is as follows: CPU # 3 → CPU # 2, CPU # 2 → CPU # 1, CPU # 1 → CPU # 0, During the normal operation of the system, such as CPU # 0 → CPU # 3, it moves so that it can travel evenly among all CPUs according to the round robin format.

  Next, the operation when a failure occurs will be described. FIG. 6 is a diagram for explaining the operation when a failure occurs in this embodiment. According to the above-described processing procedure of the present embodiment (hereinafter referred to as CPU movement processing), CPU # 3 at time T1, past CPU # 2 at time T2, CPU # 1 at time T3, and time T4 at time T2. It is assumed that the WDT counter is cleared by the WD daemon 11 on the CPU # 0. Assume that a failure due to an infinite loop or the like has occurred in task D being executed on CPU # 3 after time T4. Since the WD daemon 11 is activated by the CPU # 0 at time T4, the CPU information area of the task information is rewritten to 3 at the time of occurrence of the failure.

  However, even if the task of the WD daemon 11 comes to the head of the CPU # 3 waiting queue, the task D continues to run away while acquiring the execution right of the CPU # 3. I have to keep waiting. If the WDT is not cleared for a certain period after time T4, the WDT detects a WD timeout and notifies the OS of an interrupt (the cause of the interrupt is a WD timeout). When an interrupt is notified, the OS regards it as a failure occurrence trigger, and the dump file output unit 13 performs a process of outputting memory information for all CPUs to the dump file. Here, an example in which a failure has occurred in CPU # 3 has been described, but even if a failure has occurred in another CPU, any of the WD daemon 11 will be queued in the execution queue of the CPU that has failed due to the above-described CPU movement process. Since a task is entered, the occurrence of a failure can be detected.

  In this embodiment, the CPU movement rule is moved to the round robin method according to the descending order of the CPU numbers. However, the present invention is not limited to this. Any rule may be used as long as it is a rule that makes a round of all CPUs during a certain period, such as a system.

  As described above, in this embodiment, the operation CPU of the WD daemon is moved according to a predetermined CPU movement rule. For this reason, it is possible to detect a failure due to a WD timeout regardless of which CPU in the multiprocessor system has a failure without mounting dedicated hardware.

  FIG. 7 is a diagram illustrating an example of a functional configuration of the multiprocessor system according to the second embodiment of the present invention. As shown in FIG. 7, in the multiprocessor system of the present embodiment, a WD history recording unit 14 and a WD activation monitoring unit 15 are added to the multiprocessor system of the first embodiment. It is the same. Components having the same functions as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and description thereof is omitted.

  The OS of this embodiment has a load balance function that forcibly moves from a CPU with a large number of tasks in the execution queue to a CPU with a small number of tasks in the CPU. Assume. The CPU movement process of the present embodiment is the same as that of the first embodiment, and only the parts different from the first embodiment are described below.

  During normal operation, as in the first embodiment, the WD daemon 11 clears the WDT counter while moving the operating CPU according to the CPU movement rule. Here, as in the first embodiment, the WDT is cleared while the CPU movement rule is moved in a round-robin manner according to the descending order of the CPU numbers.

  FIGS. 8A and 8B are first and second conceptual diagrams for explaining the operation when a failure occurs according to the present embodiment. First, as shown in FIG. 8A, according to the above-described CPU movement rule, the WD daemon is on CPU # 3 at the past time T1, on CPU # 2 at time T2, and on CPU # 1 at time T3. 11, the WDT counter is cleared. At this point (after time T3), it is assumed that a failure due to an infinite loop has occurred in task D being executed on CPU # 3. When the WD daemon 11 wakes up from the standby state, the CPU information area of the task information is changed to “0”, so the task of the WD daemon 11 enters the execution waiting queue of the CPU # 0.

  For this reason, the WD daemon 11 should next clear the WDT counter on the CPU # 0. However, since the OS has a load balancing function in this embodiment, if the OS determines that there are too many tasks in the execution waiting queue of the CPU # 0, the OS is as shown in FIG. In addition, a part of the task in the execution waiting queue of CPU # 0 is forcibly moved to the execution waiting queue of another CPU. The WD daemon 11 is also subject to load balancing. Here, it is assumed that the task of the WD daemon 11 has been moved from the CPU # 0 to the execution waiting queue of the CPU # 2.

  With the above processing by the load balance function, the WD daemon 11 then clears the WDT counter on the CPU # 2, and thereafter, the CPU movement in the descending order of the CPU number is continued again. In such a situation where the forced execution waiting queue movement by the load balance function is repeated many times, the WD daemon 11 may not be able to enter the execution waiting queue of the CPU # 3 in some cases. . In this case, since the WD timeout does not occur, it cannot be detected that the task D is running out of control on the CPU # 3.

  In order to cope with such a case, in this embodiment, the CPU number for which the WDT counter clear process (hereinafter referred to as WDT clear) is performed is stored as a WD activation history, and CPUs that have not been cleared for a predetermined time are stored. It can be detected. FIG. 9 is a flowchart illustrating an example of a processing procedure of the present embodiment. First, after the WD management unit 12 rewrites the CPU information area of the task information of the WD daemon 11 (step S16 in the first embodiment), the WD management unit 12 performs the CPU number (WD) for which the WD history recording unit 14 has performed WDT clear. The CPU number on which the daemon 11 operates is notified (step S21).

  Receiving the notification, the WD history recording unit 14 obtains the current time from the system clock and writes the WD activation history in the primary storage device 2 (step S22). FIG. 10 is a diagram showing an example of table information written as a WD activation history. This table includes at least the current time and the CPU number that cleared WDT.

  When the writing to the WD activation history is completed, the WD history recording unit 14 activates the WD activation monitoring unit 15 (step S23). The WD activation monitoring unit 15 refers back to the WD activation history for a certain past time, and the period during which the WD daemon 11 is not activated among the CPUs # 0 to # 3 is a predetermined reference time (for example, 120 seconds). It is determined whether there is a CPU exceeding () (step S24). If it is determined that there is a CPU for which the WD daemon 11 has not been activated for longer than the reference time (Yes in step S24), the WD activation monitoring unit 15 determines that a failure has occurred on the CPU, and notifies the occurrence of the failure. (Step S25). Upon receiving this notification, the dump file output unit 13 generates a dump file and outputs the dump file to the output destination device 3 in the same manner as when receiving the notification of the occurrence of the WD timeout (step S26).

  FIG. 11 shows a third conceptual diagram for explaining the operation of this embodiment. As in FIG. 8A, in a situation where the task D is running out of control such as an infinite loop on the CPU # 3, the WD daemon 11 clears the WDT while moving the operating CPU. Since processing by the load balance function frequently occurs, it is assumed that the situation where the WD daemon 11 is activated only within the range of CPU # 0 to CPU # 2 continues for a while. In this example, the WD daemon 11 is not operating on the CPU # 3 from T3 when the WD daemon 11 is operated on the CPU # 1 to T11 when m seconds as the reference time has elapsed. FIG. 12 is a diagram illustrating an example of the WD activation history of the present embodiment. FIG. 12 is an example of the WD activation history at the time T11 in the example described in FIG. Thus, since it can be determined that the WD daemon has not been activated on the CPU # 3 once in m seconds from T3 to T11 (reference time: for example, 120 seconds) based on the WD activation history, Assuming that a failure has occurred on CPU # 3, WD activation monitoring unit 15 issues a failure notification.

  As described above, in this embodiment, the CPU number and the time at which the WD history recording unit 14 has performed WDT clearing are stored in the primary storage device 2 as the WD activation history, and the WD activation monitoring unit 15 is based on the WD activation history. When there is a CPU that has not been WDT cleared for a predetermined reference time, it is determined that a failure has occurred on that CPU and a failure notification is issued. For this reason, in a system in which the OS has a load balance function, even if any CPU in the multiprocessor system has a failure, failure detection by WD timeout can be performed.

  FIG. 13 is a diagram illustrating an example of a functional configuration of the multiprocessor system according to the third embodiment of the present invention. As shown in FIG. 13, the multiprocessor system of this embodiment replaces the WD management section 12 of the multiprocessor system of the first embodiment with the WD management section 12a and adds a WD behavior specifying section 16, but otherwise Is the same as in Example 1. Components having the same functions as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and description thereof is omitted.

  In the present embodiment, the WD behavior specifying unit 16 is added in order for the user to specify the CPU movement rule, and the WD management unit 12a rewrites the CPU movement rule of the primary storage device 2 based on the user's specification.

  FIG. 14 is a flowchart illustrating an example of a processing procedure according to this embodiment. FIG. 15 is a diagram illustrating the flow of the CPU on which the WD daemon 11 of this embodiment operates. First, similarly to the first embodiment, it is assumed that CPU movement rules are determined in advance and stored in the primary storage device 2. Here, it is assumed that random circulation (repeating movement of all CPUs in a random order) is set as a predetermined CPU movement rule. Then, it is assumed that the process based on the CPU movement rule is performed by the CPU movement process described in the first embodiment. In this state, the WD daemon 11 performs random patrol as shown in the period (a) of FIG.

  At this time, it is assumed that the user wants to change the CPU movement rule to a rule of “move to the round robin format and descending CPU number”. In this case, the WD behavior specifying unit 16 accepts a user instruction (in this example, an instruction to change to the rule of “round robin format and move to descending CPU number”) and notifies the WD management unit 12a of the instruction content ( Step S31). The instruction from the user is performed via an input device such as a keyboard and a mouse (not shown). Next, the WD management unit 12a that receives the notification rewrites the CPU migration rule on the primary storage device 2 based on the contents of the instruction (step S32).

  After this processing, the WD management unit 12a performs the CPU assignment for the WD daemon 11 based on the rule of “moving in a round robin format and descending CPU number” changed by the user (period (b) in FIG. )). There is no particular restriction on the timing of accepting the change of the CPU movement rule by the user.

  In the present embodiment, the WD management specifying section 16 of the multiprocessor system of the second embodiment is added by replacing the WD management section 12 of the multiprocessor system of the first embodiment with the WD management section 12a. 12 may be replaced with the WD management unit 12a, and a WD behavior specifying unit 16 may be added to perform a process of reflecting the user's change instruction for the above-described CPU movement rule.

  As described above, in this embodiment, the WD behavior specifying unit 16 receives a user change instruction for the CPU movement rule, and the WD management unit 12a rewrites the CPU movement rule stored in the primary storage device 2 based on the change instruction. I did it. For this reason, the CPU movement rule can be arbitrarily changed at any time.

  FIG. 16 is a diagram illustrating an example of a functional configuration of the multiprocessor system according to the fourth embodiment of the present invention. As shown in FIG. 16, in the multiprocessor system of this embodiment, a program running history recording unit 17 is added to the multiprocessor system of the first embodiment, and the program running history is further stored in the primary storage device 2. However, the rest is the same as in the first embodiment. Components having the same functions as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and description thereof is omitted.

  There are cases where it is difficult to investigate the cause of the failure from the dump file output by the conventional WD timeout detection. If the cause of the failure is an illegal memory access or logical contradiction that occurred in the program, the cause of the failure is identified from the information in the dump file because the address of the program that was running when the failure occurred is the failure location itself. Things are relatively easy. On the other hand, if an infinite loop occurs in the program and a WD timeout is detected, the CPU execution address information obtained from the dump file runs at the moment the WD timeout occurs in the looped address range. It was just the address I had. Therefore, it is impossible to identify the cause of the loop within the program and what caused the loop from the information remaining in the dump file. There was a problem that it was difficult to investigate. Therefore, in the case of such a failure, recording the running information of the program for each execution command can be effective information for analysis. However, it is not practical to always record such traveling information even during normal operation because it places a great load on the system.

  In order to solve the above problem, in this embodiment, a function of collecting a running history of a program for a certain period of time from the time when a failure is detected is added. The running history of a program generally includes information similar to the contents output as a dump file (for example, register information including program execution instructions, task information, stack information, time information, etc.) and information on each instruction acquired from the system. Execution time information is also included.

  Next, the operation of this embodiment will be described. FIG. 17 is a flowchart illustrating an example of a processing procedure according to this embodiment. FIG. 18 is a diagram illustrating a processing concept before and after the occurrence of a failure according to the present embodiment. First, in the normal state, the multiprocessor system of the present embodiment performs the CPU movement process according to the CPU movement rule as in the first embodiment ((a) normal operation period in FIG. 18).

  At this time, it is assumed that a program runaway or the like has occurred, and the WDT detects a WD timeout and notifies the occurrence of the failure in the same manner as the operation when the failure occurs in the first embodiment (step S41). Upon receipt of the failure occurrence notification, the OS normally collects information necessary for the dump file, and the dump file output unit 13 generates the dump file. That is, since the failure occurrence notification is a failure information collection trigger, hereinafter, the failure occurrence notification is referred to as a failure information collection trigger.

  In general, the OS that has received the failure occurrence trigger immediately starts processing to collect the contents of the memory at that time as information for outputting as a dump file. On the other hand, in this embodiment, first, when a failure occurrence trigger occurs, the OS notifies the program running history recording unit 17 of the occurrence of the failure, and the received program running history recording unit 17 is executed by each CPU. Register information including task execution instructions, task information, stack information (items similar to those normally output to a dump file) and instruction execution time information are collected and stored in the primary storage device 2 as a program running history Store (step S42 in FIG. 17, (c) travel information collection for each CPU in FIG. 18). This collection and storage continues for a predetermined time (5 seconds in the example of FIG. 18).

  Then, when an arbitrary predetermined time has elapsed, normal dump file output processing is performed (step S43). Other operations in the present embodiment are the same as those in the first embodiment.

  In this embodiment, the program running history recording unit 17 is added to the multiprocessor system of the first embodiment. However, the program running history recording unit 17 is added to the multiprocessor system of the second embodiment or the multiprocessor system of the third embodiment. As in the present embodiment, the program running history may be collected and stored in the primary storage device 2 when a failure collection trigger occurs.

  As described above, in this embodiment, when there is a failure occurrence notification, the program running history recording unit 17 stores register information including a program execution instruction, task information, stack information, time information, etc. It was collected during the time and stored in the primary storage device 2 as a program running history. For this reason, it is possible to easily identify the fault location by analyzing the stored program running history even for a fault that causes an infinite loop.

  FIG. 19 is a diagram illustrating a functional configuration example of the fifth embodiment of the multiprocessor system according to the present invention. As shown in FIG. 19, in the multiprocessor system of the present embodiment, a dump file output destination designating unit 18 is added to the multiprocessor system of the fourth embodiment. Otherwise, the multiprocessor system of the fourth embodiment is the same as the multiprocessor system of the fourth embodiment. It is the same. Components having functions similar to those of the fourth embodiment are denoted by the same reference numerals as those of the fourth embodiment, and description thereof is omitted.

  In the present embodiment, the output destination device 3 includes a disk 31 formed of a magnetic disk, a packet processing device 32 that outputs a packet and transfers it to the network, and a standard output device that includes a monitor and performs display. 33 and a memory 34 composed of a semiconductor or the like.

  FIG. 20 is a flowchart illustrating an example of a processing procedure according to this embodiment. The operation of the present embodiment is the same as the operation of the fourth embodiment, but in this embodiment, the dump file output in the fourth embodiment and the output destination of the program running information can be selected. First, it is assumed that the occurrence of a failure is notified as in the fourth embodiment (step S51). Thereafter, similarly to the fourth embodiment, the program running history recording unit 17 executes step S42 of the fourth embodiment, and then instructs the dump file output unit 13 to output a dump file (step S52). The dump file output unit 13 outputs the program running information and dump information (information collected as dump file information) to the dump file output destination designating unit 18 (step S53). Then, the dump file output destination designation unit 18 outputs the program running information and the dump file based on the designation of the output destination set in advance by the user (step S54). For example, the output device 3 outputs the data to any of the disk 31, the packet processing device 32, the standard output device 33, and the memory 34. Note that the dump file output destination designation unit 18 holds a preset output destination, and rewrites the setting contents when designated by the user.

  In the present embodiment, both the program travel history information and the normal dump file are output to the output destination device 3, but only the program travel history information may be output.

  As described above, in this embodiment, the output destination of the dump file can be specified. Therefore, the dump file can be output in a form that is easy for the user to use. Also, for example, if there is a failure in the device that outputs the dump file, the dump file cannot be recorded normally, but in this embodiment, the dump file can be recorded by changing the output destination. Become.

(Appendix 1) A multiprocessor system comprising a plurality of processors, wherein a watchdog daemon detects a failure using a watchdog timer, and notifies a failure when a failure is detected,
This is a rule for moving the processor where the identifier of the processor that operates when the waiting task acquires the execution right for each corresponding task and the processor on which the watchdog daemon operates to move around. Storage means for storing processor movement rules;
Watchdog management means for updating the processor information corresponding to the task of the watchdog daemon based on the processor movement rule, and writing the updated processor information in the storage means;
A multiprocessor system comprising:

(Appendix 2) Watchdog history recording means for storing in the storage means the identifier of the processor in which the watchdog daemon has been operated and the time when the watchdog daemon has cleared the watchdog timer in association with each other, and storing it as a watchdog activation history;
When it is determined that there is a processor for which the watchdog daemon has not been operated for a predetermined time based on the watchdog activation history, a watchdog for notifying the occurrence of a failure indicating that the processor has failed Startup monitoring means;
The multiprocessor system according to appendix 1, further comprising:

(Appendix 3) Watchdog daemon behavior designation means for accepting a change request from the user for the processor movement rule,
Further comprising
The multiprocessor system according to appendix 1 or 2, wherein the watchdog management unit rewrites the processor movement rule based on the change request.

(Supplementary Note 4) When a failure notification is made, register running information including execution instructions, task information, and program running information including stack information are collected for a predetermined period, and the collected program running information is written to the storage means Program running history recording means,
The multiprocessor system according to appendix 1, 2, or 3, further comprising:

(Supplementary note 5) The multiprocessor system according to supplementary note 4, wherein the output destination device of the program running history can be changed.

(Appendix 6) A failure detection method in a multiprocessor system comprising a plurality of processors, wherein a watchdog daemon detects a failure using a watchdog timer and notifies the occurrence of a failure when a failure is detected,
A processor movement rule storage step for storing a processor movement rule that is a rule for sequentially moving and circulating the processors on which the watchdog daemon operates;
Based on the processor movement rule, the processor information corresponding to the task of the watchdog daemon among the processor information stored for each task corresponding to the identifier of the processor that operates when the waiting task acquires the execution right Watchdog management steps to update,
A failure detection method comprising:

(Appendix 7) A watchdog history recording step for storing the identifier of the processor in which the watchdog daemon has been operated and the time when the watchdog daemon has cleared the watchdog timer in association with each other and storing it as a watchdog activation history;
When it is determined that there is a processor for which the watchdog daemon has not been operated for a predetermined time based on the watchdog activation history, a watchdog for notifying the occurrence of a failure indicating that the processor has failed A startup monitoring step;
The failure detection method according to appendix 6, further comprising:

(Appendix 8) Watchdog daemon behavior designation step for accepting a change request from a user for the processor movement rule;
Rewriting the processor migration rule based on the change request;
The failure detection method according to appendix 6 or 7, further comprising:

(Supplementary Note 9) When a failure occurrence is notified by detecting a failure by the watchdog daemon, or when a failure occurrence is notified by the watchdog activation monitoring step, register information including an execution instruction, a task Program running history recording step for collecting program running information including information and stack information for a predetermined period and recording the collected program running information;
The failure detection method according to appendix 6, 7 or 8, further comprising:

(Supplementary note 10) The failure detection method according to supplementary note 9, wherein the output destination device of the program running history can be changed.

(Supplementary Note 11) A multi-processor system including a plurality of processors, wherein a watchdog daemon detects a fault using a watchdog timer, and a fault detection program for detecting a fault,
A processor movement rule storage procedure for storing a processor movement rule, which is a rule for sequentially moving and circulating the processor on which the watchdog daemon operates, in the storage unit;
The processor movement rule is read from the storage unit, and the identifier of the processor that operates when the waiting task acquires the execution right corresponds to the task of the watchdog daemon among the processor information stored for each corresponding task. A watchdog management procedure for reading processor information from the storage unit, updating the read processor information based on the processor movement rule, and writing the updated processor information to the storage unit;
A failure detection program for causing a computer to execute

(Appendix 12) A watchdog history recording procedure in which the identifier of the processor in which the watchdog daemon has been operated and the time when the watchdog daemon clears the watchdog timer are associated with each other and stored in the storage unit as a watchdog activation history;
When the watchdog activation history is read from the storage unit, and it is determined that there is a processor that has not operated the watchdog daemon for a predetermined time based on the read watchdog activation history, a failure has occurred in that processor Watchdog activation monitoring procedure to notify the occurrence of a failure indicating
The failure detection program according to appendix 11, further comprising:

(Supplementary note 13) Watchdog daemon behavior designation procedure for accepting a change request from the user to the processor movement rule;
Rewriting the processor movement rule based on the change request;
The failure detection program according to appendix 11 or 12, further comprising:

(Supplementary Note 14) Register information including an execution instruction and task when a failure occurrence is notified by detecting a failure by the watchdog daemon or when a failure occurrence is notified by the watchdog activation monitoring procedure Program running history recording procedure for collecting information, program running information including stack information for a predetermined period, and writing the collected program running information to the storage unit,
The failure detection program according to appendix 11, 12 or 13, further comprising:

(Supplementary note 15) The failure detection program according to supplementary note 14, wherein an output destination device of the program running history can be changed.

  As described above, the multiprocessor system, the failure detection method, and the failure detection program according to the present invention are suitable for a computer system having a plurality of processors and having a failure detection function using WDT.

It is a figure which shows the function structural example of Example 1 of the multiprocessor system concerning this invention. It is a figure which shows the state transition of a task. It is a figure for demonstrating the task waiting queue. 3 is a flowchart illustrating an example of a processing procedure according to the first exemplary embodiment. FIG. 3 is a first conceptual diagram for explaining the operation of the first embodiment. FIG. 6 is a second conceptual diagram for explaining the operation of the first embodiment. FIG. 6 is a third conceptual diagram for explaining the operation of the first embodiment. FIG. 9 is a fourth conceptual diagram for explaining the operation of the first embodiment. FIG. 6 is a diagram for explaining an operation when a failure occurs according to the first embodiment. It is a figure which shows the function structural example of Example 2 of the multiprocessor system concerning this invention. FIG. 10 is a first conceptual diagram for explaining an operation when a failure occurs in the second embodiment. FIG. 10 is a second conceptual diagram for explaining an operation when a failure occurs according to the second embodiment. 10 is a flowchart illustrating an example of a processing procedure according to the second embodiment. It is a figure which shows an example of the table information written as a WD starting log | history. FIG. 10 is a third conceptual diagram for explaining the operation of the second embodiment. It is a figure which shows an example of the WD starting log | history of Example 2. FIG. It is a figure which shows the function structural example of Example 3 of the multiprocessor system concerning this invention. 10 is a flowchart illustrating an example of a processing procedure according to the third embodiment. It is a figure which shows the flow of CPU which the WD daemon of Example 3 operate | moves. It is a figure which shows the function structural example of Example 4 of the multiprocessor system concerning this invention. 10 is a flowchart illustrating an example of a processing procedure according to a fourth embodiment. It is a figure which shows the processing concept before and behind the failure generation of Example 4. It is a figure which shows the function structural example of Example 5 of the multiprocessor system concerning this invention. 10 is a flowchart illustrating an example of a processing procedure according to a fifth embodiment. It is a figure which shows the motion of WDT and WD daemon at the time of normal operation in a prior art. It is a figure which shows the motion of WDT and WD daemon at the time of the failure occurrence in a prior art. It is a figure which shows the case where WDT in a prior art cannot detect a failure.

Explanation of symbols

1 Control Unit 2 Primary Storage Device 3 Output Destination Device 10 Scheduler 11 WD Daemon 12, 12a WD Management Unit 13 Dump File Output Unit 14 WD History Recording Unit 15 WD Activation Monitoring Unit 16 WD Behavior Designation Unit 17 Program Running History Recording Unit 18 Dump File output destination designation unit 31 Disk 32 Packet processing device 33 Standard output device 34 Memory

Claims (7)

A multiprocessor system comprising a plurality of processors, wherein a watchdog daemon detects a failure using a watchdog timer, and notifies a failure when a failure is detected,
This is a rule for moving the processor where the identifier of the processor that operates when the waiting task acquires the execution right for each corresponding task and the processor on which the watchdog daemon operates to move around. Storage means for storing processor movement rules;
Watchdog management means for updating the processor information corresponding to the task of the watchdog daemon based on the processor movement rule, and writing the updated processor information in the storage means;
A multiprocessor system comprising: Watchdog history recording means for associating the storage means with the identifier of the processor on which the watchdog daemon has been operated and the time when the watchdog daemon has cleared the watchdog timer, and storing it as a watchdog activation history;
When it is determined that there is a processor for which the watchdog daemon has not been operated for a predetermined time based on the watchdog activation history, a watchdog for notifying the occurrence of a failure indicating that the processor has failed Startup monitoring means;
The multiprocessor system according to claim 1, further comprising: A watchdog daemon behavior specifying means for accepting a change request from a user to the processor movement rule;
Further comprising
The multiprocessor system according to claim 1, wherein the watchdog management unit rewrites the processor movement rule based on the change request. When a failure is notified, register running information including execution instructions, task information, and program running information including stack information are collected for a predetermined period after the occurrence of the failure, and the collected program running information is stored in the memory. Program running history recording means for writing to the means,
The multiprocessor system according to claim 1, further comprising:   5. The multiprocessor system according to claim 4, wherein an output destination device of the program running history can be changed. A failure detection method in a multiprocessor system comprising a plurality of processors, wherein a watchdog daemon detects a failure using a watchdog timer, and notifies a failure when a failure is detected,
A processor movement rule storage step for storing a processor movement rule which is a rule for sequentially moving and circulating the processors on which the watchdog daemon operates;
Based on the processor movement rule, the processor information corresponding to the task of the watchdog daemon among the processor information stored for each task corresponding to the identifier of the processor that operates when the waiting task acquires the execution right Watchdog management steps to update,
A failure detection method comprising: A failure detection program for detecting a failure in a multiprocessor system comprising a plurality of processors, wherein a watchdog daemon detects a failure using a watchdog timer,
A processor movement rule storage procedure for storing a processor movement rule, which is a rule for sequentially moving and circulating the processor on which the watchdog daemon operates, in the storage unit;
The processor movement rule is read from the storage unit, and the identifier of the processor that operates when the waiting task acquires the execution right corresponds to the task of the watchdog daemon among the processor information stored for each corresponding task. A watchdog management procedure for reading processor information from the storage unit, updating the read processor information based on the processor movement rule, and writing the updated processor information to the storage unit;
A failure detection program for causing a computer to execute

Images