JP5490067B2 - Fault management apparatus and program - Google Patents

Description

  The present invention relates to a failure management device and a program, and in particular, in a configuration in which a plurality of paths for a computer to communicate with an external device are provided for each of a plurality of external devices, a failure management device that manages a failure in each path, The present invention also relates to a failure management program for causing a computer to function as the failure management apparatus.

  Includes a computer and a storage device, performs access to the storage device (reading data stored in the storage device, writing data to the storage device, etc.) according to an external request, and returns a response to the request source In the system, when it is required to continue the operation state even when a failure occurs, the storage device itself is multiplexed, and the communication path (path) for the computer to communicate with the storage device is also multiplexed. The redundant configuration is often adopted.

  As a technology applicable to the above redundant configuration, for example, Patent Document 1 discloses a computer system including a host computer and a storage system including a physical disk and a disk controller. Provided to the host computer as one or more logical units. When the host computer detects a failure in a logical path that is an access path from the host computer to the logical unit, the host computer uses the same logical unit as the logical path in which the failure is detected. By specifying the logical path to access and executing failure detection processing for the specified logical path, it is determined whether or not the specified logical path is normal, and from among the specified path A technology for selecting a normal logical path and accessing a logical unit using the selected normal logical path is disclosed. It has been.

  Further, in Patent Document 2 related to the above, in the distributed computer system, the master process periodically updates the shared resource control file with a new time stamp time, and the process for requesting access to the shared resource is shared. The resource control file is read to determine whether another process is designated as the master. If the elapsed time from the last timestamp is longer than the preset time, the shared resource control file is invalidated. A technique for efficiently determining a master process for controlling a shared resource by creating a new shared resource control file is disclosed.

JP 2007-265243 A JP 7-13939 A

  In the above-described redundant configuration, for example, when a failure occurs in a device existing in the middle of a path from the computer to the storage device, the occurrence of the failure is detected in each of a plurality of paths passing through the device. In many cases, when a failure occurs, the occurrence of the failure is continuously notified a plurality of times. For this reason, when performing failure handling processing such as blocking a path that has been notified of the occurrence of a failure, triggered by the notification of the occurrence of the failure, a large load is placed on the computer by repeating this failure handling processing frequently. As a result, the operation of the computer system may become unstable or the computer may go down.

  To deal with the above problem, it is conceivable to apply the exclusive control described in Patent Document 2 to a failure handling process that is triggered by the notification of the occurrence of a failure. However, when the technique described in Patent Document 2 is applied to the execution of the fault handling process, a process for performing the fault handling process is generated each time the occurrence of the fault is notified, and the generated process acquires the master right. Until then, it will repeatedly try to acquire mastership. For this reason, when a failure that affects many paths occurs, such as when a failure has occurred in a device that exists in the middle of a path, and when the failure occurrence of a large number of paths is continuously notified, many processes The generation of the load adds a great load on the computer and consumes a large amount of resources such as a memory, which is not effective for stabilizing the operation of the computer.

  The present invention has been made in view of the above facts, and an object of the present invention is to obtain a failure management apparatus and a failure management program capable of stabilizing the operation of a computer when a failure affecting a plurality of paths occurs. .

To achieve the above object, the fault management apparatus according to the first aspect of the present invention is realized by a computer provided with a group of a plurality of paths for communicating with different external devices among a plurality of external devices. The plurality of paths physically share a single communication path, and when the occurrence of a failure in one of the paths is notified, the occurrence of the failure is notified Control means for performing blocking processing for blocking all paths belonging to the same group as the path is provided.

The fault management apparatus according to the first aspect of the present invention is realized by a computer provided with a group including a plurality of paths for communicating with different external devices among a plurality of external devices. Incidentally, as an external device according to the present invention, For example, although the information stored possible and storing information rewritable storage device is suitable may be other devices. In addition, the group in the present invention can be composed of a plurality of paths sharing a physical communication path, for example, as shown as “path group” in FIG. In the first aspect of the present invention, when the occurrence of a failure in one of the paths is notified, the control means blocks all the paths belonging to the same group as the path in which the occurrence of the failure is notified. A blocking process is performed.

  As described above, in the invention according to claim 1, paths belonging to the same group are often affected in the same way when a failure occurs, and the occurrence of a failure is often notified. Since all the paths that belong to the same group as the path that is notified of the occurrence of a failure are blocked, even if a failure occurs that affects each path that belongs to the same group, After the blockage process is once performed based on the notification of the occurrence of a failure in one path, the occurrence of a failure in each path blocked by the blockage process is not notified, and a plurality of paths It is possible to reduce the load applied to the computer when a failure that affects the computer occurs, and to suppress the consumption of resources such as memory. Therefore, according to the first aspect of the present invention, it is possible to realize the stabilization of the operation of the computer when a failure that affects a plurality of paths occurs.

  Also, when a failure occurs in a certain path, the cause is often a malfunction or failure of hardware (for example, HBA (Host Bus Adapter) or cable) on the path where the failure has occurred. Is removed from the operating part as a suspected part and replaced when necessary (work to recover from a failure that has occurred) .When performing this maintenance work, the hardware is used as the suspected part. In order to disconnect the hardware as the suspected part regardless of whether or not a failure has occurred in the other path, the other path is used for communication. The other path needs to be blocked so that it is not used. In the first aspect of the invention, when a failure occurrence is notified in any of the paths, a blocking process is performed to block all the paths belonging to the same group as the path for which the failure has been notified. Therefore, when performing the maintenance work, it is not necessary to block all other paths belonging to the same group as the path where the failure has occurred, and the burden on the worker who performs the maintenance work can be reduced.

Further, in the invention described in claim 2, in the invention described in claim 1, status information indicating the status of each path of the group is registered and managed in the status table, and communication between the computer and each external device is performed. Management means for performing a management process including a process for selecting a path to be used, and excluding a path in which information indicating blockage is registered in the status table as status information from a selection target in communication with an external device; and The state information of all the paths belonging to the same group as the path notified of the occurrence of the failure among the state information registered in the state table by the block processing by means is information indicating blockage. And a start control means for stopping the start of the program for closing the path when each of them is rewritten.

In the invention according to claim 2, although all paths belonging to the same group as the path notified of the occurrence of the failure are blocked, that is, all the paths are in a state where blocking processing is unnecessary. Thus, it is possible to prevent the program from being activated in vain, and to further reduce the load applied to the computer and further suppress the consumption of resources such as memory.

According to a third aspect of the present invention, there is provided a failure management program comprising: a computer provided with a group of a plurality of paths for communicating with different external devices among a plurality of external devices ; share a single communication path, when the occurrence of failure in any of the paths has been notified, closed for closing all the paths belonging to group generate notifications paths identical and of the fault It functions as a control means for performing processing.

Since the failure management program according to the invention of claim 3 is a program for causing the computer to function as the control means, the computer executes the failure management program according to the invention of claim 3 , The computer functions as the failure management device according to claim 1, and, similarly to the invention according to claim 1, realizes stabilization of computer operation when a failure that affects a plurality of paths occurs. Can do.

  In the above invention, a plurality of groups each having a plurality of paths for communicating with different external devices among a plurality of external devices are provided, and whenever a failure occurs in any of the paths is notified. Determination means for determining whether or not file information in which duplicate creation with the same attribute is restricted has already been created with a predetermined attribute corresponding to a group to which the path to which the occurrence of the failure is notified belongs, Creation means for attempting creation of the file information of the predetermined attribute when the determination means determines that the file information of the predetermined attribute has not been created, and the control means includes the creation means The blocking process may be performed when the file information having the predetermined attribute is successfully created.

  According to the first aspect described above, a plurality of paths connected to the computer (a plurality of paths belonging to different groups: as an example, FIG. 2 shows all the connections between the logical disk LU # 0 and the server computer 12 (In this specification, this state is crossed) in which some paths are blocked and there are multiple external devices with different groups to which the blocked path belongs. It is possible to avoid as much as possible from occurring. When the above cross-blocking state occurs, when performing maintenance work, if all other paths belonging to the same group as the failed path are blocked, all the paths connected to the computer are blocked. However, in the first aspect, it is possible to avoid such a situation as much as possible, and maintenance work can be avoided. It is possible to improve the probability that can be implemented.

  Further, in the first aspect, when a failure that affects a plurality of paths occurs and the occurrence of a failure in different paths belonging to the same group is notified almost simultaneously, the occurrence of the failure is notified earliest. In the process corresponding to the specific path, the block process is performed by successfully creating the file information. On the other hand, in the process corresponding to the remaining path in which the occurrence of the failure is notified after the specific path, a predetermined process is performed. If the timing for determining whether or not file information with a predetermined attribute has already been created is after creation of file information with a predetermined attribute in the processing corresponding to the specific path, the file information with a predetermined attribute has already been created. A file with a predetermined attribute in a process corresponding to a specific path is determined as to whether or not file information with a predetermined attribute has already been created. Even a before creating a broadcast, a failure that the creation of the file information to duplicate create the file information of the same attribute is restricted, closing processing is not performed. Therefore, even when a failure that affects multiple paths occurs and the occurrence of a failure in different paths belonging to the same group is notified almost simultaneously, it is ensured that the blocking process is executed in duplicate. This can prevent the load on the computer.

  Further, in the first aspect, for example, the determination unit, the creation unit, and the control unit are activated each time a failure occurrence is notified in any path of any group and executed by a computer. Preferably, the execution of the program by the computer is preferably configured to be terminated when the creation of the file information with the predetermined attribute by the creation unit fails (second mode). As a result, even when a failure that affects multiple paths occurs and the occurrence of failures in different paths belonging to the same group is notified almost simultaneously, the file information of the processing corresponding to each notification The processing that failed to be created will be terminated by the computer, reducing the load on the computer and reducing the consumption of resources such as memory compared to the case where the execution of the processing is not terminated. can do.

  In the first or second aspect, for example, when the determination unit determines that file information with a predetermined attribute has already been created, is the elapsed time since the creation of the file information equal to or greater than a threshold value? The control means deletes the file information of the predetermined attribute created by the creating means after the end of the blocking process, and the file information of the predetermined attribute has already been created by the judging means, When it is determined that the elapsed time from the creation of the file information is equal to or greater than the threshold, the file information that has already been created is deleted, and the creation means determines that the elapsed time is equal to or greater than the threshold In addition, it is preferable that the configuration is such that the creation of file information having a predetermined attribute is tried after the already created file information is deleted by the control means (third Embodiment).

  The threshold value in the third aspect is longer than a time required for a series of processes by the determination unit, the generation unit, and the control unit by a predetermined value or more and can be determined to be a failure different from the failure that has been notified of the previous occurrence. A value corresponding to time (for example, about 1 minute) is preferable. In the third aspect, the file information created by the creation means is deleted by the control means after the closing process is completed, but the file information with a predetermined attribute has already been created by the judgment means, and the file information is If it is determined that the elapsed time from the creation is equal to or greater than the threshold, the file information is likely to be file information that remains after deletion by the control means for some reason. In the third aspect, such file information is also deleted by the control means, and then the creation of the file information is attempted by the creation means. Therefore, even if the deletion of the file information by the control means fails, it is not deleted. When the file information remaining in the file is created, the creation unit can create the file information with the same attribute again as time elapses beyond the threshold, and the control unit can perform the blocking process. Can be restored.

  Further, in the third aspect, for example, the determination unit, the creation unit, and the control unit are activated and executed by the computer each time a failure occurrence is notified in any path of any group. The execution of the program by the computer may be configured to be terminated when the file information with a predetermined attribute is created by the determination unit and the elapsed time is determined to be less than the threshold value. Preferred (fourth embodiment). As a result, even when a failure that affects multiple paths occurs and the occurrence of a failure in a different path belonging to the same group is notified at a time interval that is less than the threshold, the processing corresponding to each path is performed. Of these, the processing corresponding to the path notified of the occurrence of the failure after the second is executed by the computer, and the load applied to the computer is reduced as compared with the case where the execution of the processing is not ended. In addition, the consumption of resources such as memory can be suppressed.

  Further, in the configuration in which any one of the first to fourth aspects is applied to the invention according to claim 2, the control means refers to, for example, the state table before performing the blocking process and is notified of the occurrence of the failure. Check whether a specific path among all paths belonging to the same group as the path is the last unblocked path among multiple paths corresponding to the same external device as the specific path. If the current path is the last path, the rewriting of the status information corresponding to the specific path registered in the status table is canceled in the same group as the path that is notified of the occurrence of the failure. It is preferable to configure so that all the paths to which it belongs are performed (fifth aspect). As a result, it is possible to prevent the appearance of an external device in which all paths for communicating with the computer are blocked along with the execution of the blocking process. It is possible to maintain a state in which communication with each external device is possible via the blocking path.

  In addition, in the configuration in which any one of the first to fifth aspects is applied to the invention according to claim 3, the activation control unit is configured so that each of the plurality of external devices is, for example, the result of the blocking process performed by the control unit. In addition, it is preferable that the start of the program when the occurrence of a failure is notified is stopped even when the number of unblocked paths among a plurality of corresponding paths is one remaining state. Sixth aspect). As a result, the number of unblocked paths corresponding to each external device is one remaining state, that is, it is not desirable to increase the number of blocked paths by performing blocking processing. Can be prevented from being started unnecessarily, and further reduction of the load applied to the computer and further suppression of consumption of resources such as memory can be realized.

  As described above, the present invention tries to create file information with a predetermined attribute when the occurrence of a failure in one of the paths is notified, and the file information with the predetermined attribute is successfully created. Since block processing is performed to block all paths that belong to the same group as the path that has been notified of the occurrence of a failure, computer operation can be stabilized when a failure that affects multiple paths occurs. It has an excellent effect that can be realized.

It is a schematic block diagram of the computer system which concerns on this embodiment. It is the schematic for demonstrating the access to the disk apparatus at the time of operation. It is the schematic which shows the flow of a process at the time of I / O delay generation | occurrence | production. It is a flowchart which shows the content of a path | pass block | close control process. It is a chart which shows the result of path blockage control processing for every pattern.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a computer system 10 according to the present embodiment. The computer system 10 includes a plurality of server computers 12 (only two server computers 12A and 12B are shown in FIG. 1), a plurality of logical disks LU, and individual optical fibers 22 via optical fiber cables 22. A disk device 24 is connected to each of the server computer 12 and functions as a DB (database) server. Incidentally, the individual server computer 12 to the computer according to the present invention, each logical disk LU provided in the disk unit 24 corresponds to the external device according to the present invention.

  Each server computer 12 connects the CPU 14, the memory 16, a nonvolatile storage unit 18 including a disk drive driven by a disk drive unit (DKU), and the server computer 12 to the disk device 24 via an optical fiber. A plurality of HBAs (Host Bus Adapters) 20 (only two HBAs (HBA # 0, HBA # 2) are shown in FIG. 1). The storage unit 18 of each server computer 12 has an I / O (Input / Output) request to the disk device 24 (specifically, any one of a plurality of logical disks of the disk device 24). , An operating system (OS) program that controls hardware in response to an I / O request from the application, a link management unit program, a driver program, a path blockage control shell (program ) And log trap programs are stored. The link management unit, driver, path blockage control shell, and log trap will be described later.

Each program stored in the storage unit 18 is read from the storage unit 18 at the start of operation of the server computer 12, stored in the memory 16, and executed by the CPU 14 when necessary. Incidentally, among the programs stored in the storage unit 18, path deactivation control shell Ri Contact corresponds to the fault management program according to the present invention, the server computer 12, the CPU14 executes a path deactivation control shell It functions as a failure management apparatus according to the present invention.

  On the other hand, the disk device 24 includes a large number of disk drives driven by a disk drive unit (DKU) 26. A large number of disk drives are connected to each other so as to form a plurality of redundant logical disks LU (in FIG. 1, a part of logical disks (LU # 0 to LU # provided in the disk device 24). Only # 3, LU # m to LU # m + 3, LU # n to LU # n + 3) are shown). The disk device 24 is provided with a disk controller (DKC) 28. The disk controller (DKC) 28 is provided with a plurality (two in FIG. 1) of clusters 30, and each cluster 30 includes a plurality (two in FIG. 1) of channel adapters (CHAs) 32, A plurality (two in FIG. 1) of disk adapters (DKAs) 34, and a channel switch (CHA) 32 and a cache switch (CSW) 36 for connecting the disk adapters (DKAs) 34 are provided.

  Each logical disk LU is connected to any one disk adapter (DKA) 34 of each cluster 30. A plurality of channel adapters (CHA) 32 provided in the same cluster 30 are connected to the HBA 20 of the server computer 12 and the optical fiber cable 22 which are different from each other, and the channel adapter (CHA) 32 is connected. When an I / O request is received from the server computer 12 via the optical fiber cable 22, the disk to which the logical disk LU to be accessed in the received I / O request is connected via the cache switch (CSW) 36. The I / O request is transferred to the adapter (DKA) 34.

  Next, as an operation of the present embodiment, first, each program is executed by the CPU 14 of the server computer 12 in a state where the computer system 10 is operating as a DB server and no failure has occurred in the disk device 24 or the like. The processing realized by this will be described with reference to FIG.

  In this embodiment, each server computer 12 is provided with a plurality of HBAs 20, and each HBA 20 is connected to a disk device 24 (channel adapter (CHA) 32 of a different cluster 30) via an optical fiber cable 22. Since each is connected, each server computer 12 passes through each logical disk LU of the disk device 24 and a plurality of paths (different channel adapters (CHAs) 32 as shown by broken lines in FIG. A plurality of paths), and the paths for communicating with the individual logical disks LU are made redundant (multiplexed). A plurality of paths provided between the single server computer 12 and the disk device 24 are paths (paths sharing a physical communication path) that pass through the same channel adapter (CHA) 32. They are grouped as a unit (see also “path group” shown in FIG. 2).

  In the I / O request to the disk device 24 output from the aforementioned application, the logical disk LU to be accessed is specified. This I / O request is transferred / input via the OS to the link management unit realized by the CPU 14 of the server computer 12 executing the program of the link management unit. The link management unit grasps the load on each path, selects a path for transferring the input I / O request to the logical disk LU to be accessed, so that the load on each path is distributed, The driver program is executed by the CPU 14 of the server computer 12, and the I / O request is transferred and input by designating the selected path to the driver. Control is performed so as to be transferred through the selected path. In addition, the link management unit maintains a path status management table for registering status information indicating whether the status is active (online) or inactive (blocked / offline) for each path. When selecting a path to be used for transferring the / O request, a path whose status information registered in the path status management table is “inactive (blocked): offline” is excluded from selection targets.

  Thus, the link management unit corresponds to the management unit described in claim 2, and the path status management table corresponds to the status table described in claim 2.

  Further, as shown in FIG. 2, the driver has a queue for holding the input I / O request for each path, and the I / O request input from the link management unit is sent from the link management unit. I / O requests held in the individual queues are once entered into the queue corresponding to the specified path, taken out from the individual queues in the order of entry into the queue, transferred to the corresponding HBA 20, and input to the IBA. Controls the execution order of / O requests. Further, the I / O request transferred / input from the driver to the HBA 20 is transferred to the logical disk LU to be accessed through the path determined and specified in advance by the link management unit, and the disk corresponding to the I / O request After access processing (reading or writing of data to any disk drive constituting the logical disk LU to be accessed) is performed, a response to the I / O request is transferred through the same path.

  Next, when a failure occurs in the logical disk LU to be accessed in the disk device 24 or any device existing in the middle of a specific path, the specific I transferred / input by the driver to the HBA 20 A case where a response to the / O request is delayed (when an I / O delay occurs) will be described.

  When the I / O request is transferred / input to the HBA 20, the driver starts a timer having a predetermined timer value in preparation for the occurrence of an I / O delay, and resets the count value of the retry counter to zero. If the timer corresponding to the specific I / O request has timed out before receiving a response to the specific I / O request, the count value of the retry counter is incremented by 1, and the specific I / O request is incremented. The O request is again transferred and input to the HBA 20 and the timer is started until the count value of the retry counter reaches a predetermined maximum number of retries. If a response to a specific I / O request is not received even if the count value of the retry counter reaches the maximum number of retries, the link management unit is notified of the occurrence of a failure.

  By the way, the timer value is, for example, 30 to 40 seconds, and the maximum number of retries is, for example, about 3 to 5 times. However, the timer value is 36 seconds and the maximum number of retries is 4 times. Assuming that there is an elapsed time from when the driver sends the first I / O request until the link management unit is notified of the occurrence of the failure due to the I / O delay, 36 seconds × (1 + 4) = 180 seconds. There is a problem that it takes a relatively long time after the occurrence of a failure until the link management unit recognizes the occurrence of the failure and starts executing the failure handling process.

  Further, until the driver notifies the link management unit of the occurrence of the failure, the failure that has occurred is left without being subjected to the failure handling processing, but the location where the failure has occurred is, for example, a plurality of paths In the case of a device that passes through (for example, HBA 20 or channel adapter (CHA) 32), an I / O delay occurs in each of the other paths that pass through the failure location, and an I / O request due to the I / O delay occurs. By performing each retransmission, the occurrence of a failure is continuously notified to the link management unit a plurality of times, and each time the occurrence of a failure is notified, the above-described failure handling processing is repeated, and the server -A great load is applied to the computer 12, and the operation of the computer system 10 may become unstable or the server computer 12 may be down.

  On the other hand, the driver according to the present embodiment generates an I / O delay due to a timer timeout corresponding to the I / O request before receiving a response to an arbitrary I / O request (see FIG. 3 (see also (1) in FIG. 3) indicating that an I / O delay (timer timeout) has occurred in the log information recording area provided in advance in the storage area of the memory 16 or the storage unit 18. / O delay message is recorded (see also (2) in FIG. 3). For this reason, in this embodiment, in order to solve the above-described problem, recording of an I / O delay message in the log information recording area is used, and this is used as a trigger to pass through the same channel adapter (CHA) 32. Control is performed to block paths in units of path groups.

  That is, in this embodiment, in order to realize the above control, a log trap program and a path blockage control shell are stored in the storage unit 18 of the server computer 12. There are as many log traps and path blocking control shells as there are HBAs 20 provided in a single server computer 12, and each log trap corresponds to a different HBA 20 (path group), and individual path blockings. The control shells also correspond to different HBAs 20 (path groups).

  In this embodiment, when the occurrence of an I / O delay is detected, the I / O delay message recorded in the log information recording area by the driver is not clearly indicated the path where the I / O delay has occurred. Whether the I / O delay message indicating that an I / O delay has occurred in the corresponding path group has been recorded in the log information recording area is a message that clearly indicates the path group in which the delay has occurred. It is monitoring whether or not. A log trap that detects that an I / O delay message indicating that an I / O delay has occurred in the corresponding path group is recorded in the log information recording area (see also (3) in FIG. 3) The path block control shell having the same path group is started (see also (4) in FIG. 3). The path block control process performed by the path block control shell activated by the log trap will be described below with reference to FIG.

  In the path blocking control process, first, in step 50, it is determined whether or not a lock directory having a preset name has already been created for a corresponding path group (a group of paths in which an I / O delay has occurred). . In the present embodiment, a UNIX (registered trademark) OS is used as the OS, and the UNIX (registered trademark) OS restricts (prohibits) duplicate creation of directories having the same name. The lock directory according to the present embodiment is for using the above-described characteristics of the directory in the UNIX (registered trademark) OS for exclusive control of the path blockage control shell. In this embodiment, the directory of the directory used as the lock directory is used. A name is preset for each path group, and only the path blockage control shell that has created a lock directory with a preset name for the corresponding path group is given the right to update the path status management table. The exclusive control is realized when multiple path block control shells with the same path group are activated (when multiple path block control processes are executed in parallel).

  In most cases, the path blockage control process stops the log trap that started the own shell (the log trap with the same path group) (details will be described later). After the execution, the path blocking control shell having the same corresponding path group cannot be activated twice. However, before the process of stopping the log trap is performed, for example, when an I / O delay occurs almost simultaneously in each path belonging to the same group, the corresponding path block control shell has the same path group. When it is activated again, the above-described overlapping activation occurs. Step 50 described above corresponds to the determination means according to the present invention.

  The above lock directory corresponds to file information according to the present invention. In the present invention, the OS is not limited to the UNIX (registered trademark) system, and duplicate creation of folders with the same name is also restricted (prohibited) for a folder in a WINDOWS (registered trademark) OS. Therefore, when the OS is a WINDOWS (registered trademark) system, a folder in the WINDOWS (registered trademark) system OS can be applied as file information according to the present invention. Further, the file information according to the present invention is not limited to information in which duplicate creation with the same name is restricted, and information in which duplicate creation of information with the same attribute other than the name is restricted is applied. It is also possible to do.

  If the determination in step 50 above is negative, the path blocking control shell with the same corresponding path group has not been activated twice (other path blocking control processes with the same corresponding path group are executed in parallel). Therefore, the process proceeds to step 60, and an attempt is made to create a lock directory having a preset name for the corresponding path group (the group of the path in which the I / O delay has occurred). Step 60 corresponds to the creating means according to the present invention. In the next step 62, it is determined whether or not the lock directory has been successfully created in step 60. If the creation of the lock directory is successful, it can be determined that the right to update the path state management table has been acquired. If the determination in step 62 is affirmative, the process proceeds to step 66, and the path state management table is updated in step 66 and subsequent steps. Perform blocking processing.

  That is, first, in step 66, a single path to be processed is selected from a corresponding path group (a group of paths in which an I / O delay has occurred). In the next step 68, the path status management table is referred to, and the determination target logical disk LU corresponding to the processing target path selected in step 66 (connected to the server computer 12 by the processing target path) The state of all paths connecting the logical disk LU to be determined to the server computer 12 is confirmed. In step 70, all paths connecting the logical disk LU to be determined to the server computer 12 (as an example, all paths connecting the logical disk LU # 0 to the server computer 12 are shown in FIG. Whether the number of unblocked paths is 2 or more.

  If the determination in step 70 is affirmative, the process proceeds to step 72, where the status information of the processing target path registered in the path status management table is changed from “in operation: online” to “not in operation (blocked): offline”. ”To block the processing target path, and the process proceeds to step 74. In addition, when the number of unblocked paths connecting the determination target logical disk LU to the server computer 12 is 1, that is, the determination target logical disk LU is connected to the server computer 12. In the case of the last unblocked path, if the processing target path is blocked, the server computer 12 becomes unable to communicate with the logical disk LU to be determined. If so, the process proceeds to step 74 without performing the process of step 72.

In step 74, it is determined whether or not an unprocessed path (unselected path as a processing target) exists in the corresponding path group (group of paths in which an I / O delay has occurred). If this determination is affirmative, the process returns to step 66, and steps 66 to 74 are repeated until the determination in step 74 is negative. As a result, the blocking process in steps 66 to 74 is performed for all paths belonging to the corresponding path group (the group of paths in which the I / O delay has occurred) (see also (5) in FIG. 3). . Steps 66 to step 74 corresponds to the control hand stage according to the present invention.

  As a result of the above blocking process, for example, as shown in pattern 1 in FIG. 5, all paths provided between the server computer 12 and the disk device 24 are in the “online” state, and the HBA0 system path When an I / O delay occurs in some of the paths in the group, as shown in “Contents of management table after execution of path blockage control”, all the paths in the HBA0 system path group The path status information is rewritten to "Non-operating (blocked): offline". Further, for example, as in pattern 2 shown in FIG. 5, a part of the HBA 0 system path group is in a state of “inactive (blocked): offline” and the HBA 0 system path group When an I / O delay occurs in another path, as shown in “Contents of management table after execution of path blockage control”, “in operation: online” in the group of HBA0 path. The status information of the path is rewritten to "Inactive (Blocked): offline".

  In addition, as described above, a state in which a part of the paths is “inactive (blocked): offline” indicates that the specific path is not connected due to optical communication failure or temporary attenuation in the specific path. This occurs when the HBA detects that the communication is physically impossible and the driver rewrites the corresponding path status information based on this. In the patterns 1 and 2 shown in FIG. 5, all the paths of the HBA 0 system after the execution of the path block control are “not operating (blocked): offline”, and “acting” is included in the paths belonging to the same group. Since it is avoided that a cross-blocking state where a path of `` Medium: online '' and a path of `` not working (blocked): offline '' coexist will occur, the work to recover from the failure that occurred is easy Become.

  On the other hand, for example, as shown in pattern 3 in FIG. 5, a part of the HBA2 path group (path corresponding to the logical disk LU # 0) is “inactive (blocked): offline”. If an I / O delay occurs in a part of the HBA0 path group in the state, as shown in "Contents of management table after execution of path blocking control", the HBA0 system For the paths corresponding to logical disks other than logical disk LU # 0 in the group of paths, the status information is rewritten to "inactive (blocked): offline", but it corresponds to logical disk LU # 0 The path is the last unblocked path connecting the logical disk LU # 0 to the server computer 12, and if the path is blocked, the server computer 12 is connected to the logical disk LU # 0. Communication is disabled. It from "Up: online" state information by the determination of the foregoing step 70 is negative while the maintenance of. In this case, although a cross blockage state occurs, it is possible to prevent the server computer 12 from being in a state where it cannot communicate with the logical disk LU # 0.

  Since the path blocked by the above-described blocking process is excluded from the selection targets of the path used for transferring the I / O request from the server computer 12 to the disk device 24, an I / O delay is generated by the driver. Detected and no I / O delay message is recorded in the log information recording area. For this reason, even if a failure occurs in a device or the like through which each path belonging to the same group passes, if all paths belonging to the same group can be blocked as in patterns 1 and 2 shown in FIG. Until the current failure is recovered, the path blockage control process is not performed again for the group of paths in which the current I / O delay has occurred. For a path blocked by the blocking process, an I / O request is transmitted using the path before being blocked, and a response to the transmitted I / O request times out, so that the I / O is blocked after the path is blocked. Even if the occurrence of an O delay is detected, the I / O request is not retransmitted based on the block being blocked. Therefore, it is possible to reduce the load applied to the server computer 12 when a failure that affects each path belonging to the same group occurs, and to suppress the consumption of resources such as memory. In addition, since the above blocking process is triggered by the recording of the I / O delay message in the log information recording area, that is, when the first transmission of the I / O request by the driver has timed out, the link management unit from the driver. Compared to the case where the blockage process is triggered by the notification of the occurrence of a fault, the blockage process can be performed earlier.

  If the processing of step 66 to step 74 is performed for all the paths belonging to the corresponding path group (the group of paths in which the I / O delay has occurred), the determination of step 74 is denied and the process proceeds to step 76. The path state management table is referred again, and it is determined whether or not all the paths in the corresponding path group have been blocked as a result of the processing in step 66 to step 74. For example, patterns 1 and 2 shown in FIG. 5 correspond to the determination condition. If the determination is positive, the process proceeds to step 82. If the determination in step 76 is negative, the process proceeds to step 78, and the path state management table is referenced again to count the number of unblocked paths for each logical disk of the disk device 24. In the next step 80, it is determined whether or not the number of unblocked paths for each logical disk of the disk device 24 is all one. The condition in which the determination in step 76 is negative and the determination in step 80 is affirmative is, for example, the pattern 3 shown in FIG. 5. If the determination in step 80 is affirmative, the process proceeds to step 82.

  Under the condition that the determination in step 76 or step 80 is affirmative, at least the corresponding path group (the group of the path in which the I / O delay has occurred) is blocked until the current failure is recovered. There is no need to rewrite the status information), and for example, a path that is not blocked in the corresponding path group, such as a path corresponding to the logical disk LU # 0 of the HBA0 system in the pattern 3 shown in FIG. Even if the path exists, a logical disk LU that causes the server computer 12 to be unable to communicate appears when the path is blocked. It is not desirable to perform (replacement). For this reason, in step 82, the operation of the log trap (the log trap having the same path group corresponding to the corresponding process) that started its own processing is stopped ((6) in FIG. ”(See also the column“ ”), the process proceeds to step 84.

When this log trap operation stops, the path trap control is performed by the log trap even if an I / O delay message indicating that an I / O delay has occurred in the corresponding path group is recorded in the log information recording area. The shell is not started and the path block control process is not performed. As a result, even if a failure occurs in a device or the like through which each path belonging to the same group passes, the current I / O delay will continue until the current failure is recovered after the log trap operation is stopped. Since the path blockage control process is not performed again for the group of paths that have occurred, the load on the server computer 12 can be reduced when a failure that affects each path belonging to the same group occurs, and the memory The consumption of resources such as these can also be suppressed. Steps 76 to 82 correspond to the activation control means described in claim 2 .

  On the other hand, as a condition in which the determination in step 76 and the determination in step 80 are respectively denied, for example, a case where one or more paths have failed to be blocked due to some reason. A typical example is shown as pattern 4 in FIG. Pattern 4 shown in FIG. 5 is a state in which all paths provided between the server computer 12 and the disk device 24 are “online” and a part of the HBA 0 system path group. Due to the occurrence of the I / O delay, the state information of all paths in the HBA 0 system path group was rewritten to “inactive (blocked): offline” in the same manner as pattern 1 described above. However, as shown in “Contents of management table after execution of path blockage control”, the status information of the path corresponding to logical disk LU # 0 in the HBA0 system path group is written due to an error or the like. The pattern failed to be replaced.

  In the above pattern 4, since the HBA0 system path group includes a path whose status is “active: online” (path corresponding to the logical disk LU # 0), the determination in step 76 is negative, In the logical disk LU # 0, since the corresponding HBA0 system and HBA2 system paths are “online” and the number of unblocked paths = 2, the determination in step 80 is also denied. When there is a logical disk with two or more unblocked paths among the corresponding paths as in pattern 4, there is room for blocking the corresponding path when a failure occurs for the logical disk. (Since any one of the corresponding paths is blocked when a failure occurs, communication with the server computer 12 is not disabled.) If the determination in steps 76 and 80 is negative, a log trap (Step 82 is skipped) and the process proceeds to Step 84 (see also the “Log Trap” column of Pattern 4 shown in FIG. 5).

  In step 84, the lock directory created in the previous step 60 is deleted, and the path closing process is terminated. Along with the deletion of this lock directory, it is possible to perform a blocking process that updates the path status management table when the local shell is restarted and the corresponding path grouping process is performed again for the same path group. Become.

  By the way, in the path block control process, as described above, the block process in step 66 to step 74 attempts to block all the paths in the corresponding path group, and normally (to rewrite the path status information). Since the log trap operation is also stopped (except in rare cases such as failure), there is an I / O delay in each path even if a failure occurs in a device that passes through each path belonging to the same group. If there is a time difference of a certain time or more in the timing detected by the driver, the I / O delay message is recorded in the log information recording area only for the path where the driver first detected that the I / O delay has occurred. Then, the path blocking control shell is started by the log trap and the path blocking control process is executed.

  However, when it is detected almost simultaneously by the driver that an I / O delay has occurred in each path belonging to the same group, more specifically, the log information recording area for the path where the occurrence of the I / O delay is first detected. For the other paths belonging to the same group, the path blockage control process is triggered by the recording of the I / O delay message in the log, the path status management table is updated, and log trap execution is stopped. When the occurrence of an I / O delay is detected and an I / O delay message is recorded in the log information recording area, the I / O delay message to the log information recording area for a plurality of paths belonging to the same group. Multiple path blockage control processes with the same corresponding path group are triggered by recording, and multiple corresponding path groups are the same. So that the path deactivation control processing is performed in parallel at almost the same timing. Hereinafter, the path blockage control process in this case will be described.

  As described above, in step 50 of the path blocking control process, it is determined whether or not a lock directory has already been created. As described above, in the path block control process, the created lock directory is deleted at the end of the process (step 84). Therefore, if the determination in step 50 is affirmative, the corresponding path group is the same path block. Although there is a high possibility that multiple control shells have been activated, in rare cases, the lock directory deletion failed in the path block control process that was performed previously, and the path block control process itself has ended, but the lock directory remains. The possibility that this is the cause is undeniable.

  Therefore, if the determination in step 50 is affirmed, the process proceeds to step 52, and after confirming the time stamp (lock directory creation date and time) of the lock directory that already exists, in the next step 54, it already exists. It is determined whether the elapsed time from the creation date and time of the current lock directory is equal to or greater than a predetermined threshold. As the threshold value in the determination of step 54, for example, a value corresponding to a time (for example, 1 minute) sufficiently longer than the time required for the path blockage control process can be applied. If the elapsed time from the creation date and time of the existing lock directory is less than a predetermined threshold, it is determined that there is a very high possibility that the same path block control shell with the same path group has been activated twice. Therefore, if the above determination is negative, the routine proceeds to step 56, where the message “stop for multiple activation” is output and the path blocking control process is terminated.

  If the determination in step 54 is affirmative, the lock directory that already exists has been left unsuccessfully deleted in the path block control process performed previously, and the previously performed path directory. Since it can be determined that the blocking control process itself has been completed, the process proceeds to step 58, after deleting the existing lock directory, the process proceeds to step 60 to try to create a new lock directory. If the creation of the lock directory is successful, the determination at step 62 is affirmed and the routine proceeds to step 66, where the above-described blocking process and log trap operation stop are sequentially performed.

  In addition, when the time difference in timing at which the occurrence of I / O delay is detected in each of a plurality of paths belonging to the same group is extremely small, the path blockage control process corresponding to the path where the occurrence of I / O delay is first detected Before the lock directory is created in step 1, the path blockage control process corresponding to the path where the occurrence of the I / O delay is detected after the second is performed, and the occurrence of the I / O delay is detected after the second. It is not 0 as a probability that the determination in step 50 is denied in the path blockage control process corresponding to the path.

  However, since the directory is created through the OS, even if a plurality of processes (processes) try to create the directory through the OS at the same time, there is always a time difference in the creation timing, and the corresponding path group is the same path. In the path blocking control process by the blocking control shell, the directory name created as the lock directory is the same, while in the UNIX (registered trademark) OS, the creation of duplicate directories with the same name is restricted (prohibited). On the other hand, the path block control process always fails to create a lock directory. Accordingly, if the determination in step 62 is negative in the path block control process, that is, if the creation of the lock directory attempted in step 60 has failed, another path block control process by the redundantly activated path block control shell is performed. Since it can be determined that the lock directory has been created first, the process proceeds to step 64 to output a “directory creation failure” message and terminate the path blockage control process.

  As described above, in the path blocking control process according to the present embodiment, when the lock directory already exists and the elapsed time from the creation date and time of the lock directory is less than the threshold, and the lock directory does not exist. However, if the creation of the lock directory fails, it is determined that there is a very high possibility that the same path block control shell with the same path group has been activated, and the path block control process is immediately terminated. Therefore, it is detected almost simultaneously by the driver that an I / O delay has occurred in each path belonging to the same group, and a plurality of path blockage control processes with the same group of corresponding paths are executed in parallel at substantially the same timing. Even in this case, it is possible to reliably prevent the blocking process from being executed repeatedly. It is possible to reduce the Waru load, it is possible to recover the resource consumption of memory or the like in an early stage.

  Therefore, according to the path blockage control processing according to the present embodiment, when a failure that affects a plurality of paths occurs in the computer system 10, the occurrence of an I / O delay in each path is detected. Regardless, since the load applied to the server computer 12 can be reduced and the consumption of resources such as memory can be suppressed, the server computer when a failure that affects a plurality of paths occurs in the computer system 10. Twelve operations can be stabilized.

  In the path block control process described above, after the block process is performed on the corresponding path group, the number of unblocked paths for each logical disk of the disk device 24 is all 1 (step 80). The group of the corresponding paths has stopped the operation of the same log trap, but the present invention is not limited to this. In the above processing, log traps with different path groups continue to operate, and if an I / O delay occurs on a path belonging to a group different from the corresponding path group, the path blockage control shell is activated. However, if the number of unblocked paths for each logical disk is all 1, block processing is unnecessary for all groups of paths, and it is not desirable to perform block processing. When the number of unblocked paths for each disk is all 1, the log trap operation may be stopped for all groups of paths.

  In the above description, the log trap and the path blocking control shell are provided for each path group. However, the present invention is not limited to this, and the log trap and the path blocking control shell are provided one by one. The log trap uses the recording of the I / O delay message in the log information recording area as a trigger to determine the path group corresponding to the recorded I / O delay message, and uses the path blocking control shell as an argument. The path blocking control shell that was started and started determines whether the notified group is stopped or not, and if the blocked process is not stopped, the existence confirmation of the lock directory with the name corresponding to the notified group is confirmed. An attempt to create a lock directory with the same name, block processing for each path of the notified group, and the notified group is stopped Set and stored information representative of the Rukoto may be configured to perform each process.

  In the above description, the disk device 24 including a plurality of disk drives driven by the disk drive unit (DKU) 26 has been described as an example of the external device according to the present invention. However, the external device according to the present invention is configured as described above. However, the present invention is not limited thereto, and may be a storage device including a semiconductor memory such as a flash memory, a communication device that communicates with a counterpart device other than the computer according to the present invention, and any device. Is applicable.

  Further, the number of external devices (logical disks LU), the number of path groups (the number of HBA 20 and channel adapter (CHA) 32), the number of computers (server computers 12), etc. are limited to the numbers shown in the drawings. However, the present invention can be changed as appropriate without departing from the scope of the present invention.

  In the above description, the path block control shell corresponding to the failure management program according to the present invention has been previously stored (installed) in the storage unit 18 of the server computer 12. However, the failure management program according to the present invention is It is also possible to provide the information recorded in a recording medium such as a CD-ROM or a DVD-ROM.

10 Computer system 12 Server computer 14 CPU
16 Memory 18 Storage unit 20 HBA
22 Optical fiber cable 24 Disk device

Claims (3)

A failure management device realized by a computer provided with a group of a plurality of paths for communicating with different external devices among a plurality of external devices,
The plurality of paths physically share a single communication path, and when the occurrence of a failure in one of the paths is notified, all of the paths belonging to the same group as the path in which the occurrence of the failure is notified A failure management apparatus comprising a control means for performing a blocking process for blocking a path. Status information representing the status of each of the individual paths in the group is registered and managed in a status table, and a path used for communication between the computer and each external device is selected, and the status information means blocking. A management means for performing a management process including a process of excluding a path whose information is registered in the state table from a selection target in communication with the external device;
The state information of all paths belonging to the same group as the path notified of the occurrence of the failure among the state information registered in the state table by the blocking process by the control means means blocking. An activation control means for stopping activation of a program for closing the path when each of the information is rewritten;
The failure management apparatus according to claim 1, further comprising: A computer provided with a group consisting of a plurality of paths for communicating with different external devices among a plurality of external devices,
The plurality of paths physically share a single communication path, and when the occurrence of a failure in one of the paths is notified, all of the paths belonging to the same group as the path in which the occurrence of the failure is notified A failure management program that functions as a control unit that performs block processing for blocking a path.