JP2021117547A - Failure analysis device, multi-cluster system, failure analysis program and failure analysis method - Google Patents

Abstract

To provide a failure analysis device that identifies the cause of a failure when the failure occurs in a multi-cluster system during an operation of a console device.SOLUTION: In a multi-cluster system, a failure analysis device 40 includes a first determination unit 405 and a second determination unit 406. The first determination unit determines, with respect to the input operation performed on a console device 30 via an input screen 309, whether or not a cluster state of a cluster at the time of an input operation and a cluster state of a cluster 10 on the input screen match. If the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, the second determination unit determines whether the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation.SELECTED DRAWING: Figure 2

Description

The present invention relates to a fault analysis device, a multi-cluster system, a fault analysis program, and a fault analysis method.

In recent years, a multi-cluster system including a plurality of server devices (hereinafter, simply referred to as a cluster), an SVPM (SerVice Platform Manager), and a console device has been known.

In the console device, in order to investigate the cause of the failure that occurred in the process of performing cluster operation management and monitoring control, the "console internal status log" that shows the CPU usage rate and process status history of the console device, and the actual console device The "cluster status change log" that shows the change history of the cluster status is saved.

Further, the console device stores a "log (operation log) for leaving an operation history" in order to record a history of operations performed by the operator on the console device. In addition, in order to enable investigation of what kind of screen display state the operation was performed on, "screen" capture or video recording may be performed.

A technique for storing information displayed on the screen of a console device at the time of an operation performed by an operator as a hash value is known (Patent Documents 1 and 2 below).

JP-A-2009-246750 International Publication No. 2016/051479

In such a conventional multi-cluster system, a failure may occur while the console device is performing an operation on the cluster, which may cause the following problems.

When the load inside the console device is high, the drawing process that reflects the cluster state on the screen of the console device is delayed, and the actual cluster state and the cluster state on the screen do not match. Under this phenomenon, if the operator performs an operation based on the old cluster state displayed on the screen, which is not the actual cluster state, even if the operation is appropriate for the cluster state on the screen, In some cases, operations that cannot be performed on the actual cluster state may be performed. When a failure occurs due to such an operation, the conventional multi-cluster system does not save the event that the drawing delay has occurred, so the "cluster state change log" and "operation log" that have been saved in the past are not saved. Analyze the cause of the failure based on. Therefore, when the above logs are viewed in chronological order, there is a problem that the operator is determined to have performed an inappropriate operation even though he / she has performed an appropriate operation based on the display on the screen.

Further, there is a problem that it is difficult to determine whether the failure generated during the operation is a failure caused by the operation or a failure closed inside the cluster regardless of whether or not the drawing process is delayed.

In the conventional stored information, at the time of operation, it is determined whether or not the actual cluster state and the cluster state displayed on the screen are the same, and whether or not the generated failure is caused by the operation. Can't.

In one aspect, the present invention aims to identify the cause of a failure if the multi-cluster system fails during the operation of the console device.

Therefore, in this fault analysis device, whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with respect to the input operation performed via the input screen in the console device. When the first determination unit for determining whether or not the input operation matches the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen, the input operation is the cluster of the cluster at the time of the operation. It is provided with a second determination unit that determines whether or not it corresponds to a prohibited operation for a state.

According to one embodiment, the cause of the failure that occurred in the multi-cluster system during the operation of the console device can be identified.

It is a figure which shows the structure of the multi-cluster system as an example of an embodiment. It is a figure which shows the functional structure of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the cluster state change log of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the operation log with the screen hash value of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the cluster state error list of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the cluster state change completion list of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the cluster state correspondence table of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the intermediate table A of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the intermediate table B of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the analysis table of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the operation type error table of the multi-cluster system as an example of an embodiment. It is a flowchart for demonstrating the process of acquiring the operation log with the screen hash value in the console device of the multi-cluster system as an example of embodiment. It is a figure for demonstrating the process which identifies the failure in the failure analysis apparatus of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the failure cause of four patterns in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 1 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 1 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 1 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 1 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 2 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 2 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 2 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 2 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 2 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 3 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 3 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 3 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 3 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 3 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 3 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 4 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 4 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 4 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 4 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 4 in the multi-cluster system as an example of an embodiment. It is a figure explaining the process of specifying the failure cause pattern 4 in the multi-cluster system as an example of an embodiment. It is a figure which illustrates the hardware configuration of the console apparatus of the multi-cluster system as an example of an embodiment. It is a figure which illustrates the hardware configuration of the fault analysis apparatus of the multi-cluster system as an example of an embodiment.

Hereinafter, embodiments relating to the multi-cluster system, the fault analysis device, the fault analysis method, and the fault analysis program will be described with reference to the drawings. However, the embodiments shown below are merely examples, and there is no intention of excluding the application of various modifications and techniques not specified in the embodiments. That is, the present embodiment can be variously modified and implemented within a range that does not deviate from the purpose. Further, each figure does not mean that it includes only the components shown in the figure, but may include other functions and the like.

(A) Configuration [Configuration of multi-cluster system 1]
FIG. 1 is a diagram schematically showing a configuration of a multi-cluster system 1 as an example of an embodiment.

The multi-cluster system 1 illustrated in FIG. 1 includes a plurality of clusters 10-1 and 10-2 (two in the example shown in FIG. 1), an SVPM 20, a console device 30, and a failure analysis device 40.

The clusters 10-1, 10-2, SVPM20, and console device 30 are configured to be able to communicate with each other via the network 50. The network 50 is, for example, a LAN (Local Area Network).

Although the fault analysis device 40 is not connected to the network 50, it can be connected to the console device via the network IF unit 4004 or the device connection IF unit 4007, which will be described later with reference to FIG. 37.

Clusters 10-1 and 10-2 are, for example, server computers.

Hereinafter, as the code indicating the cluster, reference numerals 10-1 and 10-2 are used when it is necessary to specify one of a plurality of clusters, but reference numeral 10 is used when referring to an arbitrary cluster.
Further, cluster 10-1 may be referred to as cluster # 0, and cluster 10-2 may be referred to as cluster # 1.

The SVPM20 is a device for integrating hardware control of a plurality of clusters.

The console device 30 is an input / output device used by connecting to the clusters 10-1 and 10-2 in order to manage and monitor and control the clusters 10-1 and 10-2.

The failure analysis device 40 is a device for analyzing a failure when a failure occurs in the cluster system and identifying the cause of the failure.

FIG. 2 is a diagram illustrating a functional configuration of the multi-cluster system 1 as an example of the embodiment.

[Functional configuration of console device 30]

As shown in FIG. 2, the console device 30 includes a cluster state change detection unit 301, a frame display control unit 302, a console state monitoring unit 303, a storage unit 304, a screen hash value storage unit 305, and a mouse / keyboard. It includes a driver 306 and a display driver 307.

The cluster 10 notifies the SVPM 20 of the state change each time the state of the cluster (power state, abnormal state, configuration, etc.) changes. Upon receiving the notification from the cluster 10, the SVPM 20 asynchronously notifies the console device 30 of the cluster state change.

The cluster state change detection unit 301 detects the state change of the cluster by the notification transmitted from the cluster 10 via the SVPM20. When the cluster state change detection unit 301 receives the notification from the SVPM 20, the cluster state change detection unit 301 stores the state change history in the storage unit 304 and notifies the frame display control unit 302 that displays the cluster state of the state change.

The frame display control unit 302 controls a plurality of frames on the display screen. The frame is a drawing section for partitioning a display screen displayed on a display (not shown) of the console device 30. Frames are provided for each function. For example, there are a clock frame for displaying the time, a processing frame for displaying button processing (frame B), a state display frame for displaying the state of the cluster (frame C), and the like.

Further, the frame display control unit 302 includes a button processing unit 310 and a screen processing unit 311 to control button operations and screen processing on the screen. The button processing unit 310 is interlocked with the buttons displayed in each frame, and when the operator operates a mouse or the like to input to press the button, the button processing unit 310 receives a pressing signal.

The screen processing unit 311 acquires an image of screen information at the time of operation (input) by the operator, and calculates a hash value of the image. For example, the screen processing unit 311 acquires an image of the screen displayed on the display at the time when the operator performs an input operation such as pressing a button or inputting a command, and obtains a hash value. Make a calculation.

In particular, the screen processing unit 311 acquires an image of the state display frame on the screen at the time of operation. The image is acquired via the display driver 307, which will be described later. The acquired image is subjected to a correction process for removing noise such as a mouse pointer by a known method, and then a hash value is calculated from the image. The hash value is calculated each time an operation input is made in the console device.

The screen processing unit 311 calculates the hash value by, for example, digitizing the image data of the acquired screen (state screen frame) and applying this numerical value to a known hash function. The method of changing the image data to a hash value is known, and detailed description thereof will be omitted.

Upon receiving the notification from the SVPM 20, the frame display control unit 302 displays the status of the cluster 10 (power status, abnormal status, configuration status, clock, etc. of the cluster 10) on the screen of the console device. As a result, the cluster state on the screen 309 is dynamically changed in synchronization with the state change of each cluster. Therefore, the operator of the console device 30 can appropriately operate, manage, and control the cluster system while always grasping the state of each cluster.

When the operator performs an operation within the processing frame displayed on the screen of the console device 30, the content on the screen is dynamically changed according to the operation. The operation history is stored in the storage unit 304 in the operation log 314 with a screen hash value, which will be described later.

The screen hash value storage unit 305 stores the hash value calculated by the screen processing unit 311 in the operation log 314 with the screen hash value of the storage unit 304.

Hereinafter, the information stored in the storage unit 304 will be described with reference to FIGS. 3 to 5.

The storage unit 304 stores the console internal state log 312, the cluster state change log 313, and the operation log 314 with the screen hash value. In FIGS. 3 to 5, each information stored in the storage unit 304 is shown in the form of a table for convenience, but the present invention is not limited to this, and various modifications may be performed. can.

The console internal state log 312 is a log showing the history of the CPU usage rate and the process state of the console device 30. The log is referenced by the console status monitoring unit that monitors the status inside the console.

The cluster state change log 313 is a log showing the history of cluster-like changes notified from the cluster 10 via the SVPM20.

FIG. 3 is a diagram illustrating a cluster state change log 313 of the multi-cluster system 1 as an example of the embodiment.

The cluster state change log 313 includes fields of "time" and "state change history". In the field "time", the time when the state of the cluster 10 changes is stored. In the field "state change history", the contents of the state change of the cluster 10 (start, completion, abnormality occurrence, etc.) are stored.

The operation log 314 with a screen hash value is a log in which the operation time and operation content of each operation input from the frame display control unit 302 are associated with the hash value stored by the screen hash value storage unit 305.

FIG. 4 is a diagram illustrating an operation log 314 with a screen hash value of the multi-cluster system 1 as an example of the embodiment.

The operation log 314 with a screen hash value includes fields of "time", "operation history", and "screen hash value at the time of operation". The time when the operation was performed is stored in the field "time". In the field "operation history", the history of operations performed by the console device on the cluster 10 is stored. In the field "Hash value of screen at the time of operation", the hash value calculated from the information of the screen at the time of operation input is stored. For example, the frame display control unit 302 records the content of the input operation performed by the operator as an operation history in the operation log 314 with a screen hash value together with the time when the input operation is performed.

The screen hash value storage unit 305 performs the input operation in the operation log 314 with the screen hash value by using the hash value calculated for the screen image displayed on the display when the operator performs the input operation. Record in association with the operation history entry where the time matches or substantially matches. As a result, in the operation log 314 with the screen hash value, the time when the input operation is performed, the operation history, and the screen hash value at the time of the operation are recorded in association with each other.

The mouse / keyboard driver 306 and the display driver 307 are software for using peripheral devices including input / output devices such as the mouse / keyboard 308 in the console device.

[Functional configuration of fault analysis device 40]
As shown in FIG. 2, the failure analysis device 40 includes a storage unit 401, a memory control unit 402, an analysis table creation unit 403, an extraction unit 404, a first determination unit 405, and a second determination unit 406. Be prepared.

The storage unit 401 stores information used for analysis of the cause of failure. In the present embodiment, the stored information is shown in the form of a list or a table for convenience, but the present invention is not limited to this, and various modifications can be made.

As shown in FIG. 2, the storage unit 401 includes a cluster status error list (cluster status error information) 407, a cluster status change completion list (cluster status change completion information) 408, and a cluster status correspondence table (cluster status correspondence information) 409. The analysis table (analysis information) 410 and the operation type error table (prohibition information) 411 are stored.

The storage control unit 402 acquires the cluster status change log 313 and the operation log 314 with the screen hash value from the console device 30, and based on these logs, the cluster status error list (cluster status error information) 407 and the cluster status change completion. The list (cluster status change completion information) 408, the cluster status correspondence table (cluster status correspondence information) 409, the analysis table (analysis information) 410, and the operation type error table (prohibition information) 411 are stored in the storage unit 304.

The cluster status error list 407 shows the history of errors that have occurred in the cluster 10. The cluster status error list 407 is a list of all status change logs of the cluster 10 that can be displayed in the cluster status change log 313. The log indicating the error status is extracted from the cluster status change log 313 and listed. It is a thing.
The cluster status error list 407 is provided for each cluster 10 provided in the multi-cluster system 1.

FIG. 5 is a diagram illustrating a cluster state error list 407 of the multi-cluster system 1 as an example of the embodiment.

The cluster status error list 407 illustrated in FIG. 5 includes a field of “cluster status error”, and information for identifying the cluster in the error status and the error content are stored in this field.

In the present embodiment, an example in which the cluster name is used as the information for identifying the cluster is shown, but the present invention is not limited to this. For example, as the information for identifying the cluster, the identification information set in advance for each cluster 10 may be used, and various modifications can be made.

The storage control unit 402 may create the cluster state change completion list 408 by, for example, extracting a log indicating that the state change is completed from the cluster state change log 313.

The cluster state change completion list 408 shows the history of state changes that have occurred and completed in the cluster 10. The cluster status change completion list 408 is a list of all cluster status change logs that can be displayed in the cluster status change log 313. The cluster state change completion list 408 is provided for each cluster 10 provided in the multi-cluster system 1.

FIG. 6 is a diagram illustrating a cluster state change completion list 408 of the multi-cluster system 1 as an example of the embodiment.

The cluster state change completion list 408 illustrated in FIG. 6 includes a field of “cluster state change completion”, and this field contains information (cluster name) that identifies the cluster for which the state change has been completed, and the state and change contents. Stored.

The cluster state correspondence table 409 is information indicating the history of the state of the cluster 10 displayed on the screen 309 when the operator inputs an operation in the console device 30.

FIG. 7 is a diagram illustrating a cluster state correspondence table 409 of the multi-cluster system 1 as an example of the embodiment.

The cluster state correspondence table 409 illustrated in FIG. 7 is configured by associating the hash value acquired from the screen hash value storage unit 305 of the console device 30 with the state of the cluster 10 displayed on the screen 309 of the console device 30. Has been done.

That is, the cluster state correspondence table 409 illustrated in FIG. 7 includes each field of "hash value", "state display frame image corresponding to the hash value", and "cluster #n state" (n is a natural number of 0 or more). .. The field "hash value" stores the hash value acquired from the screen hash value storage unit 305. In the field "state display frame image corresponding to the hash value", an image of the state display frame at the time of operation, which is the basis for calculating the hash value, is stored. In the field "cluster #n state", the state of each cluster displayed in the image of the state display frame described above is stored. That is, the value of the field "cluster #n state" indicates the state of each cluster represented by the image of the state display frame described above, and indicates the state of cluster #n represented by the state screen frame.

The hash value stored in the field "hash value" indicates the state of all clusters included in the image of the state display frame. Therefore, by referring to the cluster status correspondence table 409 based on the hash value, it is possible to specify the status of all the clusters displayed on the screen 309 of the console device 30 at the time of operation.

The analysis table creation unit 403 generates the analysis table 410. The analysis table 410 is a table used by the first determination unit 405, which will be described later, in order to identify the cause of the failure.

The method of creating the analysis table 410 by the analysis table creation unit 403 will be described below.

The analysis table creation unit 403 first creates an intermediate table A, which is the predecessor of the analysis table 410, based on the cluster state change log 313.

FIG. 8 is a diagram illustrating an intermediate table A in the multi-cluster system 1 as an example of the embodiment.

As shown in FIG. 8, the intermediate table A has each of "time", "state change history", "operation history", "screen hash value at the time of operation", and "cluster state corresponding to the screen hash value". Have a field.

The analysis table creation unit 403 displays the information stored in the “time” and “state change history” fields of the cluster state change log 313 in the “time” and “state change history” fields of the intermediate table A. Store in series. As a result, the intermediate table A is completed.

Next, the analysis table creation unit 403 creates an intermediate table B, which is the predecessor of the analysis table 410, based on the operation log 314 with the screen hash value and the cluster state correspondence table 409.

FIG. 9 is a diagram illustrating an intermediate table B in the multi-cluster system 1 as an example of the embodiment.

As shown in FIG. 9, the intermediate table B includes the same fields as the intermediate table A.

The analysis table creation unit 403 displays the "time" and "operation history" of the operation log 314 with the screen hash value in the "time", "operation history" and "screen hash value at the time of operation" fields of the intermediate table B. And the information stored in the "hash value of the screen at the time of operation" is stored in chronological order.

Further, the analysis table creation unit 403 extracts the information stored in the field “cluster # n state” (n is a natural number) from the cluster state correspondence table 409, and uses the information as the hash value of the field “screen” of the intermediate table B. Store in the corresponding cluster state.
As a result, the intermediate table B is completed.

The analysis table creation unit 403 completes the analysis table 410 by combining the intermediate gable A and the intermediate table B and arranging them in chronological order. In other words, the analysis table creation unit 403 creates the analysis table 410 by merging the intermediate gable A and the intermediate table B in chronological order.

FIG. 10 is a diagram illustrating an analysis table 410 in the multi-cluster system 1 as an example of the embodiment. Similar to the intermediate tables A and B, the analysis table 410 also has "time", "state change history", "operation history", "screen hash value at the time of operation", and "cluster state corresponding to the screen hash value". Each field is provided. Since these fields in the analysis table 410 are the same as those in the intermediate tables A and B described above, detailed description thereof will be omitted.

The operation type error table 411 is information indicating whether or not the input operation performed in the console device 30 is a prohibited operation (NG operation), and the input operation corresponds to the actual cluster state (NG operation). It is a table showing whether or not it is an operation).

Further, the operation type error table 411 is information indicating whether or not the cluster after inputting the prohibited operation (NG operation) is in the error state (error status), and when the input operation is the prohibited operation (NG operation). , A table showing whether or not an error has occurred in the actual cluster as a result of the input operation.

FIG. 11 is a diagram illustrating an operation type error table 411 in the multi-cluster system 1 as an example of the embodiment.

The operation type error table 411 includes fields of "cluster status", "operation", "operation type", and "error generated by NG operation". The field "cluster state" stores the states that the cluster can take. The field "operation" stores the operation performed in the console device 30. In the field "operation type", OK is stored when the operation is appropriate for the cluster state, and NG is stored when the operation is inappropriate. That is, the operation type error table 411 defines whether or not each input operation is a prohibited operation for each cluster state. Further, in the field "Errors that occur in the NG operation", the error contents that occur in the cluster when the operation is an NG operation are stored.

The extraction unit 404 extracts information used for identifying the cause of the failure from the analysis table 410. The information used to identify the disorder will be described later.

The first determination unit 405 uses the analysis table 410 stored in the storage unit 304 to describe the cluster state of the cluster 10 at the time of operation input and the screen 309 of the console device at the time of operation input for the operations performed in the console device 30. Judge whether or not the cluster status displayed in is the same.

Specifically, the first determination unit 405 uses the cluster state (actual cluster state) of the cluster 10 at the time of operation input and the cluster state (screen) of the cluster specified from the hash value of the screen 309 of the console device at the time of operation input. It is determined whether or not the display matches the cluster status).

The second determination unit 406 refers to the operation type error table 411 stored in the storage unit 304, and determines whether or not the input operation performed in the console device 30 corresponds to a prohibited operation for the cluster state of the cluster at the time of operation input. Is determined.

Specifically, the second determination unit 406 refers to each value of the field "cluster state" and the field "operation" of the operation type error table 411, and in the analysis table 410 determined to match by the first determination unit 405. For the same combination of the cluster state and the input operation, it is determined whether or not the operation is appropriate based on the information stored in the field "operation type" corresponding to this combination.

Further, when the input operation is a prohibited operation, the second determination unit 406 determines whether or not the cluster state of the cluster 10 after the input of the prohibited operation is an error state corresponding to the prohibited operation. That is, when the input operation is a prohibited operation, the second determination unit 406 determines whether or not the cluster state of the cluster 10 after the prohibited operation is an error state corresponding to the prohibited operation. It also functions as a department.

The error state corresponding to the prohibited operation is the error content stored in the field "Error generated in NG operation" in the operation type error table 411. Specifically, the second determination unit 406 refers to the analysis table 410 and the operation type error table 411 stored in the storage unit 304, and the error stored in the field "state change history" of the failure occurrence time in the analysis table 410. It is determined whether or not the content (error status or error message) matches the error content stored in the field "Error generated in NG operation" in the operation type error table 411.

Then, when the cluster state of the cluster 10 after the input of the prohibited operation is an error state, the second determination unit 406 determines that the cause of the failure is an erroneous operation by the operator. If it is not in an error state, it is determined that the cause of the failure is a cluster failure that does not correspond to the prohibited operation.

(B) Operation The process of acquiring the operation log with the screen hash value in the console device 30 of the multi-cluster system 1 as an example of the embodiment will be described with reference to the flowcharts (steps T1 to T4) shown in FIG.

In the following, an example will be illustrated in which the operator operates a mouse or the like to operate a button on the display screen in the console device 30 as an input operation.

In step T1, when a button in the frame on the screen 309 of the console device 30 is pressed, the signal is received by the button processing unit 310 of the frame display control unit 302.

Next, in step T2, the button processing unit 310 extracts the operation time and the operation content “press the button” from the received signal. The button processing unit 310 writes the operation time and the operation content in the operation log with the screen hash value of the storage unit 304.

Subsequently, in step T3, the screen processing unit 311 of the frame display control unit 302 acquires the image data of the state display frame on the screen 309 at the time of operation input, and calculates the hash value of the image information.

In step T4, the screen processing unit 311 stores the hash value in the screen hash value storage unit 305. The screen hash value storage unit 305 records the hash value in association with an entry in the operation history in which the time when the input operation is performed matches or substantially matches in the operation log with the screen hash value of the storage unit 304. After that, the process ends.

Next, processing when a failure occurs in the failure analysis device 40 of the multi-cluster system 1 as an example of the embodiment will be described with reference to the flowcharts (steps S1 to S7) shown in FIG.

In steps S1 to S3, the analysis table creation unit 403 creates the analysis table 410 required for the analysis of the cause of the failure.
In step S1, the analysis table creation unit 403 reads the cluster state change log 313 from the storage unit 304 and creates the intermediate table A.

In step S2, the analysis table creation unit 403 reads the operation log 314 with the screen hash value and the cluster state correspondence table 409 from the storage unit 304, and creates the intermediate table B.

In step S3, the analysis table creation unit 403 completes the analysis table 410 by combining the intermediate tables A and B and arranging them in chronological order.

In steps S4 to S7, the cause of the failure is identified using various tables.
In step S4, the extraction unit 404 extracts information (A) to (F) used for identifying the cause of the failure, which will be described later, from the analysis table.

In step S5, the first determination unit 405 determines whether or not the cluster state of the cluster 10 and the cluster state displayed on the screen 309 of the console device 30 match. If they do not match (No route in step S5), the cause of the failure is that the load inside the console device 30 is high and the drawing process that reflects the cluster status on the screen of the console device 30 is delayed. It is identified as an operation error that occurred. Hereinafter, the cause of the failure caused by the operation error due to the delay in drawing the screen due to the load of the console device 30 may be referred to as pattern 1. If, as a result of the determination in step S5, the cluster state of the cluster 10 and the cluster state displayed on the screen 309 of the console device 30 match (Yes route in step S5), the process proceeds to step S6.

In step S6, the second determination unit 406 further determines whether or not the operation performed in the console device corresponds to a prohibited operation for the cluster state of the cluster at the time of operation. If it does not correspond to the prohibited operation (No route in step S6), the cause of the failure is identified as a cluster failure not caused by the operation, which occurs even though the operation is not a prohibited operation (OK operation). Hereinafter, the cause of failure in which a cluster failure that is not caused by the operation due to the occurrence of a failure may occur in spite of such an OK operation may be referred to as pattern 2. As a result of the determination in step S6, if the operation performed in the console device corresponds to a prohibited operation for the cluster state of the cluster at the time of operation (Yes route in step S6), the process proceeds to step S7.

In step S7, the second determination unit 406 further determines whether or not the cluster state of the cluster after inputting the prohibited operation is an error state corresponding to the prohibited operation.

In the case of an error state (Yes route in step S7), it is determined that the cause of the failure is a simple erroneous operation by the operator. Hereinafter, the cause of failure caused by such a simple operation error (operation error) by the operator may be referred to as pattern 4. On the other hand, if it is not in the error state (No route in step S7), the cause of the failure is identified as a cluster failure that is not a failure due to the prohibited operation. Hereinafter, the cause of failure in which a cluster failure that is not a failure due to such a prohibited operation may occur is referred to as pattern 3.

As described above, in the failure identification process according to the present embodiment, the failure causes of the four patterns 1 to 4 are specified according to the conditional branching.

FIG. 14 is a diagram for explaining four patterns for identifying the cause of failure in a multi-cluster system as an example of an embodiment. In FIG. 14, the failure cause identification method shown in the flowchart of FIG. 13 is shown in a table format. That is, failure causes (cause candidates) classified into patterns 1 to 4 are specified according to the combination of presence / absence of corresponding items for each item shown as conditional branches 1 to 3 in the table shown in FIG.
Hereinafter, the failure cause identification process will be described for each of the failure cause patterns 1 to 4.

[Pattern 1]
15 to 18 are diagrams for explaining the process of identifying the failure cause pattern 1. In pattern 1, when the cluster state of the cluster 10 (actual machine) and the cluster state of the cluster 10 displayed on the screen 309 of the console device 30 are different, an operation is performed in the console device 30 and a failure occurs in the cluster (actual machine). Applies when

The extraction unit 404 extracts information (A) to (F) necessary for identifying the cause of the failure by using the analysis table 410, the cluster state error list 407, and the cluster state change completion list 408 (FIG. See step S4 of 13).
(A) Failure time (B) Time when the cluster status changes to the actual machine at the failure occurrence time (C) Cluster status on the actual machine at the failure occurrence time (D) Operation for the cluster on the screen at the failure occurrence time Time (E) Operation for the cluster on the screen at the time of failure (F) Cluster status on the screen at the time of failure

The extraction unit 404 detects the occurrence of an abnormality in the cluster state by performing a character string search for each word included in the cluster state error list 407 in the field “state change history” of the analysis table 410.

In the example shown in FIG. 15, "cluster # 0 abnormality" is detected in the "state change history" (see reference numeral P1). The extraction unit 404 extracts the time “2019/01/01 00:00:15” corresponding to the history of the detected cluster abnormality occurrence as the failure occurrence time {information (A)} (see reference numeral P2).

Next, the extraction unit 404 detects the occurrence of a state change in the cluster 10 by performing a character string search for each word included in the cluster state change completion list 408 in the field “state change history” of the analysis table 410. do.
In the example shown in FIG. 16, as a result of the search, "completion of change to cluster # 0 state B" is detected in the "state change history" (see reference numeral P3).

In the cluster state change completion list 408 shown in FIG. 16, x of “cluster # x” indicates a variable, and the extraction unit 404 performs a search while sequentially changing the value of this x. The extraction unit 404 also performs a search in the following patterns 2 to 4.

The extraction unit 404 sets the time "2019/01/01 00:00:09" corresponding to the history of the detected state change as the time {information (B)} when the failure occurred and changed to the cluster state on the actual machine. Extract (see reference numeral P4). Further, the extraction unit 404 extracts the "change completed to cluster # 0 state B" of the state change history corresponding to the detected history as the cluster state {information (C)} on the actual machine at the time of failure occurrence. (See reference numeral P5). That is, the extraction unit 404 extracts the cluster state on the actual machine immediately before the error occurs in the cluster 10.

Next, the extraction unit 404 searches the analysis table 410 for the latest operation at the failure occurrence time. Specifically, the extraction unit 404 searches for the input operation performed in the latest past of the failure occurrence time {information (A)} with reference to the operation history of the analysis table 410, and the operation time related to the input operation. ， Acquires the operation details and cluster status.

In the example shown in FIG. 17, the extraction unit 404 is set to the time “2019/01/01 00:” when the most recent operation in the past of “2019/01/01 00:00:15” when the cluster abnormality was detected was performed. "00:11" is extracted as information (D) (see reference numeral P6).

Then, the extraction unit 404 extracts the operation content "operation Y execution" performed at this time "2019/01/01 00:00:11" as information (E) with reference to the operation history of the analysis table 410. (See reference numeral P7). Further, the extraction unit 404 refers to the hash value of the screen at the time of operation of the analysis table 410, and the hash value “123abc” of the screen at the time of operation displayed at this time “2019/01/01 00:00:11”. Is extracted. Further, the extraction unit 404 refers to the cluster state corresponding to the hash value on the screen of the analysis table 410, and indicates the state of the cluster 10 at this time “2019/01/01 00:00:11”, “cluster # 0”. "State A" is extracted as information (F) (see reference numeral P8).

The first determination unit 405 displays the cluster status of cluster # 0 and the screen 309 of the console device 30 in order to identify the cause of the abnormality in cluster # 0 based on the information (A) to (F). It is determined whether or not the cluster state of the cluster # 0 is the same (see step S5 in FIG. 13).

In the example shown in FIG. 18, according to the analysis table 410, at the time “2019/01/01 00:00:09” immediately before the operation, the cluster # 0 has been completely changed to the state B. However, at the operation time "2019/01/01 00:00:11", the state of cluster # 0 displayed on the screen 309 of the console device 30 is "state A", and the cluster (actual machine) cluster. It is recognized that the state does not match (see reference numeral P9). This means that the state change history notified from the cluster 10 every time the state of the cluster 10 changes is not processed by the console device 30. Therefore, since the cluster states do not match, the cause of the failure is that the load inside the console device 30 is high, so that the drawing process for reflecting the cluster state on the screen of the console device 30 is delayed. Is specified.

[Pattern 2]
19 to 23 are diagrams for explaining the process of identifying the failure cause pattern 2. Pattern 2 is an operation appropriate for the cluster state of the cluster 10 (actual machine) when the cluster state of the cluster 10 (actual machine) and the cluster state of the cluster 10 displayed on the screen 309 of the console device 30 match. This corresponds to the case where a failure occurs in the cluster of cluster 10 (actual machine) even though (OK operation) is executed.

Similar to pattern 1, the extraction unit 404 uses the analysis table 410, the cluster state error list 407, and the cluster state change completion list 408 to use information (A) to (F) for identifying the cause of the failure. Is extracted (see step S4 in FIG. 13). Since the contents of the information (A) to (F) are the same as those described above, detailed description thereof will be omitted.

The extraction unit 404 detects the occurrence of an abnormality in the cluster state by performing a character string search for each word included in the cluster state error list 407 in the field “state change history” of the analysis table 410.

In the example shown in FIG. 19, "cluster # 0 command execution failure" is detected in the "state change history" (see reference numeral P1). The extraction unit 404 extracts the time “2019/01/01 00:00:15” corresponding to the history of occurrences of the detected clusters or more as the information generation time {information (A)} (see reference numeral P2).

Next, the extraction unit 404 detects the occurrence of a state change in the cluster 10 by performing a character string search for each word included in the cluster state change completion list 408 in the field “state change history” of the analysis table 410. do.

In the example shown in FIG. 20, as a result of the search, "cluster # 0 ON completed" is detected in the "state change history" (see reference numeral P3). The extraction unit 404 sets the time "2019/01/01 00:00:09" corresponding to the history of the detected state change as the time {information (B)} when the failure occurred and changed to the cluster state on the actual machine. Extract (see reference numeral P4). Further, the extraction unit 404 extracts "cluster # 0 ON completion" of the state change history corresponding to the detected history as information (C) (see reference numeral P5). That is, the extraction unit 404 extracts the cluster state on the actual machine immediately before the error occurs in the cluster 10.

Next, the extraction unit 404 searches the analysis table 410 for the latest operation at the failure occurrence time. Specifically, the extraction unit 404 searches for the input operation performed in the latest past of the failure occurrence time {information (A)} with reference to the operation history of the analysis table 410, and the operation time related to the input operation. ， Acquires the operation details and cluster status.

In the example shown in FIG. 21, the extraction unit 404 is set to the time “2019/01/01 00:” when the most recent operation in the past of “2019/01/01 00:00:15” when the cluster abnormality was detected was performed. "00:11" is extracted as information (D) (see reference numeral P6).

Then, the extraction unit 404 extracts the operation content "operation Y execution" performed at this time "2019/01/01 00:00:11" as information (E) with reference to the operation history of the analysis table 410. (See reference numeral P7). Further, the extraction unit 404 refers to the hash value of the screen at the time of operation of the analysis table 410, and the hash value “456def” of the screen at the time of operation displayed at this time “2019/01/01 00:00:11”. Is extracted. Further, the extraction unit 404 refers to the cluster state corresponding to the hash value on the screen of the analysis table 410, and indicates the state of the cluster 10 at this time “2019/01/01 00:00:11”, “cluster # 0”. "ON" is extracted as information (F) (see reference numeral P8).

The first determination unit 405 displays the cluster status of cluster # 0 and the screen 309 of the console device 30 in order to identify the cause of the abnormality in cluster # 0 based on the information (A) to (F). It is determined whether or not the cluster state of the cluster # 0 is the same (see step S5 in FIG. 13).

In the example shown in FIG. 22, according to the analysis table 410, the state change of cluster # 0 is completed ON at the time “2019/01/01 00:00:09” immediately before the operation. Then, even at the operation time "2019/01/01 00:00:11", the state of cluster # 0 displayed on the screen 309 of the console device 30 is "ON", and the cluster (actual machine) cluster. It is recognized that the state is consistent (see reference numeral P9). Therefore, from the matching of the cluster states, it can be seen that the cause of the failure is other than the delay in the drawing process of the screen of the console device 30 due to the high load inside the console device 30.

Therefore, the second determination unit 406 further determines whether or not the operation performed in the console device 30 corresponds to the prohibited operation for the cluster state of the cluster # 0 at the time of operation based on the operation type error table 411. (See step S6 in FIG. 13).

In the example shown in FIG. 23, according to the analysis table 410, the cluster state {information (C)} of the cluster (actual machine) at the time of failure occurrence is "cluster # 0 ON completed", and a command execution failure is detected. The operation content {information (E)} performed most recently in the past at the specified time is "OFF execution".

The second determination unit 406 refers to the operation type error table 411 in order to determine whether or not the operation content {information (E)} for the cluster state {information (C)} is appropriate. According to the operation type error table 411, the "operation type" corresponding to the "operation" and "OFF" performed for the "cluster state" and "ON" is "OK operation". This means that the operation does not cause an error state in the cluster (actual machine) (see reference numeral P10). Therefore, even though the cluster states are the same and the operation is an NG operation, a failure has occurred. Therefore, the cause of the failure is identified as a failure in the cluster 10 that is not caused by the operation. NS.

[Pattern 3]
24 to 29 are diagrams for explaining the process of identifying the failure cause pattern 3. Pattern 3 is inappropriate for the cluster state of cluster 10 (actual machine) when the cluster state of cluster 10 (actual machine) and the cluster state of cluster 10 displayed on the screen 309 of the console device 30 match. This corresponds to the case where the operation (NG operation) is executed and a failure that does not correspond to the NG operation occurs in the cluster 10 (actual machine).

Similar to patterns 1 and 2, the extraction unit 404 uses the analysis table 410, the cluster state error list 407, and the cluster state change completion list 408 to use information (A) to () for identifying the cause of the failure. F) is extracted (see step S4 in FIG. 13). Since the contents of the information (A) to (F) are the same as those described above, detailed description thereof will be omitted.

The extraction unit 404 detects the occurrence of an abnormality in the cluster state by performing a character string search for each word included in the cluster state error list 407 in the field “state change history” of the analysis table 410.

In the example shown in FIG. 24, "cluster # 0 startup failure" is detected in the "state change history" (see reference numeral P1). The extraction unit 404 extracts the time “2019/01/01 00:00:15” corresponding to the history of the detected cluster abnormality occurrence as the failure occurrence time {information (A)} (see reference numeral P2).

Next, the extraction unit 404 detects the occurrence of a state change in the cluster 10 by performing a character string search for each word included in the cluster state change completion list 408 in the field “state change history” of the analysis table 410. do.

In the example shown in FIG. 25, as a result of the search, "cluster # 0 ON completed" is detected in the "state change history" (see reference numeral P3). The extraction unit 404 extracts the time "2019/01/01 00:00:09" corresponding to the history of the detected state change as the time information (B) that changed to the cluster state on the actual machine at the time when the failure occurred. (See reference numeral P4). Further, the extraction unit 404 extracts "cluster # 0 ON completion" of the state change history corresponding to the detected history as the cluster state {information (C)} on the actual machine at the time of failure occurrence (reference numeral P5). reference). That is, the extraction unit 404 extracts the cluster state on the actual machine immediately before the error occurs in the cluster 10.

Next, the extraction unit 404 searches the analysis table 410 for the latest operation at the failure occurrence time. Specifically, the extraction unit 404 searches for the input operation performed in the latest past of the failure occurrence time {information (A)} with reference to the operation history of the analysis table 410, and the operation time related to the input operation. ， Acquires the operation details and cluster status.

In the example shown in FIG. 26, the extraction unit 404 is set to the time “2019/01/01 00:” when the most recent operation in the past of “2019/01/01 00:00:15” when the cluster abnormality was detected was performed. "00:11" is extracted as information (D) (see reference numeral P6).

Then, the extraction unit 404 extracts the operation content "ON execution" performed at this time "2019/01/01 00:00:11" as information (E) with reference to the operation history of the analysis table 410. (See reference numeral P7).

Further, the extraction unit 404 refers to the hash value of the screen at the time of operation of the analysis table 410, and the hash value “456def” of the screen at the time of operation displayed at this time “2019/01/01 00:00:11”. Is extracted. Further, the extraction unit 404 refers to the cluster state corresponding to the hash value on the screen of the analysis table 410, and indicates the state of the cluster 10 at this time “2019/01/01 00:00:11”, “cluster # 0”. "ON" is extracted as information (F) (see reference numeral P8).

The first determination unit 405 displays the cluster status of cluster # 0 and the screen 309 of the console device 30 in order to identify the cause of the abnormality in cluster # 0 based on the information (A) to (F). It is determined whether or not the cluster state of the cluster # 0 is the same (see step S5 in FIG. 13).

In the example shown in FIG. 27, according to the analysis table 410, the state change of cluster # 0 is completed ON at the time “2019/01/01 00:00:09” immediately before the operation. Then, even at the operation time "2019/01/01 00:00:11", the state of cluster # 0 displayed on the screen 309 of the console device 30 is "ON", and the cluster (actual machine) cluster. It is recognized that the state is consistent (see reference numeral P9). Therefore, from the matching of the cluster states, it can be seen that the cause of the failure is other than the delay in the drawing process of the screen of the console device 30 due to the high load inside the console device 30.

Therefore, the second determination unit 406 further determines whether or not the operation performed in the console device 30 corresponds to the prohibited operation for the cluster state of the cluster # 0 at the time of operation based on the operation type error table 411. (See step S6 in FIG. 13).

In the example shown in FIG. 28, according to the analysis table 410, the cluster status {information (C)} of the cluster (actual machine) at the time of failure occurrence is "cluster # 0 ON completed", and a command execution failure is detected. The operation content {information (E)} performed most recently in the past at the specified time is "ON execution".

The second determination unit 406 refers to the operation type error table 411 in order to determine whether or not the operation content {information (E)} for the cluster state {information (C)} is appropriate. According to the operation type error table 411, the "operation type" corresponding to the "operation" "ON" performed for the "cluster state" "ON" is "NG operation". This means that the operation causes an error state in the cluster (actual machine) (see reference numeral P10). Therefore, since the matching of the cluster states and the operation are NG operations, the cause of the failure is identified as other than the failure in the cluster 10 which is not caused by the operation.

Therefore, the second determination unit further determines whether or not the cluster state of the cluster 10 after inputting the prohibited operation is an error state corresponding to the prohibited operation based on the operation type error table 411 (FIG. 13). See step S7).

In the example shown in FIG. 29, according to the analysis table 410, the cluster state at the failure occurrence time “2019/01/01 00:00:15” is “cluster # 0 startup failure”, and a startup failure is detected. The operation content {information (E)} performed most recently in the past at the specified time is "ON execution".

The second determination unit 406 refers to the operation type error table 411 in order to determine whether or not the cluster state corresponding to the operation content {information (E)} is appropriate. According to the operation type error table 411, the "operation type" corresponding to the "operation" and "ON" performed for the "cluster state" and "ON" is "NG operation", and the error generated by the operation is "Command execution failed". On the other hand, the cluster state in the analysis table 410 is "start failure", which does not match the error content in the operation type error table 411 (see reference numeral P11). Therefore, since the operation is an NG operation and does not match the error content in the operation type error table 411, the cause of the failure is identified as a failure in the cluster (actual machine) that does not correspond to the NG operation.

[Pattern 4]
30 to 35 are diagrams for explaining the process of identifying the failure cause pattern 4. Pattern 4 is an operation inappropriate for the cluster state of cluster 10 (actual machine) when the cluster state of cluster 10 (actual machine) and the cluster state of the cluster displayed on the screen 309 of the console device 30 match. This corresponds to the case where (NG operation) is executed and a failure corresponding to the NG operation occurs in the cluster 10 (actual machine).

Similar to patterns 1 to 3, the extraction unit 404 uses the analysis table 410, the cluster state error list 407, and the cluster state change completion list 408 to use information (A) to () for identifying the cause of the failure. F) is extracted (see step S4 in FIG. 13). Since the contents of the information (A) to (F) are the same as those described above, detailed description thereof will be omitted.

The extraction unit 404 detects the occurrence of an abnormality in the cluster state by performing a character string search for each word included in the cluster state error list 407 in the field “state change history” of the analysis table 410.

In the example shown in FIG. 30, "cluster # 0 command execution failure" is detected in the "state change history" (see reference numeral P1). The extraction unit 404 extracts the time “2019/01/01 00:00:15” corresponding to the history of the detected cluster abnormality occurrence as the failure occurrence time {information (A)} (see reference numeral P2).

Next, the extraction unit 404 detects the occurrence of a state change in the cluster 10 by performing a character string search for each word included in the cluster state change completion list 408 in the field “state change history” of the analysis table 410. do.

In the example shown in FIG. 31, as a result of the search, "cluster # 0 command execution failure" is detected in the "state change history" (see reference numeral P3). The extraction unit 404 sets the time "2019/01/01 00:00:09" corresponding to the history of the detected state change as the time {information (B)} when the failure occurred and changed to the cluster state on the actual machine. Extract (see reference numeral P4). Further, the extraction unit 404 extracts the "cluster # 0 command execution failure" of the state change history corresponding to the detected history as the cluster state {information (C)} on the actual machine at the time of failure occurrence (reference numeral (code). See page 5). That is, the extraction unit 404 extracts the cluster state on the actual machine immediately before the error occurs in the cluster 10.

Next, the extraction unit 404 searches the analysis table 410 for the latest operation at the failure occurrence time. Specifically, the extraction unit 404 searches for the input operation performed in the latest past of the failure occurrence time {information (A)} with reference to the operation history of the analysis table 410, and the operation time related to the input operation. ， Acquires the operation details and cluster status.

In the example shown in FIG. 32, the extraction unit 404 is set to the time “2019/01/01 00:” when the most recent operation in the past of “2019/01/01 00:00:15” when the cluster abnormality was detected was performed. "00:11" is extracted as information (D) (see reference numeral P6).

Then, the extraction unit 404 extracts the operation content "ON execution" performed at this time "2019/01/01 00:00:11" as information (E) with reference to the operation history of the analysis table 410. (See reference numeral P7).

Further, the extraction unit 404 refers to the hash value of the screen at the time of operation of the analysis table 410, and the hash value “456def” of the screen at the time of operation displayed at this time “2019/01/01 00:00:11”. Is extracted. Further, the extraction unit 404 refers to the cluster state corresponding to the hash value on the screen of the analysis table 410, and indicates the state of the cluster 10 at this time “2019/01/01 00:00:11”, “cluster # 0”. "ON" is extracted as information (F) (see reference numeral P8).

The first determination unit 405 displays the cluster status of cluster # 0 and the screen 309 of the console device 30 in order to identify the cause of the abnormality in cluster # 0 based on the information (A) to (F). It is determined whether or not the cluster state of the cluster # 0 is the same (see step S5 in FIG. 13).

In the example shown in FIG. 33, according to the analysis table 410, the state change of cluster # 0 is completed ON at the time “2019/01/01 00:00:09” immediately before the operation. Then, even at the operation time "2019/01/01 00:00:11", the state of cluster # 0 displayed on the screen 309 of the console device 30 is "ON", and the cluster (actual machine) cluster. It is recognized that the state is consistent (see reference numeral P9). Therefore, from the matching of the cluster states, it can be seen that the cause of the failure is other than the delay in the drawing process of the screen of the console device 30 due to the high load inside the console device 30.

Therefore, the second determination unit 406 further determines whether or not the operation performed in the console device 30 corresponds to the prohibited operation for the cluster state of the cluster # 0 at the time of operation based on the operation type error table 411. (See step S6 in FIG. 13).

In the example shown in FIG. 34, according to the analysis table 410, the cluster state {information (C)} of the cluster (actual machine) at the time of failure occurrence is "cluster # 0 ON completed", and a command execution failure is detected. The operation content {information (E)} performed most recently in the past at the specified time is "ON execution".

The second determination unit 406 refers to the operation type error table 411 in order to determine whether or not the operation content {information (E)} for the cluster state {information (C)} is appropriate. According to the operation type error table 411, the "operation type" corresponding to the "operation" "ON" performed for the "cluster state" "ON" is "NG operation". This means that the operation causes an error state in the cluster (actual machine) (see reference numeral P10). Therefore, since the matching of the cluster states and the operation are NG operations, the cause of the failure is identified as other than the failure in the cluster 10 which is not caused by the operation.

Therefore, the second determination unit further determines whether or not the cluster state of the cluster 10 after inputting the prohibited operation is an error state corresponding to the prohibited operation based on the operation type error table 411 (FIG. 13). See step S7).

In the example shown in FIG. 35, according to the analysis table 410, the cluster state at the failure occurrence time “2019/01/01 00:00:15” is “cluster # 0 command execution failure”, and the command execution failure occurs. The operation content {information (E)} performed most recently in the past at the detected time is "ON execution".

The second determination unit 406 refers to the operation type error table 411 in order to determine whether or not the cluster state corresponding to the operation content {information (E)} is appropriate. According to the operation type error table 411, the "operation type" corresponding to the "operation" and "ON" performed for the "cluster state" and "ON" is "NG operation", and the error generated by the operation is "Command execution failed".

The cluster state in the analysis table 410 is "command execution failure", which matches the error content in the operation type error table 411 (see reference numeral P11). Therefore, since the operation is an NG operation and matches the error content of the operation type error table 411, the cause of the failure is identified as a simple operation error.

(C) Effect As described above, according to the multi-cluster system 1 as an example of the embodiment, regarding the input operation performed by the console device 30 via the input screen in the first determination unit 405 of the failure analysis device 40. It is determined whether or not the cluster state (real cluster state) of the cluster at the time of the input operation and the cluster state of the cluster 10 on the input screen (cluster state on the screen display) match.

As a result, when the cluster states do not match, the cause of the failure in the multi-cluster system 1 is the drawing process that reflects the cluster state on the screen of the console device 30 due to the high load inside the console device 30. It is identified that a delay has occurred. Therefore, since it is known that the operator has performed the operation based on the information on the screen 309 of the console device 30, it is possible to prevent the misunderstanding that the operator makes an erroneous operation.

Further, in the second determination unit 406 of the fault analysis device 40, the cluster state of the cluster (actual cluster state) at the time of the input operation and the cluster state of the cluster 10 on the input screen (cluster state on the screen display) match. In this case, it is determined whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation.

Thereby, when the operation does not correspond to the prohibited operation (OK operation), the cause of the failure in the multi-cluster system 1 can be identified as a cluster failure not caused by the operation.

Further, in the second determination unit 406 of the failure analysis device 40, when the input operation is a prohibited operation, it is determined whether or not the cluster state of the cluster 10 after the prohibited operation is an error state corresponding to the prohibited operation. do.

As a result, if the error state does not correspond to the prohibited operation, the cause of the failure in the multi-cluster system 1 is identified as the failure in the cluster (actual machine) that does not correspond to the prohibited operation.

On the other hand, when the error state does not correspond to the prohibited operation, the cause of the failure in the multi-cluster system 1 is identified as a simple operation error.

Analysis that associates the state change history information indicating the state change of the cluster, the operation history information indicating the operation performed in the console device, and the screen information of the console device at the time of the operation in the storage unit of the failure analysis device 40. Store table 410.

As a result, it is possible to associate the actual cluster state of the cluster before and after the operation with the operation content performed by the console device on the cluster and the cluster state of the cluster on the screen of the console device at the time of operation.

In this embodiment, the screen information is calculated as a hash value. As an image acquisition method, there is also a method of capturing a "screen" or shooting a video. Compared with these methods, the method of calculating the hash value has a small amount of data of screen information to be saved, and is highly secure in terms of security. In addition, the operation of the operator and the operation time saved as a history match or substantially match.

In the storage unit of the fault analysis device 40, the cluster state correspondence table 409 that associates the screen information of the console device 30 at the time of operation with the cluster state of the cluster 10 identified from the screen information is stored.

As a result, the hash value of the screen can be associated with the cluster state of the cluster on the screen of the console device at the time of operation.

In the storage unit of the fault analysis device 40, the operation type error table 411 in which the cluster state and the prohibited operation are associated with each other is stored.

Thereby, the cluster state of the cluster 10 can be associated with the operation performed by the console device 30.

Further, in the second determination unit 406 of the fault analysis device 40, the input operation can be performed on the cluster at the time of operation by referring to the operation type error table 411 based on the cluster state (actual cluster state) of the cluster at the time of input operation. Determine if it corresponds to a prohibited operation for the cluster status.

Thereby, it is possible to identify whether or not the operation is appropriate for the cluster state and the error state that occurs in the cluster 10 according to the type of operation (OK operation / NG operation).

Further, the analysis table creation unit 403 of the failure analysis device 40 creates the analysis table 410 by arranging the state change history information and the operation history information in chronological order.

Thereby, the information used for identifying the obstacles of the above patterns 1 to 4 can be extracted.

In the first determination unit 405 of the fault analysis device 40, referring to the analysis table 410, the cluster state of the cluster at the time of the input operation (real cluster state) and the cluster state of the cluster 10 on the input screen (cluster state on the screen display). Determine if they match.

As a result, when the cluster states do not match, the cause of the failure in the multi-cluster system 1 is the drawing process that reflects the cluster state on the screen of the console device 30 due to the high load inside the console device 30. It is identified that a delay has occurred.

The analysis table further includes screen information of the console device 30 during operation.

Further, in the analysis table creation unit 403 of the failure analysis device 40, the cluster state of the cluster specified from the screen information is acquired by referring to the cluster state correspondence information based on the screen information of the console device 30.

Thereby, the cluster state of the cluster 10 (cluster state on the screen display) on the input screen of the console device 30 can be specified.

(D) Others [Hardware configuration of console device 30]
FIG. 36 is a diagram illustrating a hardware configuration of the console device 30 of the multi-cluster system 1 as an example of the embodiment.

The console device 30 is, for example, a computer having a server function, and is a CPU (Central Processing Unit; processor) 3001, a memory 3002, a storage unit 3003, a network IF (Interface) unit 3004, an output unit 3005, an input unit 3006, and a device. A connection IF unit 3007 may be provided. These devices are interconnected by a network 3008.

The CPU 3001 executes an OS (Operating System) or an arbitrary program stored in a storage unit 3003 or the like described later, and performs an operation on the cluster 10 described later, for example.

The memory 3002 is an example of hardware for storing various data and programs. Examples of the memory 3002 include volatile memories such as RAM (Random Access Memory) and non-volatile memories such as flash memory, SCM, and ROM (Read Only Memory).

The storage unit 3003 is an example of hardware for storing various data, programs, and the like. For example, the storage unit 3003 may be used as a secondary storage device of the console device 30, and may store programs such as an OS, firmware, and applications, and various data. Examples of the storage unit 3003 include a magnetic disk device such as an HDD (Hard Disk Drive) and a semiconductor storage device such as an SSD (Solid State Drive) and an SCM (Storage Class Memory). Further, the storage unit 3003 may store a program that realizes all or a part of various functions of the console device 30.

The network IF unit 3004 is an example of a communication interface that controls connection and communication between the cluster 10, the SVPM20, and the fault analysis device 40 via a network (not shown). For example, the network IF unit 3004 includes an adapter compliant with Ethernet (registered trademark), optical communication (for example, Fiber Channel), and the like.

The output unit 3005 may include, for example, at least one of a display 3015a and an output device such as a projector, a speaker, and a printer.

The input unit 3006 may include, for example, at least one of an input device such as a keyboard 3016a (308), a mouse 3016b (308), a touch panel, and operation buttons.

The device connection IF unit 3007 is a communication interface for connecting peripheral devices to the console device 30. For example, a memory device 3017a or a memory reader / writer 3017b can be connected to the device connection IF unit 3007. The memory device 3017a is a non-temporary recording medium equipped with a communication function with the device connection IF unit 3007, for example, a USB (Universal Serial Bus) memory. The memory reader / writer 3017b writes data to the memory card 3017c or reads data from the memory card 3017c. The memory card 3017c is a card-type non-temporary recording medium.

In the present embodiment, the console device 30 may transmit various logs stored in the storage unit 3003 to the failure analysis device 40 via a wired or wireless network. Alternatively, the console device 30 may write the various logs to the memory device 3017a or the memory card 3017c.

[Hardware configuration of fault analysis device 40]
FIG. 37 is a diagram illustrating a hardware configuration of the failure analysis device 40 of the multi-cluster system 1 as an example of the embodiment.

The fault analysis device 40 is, for example, a computer having a server function, and is a CPU (Central Processing Unit; processor) 4001, a memory 4002, a storage unit 4003, a network IF (Interface) unit 4004, an output unit 4005, an input unit 4006, and The device connection IF unit 4007 may be provided. These devices are interconnected by a network 4008.

The CPU 4001 executes an OS (Operating System) or an arbitrary program stored in a storage unit 4003 or the like described later, and performs an operation on the cluster 10 described later, for example.

The memory 4002 is an example of hardware for storing various data and programs. Examples of the memory 4002 include volatile memories such as RAM (Random Access Memory) and non-volatile memories such as flash memory, SCM, and ROM (Read Only Memory).

The storage unit 4003 is an example of hardware for storing various data, programs, and the like. For example, the storage unit 4003 may be used as a secondary storage device of the console device 30, and may store programs such as an OS, firmware, and applications, and various data. Examples of the storage unit 4003 include a magnetic disk device such as an HDD (Hard Disk Drive) and a semiconductor storage device such as an SSD (Solid State Drive) and an SCM (Storage Class Memory). Further, the storage unit 4003 may store a program that realizes all or a part of various functions of the console device 30.

The network IF unit 4004 is an example of a communication interface that controls connection and communication between the cluster 10, the SVPM20, and the fault analysis device 40 via a network (not shown). For example, the network IF unit 4004 includes an adapter compliant with Ethernet (registered trademark), optical communication (for example, Fiber Channel), and the like.

The output unit 4005 may include, for example, at least one of a display 4015a and an output device such as a projector, a speaker, and a printer.

The input unit 4006 may include, for example, at least one of an input device such as a keyboard 4016a, a mouse 4016b, a touch panel, and operation buttons.

The device connection IF unit 4007 is a communication interface for connecting peripheral devices to the console device 30. For example, a memory device 4017a or a memory reader / writer 4017b can be connected to the device connection IF unit 4007. The memory device 4017a is a non-temporary recording medium equipped with a communication function with the device connection IF unit 3007, for example, a USB (Universal Serial Bus) memory. The memory reader / writer 4017b writes data to the memory card 4017c or reads data from the memory card 4017c. The memory card 4017c is a card-type non-temporary recording medium.

In the present embodiment, the fault analysis device 40 may transmit the information stored in the storage unit 4003 to the console device 30 via a wired or wireless network. Alternatively, the fault analysis device 40 may write the various logs to the memory device 4017a or the memory card 4017c.

The disclosed technique is not limited to the above-described embodiment, and can be variously modified and implemented without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected as necessary, or may be combined as appropriate.

Regardless of the above-described embodiment, various modifications can be made without departing from the spirit of the present embodiment.

Further, according to the above-mentioned disclosure, it is possible for a person skilled in the art to carry out and manufacture the present embodiment.

(E) Additional notes The following additional notes will be further disclosed with respect to the above embodiments.

(Appendix 1)
A multi-cluster system including a cluster, a console device that controls the cluster, and a failure analysis device that analyzes failures.
The fault analysis device
First determination for determining whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with respect to the input operation performed via the input screen in the console device. Department and
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A multi-cluster system comprising a second determination unit for determining.

(Appendix 2)
The fault analysis device
When the input operation is a prohibited operation, a third determination unit for determining whether or not the cluster state of the cluster after the prohibited operation is an error state corresponding to the prohibited operation is provided. The multi-cluster system according to Appendix 1.

(Appendix 3)
The fault analysis device
A storage unit that stores analysis information that associates state change history information indicating a state change of the cluster, operation history information indicating an operation performed in the console device, and screen information of the console device at the time of the operation is provided. ,
The first determination unit refers to the analysis information and determines whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen. , The multi-cluster system according to Appendix 1 or 2.

(Appendix 4)
The fault analysis device
The multi-cluster system according to Appendix 3, further comprising an analysis information creating unit that creates the analysis information by arranging the state change history information and the operation history information in chronological order.

(Appendix 5)
The analysis information further includes screen information of the console device at the time of the operation.
The analysis information creation unit obtains the cluster state of the cluster specified from the screen information by referring to the cluster state correspondence information based on the screen information of the console device. Described multi-cluster system.

(Appendix 6)
The fault analysis device
A storage unit for storing prohibited information in which the cluster state and the prohibited operation are associated with each other is provided.
The second determination unit refers to the prohibition information based on the cluster state of the cluster at the time of the input operation, and whether or not the input operation corresponds to the prohibition operation for the cluster state of the cluster at the time of the operation. The multi-cluster system according to any one of Appendix 1 to 5, wherein the multi-cluster system is characterized in that.

(Appendix 7)
It is a failure analysis device provided in a multi-cluster system including a cluster and a console device that controls the cluster, and analyzes a failure that occurs in the multi-cluster system.
First determination for determining whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with respect to the input operation performed via the input screen in the console device. Department and
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A fault analysis device comprising a second determination unit for determining.

(Appendix 8)
When the input operation is a prohibited operation, a third determination unit for determining whether or not the cluster state of the cluster after the prohibited operation is an error state corresponding to the prohibited operation is provided. The fault analysis device according to Appendix 7.

(Appendix 9)
A storage unit that stores analysis information that associates state change history information indicating a state change of the cluster, operation history information indicating an operation performed in the console device, and screen information of the console device at the time of the operation is provided. ,
The first determination unit refers to the analysis information and determines whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen. , The fault analysis apparatus according to Appendix 7 or 8.

(Appendix 10)
The failure analysis apparatus according to Appendix 9, further comprising an analysis information creating unit that creates the analysis information by arranging the state change history information and the operation history information in chronological order.

(Appendix 11)
The analysis information further includes screen information of the console device at the time of the operation.
The analysis information creation unit is described in Appendix 10 characterized in that the cluster state of the cluster specified from the screen information is acquired by referring to the cluster state correspondence information based on the screen information of the console device. Fault analysis device.

(Appendix 12)
A storage unit for storing prohibited information in which the cluster state and the prohibited operation are associated with each other is provided.
The second determination unit refers to the prohibition information based on the cluster state of the cluster at the time of the input operation, and whether or not the input operation corresponds to the prohibition operation for the cluster state of the cluster at the time of the operation. The fault analysis apparatus according to any one of Supplementary note 7 to 11, wherein the fault analysis apparatus is characterized by determining whether or not the data is used.

(Appendix 13)
A processor of a failure analysis device provided in a multi-cluster system including a cluster and a console device for controlling the cluster and analyzing a failure occurring in the multi-cluster system.
With respect to the input operation performed via the input screen in the console device, it is determined whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen.
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A failure analysis program that executes the process of determining.

(Appendix 14)
When the input operation is a prohibited operation, the processor is made to execute a process of determining whether or not the cluster state of the cluster after the prohibited operation is an error state corresponding to the prohibited operation. The fault analysis program described in.

(Appendix 15)
The state change history information indicating the state change of the cluster, the operation history information indicating the operation performed in the console device, and the analysis information relating to the screen information of the console device at the time of the operation are stored.
Appendix 13 or 14 for causing the processor to execute a process of determining whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with reference to the analysis information. The fault analysis program described in.

(Appendix 16)
The failure analysis program according to Appendix 15, which causes the processor to execute a process of creating analysis information by arranging the state change history information and the operation history information in chronological order.

(Appendix 17)
The analysis information further includes screen information of the console device at the time of the operation.
The failure analysis program according to Appendix 16, which causes the processor to execute a process of referring to cluster state correspondence information based on the screen information of the console device and acquiring the cluster state of the cluster specified from the screen information.

(Appendix 18)
The prohibition information associated with the cluster state and the prohibition operation is stored, and the prohibition information is stored.
By referring to the prohibition information based on the cluster state of the cluster at the time of the input operation, the process of determining whether or not the input operation corresponds to the prohibition operation for the cluster state of the cluster at the time of the operation is described. The failure analysis program according to any one of Appendix 13 to 17, which is executed by a processor.

(Appendix 19)
A failure analysis method in a multi-cluster system including a cluster and a console device that controls the cluster.
Regarding the input operation performed via the input screen in the console device, it is determined whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen.
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A failure analysis method characterized by determining.

(Appendix 20)
When the input operation is a prohibited operation, it is described in Appendix 19 that it is determined whether or not the cluster state of the cluster after the prohibited operation is an error state corresponding to the prohibited operation. Failure analysis method.

(Appendix 21)
The state change history information indicating the state change of the cluster, the operation history information indicating the operation performed in the console device, and the analysis information relating to the screen information of the console device at the time of the operation are stored.
19 or 20, wherein it is determined whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with reference to the analysis information. Failure analysis method.

(Appendix 22)
The failure analysis method according to Appendix 21, wherein analysis information is created by arranging the state change history information and the operation history information in chronological order.

(Appendix 23)
The analysis information further includes screen information of the console device at the time of the operation.
The failure analysis method according to Appendix 22, wherein the cluster state correspondence information is referred to based on the screen information of the console device, and the cluster state of the cluster specified from the screen information is acquired.

(Appendix 24)
The prohibition information associated with the cluster state and the prohibition operation is stored, and the prohibition information is stored.
By referring to the prohibition information based on the cluster state of the cluster at the time of the input operation, it is determined whether or not the input operation corresponds to the prohibition operation for the cluster state of the cluster at the time of the operation. The fault analysis method according to any one of Appendix 19 to 23.

1 Multi-cluster system 10-1 Cluster # 0
10-2 Cluster # 1
20 SVPM
30 Console device 40 Fault analysis device 50 LAN network 301 Cluster status change detection unit 302 Frame display control unit 303 Console status monitoring unit 304 Storage unit 305 Screen hash value storage unit 306 Mouse / keyboard driver 307 Display driver 308 Keyboard / mouse 309 Screen 310 Button processing unit 311 Screen processing unit 312 Console internal status log 313 Cluster status change log 314 Operation log with screen hash value 401 Storage unit 402 Storage control unit 403 Analysis table creation unit 404 Extraction unit 405 First judgment unit 406 Second judgment unit 407 Cluster status error list 408 Cluster status change completion list 409 Cluster status correspondence table 410 Analysis table 411 Operation type error table 3001 Processor (CPU)
3002 Memory 3003 Storage 3004 Network IF (Interface) 3005 Output 3006 Input 3007 Device connection IF 3008 Network 3015a Display 3016a Keyboard 3016b Mouse 3017a Memory device 3017b Memory reader / writer 3017c Memory card 4001 Processor (CPU)
4002 Memory 4003 Storage 4004 Network IF (Interface) 4005 Output 4006 Input 4007 Device connection IF 4008 Network 4015a Display 4016a Keyboard 4016b Mouse 4017a Memory device 4017b Memory reader / writer 4017c Memory card

Claims (9)

It is a failure analysis device provided in a multi-cluster system including a cluster and a console device that controls the cluster, and analyzes a failure that occurs in the multi-cluster system.
First determination for determining whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with respect to the input operation performed via the input screen in the console device. Department and
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A fault analysis device comprising a second determination unit for determining. When the input operation is a prohibited operation, a third determination unit for determining whether or not the cluster state of the cluster after the prohibited operation is an error state corresponding to the prohibited operation is provided. The fault analysis device according to claim 1. A storage unit that stores analysis information that associates state change history information indicating a state change of the cluster, operation history information indicating an operation performed in the console device, and screen information of the console device at the time of the operation is provided. ,
The first determination unit refers to the analysis information and determines whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen. , The fault analysis apparatus according to claim 1 or 2. The failure analysis apparatus according to claim 3, further comprising an analysis information creating unit that creates analysis information by arranging the state change history information and the operation history information in chronological order. The analysis information further includes screen information of the console device at the time of the operation.
The fourth aspect of the present invention is that the analysis information creation unit acquires the cluster state of the cluster specified from the screen information by referring to the cluster state correspondence information based on the screen information of the console device. The fault analyzer described. A storage unit for storing prohibited information in which the cluster state and the prohibited operation are associated with each other is provided.
The second determination unit refers to the prohibition information based on the cluster state of the cluster at the time of the input operation, and whether or not the input operation corresponds to the prohibition operation for the cluster state of the cluster at the time of the operation. The fault analysis apparatus according to any one of claims 1 to 5, wherein the fault analysis apparatus is characterized by determining whether or not the data is used. A multi-cluster system including a cluster, a console device that controls the cluster, and a failure analysis device that analyzes failures.
The fault analysis device
First determination for determining whether or not the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match with respect to the input operation performed via the input screen in the console device. Department and
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A multi-cluster system comprising a second determination unit for determining. A processor of a failure analysis device provided in a multi-cluster system including a cluster and a console device for controlling the cluster and analyzing a failure occurring in the multi-cluster system.
With respect to the input operation performed via the input screen in the console device, it is determined whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen.
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A failure analysis program that executes the process of determining. A failure analysis method in a multi-cluster system including a cluster and a console device that controls the cluster.
Regarding the input operation performed via the input screen in the console device, it is determined whether or not the cluster state of the cluster at the time of the input operation matches the cluster state of the cluster on the input screen.
When the cluster state of the cluster at the time of the input operation and the cluster state of the cluster on the input screen match, whether or not the input operation corresponds to a prohibited operation for the cluster state of the cluster at the time of the operation. A failure analysis method characterized by determining.

Images