WO2016120989A1

WO2016120989A1 - Management computer and rule test method

Info

Publication number: WO2016120989A1
Application number: PCT/JP2015/052164
Authority: WO
Inventors: 淳北脇; 峰義増田; 裕工藤
Original assignee: 株式会社日立製作所
Priority date: 2015-01-27
Filing date: 2015-01-27
Publication date: 2016-08-04

Abstract

A management computer for testing a rule for monitoring a system comprising a plurality of devices, wherein a storage unit of said management computer stores event information for managing past failure events and configuration change events in the system, configuration information for managing the current configuration of the system, and a topology snapshot representing a past configuration of the system, and wherein said management computer is provided with: a configuration information reproduction unit which reproduces a past configuration of the system by use of the aforementioned configuration information and information about the configuration change events, and causes the reproduced past configuration to be reflected in the topology snapshot; and a rule testing unit which applies a cause analysis algorithm to the reproduced past configuration, deduces the cause of a failure using a rule to be tested, and outputs a determination as to whether the rule to be tested can be used to detect the failure.

Description

Management computer and rule test method

The present invention relates to a test apparatus for testing a rule for monitoring a computer system.

Since there are many nodes to be managed in a large-scale environment such as a data center, in many cases, faults are monitored using RCA (Route Cause Analysis). By using RCA, a failure can be found or a failure can be easily detected.

The administrator can detect failures with RCA, but failures that cannot be detected with RCA also occur. Therefore, by creating a new rule or modifying an existing rule with RCA, it is possible to improve the accuracy of detecting a failure and construct a system that can respond to the failure more quickly.

∙ Whether a new rule or a modified rule can detect a failure that could not be detected before must be executed in the environment where the failure occurred. However, since the configuration changes every moment while the system is in operation, the exact same environment cannot be prepared. Also, testing new or modified rules on a running system can affect the services being offered. Therefore, a technique for preparing a test environment that is the same as the production environment separately from the production environment and testing the rules has been proposed.

As background arts of the present technology, there are JP 2012-185599 A (Patent Document 1) and JP 2010-231568 A (Patent Document 2). Patent Document 1 includes a work information storage unit that stores work information including one or more execution information including a command executed for program modification in a test environment and result information that is information related to a command execution result. A program storage unit for storing a program in a production environment, a command execution unit for modifying the program by executing each command included in the work information for the program stored in the program storage unit, and command execution An information processing system comprising: a determination unit that determines whether a command execution result by a unit matches result information included in the same execution information as the command; and an output unit that outputs a determination result by the determination unit Has been. Patent Document 2 describes a system for determining an event by using only an event log accumulated in the past.

JP 2012-185599 A JP 2010-231568 A

In a small environment, it is possible to physically construct the same test environment as the production environment, but in a large and complex environment, it is difficult to construct the same test environment as the production environment. In general, the test environment has a small configuration that is different from the production environment. The load on the system and the system behavior are different from the production environment, so there was no problem in the production environment because there was no problem in the test environment. It cannot be said that there is no.

Therefore, it is a technical challenge to prepare an environment for testing whether a new rule or a corrected rule can be detected for a failure that could not be detected in a large-scale and complex environment.

Also, rules for faults that have not been created as events in the past cannot be tested using past events.

A typical example of the invention disclosed in the present application is as follows. That is, a management computer for testing a rule for monitoring a system composed of a plurality of devices, comprising a storage unit and a processor that refers to the storage unit, the storage unit in the system in the past The management computer stores event information for managing a failure event that has occurred and a configuration change event for the system, configuration information for managing a current configuration of the system, and a topology snapshot representing a past configuration of the system. Reconstructing the past configuration of the system using the configuration information and the configuration change event, and applying a cause analysis algorithm to the reconstructed past configuration, and a configuration information reproducing unit that records the configuration snapshot in the topology snapshot. A rule test section that estimates the cause of the failure using the rule to be tested and outputs whether the failure can be detected. That.

According to one aspect of the present invention, the same test environment as the production environment can be logically configured, and a new rule can be tested using event information generated in the production environment. Problems, configurations, and effects other than those described above will become apparent from the description of the following embodiments.

It is a block diagram which shows the structure of the rule test apparatus of 1st Example which concerns on this invention, and the relationship between a rule test apparatus, a test environment, and a production environment. It is a flowchart of operation | movement of the rule test in the rule test apparatus of 1st Example of this invention. It is a flowchart of the detail of the process which determines the structure of the past data used for the test of 1st Example of this invention. It is a flowchart of the detail of the process which tests the RCA rule of 1st Example of this invention. It is a flowchart of the detail of the failure event process of 1st Example of this invention. It is a figure which shows the structural example of the event table of 1st Example of this invention. It is a figure which shows the structural example of the co-occurrence | waiting event table of RCA of 1st Example of this invention. It is a flowchart of the detail of the structure event process of 1st Example of this invention. It is a figure which shows the structural example of the RCA rule table of 1st Example of this invention. It is a figure which shows the example of the input screen of 1st Example of this invention. It is a figure which shows the example of the output screen of 1st Example of this invention. It is a figure which shows the structural example of the RCA rule test result table of 1st Example of this invention. It is a figure which shows the method of determining the line which starts the analysis of the RCA rule of 1st Example of this invention. It is a block diagram which shows the structure of the rule test apparatus of 2nd Example of this invention, and the relationship between a rule test apparatus, a test environment, and a production environment. It is a figure which shows the structural example of the rule history table of 2nd Example of this invention. It is a figure which shows the example of the output screen of 2nd Example of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be denoted by the same numbers. The accompanying drawings show specific embodiments and implementation examples based on the principle of the present invention, but these are for understanding the present invention and are not intended to limit the present invention. Not used.

The present embodiment has been described in sufficient detail for those skilled in the art to practice the present invention, but other implementations and forms are possible without departing from the scope and spirit of the technical idea of the present invention. It is necessary to understand that the configuration and structure can be changed and various elements can be replaced. Therefore, the following description should not be interpreted as being limited to this.

Furthermore, as will be described later, the embodiment of the present invention may be implemented by software running on a general-purpose computer, or may be implemented by dedicated hardware or a combination of software and hardware.

In the following description, each information of the present invention will be described in a “table” format. However, the information does not necessarily have to be expressed in a data structure by a table. It may be expressed in other ways. Therefore, “table”, “matrix”, “list”, “DB”, “queue”, and the like may be simply referred to as “information” in order to indicate that they do not depend on the data structure.

In addition, when explaining the contents of each information, the expressions “identification information”, “identifier”, “name”, “name”, “ID” can be used, and these can be replaced with each other. It is.

In the following, each process in the embodiment of the present invention will be described using “program” as the subject (operation subject). However, the program is executed by the processor, and the process determined by the memory and the communication port (communication control device) Since it is performed while being used, the description may be made with the processor as the subject. Further, the processing disclosed with the program as the subject may be processing performed by a computer such as a management server or an information processing apparatus. Part or all of the program may be realized by dedicated hardware, or may be modularized. Various programs may be installed in each computer by a program distribution server or a storage medium.

<First embodiment>
First, the test environment of the first embodiment will be described with reference to FIGS.

FIG. 1 is a block diagram showing the configuration of the rule test apparatus 100 of the first embodiment and the relationship between the rule test apparatus 100, the test environment, and the production environment.

As shown in FIG. 1, the test environment of the present embodiment has a rule test apparatus 100.

The rule test apparatus 100 is connected to the RCA server 500 through the test network 200. In this embodiment, the rule test apparatus 100 is connected to the RCA server 500 via the test network 200, but may be connected to the RCA server 500 via the management network 300. The rule test apparatus 100 is connected to the terminal 800 via the test network 200. The terminal 800 displays an input screen and an output screen, which will be described later, in accordance with instructions from the rule testing apparatus 100.

The rule test apparatus 100 includes a CPU 110, a memory 111, a communication device 112, an input device 113, an output device 114, a media reading device 115, and an auxiliary storage device 120.

The CPU 110 is a processor that performs various calculations by executing a program. The memory 111 includes a ROM that is a nonvolatile storage element and a RAM that is a volatile storage element. The ROM stores an immutable program (for example, BIOS). The RAM is a high-speed and volatile storage element such as a DRAM (Dynamic Random Access Memory), temporarily stores a program stored in the auxiliary storage device 120 and data used when the program is executed. It becomes a work area. The communication device 112 is a network interface device that controls communication with the RCA server 500 in accordance with a predetermined protocol.

The input device 113 is a user interface (for example, a keyboard, a mouse, etc.) for the user to input data and instructions to the rule testing device 100. The output device 114 is a user interface (for example, a display device or a printer) for presenting the execution result of the program to the user. The media reader 115 is an interface device that reads data stored in a storage medium such as a DVD drive. Note that the media reading device 115 may not be provided.

The auxiliary storage device 120 is a large-capacity and nonvolatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 120 stores a configuration information table 130, an event table 140, a co-occurrence waiting event table 150, an RCA rule table 160, a topology snapshot 170, and an RCA rule test result table 180.

The configuration information table 130 stores configuration information of the server 600 and the storage device 700 that are connected to and monitored by the RCA server 500 via the management network 300. The event table 140 stores event information created based on performance information and configuration information collected from the server 600 and storage device 700 monitored by the RCA server 500 (see FIG. 6). The co-occurrence wait event table 150 stores events within the retention period (see FIG. 7). The RCA rule table 160 stores RCA rules held by the RCA server 500 (see FIG. 9). The topology snapshot 170 is information that reproduces the configuration information of the server 600 and the storage apparatus 700 at an arbitrary time. The RCA rule test result table 180 is a table in which RCA rules and information on events that have occurred are registered (see FIG. 12). A configuration example of each table 140, 150, 160, 180 stored in the auxiliary storage device 120 will be described later.

The memory 111 performs an input reception program 102 that performs external input processing, an output control program 103 that performs external output processing, an event management program 104 that processes event information retrieved from the event table 140, and tests RCA rules. A rule test program 105 and a configuration information reproduction program 107 that reproduces the configuration information of the production environment based on the configuration information extracted from the configuration information table 130 are stored. These programs are stored in the auxiliary storage device 120, read out from the auxiliary storage device 120 at the time of execution, copied into the memory 111, and executed by the CPU 110.

The program executed by the CPU 110 is provided to the rule test apparatus 100 via a removable medium (CD-ROM, flash memory, etc.) or a network, and stored in the auxiliary storage device 120 which is a non-temporary storage medium.

The rule test apparatus 100 according to the present embodiment is a computer system configured on a single physical computer or a plurality of logically or physically configured computers. It may operate on a thread, or may operate on a virtual computer constructed on a plurality of physical computer resources.

Next, the operation of the rule test in the rule test apparatus 100 of the present embodiment will be described according to the flowchart shown in FIG.

First, the user changes the RCA rule provided by the RCA server 500 (S1). The change of the RCA rule includes editing and addition of the RCA rule, new application of the RCA rule that has not been applied until now, and exemption of application of the RCA rule that has been applied until now. Next, the user determines the configuration of past data used for the rule test, and the rule testing apparatus 100 accepts the determined configuration of past data (S2). Then, the rule test apparatus 100 executes a test of the target RCA rule and presents the result to the user (S3). The RCA rule editing (S1) performed by the user will be described later.

FIG. 3 is a detailed flowchart of the process of determining the configuration of past data used for the test (S2 in FIG. 2).

In the process of determining the configuration of past data used for the test (S2), first, it is determined whether the change of the RCA rule by the user is creation of a new rule or improvement of an existing rule (S11). When it is an improvement of the existing rule, the rule test apparatus 100 makes an inquiry to the RCA server 500 and determines whether an RCA snapshot remains. The RCA snapshot is a result of analyzing the root cause using the same RCA rule in the past. When the RCA server 500 holds the RCA snapshot, the process proceeds to step S14. On the other hand, when the RCA server 500 does not hold the RCA snapshot, the process proceeds to step S13 (S12).

In step S13, it is determined whether the conditional expression of the RCA rule newly created or edited by the user includes an already defined event (S13). If the conditional expression does not include an already defined event, the process proceeds to step S17. If the conditional expression includes an already defined event, the process proceeds to step S14. In step S14, the corresponding event is added to the test candidate. For example, an event may be added to the test candidate event table temporarily created on the memory (S14).

Next, the target system and the event that is insufficient in the conditional expression are presented to the user. For events that are insufficient in the conditional expression of the RCA rule, if one of the events constituting the conditional expression of the rule is not found from the event table 140, a co-occurrence is detected between the start date and time and the end date and time. It is not an event. Here, the start date and time is “date and time when the event holding time is subtracted from the date and time when the first event occurs”, and the end date and time is “date and time when the event holding time is added to the date and time when the last event occurs”. The target system is a part of the system to be tested, specifically, a physical or logical unit that operates on a system such as a VM (virtual machine), an HV (hypervisor), or a storage device, and an RCA rule. Is a unit to which the device is applied, and is a device type related to the event defined by the rule (S15).

Next, in the selection field of the screen for selecting the test target system and the RCA rule related to the test target system displayed in step S15, the user selects the test target RCA rule (S16). Then, the user is prompted to input whether to automatically create an event lacking in the test (S17). If the user wishes to create an event automatically, the process proceeds to step S18 to create a test event. On the other hand, if the event is not automatically created, the process ends.

The test event is an event in which an event that is insufficient in the conditional expression presented in step S15 is added as a temporary event. In step S18, (1) before the occurrence of the first event of the n rules constituting the rule to be applied, (2) between the first event and the next event, (n) n− Between the first event and the last event, an event is created at each occurrence timing after the occurrence of the (n + 1) last event.

In step S18, the rule test program 105 creates an event with a value that makes the event ID 141 unique in the event table 140 described later with reference to FIG. Further, as described above, the rule test program 105 sets the event occurrence time 142 as a time for testing various contexts with other events constituting the rule. Further, the rule test program 105 is already held in the event table 140, and the identification information of the device indicated by the remaining events correctly obtained from the production environment and the identification information of the devices that are topologically related are included in the target device. ID 143. Further, the rule test program 105 sets a value corresponding to the type obtained from the ID 143 of the target device to the type 144 of the target device.

Also, the rule test program 105 sets the value assigned at the time of rule creation as the event type 145. The rule test program 105 also sets values given by the user when creating the rules for the failure flag 146 and the configuration change flag. Here, the method in which the user sets the values of the failure flag 146 and the configuration change flag 147 is specified in the uploaded rule file when uploading a rule (for example, in the input screen 10 of FIG. 10 described later). Alternatively, the user may specify the values of the failure flag 146 and the configuration change flag 147 using a separately provided GUI.

Also, the rule test program 105 sets a unique value that does not overlap with the co-occurrence wait event ID 151 of the co-occurrence wait event table 150 in the co-occurrence wait event ID 148.

FIG. 4 is a detailed flowchart of the process for testing the RCA rule (S3 in FIG. 2).

When testing the RCA rule, first, as an initialization process, the position of the event to start reading, the reading direction, and the reading period are determined (S21). The position of the event at which reading is started is a line in the event table 140 where analysis of the RCA rule is started. A method of determining a line for starting the analysis of the RCA rule will be described later. The reading direction indicates whether reading is to be performed upward or downward from the position of the event at which reading of the event table 140 is started. That is, whether the event is read back in time or the event is read as time elapses. The reading period 84 (see FIG. 13) is a time for reading an event from the position where the reading of the event is started. A set of failure events to which the RCA rules should be applied can be obtained by reading events that occurred during the reading period from the position of the event at which reading is started in the reading direction.

FIG. 13 is a diagram showing a method for determining a line for starting the analysis of the RCA rule.

In operation management sites, all data of operation management software is often backed up regularly. Therefore, rather than starting the analysis from the oldest event in the event table 140, the data backup point (81a or 81b) closest to the failure event occurrence time 82 to be tested is set to the line where the analysis of the RCA rule is started. Extracting information can speed up processing. Further, in order to reach the failure event occurrence time 82 to be tested, a data backup that is newer than the failure event occurrence time 82 to be tested from the data backup time 81a that is older than the failure event occurrence time 82 to be tested. There may be fewer events 85 to analyze at time 81b. In this case, the event information may be extracted by setting the new data backup time 81b in the line where the analysis of the RCA rule is started.

In this way, the event reading start time and reading direction are determined using the relationship between the backup point 81, the event holding time 83, the occurrence time of the failure event 85, and the number of events 85 to be analyzed. The processing time for reproducing can be shortened.

Referring back to FIG. 4, next, in the RCA rule test process S3, the configuration information reproduction program 107 creates a base of topology information necessary for RCA analysis (S22). Specifically, the configuration information reproduction program 107 selects the backup time with the smallest number of events between the backup of the topology information from the RCA server 500 and the position of the event to start reading, and starts reading. Apply the configuration change event one by one up to the event position to create a base of topology information. When going back in time, the configuration information reproduction program 107 applies the reverse operation of the configuration change event one by one, updates the configuration information, and creates the topology information base. The created topology information base is stored in the topology snapshot 170.

Thus, since the past state is reproduced by applying the change history in the event reading direction from the backup point 81, the processing time for reproducing the past configuration information can be shortened.

Next, in the RCA rule test process S3, one event is read, and the position of the read event is updated one new or old along the reading direction (S23). Steps S23 to S28 are processes in a loop for analyzing events one by one.

First, the event has information shown in the event table 140 (see FIG. 6). Details of the contents of the event table 140 will be described later. The RCA rule test process S3 compares the event reading period set in step S21 with the event occurrence time 142 of the event table 140. If the event occurrence time 142 of the read event is outside the event reading period, it is not necessary to read further events from the event table 140, and the process proceeds to step S29. If the event occurrence time 142 of the read event is within the event reading period, the event needs to be analyzed, and the process proceeds to step S25 (S24).

Next, in the RCA rule test processing S3, the failure flag 146 of the read event is evaluated, and if the failure flag 146 is ON, the failure event processing S26 is executed. On the other hand, if the failure flag 146 is OFF, the process proceeds to step S27 (S25). Details of the processing in step S26 will be described later with reference to FIG.

Similarly, in the RCA rule test process S3, the configuration change flag 147 of the read event is evaluated. If the configuration change flag 147 is ON, the configuration change event process S28 is executed. On the other hand, if the configuration change flag 147 is OFF, the loop is terminated and the process returns to step S23 (S27). Details of the processing in step S28 will be described later with reference to FIG.

FIG. 5 is a detailed flowchart of the failure event process (S26 in FIG. 4).

First, in the failure event process S26, it is determined whether the occurrence time of the failure event is within the event holding period 83 (S31). If the occurrence time of the failure event is within the event holding period of the RCA, the corresponding information in the event table 140 is registered in the co-occurrence waiting event table 150 (S32). Specifically, the value of the event occurrence time 142 in the event table 140 is registered in the occurrence time 152 of the co-occurrence waiting event table 150, the value of the target device ID 143 is registered in the node ID 153, and the event type 145 value is changed to the event. Register to type 154. Then, the value of the co-occurrence wait event ID 151 of the co-occurrence wait event table 150 is registered in the co-occurrence wait event ID 148 of the event table 140.

FIG. 6 is a diagram illustrating a configuration example of the event table 140.

The event table 140 is a table for storing events. The event ID 141, the event occurrence time 142, the node ID 143, the node type 144, the event type 145, the failure flag 146, and the configuration change flag 147 are co-occurrence. And a waiting event ID 148.

The event ID 141 is identification information for uniquely identifying an event. The event occurrence time 142 is the occurrence time of the event. The node ID 143 is identification information of the device that is the target of the event. The node type 144 is the type of the device that is the target of the event. For example, “HV” indicating that the device is a hypervisor and “VM” indicating that the device is a virtual machine are set. In addition to the illustrated types, there may be types representing storage devices, fiber channel switches, network switches, and routers. The event type 145 is identification information representing the event type. The failure flag 146 is a flag indicating whether or not the event is a failure event. The configuration change flag 147 is a flag indicating whether the event is a configuration change event. The co-occurrence wait event ID 148 is identification information for uniquely identifying the corresponding event in the co-occurrence wait event table 150.

The rule test apparatus 100 receives the update information from the database event of the RCA server 500 at the timing when the database of the RCA server 500 is updated, and synchronizes the event table 140 with the event database of the RCA server 500.

FIG. 7 is a diagram showing a configuration example of the RCA co-occurrence waiting event table 150.

The co-occurrence wait event table 150 includes a co-occurrence wait event ID 151, a time 152, a node ID 153, and an event type 154.

The co-occurrence wait event ID 151 is identification information for uniquely identifying the co-occurrence wait event. Time 152 is the time when the co-occurrence waiting event occurs. The node ID 153 is identification information of the device that is the target of the event. The event type 154 is identification information representing the event type.

FIG. 8 is a detailed flowchart of the configuration event process (S28 in FIG. 4).

First, in the configuration change event process S28, the configuration information of the topology snapshot 170 is changed according to the contents of the configuration change event (S41).

Next, it is determined whether the current configuration change is a change to the topology. Here, changing the topology means that the relationship between devices changes regardless of physical or virtual. For example, a virtual server operating on a physical server moves to another physical server. If there is a change in the topology, the co-occurrence wait event table 150 is reset (S43). On the other hand, if there is no change in the topology, the process is terminated (S42).

FIG. 9 is a diagram illustrating a configuration example of the RCA rule table 160.

The RCA rule table 160 is generated based on the RCA rule copied from the RCA server 500, and is updated when the rule is edited on the rule test apparatus 100. The RCA rule table 160 includes a rule ID 161 for uniquely identifying a rule, a rule usage state 162 indicating whether the user is using the rule, a device type 163 related to the rule, and a rule content 164. including.

Return to FIG. Next, in the RCA rule test process S3, the failure event is analyzed based on the RCA rule (S29). Specifically, first, the information of the topology snapshot 170 is fetched. Thereafter, the RCA rule is applied to the remaining co-occurrence waiting event. By applying the same processing algorithm as that of the RCA server 500 to the RCA rule, the same root cause location as that in the production environment is estimated. If there is an event that conforms to the RCA rule, the RCA rule and information on the event that has occurred are registered in the RCA rule test result table 180.

In the failure event analysis S29 based on the RCA rule, first, an event occurring at the corresponding occurrence time in the event table 140 is compared with the RCA test result table 180, and the event occurs in the RCA server 500 and the rule testing apparatus 100. Or an event that occurred in the rule testing apparatus 100 but did not occur in the RCA server 500, or an event that occurred in the RCA server 500 but did not occur in the rule testing apparatus 100. A value (“+” in this embodiment) indicating an event newly generated by a new RCA rule, a value indicating an event that has occurred conventionally and also occurred in a new RCA rule (this embodiment) In the example, it is blank), but the event change information 181 is set to one of the values ("-" in the present embodiment) indicating that it has occurred in the past but has not occurred in the new RCA rule. In this embodiment, the event change information is represented as characters, but may be displayed by another method such as changing the color of the displayed event or changing the frame line.

FIG. 12 is a diagram illustrating a configuration example of the RCA rule test result table 180.

The RCA rule test result table 180 adds the event change information and the ID of the applied RCA rule to the contents of the event table 140, and deletes the failure flag 146, the configuration change flag 147, and the ID 148 of the corresponding event in the co-occurrence waiting event table 150. Is. The RCA rule test result table 180 includes event change information 181, RCA rule ID 182, event occurrence time 183, target device ID 184, target device type 185, and ID 186 representing an event.

The event change information 181 exists in the event table 140 but does not exist in the RCA rule test result table 180 (“-” in this embodiment), and exists in both the event table 140 and the RCA rule test result table 180. This is information for discriminating between events to be performed (in this embodiment (blank)) and events that do not exist in the event table 140 but exist in the RCA rule test result table 180 (“+” in this embodiment).

The RCA rule ID 182 is information for uniquely specifying the RCA rule. The event occurrence time 183 is the time when the event occurred. The target device ID 184 is identification information of a device that is an event generation source. The target device type 185 is the type of the device that is the source of the event. ID 186 representing an event is information for uniquely identifying the event type.

Return to FIG. Finally, in the RCA rule test process S3, an analysis result is output (S30).

FIG. 10 is a diagram illustrating an example of the input screen 10 displayed on the terminal 800 by the output control program 103. The input screen 10 illustrated in FIG. 10 is a screen configured as a Web application, but may be a screen configured by a native application executed on the rule test apparatus 100.

The input screen 10 includes a title bar 11 for displaying a GUI name,

address bars

1 and 2 for displaying a URL (Uniform Resource Locator), an event list area 13 for displaying an event read from the event database of the RCA server 500, A rule list area 21 that displays RCA rules read from the RCA server 500, a “details” button 31 used to determine details of the RCA rule test method, and an RCA rule used to actually apply the event to the event. And an “Apply” button 41. When a URL is not used like a native application instead of a Web application, or when a URL is not displayed even with a Web application, the address bar 12 may not be provided.

The event list area 13 includes fields for displaying an event message 16, an event occurrence time 17, a device (source) 18 in which an event indicated by the event has occurred, and a device type 19 of the event generation device.

Event information stored in the event table 140 is set in the event list area 13. In the event message 16, the name of the event stored in the event type 145 is set. In the event type 145, a table in which event names and identification information are associated with each other so as to be easily understood by a person may be prepared, and identification information obtained by converting the name using the table may be stored. The event identification information stored in may be set directly. At time 17, a value stored at the event occurrence time 142 in the event table 140 is set. The source 18 is set with the name of the device in which the event indicated by the event, which is stored in the node ID 143, has occurred. The source 18 may prepare a table in which device names and identification information are associated with each other so as to be easily understood by a person, and store identification information obtained by converting the name using the table, or a node ID 143. The identification information of the target device stored in may be set. In the device type 19, the value of the type (node type 144) of the device in which the event indicated by the event has occurred is set.

The rule list area 21 includes fields for displaying a rule usage state 22, a rule name 23, a target 24 to which the rule is applied, and a rule detail 25 indicating the details of the rule. Further, the rule list area 21 includes an “upload rule” button 26 that is used when a new rule that has not been read in the RCA rule table 160 is uploaded to the rule testing apparatus 100 and an existing rule that is read in the RCA rule table 160. And a “delete rule” button 27 used when deleting the rule.

In the rule list area 21, information stored in the RCA rule table 160 is set. In the rule usage state 22RCA, the value of the rule usage state 162 of the rule table 160 is set. In the rule name 23, the name of the rule stored in the rule ID 161 is set. In the rule name 23, a table in which the name of the rule is associated with the identification information so as to be easily understood by a person may be prepared, and the identification information obtained by converting the name using the table may be stored. The identification information stored in the ID 161 may be set directly. In the target 24, a value of the device type 163 related to the rule is set. In the rule details 25, the content 164 of the currently selected rule stored in the rule content 164 is set. The field of the rule details 25 has a function of editing text, and the user can edit the rule. When the “upload rule” button 26 is operated, a screen for selecting a file is displayed, and the user can select a rule definition file to be added to the rule list. When the “delete rule” button 27 is operated, the user can delete the currently selected rule from the rule list. The operations using the rule details 25, the “upload rule” button 26, and the “delete rule” button 27 correspond to the editing of the RCA rule (S1 in FIG. 2).

When the “Details” button 31 is operated, a screen for performing detailed settings when testing the rule is displayed. On the detailed setting screen, the time for storing event information in the co-occurrence waiting event table 150 can be adjusted, and the event reading period can be set. When the event reading period is changed, the reading period determined in step S21 of the RCA test S3 is changed.

When the “Apply” button 41 is operated, a test using the currently used rule based on the event occurrence time selected in the event list area 13 or the event reading period set on the detailed setting screen. Is executed.

FIG. 11 shows an example of the output screen 50 displayed on the terminal 800 by the output control program 103. The output screen 50 illustrated in FIG. 11 is a screen configured as a Web application similarly to the input screen 10, but may be a screen configured by a native application executed on the rule testing apparatus 100.

The output screen 50 includes a title bar 51 that displays the GUI name, an address bar 52 that displays the URL, an RCA rule application result area 53 that displays a list of events obtained as a test result of the RCA rule, and “to input screen” And a “return” button 61. Similar to the input screen 10, the address bar 52 may not be provided.

The RCA rule application result area 53 includes event change information 54, an application rule ID 55, an event message 56, an event occurrence time 57, a device (source) 58 in which the event has occurred, and a device in which the event has occurred. Device type 59. In the RCA rule application result area 53, information stored in the RCA rule test result table 180 obtained by extending the event table 140 is displayed in time series.

The event change information 54 displays the value of event change information in the RCA rule test result table 180. The applied rule ID 55 displays the name of the rule. The applied rule ID 55 may be a table in which the name of the rule is associated with the identification information so as to be easily understood by a person, and the name obtained by converting the identification information using the table may be displayed. The identification information may be displayed as it is. The message 56 displays a character string corresponding to the value of ID 145 representing the event. The message 56 may display a name obtained by converting the identification information using the table by preparing a table in which the event name and the identification information are associated with each other so as to be easily understood by a person. Information (event ID 145) may be displayed as it is. At time 57, the value of the event occurrence time 142 in the RCA rule test result table 180 is displayed. The source 58 displays the name of the device in which the event indicated by the event has occurred. The source 58 displays the name of the target device in the event table 140. The source 58 may prepare a table in which device names and identification information are associated with each other so as to be easily understood by a person, and display the name obtained by converting the identification information using the table. Information may be displayed as it is. The source device type 59 displays the value of the device type 144 in which the event indicated by the event has occurred.

As described above, according to the first embodiment, since the new RCA rule is tested using the event information generated in the production environment in the same configuration information as the production environment, the RCA rule is introduced before the RCA rule is introduced into the production environment. Can confirm whether the expected failure event can be detected.

In addition, since an event occurring within a predetermined period is set as an event causing a failure, the event causing the failure can be accurately selected with a simple process.

In addition, according to the first embodiment, even if an event that matches the condition of the RCA rule has not occurred in the past, an event that matches the condition is automatically created and the operation of the RCA rule is confirmed accurately. Can do.

In the first embodiment, the root cause by the RCA is estimated when an event occurs, but the root cause by the RCA may be estimated when a change in configuration information is detected.

<Second embodiment>
Next, a second embodiment of the test environment according to the present invention will be described. In the second embodiment, the same configurations and processes as those in the first embodiment are denoted by the same reference numerals, description thereof will be omitted, and only different portions will be described.

FIG. 14 is a block diagram illustrating the configuration of the rule test apparatus 100 according to the second embodiment and the relationship between the rule test apparatus 100, the test environment, and the production environment.

The test environment of the second embodiment has the rule test apparatus 100 as in the test environment of the first embodiment.

The rule test apparatus 100 of the second embodiment has almost the same configuration as the rule test apparatus 100 of the first embodiment, but the RCA rule history management program 108 on the memory 111 and the rule history table on the auxiliary storage device 120. 190 is stored.

In the second embodiment, when the RCA rule used on the RCA server 500 is changed, such as addition, deletion, or editing, the RCA server 500 receives a change event transmitted from the database holding the rule. As a result, the RCA rule history management program 108 of the rule testing apparatus 100 records the rule change history in the rule history table 190.

FIG. 15 is a diagram illustrating a configuration example of the rule history table 190.

The rule history table 190 is a table for storing the RCA rule change history, the time 191 when the RCA rule was changed, the identification information (rule ID) 192 of the changed RCA rule, and addition, deletion, and editing. And the change type 193 such as.

FIG. 16 shows an example of the output screen 50 displayed on the terminal 800 by the output control program 103. The output screen 50 illustrated in FIG. 16 is a screen configured as a Web application similarly to the input screen 10, but may be a screen configured by a native application executed on the rule testing apparatus 100.

In the output screen 50 of the second embodiment, the same components as those of the output screen 50 of the first embodiment are denoted by the same reference numerals, description thereof is omitted, and only different portions are described. In the RCA rule application result area 53, failure events and RCA rule change events that are not failures are displayed in time series. Therefore, an event type 71 is added to the RCA rule application result area 53 instead of the event change information 54. In the second embodiment, when the RCA rule is changed, the event type 71 displays a character (“!” In this embodiment) indicating that the event is not a failure. The ID of the RCA rule with the change is displayed in the application rule ID 55, and the content of the change to the RCA rule (in this example, “RCA rule added” (RCA rule has been added)) is displayed in the message 56. The time when the change to the RCA rule is performed is displayed at time 57. Since the source 58 and the device 59 have no corresponding information, an empty character string is displayed. In this embodiment, a server resource shortage event is generated next to the RCA rule change event, and then a server down event is generated. Originally, if only the correct RCA rule is applied, only the resource shortage event is displayed as the root cause, so that the user can quickly understand the cause and deal with the problem. However, since the server down event is also displayed at present, it is necessary to investigate the cause other than the server resource shortage as a possibility. Before reaching this situation, the user knows that a rule has been added as an RCA rule change event, and can consider the possibility that the added RCA rule is inappropriate.

As described above, according to the second embodiment, the RCA rule changed before the time when the failure event occurred is displayed along with the failure event based on the past event information and the RCA rule change information. The user can be given information to determine whether the reason why the event could not be successfully applied is related to a change in the RCA rules.

The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to those having all the configurations described. A part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Moreover, you may add the structure of another Example to the structure of a certain Example. In addition, for a part of the configuration of each embodiment, another configuration may be added, deleted, or replaced.

In addition, each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, with an integrated circuit, and the processor realizes each function. It may be realized by software by interpreting and executing the program to be executed.

Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, and an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, and a DVD.

Also, the control lines and information lines indicate what is considered necessary for the explanation, and do not necessarily indicate all control lines and information lines necessary for mounting. In practice, it can be considered that almost all the components are connected to each other.

Claims

A management computer for testing rules for monitoring a system composed of a plurality of devices,
A storage unit;
A processor that refers to the storage unit,
The storage unit includes event information for managing fault events that have occurred in the system in the past and configuration change events for the system, configuration information for managing the current configuration of the system, and a topology that represents the past configuration of the system. Keep snapshots and
The management computer is
A configuration information reproduction unit that reproduces the past configuration of the system using the configuration information and the configuration change event, and records it in the topology snapshot;
A management computer comprising a rule test unit that applies a cause analysis algorithm to the reproduced past configuration, estimates a cause of a failure using a rule to be tested, and outputs whether the failure can be detected.
The management computer according to claim 1,
The storage unit stores rule history information for managing a rule change history,
The rule test unit outputs data to be displayed in time series so that a failure event whose cause could not be estimated and a change history of the rule to be tested are related.
The management computer according to claim 1,
Manage the retention period of related events,
The rule test unit starts acquiring an event from the event information based on the relationship between the time when the state information of the system is backed up, the retention period of the related event, and the failure event whose cause should be estimated. A management computer that determines a time to perform and a method for acquiring an event.
The management computer according to claim 3,
The configuration information reproduction unit reproduces the configuration of the system when a failure event occurs by applying the configuration change event to the configuration information of the system backed up at the time when the acquisition of the event is started. Management computer.
The management computer according to claim 1,
The storage unit holds co-occurrence event information for managing information on events that occur within the related event holding period,
The rule test unit outputs an event managed by the co-occurrence event information as an event that causes the failure event.
The management computer according to claim 5,
The configuration information reproducing unit creates an event which is described in the test target rule but is not recorded in the co-occurrence event information, and records the event information in the event information.
A method for testing a rule for monitoring a system composed of a plurality of devices using a management computer,
The management computer includes a storage unit and a processor that refers to the storage unit,
The storage unit includes event information for managing fault events that have occurred in the system in the past and configuration change events for the system, configuration information for managing the current configuration of the system, and a topology that represents the past configuration of the system. Keep snapshots and
The method
The management computer reproduces the past configuration of the system using the configuration information and the configuration change event, records the topology snapshot, and stores the configuration information in the memory, and
A rule test step in which the management computer applies a cause analysis algorithm to the reproduced past configuration, estimates a cause of the failure using a rule to be tested, and outputs whether the failure can be detected. A test method characterized by
The test method according to claim 7, wherein
The storage unit stores rule history information for managing a rule change history,
In the rule test step, a test method is characterized by outputting data for displaying in a time series so that a failure event whose cause could not be estimated and a change history of the rule to be tested are related.
The test method according to claim 7, wherein
The management computer manages a retention period of related events,
In the rule test step, the acquisition of the event is started from the event information based on the relationship between the time when the state information of the system was backed up, the retention period of the related event, and the failure event whose cause should be estimated. And determining a method for acquiring a time and an event to be performed.
The test method according to claim 9, wherein
In the configuration information reproduction step, the configuration change event is applied to the system configuration information backed up at the time when the acquisition of the event is started to reproduce the system configuration at the time of occurrence of the failure event. Test method.
The test method according to claim 7, wherein
The storage unit holds co-occurrence event information for managing information on events that occur within the related event holding period,
In the rule testing step, an event managed by the co-occurrence event information is output as an event that causes the failure event.
The test method according to claim 11, comprising:
In the configuration information reproducing step, an event described in the rule to be tested but not recorded in the co-occurrence event information is created and recorded in the event information.