JP5267109B2 - Failure detection system verification apparatus, failure detection system verification method, and failure detection system verification control program - Google Patents

Description

  The present invention relates to a failure detection system verification device, a failure detection system verification method, and a failure detection system verification control program. For example, the present invention monitors the operation of a system that provides a predetermined service by combining a plurality of devices connected to a network. Used to verify the quality of the adjustment result when the function is adjusted in the fault detection system and report it to the administrator etc. The present invention relates to a fault detection system verification apparatus, a fault detection system verification method, and a fault detection system verification control program.

A system that provides a predetermined service by combining multiple devices connected to the network (referred to as a “service provision system”) has been constructed. When a failure occurs, the operation of this service provision system is monitored. There is a fault detection system that reports to the administrator.
This kind of failure detection system is difficult to always operate properly for any service provision system, so the rules for setting thresholds and analyzing the cause of failure are determined. It is necessary to make adjustments from time to time in response to the service provision system to be monitored by learning and the like. Further, when adjusting the failure detection system, it is necessary to verify that a failure can be detected correctly and that a non-failure is not erroneously detected as a failure based on the adjusted operation result. For this reason, there is a demand for a failure detection system verification device for verifying the quality of the adjustment result of the failure detection system.

As this type of related technology, for example, there is a test data management system described in Patent Document 1.
As shown in FIG. 9, this system includes an input device 1, an output device 2, a data processing device 10, and a storage device 20. The data processing apparatus 10 includes a condition setting unit 11, a test data generation unit 12, a test execution unit 13, and a result comparison unit 14. The storage device 20 includes a test ledger unit 21 and a system state storage database 22. The test execution unit 13 is connected with a failure detection system 30 to be tested.

In this system, the type of failure case is designated by the input device 1 and set in the condition setting unit 11. Further, the test data generation unit 12 acquires from the system state accumulation database 22 the past system state when the type of failure set in the condition setting unit 11 occurs, and simulates the system state at the time of this failure occurrence. The test data is generated. The test data is given to the failure detection system 30 by the test execution unit 13. The result comparison unit 14 verifies whether the conclusion derived by the failure detection system 30 is equal to the type of failure set in the condition designating unit 11, and outputs the verification result to the output device 2. That is, the failure detection system 30 is detected as a failure as an analysis result of the failure detection system 30 by applying the system state at the time of an actual failure case as test data, or the analysis result is actually It is confirmed whether or not the failure detection system 30 is operating correctly by checking whether or not the failure is equal to the failure that occurred.
Japanese Unexamined Patent Publication No. 2004-220356 (Abstract, FIG. 1)

However, the technique described in the above document has the following problems.
That is, in the test data management system of FIG. 9, in order to increase the test accuracy of the failure detection system 30, it is necessary to perform a test using a large amount of data. In this case, the operation result of the failure detection system 30 is also large. Thus, the burden of verification work by the person in charge increases. For this reason, there is a problem that the person in charge cannot efficiently verify the operation result of the failure detection system.

  The present invention has been made in view of the above circumstances, and provides a failure detection system verification device, a failure detection system verification method, and a failure detection system verification control program for efficiently verifying the adjustment result of the failure detection system The purpose is to do.

  In order to solve the above-described problem, a first configuration of the present invention relates to a failure detection system verification device, and relates to a failure detection system that detects a failure of a service providing system that provides a predetermined service and outputs a failure detection result. State model for each failure type indicating statistical data of the state information of the service providing system for each type of failure based on the state information of the service providing system and the failure detection result output from the failure discovery system as a test target For each discovered failure case that constitutes the failure detection result, a degree of deviation indicating the degree of deviation of the state information of the service providing system from the state model according to the failure type is obtained, and the degree of deviation is large. It is characterized in that the detection failure cases are presented in order.

  A second configuration of the present invention relates to a failure detection system verification method, and a failure detection system that detects a failure of a service providing system that provides a predetermined service and outputs a failure detection result is tested. A state model for each failure type indicating statistical data of the state information of the service providing system for each type of failure based on the state information of the service providing system and the failure detection result output from the failure discovery system. For each discovered failure case that constitutes the failure detection result, the degree of deviation indicating the degree of deviation of the state information of the service providing system from the state model according to the failure type is obtained, and the order of the deviation from the largest The discovery failure case is presented.

  A third configuration of the present invention relates to a computer-readable failure detection system verification control program, and targets a failure detection system that detects a failure of a service providing system that provides a predetermined service and outputs a failure detection result as a test target. The failure detection system verification device configured by a computer, based on the state information of the service providing system and the failure detection result output from the failure detection system, stores the state information of the service providing system for each type of failure. A failure type state model indicating statistical data is generated, and for each discovered failure case constituting the failure detection result, a degree of deviation indicating a degree of deviation of the state information of the service providing system from the state model by failure type is indicated. And processing for presenting the detected failure cases in descending order of the degree of deviation. It is characterized by rows.

  According to the configuration of the present invention, a failure detection case in which the state of the system is greatly deviated from the other detection cases determined to be the same type of failure is preferentially presented, and the administrator can It is possible to efficiently recognize failure detection cases that are likely to have errors. Thereby, the administrator can efficiently verify the operation result of the failure detection system.

  The failure detection system is configured to perform a predetermined adjustment for detecting a failure in the service providing system, and is output from the failure detection system to which test data in which status information of the service providing system is accumulated is given. The failure detection result is classified for each failure type, and statistical data of each state parameter indicating the state of the service providing system is obtained for each failure type based on the test data. A model is generated, and for each discovered failure case constituting the failure detection result, state information of the service providing system corresponding to the time and place where each failure occurs is extracted from the test data, and the state information is extracted. Is compared with the state model according to the failure type corresponding to the failure type of the detected failure case, the state model for the failure type Seek out level indicating the degree of release, to provide a trouble detection system verification device for presenting the discovery failure cases in order from the 該外Re degree large.

  Also, in a preferred embodiment of the present invention, the failure detection system verification device obtains statistical data of each state parameter indicating a normal state of the service providing system based on the test data, and obtains a normal state model. When generating the above-mentioned degree of deviation, among the state parameters constituting the system state, the behavior in the state model by failure type is the state model by failure type other than the state model and the state model at normal time. The state model having characteristic parameters as compared with the above behavior is preferentially used.

  According to a preferred aspect of the present invention, the failure detection system verification device includes failure type classification means for classifying the failure detection result output from the failure detection system to which the test data is given for each type of failure. A failure type state model generation means for obtaining statistical data of each state parameter indicating the state of the service providing system for each failure type based on the test data, and generating a failure type state model; and the failure For each detected failure case constituting the detection result, the state information of the service providing system corresponding to the time and place where each failure occurred is extracted from the test data, and the state information is used as the failure type of the detected failure case. Compared with the state model according to the failure type corresponding to the above, the deviance meter for calculating the degree of divergence indicating the degree of deviation from the state model according to the failure type And means, and a classifying extracting means for presenting classified extracted in order from the larger the discovery failure cases of the out-degree.

  In a preferred embodiment of the present invention, the failure detection system verification device obtains statistical data of each state parameter indicating a normal state of the service providing system based on the test data and the failure detection result. The normal state model generation means for generating the normal state model, and the state model for the failure type among the state parameters constituting the system state, the state model for the failure type other than the state model and the normal state A state model having a characteristic parameter compared to the behavior of the state model is selected as a characteristic state model, and the characteristic state model is preferentially used when the degree of deviation is calculated by the degree of deviation calculation unit. Failure type-specific determination parameter selection means to be provided.

Embodiment 1

FIG. 1 is a block diagram showing the configuration of a failure detection system verification apparatus according to the first embodiment of the present invention.
As shown in the figure, the failure detection system verification device 40 of this embodiment is connected to the failure detection system 30 to be tested similar to that in FIG. 9 and is connected to the display device 50. The failure detection system 30 detects a failure in a service providing system (not shown) that provides a predetermined service, and outputs a failure detection result. The failure detection system 30 is configured to perform a predetermined adjustment for detecting a failure of the service providing system. The display device 50 is composed of, for example, a liquid crystal display device.

  The failure detection system verification device 40 outputs the failure detection system 30 to which the failure detection system 30 is given a test data td configured by integrating the status information of the service providing system. A discovered failure list ta (failure detection result) is input. Then, the failure detection system verification device 40 generates a failure type state model indicating statistical data of the state information of the service providing system for each failure type based on the test data td and the detected failure list ta. For each discovered failure case that constitutes the discovered failure list ta, the degree of divergence indicating the degree of deviation of the state of the service providing system from the state model by failure type is obtained, and the detected failure cases are presented in descending order of deviance. And displayed on the display device 50.

  In particular, in this embodiment, the failure detection system verification device 40 is configured by a computer that operates based on a failure detection system verification control program, and includes a system state reception unit 41, a detection failure list reception unit 42, and a failure type classification unit. 43, a failure type-specific state model generation unit 44, a degree-of-displacement calculation unit 45, a sorting unit 46, and an output unit 47. The system state reception unit 41 receives the test data td used when operating the failure detection system 30 as the system state list sm. The discovered failure list accepting unit 42 accepts the discovered failure list ta that is a result of the failure discovery system 30 operating based on the test data td, and outputs it as the discovered failure list tb. The failure type classification unit 43 (failure type classification means) classifies the detected failure list tb output from the detected failure list reception unit 42 for each type of detected failure, and outputs a failure-specific list tc.

  The failure type-specific state model generation unit 44 (failure type-specific state model generation means) is based on the system state list sm sent from the system state reception unit 41 and the failure-specific list tc sent from the failure type classification unit 43. A state model sn for each failure type is generated by obtaining statistical data of each state parameter indicating the state of the service providing system for each type of failure. In this case, the failure type-specific state model generation unit 44 obtains the time and place where the failure occurred for each case included in the failure type list tc sent from the failure type classification unit 43. The system status corresponding to this is extracted from the system status list sm sent from the system status accepting unit 41, so that a list of system status at that time is created for each type of failure. Further, the state model generation unit 44 for each type of failure is a model showing the statistical behavior of various parameters indicating the system state from the list of system states, that is, what probability each parameter has at what probability. A fault type-specific state model sn indicating whether to take a value is generated.

  The detachment degree calculation unit 45 (displacement degree calculation means) corresponds to the time and place where each failure occurs from the system state list sm (test data td) for each discovered failure case constituting the discovered failure list tb. The state information of the service providing system is extracted, and the state information is compared with the state model sn according to the failure type corresponding to the failure type of the detected failure case, and the degree of deviation indicating the degree of deviation from the state model sn according to the failure type tf is obtained. The sorting unit 46 (classification extracting unit) creates a presentation list tm by sequentially classifying and extracting from the detected failure cases having a large degree of detachment tf calculated by the degree of detachment calculating unit 45. The output unit 47 sends the presentation list tm created by the sorting unit 46 to the display device 50 for display.

FIG. 2 is a flowchart for explaining the operation of the failure detection system verification apparatus of FIG.
With reference to this figure, the processing content of the failure detection system verification method used for the failure detection system verification apparatus of this form is demonstrated.
In this failure detection system verification device, test data td in which status information of the service providing system is accumulated is given to the failure detection system 30 and the output failure list ta (failure detection result) is classified for each type of failure. Is done. Based on the test data td, statistical data of each state parameter indicating the state of the service providing system is obtained for each type of failure, and a state model sn for each failure type is generated. For each discovered failure case constituting the discovered failure list ta, the status information of the service providing system corresponding to the time and place where each failure occurred is extracted from the test data td, and the status information is the failure of the discovered failure case. Compared with the state model sn according to the type of failure corresponding to the type, an outlier tf is obtained, and found failure cases are displayed in descending order of the outlier tf.

  That is, as shown in FIG. 2, a test result td is used to operate the test target fault detection system 30 and a list of operation results (discovery fault list) indicating when, where, and what fault has occurred. ta) is generated (step A1). Next, the discovered failure list reception unit 42 receives the list of operation results (discovery failure list ta) generated in step A1 (step A2), while the system state reception unit 41 receives test data td (step A2). Step A3). The failure type classification unit 43 divides the operation result list (discovered failure list tb) acquired in step A2 for each failure type of the analysis result (failure type classification process), and for each failure occurrence case in which the same type of failure has occurred. A list (fault-specific list tc) summarized in (1) is created (step A4). Further, the failure type-specific state model generation unit 44 provides, for each failure type divided in step A4, the test data td (system state) that received the system state at that time for all the operation result cases. Take out from list sm) (step A5). Further, the failure type-specific state model generation unit 44 uses the system state list (system state list sm) for each failure type extracted in step A5 to statistically analyze the system state when the failure occurs. A model (state model sn for each failure type) showing a behavior is generated (step A6, state model generation processing for each failure type).

  Next, the outlier calculation unit 45 extracts one of the individual operation result cases included in the operation result list (discovery failure list ta) obtained in step A2 (step A7), and extracts it in step A7. The system state model (failure type-specific state model sn) corresponding to the type of failure generated in step A6 corresponding to the type of failure in the operation result example is extracted (step A8). Further, the outlier calculation unit 45 extracts the system state at the time of occurrence of the operation result example extracted in step A7 from the test data td (system state list sm) acquired in step A3 (step A9), and step A9. The system state extracted in step A8 is compared with the system state model (failure type-specific state model sn) for each type of failure extracted in step A8, and the degree of deviation is calculated (step A10, degree of deviation calculation processing). Then, the processing from step A7 to step A10 is performed for all the operation cases in the operation result list obtained in step A2 (step A11). Finally, the sorting unit 46 sorts (classifies and extracts) the operation cases in descending order of the degree of detachment tf calculated in step A10 to create a presentation list tm (step A12, category extraction processing). The operation examples are displayed on the display device 50 based on the presentation list tm in the order sorted in step A12 (step A13).

  As described above, in the first embodiment, for each discovery failure case constituting the discovery failure list ta output from the failure discovery system 30, the test data td is used to indicate the time and place where each failure occurs. The state information of the corresponding service providing system is extracted, and the state information is compared with the state model sn for each failure type corresponding to the failure type of the found failure case to obtain the outlier tf. Since the detected failure cases are displayed in order, the failure detection cases whose system status is significantly different from other detection cases judged to be the same type of failure are preferentially presented. It is possible to efficiently recognize failure detection cases that are likely to have errors in the results.

Embodiment 2

FIG. 3 is a block diagram showing the configuration of the failure detection system verification apparatus according to the second embodiment of the present invention. Elements common to the elements in FIG. 1 showing the first embodiment are denoted by common reference numerals. It is attached.
In the failure detection system verification device 40A of this embodiment, as shown in FIG. 3, in addition to the configuration of the failure detection system verification device 40 in FIG. 1, a normal state model generation unit 48 and failure type determination parameter selection A portion 49 is provided. The normal state state model generation unit 48 (normal state state model generation means) is configured so that the service providing system is normal based on the system state list sm (test data td) and the detected failure list tb transmitted from the system state reception unit 41. Statistical data of each state parameter indicating the state of the hour is obtained, and a normal state model sp is generated. In this case, the normal state state model generation unit 48 receives the system state information about the time and place that are not included in the discovery failure list ta received by the discovery failure list reception unit 42 when no failure has occurred. The unit 41 extracts from the test data td received, and creates a statistical model of the system state (normal state model sp) when no failure has occurred.

  The failure type determination parameter selection unit 49 (failure type determination parameter selection means) has a failure type state model sn behavior of a failure type other than the state model among the state parameters constituting the system state. When a state model having a characteristic parameter compared to the behavior of the model and the normal state model sp is selected as the characteristic state model sq, and the outlier calculation unit 45 obtains the outlier tf, the characteristic state The model sq is used preferentially. The other configuration is the same as that shown in FIG.

FIG. 4 is a flowchart for explaining the operation of the failure detection system verification apparatus of FIG.
With reference to this figure, the processing content of the failure detection system verification method used for the failure detection system verification apparatus of this form is demonstrated.
This failure detection system verification apparatus is different from the first embodiment in the following points.
That is, in the detected failure case presentation process, the normal state model sp is generated by obtaining statistical data of each state parameter indicating the normal state of the service providing system based on the test data td and the detected failure list ta. In addition, among the state parameters constituting the system state, the behavior in the state model sn for each failure type is a characteristic parameter compared with the behavior in the state model for each failure type other than the state model and the normal state model sp. Is selected as the characteristic state model sq, and the characteristic state model sq is preferentially used when the outlier calculation unit 45 calculates the outlier tf.

  In this case, as shown in FIG. 4, in Steps B1 to B6, the same processing as Steps A1 to A6 in FIG. 2 showing the first embodiment is performed. Thereafter, the normal state model generation unit 48 uses the system state other than the data extracted in step B5 from the test data td received in step B3 to show the statistical behavior of the normal system state. A model (normal state model sp) is generated (step B7, normal state model generation process). Next, the failure type-specific determination parameter selection unit 49 extracts one model of each failure type generated in step B6 (step B8), and the parameters constituting the system state for the model selected in step B8. Is selected (step B9).

  Further, in the type-specific determination parameter selection unit 49, the statistical behavior of the parameter selected in step B9 is the same in the model of the type of failure other than that selected in step B8 and the normal state model sp generated in step B7. If it is compared with the statistical behavior of the parameter (step B10) and there is a large difference in the statistical behavior (step B11), the parameter selected in step B9 is used as the parameter for evaluating the model of the type of fault selected in step B8. (Step B12, failure type-specific determination parameter selection process). Then, the processing from step B9 to step B12 is executed for all parameters constituting the system state (step B13). Further, the processing from step B8 to step B13 is executed for all the failure type state models generated in step B6 (step B14).

  Next, the outlier calculation unit 45 takes out one of the individual operation result cases included in the operation result list (discovery failure list ta) obtained in step B2 (step B15), and further performs step B15. The model of the system state for each type of failure generated in step B6 corresponding to the type of failure in the operation result example extracted in step S16 is extracted (step A16). Further, the detachment degree calculation unit 45 extracts the system state corresponding to the operation result example extracted in step B15 from the test data td acquired in step B3 (step B17), and out of the system states extracted in step B17. Then, using the parameter specified as the evaluation parameter in step B12, the model of the system state for each type of failure taken out in step B15 is compared, and the degree of deviation tf is calculated (step A18). Steps B15 to B18 are performed for all operation cases in the operation result list obtained in step B2 (step B19). Finally, the sorting unit 46 sorts the operation examples in descending order of the degree of detachment tf calculated in step B18 (classification extraction) to create a presentation list tm (step B20), and the output unit 47 performs step B20. In the sorted order, the operation examples are displayed on the display device 50 based on the presentation list tm (step B21).

  As described above, in the second embodiment, the normal state model sp is generated by the normal state model generation unit 48, and the characteristic state model sq is selected by the failure type determination parameter selection unit 49. Since the characteristic state model sq is preferentially used when the outlier calculation unit 45 calculates the outlier tf, the calculation result of the outlier tf becomes larger than that in the first embodiment, and the administrator Thus, it is possible to more efficiently recognize a failure detection case that is likely to have an error in the detection result.

Embodiment 3

FIG. 5 is a schematic diagram showing an environment in which the failure detection system verification apparatus according to the third embodiment of the present invention is used. FIG. 1 shows the first embodiment, and FIG. 3 shows the second embodiment. Elements common to the elements inside are given common reference numerals.
In the environment of this form, as shown in FIG. 5, a service providing system 60, a system state information storage DB (database) 61, a failure detection system 30, a failure detection system verification device 40 (or 40A), a display A device 50 is provided. The service providing system 60 provides a predetermined service by combining a plurality of devices connected to a network, for example. In the service providing system 60, a failure occurs a plurality of times over a long period of time. The system state information storage DB 61 periodically collects and stores system state information of the service providing system 60 to be monitored by the failure detection system 30 to be tested.

FIG. 6 is a diagram illustrating an example of the discovered failure list ta, FIG. 7 is a diagram illustrating an association between the discovered failure list ta and the system state information DB 61, and FIG. 8 is a diagram illustrating an example of a state model for each failure type. It is.
With reference to these drawings, processing contents of the fault detection system verification method used in the fault detection system verification apparatus of this embodiment will be described.
The failure detection system 30 to be tested operates based on the system state information (test data td) stored in the system state information storage DB 61. As a result, the failure discovery system 30 finds a failure that occurred during the period stored in the system state information storage DB 61, and takes out the list of discovered failures as a discovery failure list ta. In the discovered failure list ta, for example, as shown in FIG. 6, each discovered failure case includes “ID”, “time”, “location”, and “detection result” that identify each case.

  The system state information (test data td) stored in the system state information storage DB 61 includes, for example, as shown in FIG. 7, “CPU load”, “memory usage” corresponding to the ID of each detected failure case. , “Network usage” and “disk usage”. The failure detection system verification device 40 (40A) collects system states corresponding to these failure detection cases for each type of failure and creates system state statistical data (failure type-specific state model) for each failure type. For example, in FIG. 6, IDs 1, 2, and 4 have the same failure type “Fault A”, so they are grouped together as one group, and IDs 3 and 6 also have the same failure type “Fault B”. Because it is, it is put together in one group. Then, a state model for each failure type is created for each failure type. As shown in FIGS. 8A and 8B, the state model corresponding to the above “failure B” has the CPU load and the memory usage amounting to near 100%, while FIGS. 8C and 8D. As shown in (2), the network usage and disk usage are widely distributed. Therefore, it is understood that the CPU load and the memory usage are characteristic in “failure B”.

  Comparing this state model with the system state corresponding to the detected failure case, in “failure b”, IDs 1 and 2 are not greatly deviated from the state model, but the CPU load is strongly biased near 100%. Compared with the state model, the CPU load of the system state in ID4 is as high as 77%, but it is out of the state model distribution. That is, the degree of detachment of ID4 is large. As a result, ID 4 is displayed near the top on the display device 50, and the administrator of the failure detection system 30 first determines whether the detected failure case of ID 4 is correctly analyzed, that is, “9 /” in FIG. At 21, 20:12 ", whether or not" Fault A "has occurred in the server D is confirmed by checking the actual record. Thus, the closer to the top of the discovered failure list displayed on the display device 50, the higher the possibility that the analysis result is wrong. It is possible to find a case where the detection result of the failure detection system 30 is different from the actual failure.

  As described above, in this third embodiment, when the detection result by the failure detection system 30 becomes enormous, even if the administrator does not know all the failure cases included in the detection result in advance, The time required for the verification work is reduced, and the verification work is performed efficiently.

The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiment, and even if there is a design change without departing from the gist of the present invention, Included in the invention.
For example, in each of the embodiments described above, the failure detection system verification devices 40 and 40A are configured by a computer, but each unit configuring the inside may be configured by an individual unit. Further, the failure detection system verification devices 40 and 40A may be provided with a means for generating an alarm or the like when the detachment degree tf calculated by the detachment degree calculation unit 45 is larger than a predetermined threshold.

  The present invention can be applied to all devices for verifying the operation of a failure detection system that requires adjustment as needed in response to a service providing system to be monitored, and is effective in efficiently adjusting the failure detection system. is there.

It is a block diagram which shows the structure of the failure discovery system verification apparatus which is 1st Embodiment of this invention. It is a flowchart explaining operation | movement of the failure discovery system verification apparatus of FIG. It is a block diagram which shows the structure of the failure discovery system verification apparatus which is 2nd Embodiment of this invention. It is a flowchart explaining operation | movement of the failure discovery system verification apparatus of FIG. It is a schematic diagram which shows the environment where the failure discovery system verification apparatus which is 3rd Embodiment of this invention is used. It is a figure which shows the example of the discovery failure list | wrist ta. It is a figure which shows matching with the discovery failure list | wrist ta and system status information DB61. It is a figure which shows the example of the state model according to a failure type. It is a block diagram of a test data management system.

Explanation of symbols

30 Failure detection system 40, 40A Failure detection system verification device 41 System state reception unit (part of failure detection system verification device)
42 Discovery fault list reception part (part of fault discovery system verification device)
43 Failure type classification part (failure type classification means, part of failure detection system verification device)
44 Fault type-specific state model generation unit (failure type-specific state model generation means, part of fault detection system verification device)
45 Degree of calculation part (displacement degree calculation means, part of failure detection system verification device)
46 Sorting part (part of classification extraction means, part of fault detection system verification device)
47 Output unit (part of classification extraction means, part of fault detection system verification device)
48 Normal state model generation unit (normal state model generation means, part of fault detection system verification device)
49 Parameter selection unit for determination by type of failure (parameter selection means for determination by type of failure, part of failure detection system verification device)
50 display device 60 service providing system

Claims (15)

  A failure detection system that detects a failure of a service providing system that provides a predetermined service and outputs a failure detection result is used as a test target, and the status information of the service providing system and the failure detection result output from the failure detection system Based on the failure type, the state information of the service providing system is generated for each type of failure that constitutes the failure detection result. A failure discovery system verification device characterized in that a degree of deviation indicating a degree of deviation from the failure type state model is obtained, and the detected failure cases are presented in descending order of the degree of deviation. The failure detection system is configured such that a predetermined adjustment for detecting a failure of the service providing system is performed,
The fault detection system verification device
The failure detection results output from the failure detection system provided with test data in which status information of the service providing system is accumulated are classified for each type of failure, and based on the test data, the failure detection result is classified. For each type, the statistical data of each state parameter indicating the state of the service providing system is obtained to generate the state model for each failure type, and for each detected failure case constituting the failure detection result, from the test data The state information of the service providing system corresponding to the time and place where each failure has occurred is extracted, and the state information is compared with the state model for each failure type corresponding to the failure type of the detected failure case. The degree of deviation indicating the degree of deviation from another state model is obtained, and the detected failure cases are presented in descending order of the degree of deviation. And trouble detection system verification apparatus according to claim 1, wherein the are.   Based on the test data, the service providing system obtains statistical data of each state parameter indicating a normal state, generates a normal state model, and configures a system state when determining the degree of detachment Among the state parameters, the behavior of the state model according to the failure type is preferential to the state model having a characteristic parameter compared to the state model according to the failure type other than the state model and the behavior of the normal state model. The failure detection system verification device according to claim 2, wherein the failure detection system verification device is used. Fault type classification means for classifying the fault detection results output from the fault detection system to which the test data is given for each type of fault;
State model generation means for each failure type that generates statistical data for each state parameter by obtaining statistical data of each state parameter indicating the state of the service providing system for each type of failure based on the test data;
For each discovered failure case constituting the failure detection result, the status information of the service providing system corresponding to the time and place where each failure occurred is extracted from the test data, and the status information is extracted from the detected failure case. Compared with the state model for each failure type corresponding to the failure type, an outlier calculation means for obtaining a degree of deviation indicating the degree of deviation from the state model for each failure type;
The failure detection system verification apparatus according to claim 2, further comprising: a category extraction unit that sequentially extracts and presents the found failure cases having a large degree of deviation. Based on the test data and the failure detection result, a normal state model generation unit that generates a normal state model by obtaining statistical data of each state parameter indicating a normal state of the service providing system;
Among the state parameters constituting the system state, the behavior in the state model for each failure type has characteristic parameters as compared with the behavior in the state model for each failure type other than the state model and the normal state model. When selecting a state model as a characteristic state model, and determining the degree of deviation by the degree of deviation calculation means, there is provided a parameter selection means for determination by failure type that preferentially uses the characteristic state model. The fault detection system verification apparatus according to claim 4, wherein:   A failure detection system that detects a failure of a service providing system that provides a predetermined service and outputs a failure detection result is a test target, and the failure detection system verification device outputs the status information of the service providing system and the failure detection system Based on the failure detection result to be generated, for each type of failure, to generate a state model for each failure type indicating the statistical data of the state information of the service providing system, for each discovery failure case constituting the failure detection result, A failure discovery system verification method characterized by obtaining a degree of deviation indicating a degree of deviation of the state information of the service providing system from the state model for each failure type, and presenting the detected failure cases in descending order of the degree of deviation .   The failure detection system is configured to perform a predetermined adjustment for detecting a failure in the service providing system, and the failure detection system verification device includes test data in which status information of the service providing system is accumulated. Classifying the failure detection results output from the failure discovery system to each type of failure, and for each status parameter indicating the status of the service providing system for each type of failure based on the test data Providing the service corresponding to the time and place where each failure occurred from the test data for each detected failure case that constitutes the failure detection result while generating statistical data and generating a state model for each failure type State information of the system is extracted, and the state information corresponding to the failure type of the found failure case 7. The failure detection system verification according to claim 6, wherein a degree of deviation indicating a degree of divergence with respect to the state model for each failure type is obtained, and the detected failure cases are presented in descending order of the degree of deviation. Method.   Based on the test data, the failure detection system verification device obtains statistical data of each state parameter indicating a normal state of the service providing system, generates a normal state model, and calculates the degree of detachment At the time, among the state parameters constituting the system state, the behavior in the state model according to the failure type is characteristic compared with the behavior in the state model according to the failure type other than the state model and the normal state model. The failure detection system verification method according to claim 7, wherein a state model having parameters is preferentially used. Fault type classification processing for classifying the fault detection results output from the fault detection system to which the test data is given for each fault type;
State model generation processing for each fault type for generating statistical data for each type of fault parameter by obtaining statistical data of each status parameter indicating the status of the service providing system for each type of fault based on the test data;
For each discovered failure case constituting the failure detection result, the status information of the service providing system corresponding to the time and place where each failure occurred is extracted from the test data, and the status information is extracted from the detected failure case. Compared with the state model for each failure type corresponding to the failure type, an outlier calculation process for obtaining a degree of deviation indicating the degree of deviation from the state model for each failure type;
The failure detection system verification method according to claim 7, wherein a classification extraction process is performed in which classification extraction is performed in order from the detected failure cases with a large degree of deviation. Based on the test data and the failure detection result, a normal state model generation process for generating a normal state model by obtaining statistical data of each state parameter indicating the normal state of the service providing system;
Among the state parameters constituting the system state, the behavior in the state model for each failure type has characteristic parameters as compared with the behavior in the state model for each failure type other than the state model and the normal state model. A state model is selected as a characteristic state model, and when determining the degree of deviation in the degree of deviation calculation processing, a parameter selection process for determination by failure type that preferentially uses the characteristic state model is performed. The failure detection system verification method according to claim 9. A failure detection system that detects a failure of a service providing system that provides a predetermined service and outputs a failure detection result is used as a test target.
A failure detection system verification device configured by a computer, based on the state information of the service providing system and the failure detection result output from the failure detection system, statistics of the state information of the service providing system for each type of failure A state model for each failure type indicating data is generated, and a degree of deviation indicating a degree of deviation of the state information of the service providing system from the state model for each failure type is obtained for each detected failure case constituting the failure detection result. A computer-readable failure detection system verification control program for executing processing for presenting the detected failure cases in descending order of the degree of deviation.   The failure detection system is configured such that a predetermined adjustment for detecting a failure in the service providing system is performed, and the failure detection system is provided with test data in which state information of the service providing system is integrated Classifying the failure detection results output from each failure type, and obtaining statistical data of each state parameter indicating the state of the service providing system for each failure type based on the test data; Generating another state model, and for each discovered failure case constituting the failure detection result, taking out the state information of the service providing system corresponding to the time and place where each failure occurred from the test data; Comparing the state information with the state model by failure type corresponding to the failure type of the detected failure case, the state model by failure type 12. The fault detection system verification control according to claim 11, wherein the fault detection system verification apparatus is configured to cause the fault detection system verification apparatus to execute a process of obtaining a faulty degree indicating a degree of divergence with respect to each other and presenting the detected fault cases in descending order of the deviation degree. program.   Based on the test data, the service providing system obtains statistical data of each state parameter indicating a normal state, generates a normal state model, and configures a system state when determining the degree of detachment Among the state parameters, the behavior of the state model according to the failure type is preferential to the state model having a characteristic parameter compared to the state model according to the failure type other than the state model and the behavior of the normal state model. 13. The fault detection system verification control program according to claim 12, which causes the fault detection system verification apparatus to execute processing used for the fault detection system. Fault type classification processing for classifying the fault detection results output from the fault detection system to which the test data is given for each fault type;
State model generation processing for each fault type for generating statistical data for each type of fault parameter by obtaining statistical data of each status parameter indicating the status of the service providing system for each type of fault based on the test data;
For each discovered failure case constituting the failure detection result, the status information of the service providing system corresponding to the time and place where each failure occurred is extracted from the test data, and the status information is extracted from the detected failure case. Compared with the state model for each failure type corresponding to the failure type, an outlier calculation process for obtaining a degree of deviation indicating the degree of deviation from the state model for each failure type;
13. The fault discovery system verification control program according to claim 12, wherein the fault discovery system verification control program causes the fault discovery system verification apparatus to execute classification extraction processing for classifying and presenting the detected fault cases in descending order of the degree of deviation. Based on the test data and the failure detection result, a normal state model generation process for generating a normal state model by obtaining statistical data of each state parameter indicating the normal state of the service providing system;
Among the state parameters constituting the system state, the behavior in the state model for each failure type has characteristic parameters as compared with the behavior in the state model for each failure type other than the state model and the normal state model. When selecting a state model as a characteristic state model and determining the degree of detachment in the detachment degree calculation process, a failure type determination parameter selection process that preferentially uses the characteristic state model, the failure discovery 15. The failure detection system verification control program according to claim 14, which is executed by a system verification device.

Images