JP6503223B2 - Device, system, program and method for evaluating failure prediction result in consideration of operation loss - Google Patents

Description

  The present invention relates to a technology for evaluating prediction results by failure prediction means for predicting the occurrence of a failure in a business system.

  For today's communication service providers and Internet related service providers, predicting the occurrence of a failure in an NW facility or the like installed in a communication network (NW) system realizes an NW operation that enables quick failure recovery. To be important.

  In fact, to date, various failure prediction techniques have been researched and developed in various business systems. For example, in the techniques described in Patent Documents 1 to 3, as a classification learning algorithm, naive Bayes (see, for example, Non-patent Document 1), support vector machines (see, for example, Non-patent document 2), and decision trees (see, for example, Non-patent Document 3). Etc.) and predict the occurrence of failure based on data acquired by the monitoring equipment.

  Here, it is extremely important to evaluate to what extent the prediction result by these fault prediction techniques is actually effective and to decide what fault prediction method should be applied to the business system such as the target communication network. It is important.

が挙げられる。上式（１）において、kは事象種別を指定する指数であり、Kは全事象種別を含む集合である。また、Accuracyは予測全体の正解率を示し、F-measure_kは事象種別k毎の予測性能を示す。さらに、Precision_kは、ある事象種別kであると予測された場合に正解もその事象種別kである割合を示し、Recall_kは、ある事象種別kが正解である場合に予測結果もその事象種別kである割合を示す。このうち、F-measure_kはPrecision_kとRecall_kとの調和平均として算出される。 As a conventional index to measure the effectiveness of the prediction result of such an event that may occur, Accuracy, F-measure, Precision and Recall described in Non-Patent Document 4, that is,

Can be mentioned. In the above equation (1), k is an index specifying an event type, and K is a set including all event types. Also, “Accuracy” indicates the accuracy rate of the entire prediction, and “F-measure _k” indicates the prediction performance for each event type k. Furthermore, when Precision _k is predicted to be a certain event type k, it indicates the ratio that the correct answer is also that event type k, and Recall _k shows that the predicted result is also that event type when a certain event type k is correct. Indicates the proportion k. Among these, F-measure _k is calculated as a harmonic mean of Precision _k and Recall _k .

These indices are calculated from TP (True Positive), FP (False Positive), FN (False Negative), and TN (True Negative), as shown in the above equation (1). Here, since evaluation of the prediction result of an event is based on grasping the correct answer rate for each event type k, TP, FP, FN, and TN are defined as follows and their values are acquired.
TP _k : Number of event types k correctly predicted as event type k
FP _k : Number of failures that are not event types k and are incorrectly predicted as event types k
FN _k : Number of cases where event type k was mistakenly predicted as failure not event type k
TN _k : Number of failures that are not event type k predicted as failures that are not correctly event type k

Unexamined-Japanese-Patent No. 2012-128583 JP, 2013-191672, A JP, 2013-13075, A

Pedro Domingos & Michael Pazzani, "On the Optimality of the Bayesian Classifier under Zero-One Loss", Machine learning, Vol. 29, pp. 103-130, 1997 Corinna Cortes and Vladimir Vapnik, "Support-Vector Networks", Machine Learning, Vol. 20, pp. 273-297, 1995 Rokach, Lior and Maimon, O., "Data mining with decision trees: theory and applications", World Scientific Pub Co Inc., 2007 Hiroshi Motoda, Shusaku Tsumoto, Takahira Yamaguchi, and Masayuki Numao, "Basics of IT Text Data Mining", Ohm, 2006

However, even if the prediction result by the failure prediction technology is evaluated using the above-mentioned indexes such as conventional Accuracy, F-measure _k , Precision _k and Recall _k mentioned above, it is still realistic by failure prediction in the operation of the business system. It is difficult to evaluate the merits and demerits.

In fact, the indicators such as Accuracy, F-measure _k , Precision _k, and Recall _k described above are calculated based on a simple total number of prediction results regarding event types such as TP _k , FP _k , FN _k, and TN _k . On the other hand, in the failure prediction techniques described in Patent Documents 1 to 3, circumstances in the operation aspect such as the operation cost of the business system are not reflected at all in the prediction result. Therefore, the evaluation results by these indicators can not be considered in consideration of such operational circumstances.

  On the other hand, in the actual operation of the business system, for example, when the occurrence of a failure is predicted, the cost required to replace the broken equipment and the cost required for dispatching to the site where the failure occurred, etc. are taken into consideration. The outcome of this prediction needs to be evaluated. In addition, when dealing with failures based on the prediction results, it is also possible to take into consideration the degree of operationally occurring loss, such as how much the service provided will deteriorate and how long it will take to recover from the failure. Is also desired. However, it has never been proposed to evaluate failure prediction results from the viewpoint of operation loss including such operation costs.

  Therefore, the present invention aims to provide an apparatus, a system, a program, and a method capable of evaluating the prediction result of failure occurrence in consideration of the operation loss of the business system.

According to the present invention, there is provided a failure prediction and evaluation device for evaluating the prediction result by the failure prediction means for predicting the occurrence of a failure in a business system,
For each of a plurality of failure types as prediction results by the failure prediction means, the number of times the failure type is actually predicted as a predicted failure type for each correct failure type that is a failure type that was correct when predicted Forecasting result acquiring means for acquiring such prediction frequency information;
Acquisition of operation loss, which is information related to operation loss including the operation cost that occurs when each correct failure type is dealt with or left according to the content of the predicted failure type for each of the predicted failure types Means,
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type There is provided a failure prediction and evaluation device having evaluation value calculation means for calculating an evaluation value serving as an index of evaluation of a prediction result by the failure prediction means based on a value weighted by operation loss information.

Here, the business system targeted for failure occurrence prediction in the present invention is provided with at least one facility,
The prediction result acquisition means acquires the predicted number of times information on the type of failure indicating that the facility needs replacement or repair, and the type of failure indicating that the facility need not be replaced or repaired.
The operation loss acquiring unit is used as operation loss information corresponding to the correctness prediction type indicating that the equipment need not be replaced or repaired, and the predicted failure type indicating that the facility needs to be exchanged or repaired. It is also preferable to adopt a value to which the cost for replacing or repairing the facility is added.

Furthermore, the business system targeted for failure occurrence prediction has at least one site that needs to be responded on the spot when a failure occurs,
The prediction result acquisition means acquires the number of times of prediction about the failure type to the effect that the local response is required and the failure type to the effect that the local response is unnecessary.
The operation loss acquiring means is used as operation loss information corresponding to the correct answer prediction type indicating that the response in the field is unnecessary and the predicted failure type indicating that the error in the field requires the response. It is also preferable to adopt a value to which the dispatch cost for the field is added.

  Furthermore, it is also preferable that the business system to be subjected to failure occurrence prediction is a communication network system provided with at least one facility and having at least one site that may need to be dealt with on the spot when a failure occurs.

  Further, in an embodiment in which the communication network system is the failure occurrence prediction target, the prediction result for deriving the prediction number information acquired by the prediction result acquisition means is the equipment from which the failure prediction means is concerned or the network in advance. Prediction acquired by inputting alarm information to be predicted to a learning device constructed by learning alarm information related to the network collected from the installed monitoring equipment and the correct failure type as a teacher data set It is also preferable that it is a result.

Furthermore, as one embodiment of the failure prediction and evaluation apparatus according to the present invention, the plurality of failure types are (a) failure types indicating that the failure is an unnecessary failure;
(B) Failure type indicating that the remote reset needs to be performed,
(C) selected from the group consisting of a failure type indicating that the business system requires a local response, and (d) a failure type requiring replacement or repair of facilities of the business system. It is also preferable to include at least one.

Furthermore, as another embodiment of the failure prediction and evaluation apparatus according to the present invention, the operation loss information is
(A) An increase or decrease in the operation cost compared to the operation cost required when there is no failure prediction
(B) Degree of improvement or deterioration of service caused by failure or failure of failure prediction, and (c) increase or decrease of failure recovery time compared with time for failure recovery required without failure prediction It is also preferable to include at least one selected from

Further, in the present invention, the operation loss information takes a negative value or zero when the corresponding correct error type and the predicted failure type match, and takes a positive value or zero when they do not match, and the evaluation value is It is also preferable to show an evaluation that the effectiveness of the prediction result by the failure prediction means is higher as the absolute value is larger when taking a negative value.

  Furthermore, in the embodiment in which the business system includes at least one facility, the evaluation value calculation means corresponds to each correct fault type in the predicted fault type to the effect that it is a fault requiring replacement or repair of the facility. It is also preferable to calculate an evaluation value for evaluating the effectiveness of the prediction result to the effect that replacement or repair of the facility is necessary based on a value obtained by weighting the prediction frequency information by the operation loss information.

According to the present invention, there is also provided failure prediction means for predicting the occurrence of a failure in a communication network system provided with at least one facility, and failure prediction and evaluation device for evaluating the prediction result by this failure prediction means. The failure prediction and evaluation system is
A plurality of failure types as prediction results by the failure prediction means, that is a failure type to the effect that the equipment needs replacement or repair, and a failure type to the effect that the facility replacement or repair is unnecessary. For each of a plurality of failure types including, for each correct error type that is the failure type that was the correct answer when predicted, prediction frequency information is acquired regarding the number of times the failure type is actually predicted as a predicted failure type. A prediction result acquisition means,
It is an operation cost that occurs when dealing with or leaving each of the correct failure types according to the content of the predicted failure type for each of the correct failure types, that is, a correct answer indicating that replacement or repair of equipment is unnecessary. An operation loss including an operation cost that corresponds to the prediction type and the predicted failure type indicating that the equipment needs replacement or repair, and the cost for replacement or repair of the equipment is added Operation loss acquisition means for acquiring operation loss information which is information related to
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type There is provided a failure prediction system including: evaluation value calculation means for calculating an evaluation value serving as an index of evaluation of a prediction result by the failure prediction means based on a value weighted by operation loss information.

According to the present invention, there is further provided a program for causing a computer mounted on a device for evaluating the prediction result by the failure prediction means for predicting the occurrence of a failure in a business system to function.
For each of a plurality of failure types as prediction results by the failure prediction means, the number of times the failure type is actually predicted as a predicted failure type for each correct failure type that is a failure type that was correct when predicted Forecasting result acquiring means for acquiring such prediction frequency information;
Acquisition of operation loss, which is information related to operation loss including the operation cost that occurs when each correct failure type is dealt with or left according to the content of the predicted failure type for each of the predicted failure types Means,
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type There is provided a failure prediction and evaluation program that causes a computer to function as an evaluation value calculation unit that calculates an evaluation value that is an index of evaluation of a prediction result by the failure prediction unit based on a value weighted by operation loss information.

According to the present invention, there is further provided a method of evaluating the prediction result by the failure prediction means for predicting the occurrence of a failure in a business system,
For each of a plurality of failure types as prediction results by the failure prediction means, the number of times the failure type is actually predicted as a predicted failure type for each correct failure type that is a failure type that was correct when predicted Acquiring such prediction number information;
Acquiring operation loss information, which is information relating to operation losses including operation costs generated when taking measures or leaving according to the contents of the predicted failure type for each of the correct failure types,
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type There is provided a failure prediction evaluation method comprising the steps of: calculating an evaluation value serving as an index of evaluation of a prediction result by the failure prediction means based on a value weighted by operation loss information.

  According to the failure prediction and evaluation apparatus, system, program, and method of the present invention, it is possible to evaluate the prediction result of failure occurrence taking into consideration the operation loss of the business system.

FIG. 1 is a schematic view showing an embodiment of a failure prediction and evaluation system including a failure prediction and evaluation device according to the present invention. It is a schematic diagram for demonstrating roughly the operation | movement phase of a failure prediction apparatus. It is a functional block diagram which shows the function structure in one Embodiment of the failure prediction evaluation apparatus by this invention. It is a table figure showing one embodiment of a fault classification table. It is a table figure showing one embodiment of an operation cost table. It is a table figure for describing one Example of the failure prediction evaluation process by this invention. It is a table figure for describing one Example of the failure prediction evaluation process by this invention. It is a table figure for describing one Example of the failure prediction evaluation process by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Failure prediction and evaluation system
FIG. 1 is a schematic view showing an embodiment of a failure prediction and evaluation system including a failure prediction and evaluation device according to the present invention.

In FIG. 1, the failure prediction and evaluation system of this embodiment is
(A) Failure prediction device 2 as failure prediction means for predicting the occurrence of a failure in a communication network (NW) system 3 as a business system;
(B) A failure prediction and evaluation device 1 for evaluating the prediction result output from the failure prediction device 2 is provided.

  The communication NW system 3 is a carrier communication network, and may be, for example, a wireless access network such as LTE (Long Term Evolution) or WiMAX (Worldwide Interoperability for Microwave Access), which is a mobile communication network, or an optical fiber It may be a fixed access network such as a network. The communication NW system 3 includes a plurality of communication facilities 31, and may further include a monitoring facility 32 that monitors the state of the network. The monitoring equipment 32 receives various measurement item data and various alarms in the communication NW system 3 from the communication equipment 31 etc., and collects them and transmits them to the failure prediction device 2.

  The failure prediction device 2 receives measurement item data and alarms in the communication NW system 3 from the monitoring equipment 32 and the communication equipment 31, and predicts the occurrence of a failure in the communication NW system 3 based on the received data. Generate and output. Here, the alarm is information related to an abnormality or a defect in the operation / operation of the communication NW system 3, and examples thereof include information on an NW link down error and information on a signal synchronization processing error.

  Further, in the present embodiment, the prediction result by the failure prediction device 2 is a failure type that is the type of failure to be predicted. As failure types, for example, there may be mentioned “a failure requiring no coping”, “a failure requiring a remote reset”, “a failure requiring on-site handling”, “a failure requiring equipment replacement” and the like. Here, for example, when there are a plurality of communication facilities 31 that can be replaced, “fault requiring replacement of facility A”, “fault requiring replacement of facility B”, “facility A and facility C” as failure types. It is also preferable that the type of each equipment or each set of equipment is listed, such as “failure requiring replacement”.

  Furthermore, for example, when there are a plurality of locations where communication in response to a failure is required in the communication NW system 3, as the failure type, "a failure requiring a response to D" or "a failure requiring a response to E" It is also preferable that the type of each field or each group of fields be listed, such as “a failure requiring a response to the field D and the field F”.

  Next, the failure prediction and evaluation device 1 receives the failure type as the prediction result from the failure prediction device 2. Here, the failure prediction and evaluation device 1 and the failure prediction device 2 are communicably connected by a wired or wireless network or via an event person communication network and the Internet. Further, as a modification, the failure prediction evaluation device 1 and the failure prediction device 2 may be incorporated in one device.

  Furthermore, when one failure type is acquired, the failure prediction and evaluation device 1 also acquires information on a correct failure type that is a failure type that was actually the correct answer. The correct failure type is, for example, a business operator (NW operator) investigates the actual situation of failure occurrence in the communication NW system 3 for each failure prediction occurrence or after performing a countermeasure and collects and records Can be obtained by Further, as this correct answer failure type, the correct answer failure type as teacher data to be used later in the learning phase of the failure prediction device 2 shown in FIG. 2 (A) may be used.

The failure prediction and evaluation apparatus 1 further
(A) For each of the plurality of failure types as the prediction result, “predicted number information” is acquired for the number of times the failure type is actually predicted as the predicted failure type, for each correct error type when predicted.
(B) For each correct failure type, for each of the predicted failure types, obtain “operation loss information” that is information related to the operation loss including the operation cost that occurs when handling or leaving according to the content of the predicted failure type ,
(C) For each set of the correct failure type and the predicted failure type, based on the value obtained by weighting the corresponding “predicted number of times information” by the “operation loss information”, it becomes an index of evaluation of the prediction result by the failure prediction device 2 Calculate the "evaluation value".

Here, the “operation loss information” in the above (B) is
(B1) An increase or decrease in the operation cost (for example, the amount of money) compared to the operation cost required when there is no failure prediction
(B2) Degree of improvement or deterioration of service caused by failure or failure of failure prediction, or (B3) increase or decrease failure recovery time as compared with time for failure recovery required without failure prediction be able to. Furthermore, it is also possible to make all of two selected from the above (B1) to (B3) or all of the above (B1) to (B3) as "operation loss information".

  As described above, according to the failure prediction and evaluation device 1, it is possible to evaluate the prediction result by the failure prediction device 2 also in consideration of the information related to the actual operation of the communication NW system 3. That is, it becomes possible to evaluate the practical merits and demerits of failure prediction in the operation of the business system.

  In addition, by evaluating the prediction result in consideration of such an operation loss, as described later, it is possible to calculate the operation cost (amount of money) generated when the operation countermeasure is performed or left according to the prediction result. . Furthermore, the failure prediction device 2 can also be evaluated using this calculated operation cost as an evaluation index. That is, it is also possible to determine an optimal failure prediction device (system) from the viewpoint of the operation cost. Furthermore, it is also possible to evaluate the failure prediction device (system) in consideration of the degree of importance in terms of operation costs for each prediction result.

In the calculation of the "evaluation value" in (C) above, the corresponding "predicted number of times information" is weighted by the "operation loss information".
(C1) Besides being performed for each pair of the correct failure type and the predicted failure type,
(C2) It may be performed for each correct failure type in one predicted failure type, or (C3) for each predicted failure type in one correct error type. In the case of (C1) to (C3), evaluation values corresponding to Accuracy, Precision, and Recall are calculated, as described in detail later. As described above, according to the failure prediction and evaluation apparatus 1, it is also possible to calculate an "evaluation value" corresponding to a desired evaluation standard.

Failure prediction device
FIG. 2 is a schematic diagram for schematically explaining the operation phase of the failure prediction device 2. The operation phase in the failure prediction device 2 includes an “operation phase”, a “learning device evaluation phase”, and a “prediction phase”.

According to FIG. 2A, in the learning phase of the failure prediction device 2,
(A) Alarms generated from communication equipment 31 etc. collected in the past or at the present time,
(B) A correct failure type, which is the type of failure that actually occurred when the alarm is generated, is used as a teacher data set, and a plurality of teacher data sets generated from the alarm group according to (a) are failure prediction devices Input 2 to build a machine learning machine.

  Here, the learning device to be constructed differs depending on the learning method (type of machine learning, parameters of machine learning, format of teacher data set, etc.). As a type of machine learning, various algorithms such as naive Bayesian, support vector machine (SVM), or decision tree may be adopted as long as they perform supervised learning.

  Next, according to FIG. 2 (B), in the learning device evaluation phase, the alarm group used as teacher data in the learning phase is input to the constructed learning device, and the prediction result for each input alarm is experimentally input ( Output failure type). Next, the output prediction result is compared with failure type data used as teacher data in the learning phase, and for example, the matching rate between the two is calculated to evaluate the prediction performance of the constructed learning device. Here, when the evaluation result is not good (for example, when the matching rate is less than a predetermined value), it is also preferable to reconstruct the learning device.

  Furthermore, according to FIG. 2C, in the prediction phase, the alarm (group) to be predicted is input to the learning device that has undergone the processing in the learning device evaluation phase, and the failure type as the prediction result is output.

  As described above, the prediction result from the failure prediction device 2 acquired by the failure prediction evaluation device 1 is alarm information related to a network in which the failure prediction device 2 is collected from the communication facility 31, the monitoring facility 32, etc. It is the prediction result output by inputting the alarm information which is the prediction target to the learning device which is constructed by learning the correctness type and the failure type as the teacher data set.

[Failure prediction and evaluation system]
FIG. 3 is a functional block diagram showing a functional configuration in an embodiment of a failure prediction and evaluation apparatus according to the present invention.

  According to FIG. 3, the failure prediction and evaluation apparatus 1 has a communication interface unit 101, a prediction result storage unit 102, an operation cost storage unit 103, an evaluation result storage unit 104, and a processor memory. Here, the processor memory implements a function of evaluating the effectiveness of the prediction result by the failure prediction device 2 by executing a program that causes a computer installed in the failure prediction evaluation device 1 to function.

  The processor memory also includes a prediction result acquisition unit 111, an operation cost acquisition unit 112 as an operation loss acquisition unit, an evaluation value calculation unit 113, and a communication control unit 121. Here, the operation cost acquisition unit 112 includes a facility replacement cost calculation unit 112 a and a local dispatch cost calculation unit 112 b. Furthermore, the evaluation value calculation unit 113 includes an accuracy correspondence evaluation value calculation unit 113a, a precision correspondence evaluation value calculation unit 113b, and a recall correspondence evaluation value calculation unit 113c. The flow of processing in which each functional component in FIG. 3 is connected by an arrow is also understood as one embodiment of the failure prediction evaluation method according to the present invention.

The communication interface unit 101 receives
(A) receiving information on a failure type as a prediction result and information on a failure type that was correct when outputting the prediction result;
(B) The evaluation result including the evaluation value calculated by the evaluation value calculation unit 113 described later is transmitted to an external notification destination, for example, an information processing apparatus of a provider who manages the communication NW system 3.

  The communication control unit 121 outputs the predicted failure type information and the correct failure type information received via the communication interface unit 101 to the prediction result acquisition unit 111, while the failure types listed as the predicted failure type and the correct failure type. Is output to the operation cost acquisition unit 112. Further, it outputs the evaluation result including the evaluation value calculated by the evaluation value calculation unit 113 described later to the communication interface unit 101 so as to be transmitted to the outside.

  The prediction result acquisition unit 111 actually predicts the failure of the failure type for each correct failure type that is the failure type that was correct when the prediction was made for each of the plurality of failure types as the prediction result by the failure prediction device 2 Information on the number of times of prediction associated with the number of times predicted as a type is acquired, and a failure type table 111t is generated. It is also preferable that the generated failure type table 111 t be accumulated in the prediction result accumulation unit 102.

  The prediction result acquisition unit 111 acquires correct failure type information corresponding to the predicted failure type by an input by an evaluator (not shown) provided on the device 1, for example, a touch panel or a keyboard. May be

  FIG. 4 is a table diagram showing an embodiment of the failure type table 111t.

  According to FIG. 4, in the failure type table 111 t, N predicted failure types as prediction results acquired from the failure prediction device 2 are arranged as column items, and N correct answers that are actual failure types at the time of prediction Failure types are arranged as line items. For each of the predicted failure type and the correct error type listed, "Correction-unnecessary failure", "Failure requiring remote reset", "Failure requiring local response", and "Replacement of equipment (or repair)" are required. Failure is included.

Also, the data x _ij (i = 1, 2,..., N, j = 1, 2,..., N) in each column of the table 111t correspond to the row items and column items of the column. Indicates the number of occurrences of For example, column A of table 111t is predicted to be "a failure that requires equipment replacement", but corresponds to a situation in which the correct answer is "a failure that requires no action" and failure to cope with. The value x _{1 N} indicates the number of occurrences of such a forecast failure situation. On the other hand, the B column in the table 111t corresponds to the predicted success that the predicted “fault requiring replacement of equipment” was actually correct, and the value x _{NN in the} B column is such a prediction Indicates the number of successful events that occurred.

The predicted failure type and the correct failure type are not limited to those listed in the table 111t. The predicted failure type and the correct failure type are respectively "action-free failure", "remote-reset required failure", "local-needed failure" and "equipment required replacement (or repair) failure" It may be in a form including at least one, and as described above, it is also preferable that failure types for each facility or set of facilities, or fields or types of faults for each site are individually adopted.

Further, the unit of the data x _ij in each column of the table 111t may not be the number, but may be, for example, a ratio of the number in the table 111t. However, by setting this unit as the number of cases, it is possible to set an evaluation value to be calculated later as, for example, the operation cost amount (amount of money) and to make the meaning easy to interpret.

  Returning to FIG. 3, the operation cost acquisition unit 112 is an operation cost that is information relating to an operation cost that occurs when each correct failure type is dealt with or left according to the content of the predicted failure type for each of the predicted failure types. Information is acquired, and the operation cost table 112t is generated. It is also preferable that the generated operation cost table 112 t be accumulated in the operation cost accumulation unit 103.

  Here, in the present embodiment, in the failure type table 111t (FIG. 4), a failure type to the effect that the replacement (or repair) of the equipment is necessary and a failure that the replacement (or repair) of the equipment is unnecessary The failure types of the subject are listed as the predicted failure type and the correct failure type, and prediction frequency information corresponding to these is acquired. In such an embodiment, it is preferable that the operation cost acquisition unit 112 also have a facility replacement cost calculation unit 112 a.

  The equipment replacement cost calculation unit 112a copes with the correctness prediction type that the equipment replacement (or repair) is unnecessary and the prediction failure type that the equipment replacement (or repair) is necessary. A value (for example, the amount of money) to which the cost amount related to the replacement (or repair) of the facility is added is adopted as the operation cost information to be performed.

  Furthermore, in the present embodiment, in the failure type table 111t (FIG. 4), the failure type to the effect that the on-site response is required and the failure type to the on-site response are unnecessary. The failure type and the correct failure type are listed, and prediction number information corresponding to these is acquired. In such an embodiment, the operation cost acquisition unit 112 preferably includes the on-site dispatch cost calculation unit 112 b.

  The on-site dispatch cost calculation unit 112 b operates as operation cost information corresponding to the correct answer prediction type indicating that the on-site response is unnecessary and the predicted failure type indicating that the on-site response is necessary. The value (for example, the amount of money) to which the dispatch cost for the field concerned is added is adopted.

  FIG. 5 is a table diagram showing an embodiment of the operation cost table 112t.

  According to FIG. 5, in the operation cost table 112t, N predicted failure types and N correct failure types identical to the failure type table 111t (FIG. 4) are arranged as column items and row items.

Also, data y _ij (i = 1, 2,..., N, j = 1, 2,..., N) in each column in the table 112t correspond to the row item and the column item of the column. Indicates the increase or decrease (for example, the amount of money) of the operation cost.

For example, column C of table 112t is predicted to be "a failure that requires equipment replacement", but corresponds to a situation in which the correct answer is "a failure requiring no action", and it corresponds to a situation in which failure occurs. The value y _{1 N} indicates an increase (a positive monetary value) in the operation cost due to the originally unnecessary equipment replacement being performed in accordance with this erroneous prediction result. On the other hand, the D column in the table 112t corresponds to the predicted success that the predicted “fault requiring replacement of equipment” was actually correct, and the value y _{NN in the} D column is the correct prediction result. Indicates the reduction of the operation cost originally necessary due to performing the necessary equipment replacement according to the above (for example, the investigation cost until it is judged that the equipment replacement is necessary in the situation where the prediction result is not obtained). In this case, y _NN is a negative amount as a reduction.

  Thus, the operation cost (operation loss) information takes a negative value (or zero) if the corresponding correct failure type and the predicted failure type match, and takes a positive value or (zero) if they do not match. Set to As a result, it is possible to give direction as a value for evaluating the effectiveness of the prediction result to the evaluation value derived later.

In the present embodiment, in the case where the corresponding measures are taken (or the corresponding neglected measures are taken) according to each prediction result, how much the operation cost increases as compared with the operation cost originally required, or It is assumed that the amount of money to be reduced (yij) is estimated in advance. For example, the following estimation method can be adopted as a method of determining the operation cost increase / decrease amount yij.
(A) Reduction in cost of handling based on man-hours saved when only correct handling is implemented.
(B) Facility cost based on the price of the facility and the installation cost when the facility replacement is performed accidentally.
(C) Repair cost based on the contract with the vendor when the facility is sent to the vendor for repair by mistake.
(D) Dispatch cost based on the distance to the site, the number of people, the amount of work, and the time zone, when the site work is performed by mistake.
(E) Loss costs based on service impact time and impact numbers if you have inadvertently omitted the necessary actions.

  The operation cost acquisition unit 112 acquires the above information (a) to (e) by the input by the evaluator via an input interface (not shown) provided on the device 1, for example, a touch panel or a keyboard. May be

  In addition, as a modification, along with or instead of the operation cost, the degree of improvement or deterioration of service caused by failure or failure of failure prediction, and / or for failure recovery required when there is no failure prediction. Quantify operational loss (operational risk) such as increase or decrease in failure recovery time compared with time, for example, using monetary amount, generate operation loss table in the same format as the operation cost table 112t, and evaluate the prediction result It can also be used for In this case, it is possible to evaluate the failure prediction result in consideration of service quality and failure recovery time.

Returning to FIG. 3, the evaluation value calculation unit 113
(C1) For each pair of the correct failure type and the predicted failure type,
(C2) Based on a value obtained by weighting and adding prediction frequency information corresponding to each correct failure type in one predicted failure type or (C3) each predicted failure type in one correct error type with operation loss information, An evaluation value which is an index of evaluation of a prediction result by the failure prediction device 2 is calculated. The calculated evaluation value is an identifier given to the learning device of the device 2 to be evaluated, or a learning method in the learning device of the device 2 to be evaluated (type of machine learning, parameters of machine learning, It is also preferable to be stored in the evaluation result storage unit 104 in association with an identifier given for each format of the teacher data set and the like.

Specifically, the evaluation value calculation unit 113 calculates an evaluation value by combining the data y _ij fault type table 111t data x _ij and operational cost table 112t (FIG. 4) (Fig. 5).

Here, the accuracy correspondence evaluation value calculation unit 113 a of the evaluation value calculation unit 113 is the Accuracy (accuracy) correspondence evaluation value E _a in the case of the above (C 1), that is,
(2) E _a = S _{i = 1} ^{N S} _{j = 1} ^N x _ij y _ij
Calculate In the above equation (2), _{i i = 1} ^N is the sum from 1 to N for i, and _{j j = 1} ^N is the sum from 1 to N for j. The evaluation value E _a is the evaluation target of the range corresponding to the so-called Accuracy in the failure type table 111 t, and the corresponding countermeasure is performed according to all the prediction results (when the corresponding non-disposal treatment is taken), It is a value (amount of money) that indicates the total sum of the increase and decrease in the operation cost when compared with the originally required operation cost.

In fact, the increase or decrease in the operation cost y _ij (FIG. 5) takes a negative value or zero if the corresponding correct failure type and the predicted failure type match, and takes a positive value or zero if they do not match. The value E _a indicates an evaluation that the effectiveness of the prediction result by the failure prediction device 2 is higher as the absolute value as the negative value is larger.

Further, the precision corresponding evaluation value calculation unit 113 b of the evaluation value calculation unit 113 is a precision (precision) corresponding evaluation value E _pj in the case of (C2), that is,
(3) E _pj = Σ _{i = 1} ^N x _ij y _ij
Calculate In the above equation (3), _{i i = 1} ^N is the sum of 1 to N for i. The evaluation value E _pj is evaluated for a range corresponding to so-called precision in the failure type table 111 t as an evaluation target, and when the corresponding countermeasure is performed according to one prediction result (the corresponding undiscarded treatment is taken), It is a value (amount of money) indicating the sum of the increase and decrease in the operation cost when compared with the originally required operation cost.

According to the precision corresponding evaluation value calculation unit 113b, for example, the operation cost increases when the facility is replaced (or repaired) according to the predicted failure type of "a failure requiring replacement (or repair) of the facility". An evaluation value _Epj indicating the sum of the reductions is obtained. In this case, the larger the absolute value of the obtained evaluation value E _{pj as} a negative value, the higher the effectiveness in consideration of the operation cost in the prediction result of "a failure requiring equipment replacement (or repair)". become.

Furthermore, the recall-response evaluation value calculation unit 113c of the evaluation value calculation unit 113 is a Recall-response evaluation value E _r in the case of (C3), that is,
(4) E _ri = Σ _{j = 1} ^N x _ij y _ij
Calculate In the above equation (4), _{j j = 1} ^N is the sum of 1 to N for j. The evaluation value E _ri is a evaluation target of a range corresponding to so-called Recall in the failure type table 111t, and in a situation where there is one correct prediction result, the corresponding countermeasure is performed according to the prediction result In the case where the action is taken, this is a value (amount of money) indicating the total of the increase and decrease in the operation cost when compared with the operation cost originally required.

Here, for comparison, calculation of Accuracy, Precision and Recall as conventional evaluation indexes described in Non-Patent Document 4 will be considered. These conventional evaluation indexes are calculated only by the failure type table 111t (FIG. 4) without considering the operation cost at all and therefore without using the operation cost table 112t (FIG. 5). Specifically, Accuracy indicates that the diagonal component of the table 111t indicates the prediction correct solution.
(5) Accuracy = _{i i = 1} ^N x _ii / _{i i = 1} ^N _{j j = 1} ^N x _ij
Calculated in the form of Also, for example, when focusing on “failure requiring equipment replacement”, Precision
(6) Precision = x _NN / _{i i = 1} ^N x _iN
And Recall is
(7) Recall = x _NN / _{j j = 1} ^N x _Nj
Calculated in the form of

  However, in the actual NW operation, the degree of importance in terms of operation cost differs depending on the type of failure. For this reason, even if, for example, the Accuracy of the prediction result by one failure prediction device is high, there are cases where this failure prediction device or the prediction result is not suitable from the viewpoint of the operation cost. This also applies to the above-mentioned evaluation by Precision and Recall.

  On the other hand, in the failure prediction and evaluation device 1, when the corresponding operation measures are performed (or the corresponding left measures are taken) according to the obtained prediction result of the failure occurrence, compared with the operation cost originally required. By setting in advance information on how much the operation cost is incurred or reduced, the evaluation of the effectiveness of the failure prediction result is also performed in consideration of the operation cost. As a result, it becomes possible to evaluate the merits and demerits in terms of the operation cost in the present failure prediction in the business system. Hereinafter, the effect of the failure prediction evaluation according to the present invention in comparison with the conventional evaluation index will be described using an example.

[Example of failure prediction processing]
FIG. 6, FIG. 7 and FIG. 8 are table diagrams for explaining an embodiment of failure prediction evaluation processing according to the present invention.

  In the embodiments shown in FIGS. 6-8, first, learning is performed on five types of faults by different learning methods of the learning method α and the learning method β with respect to certain learning target data, and learning is performed using the respective methods. Built the vessel. Next, the same prediction target data was input to each of the constructed learning devices, and prediction processing was performed multiple times. FIG. 6 shows the prediction result by the learning device constructed by the learning method α in the form of a failure type table. Further, FIG. 7 also shows prediction results by the learning device constructed by the learning method β in the form of a failure type table. Here, the numerical values in each column in the tables of FIG. 6 and FIG. 7 are the corresponding number of times (unit: times). Also, in any of the tables, “prediction and correct answer failure type” includes “failure requiring replacement of facility A” and “failure requiring replacement of facility B,” which are failure types for individual facilities. There is.

  Further, in FIG. 8, when the corresponding operation measures are performed according to the predicted result of the present embodiment (corresponding left measures are taken), an operation cost is generated in comparison with the originally required operation cost. What is summarized or reduced is shown in the form of an operation cost table. Here, the numerical values in the respective columns in the table of FIG. 8 are the corresponding amounts (unit: 10,000 yen).

Then, using the results of the failure type tables (FIGS. 6 and 7) and the operation cost table (FIG. 8) shown above, evaluation values taking into consideration the operation costs for each of the learning methods α and β It derived using 2). Further, for comparison, Accuracy for each of the learning methods α and β was calculated as the conventional index using the above equation (5). as a result,
As an evaluation for failure prediction using the learning method α,
(8) E _a = −230 (10,000 yen)
(9) Accuracy = 0.805
Was calculated. Also,
As an evaluation for failure prediction using the learning method β
(10) E _a = − 385 (10,000 yen)
(11) Accuracy = 0.791
Was calculated.

Here, when comparing the learning method α with the learning method β, it is the learning method α that the Accuracy, which is the conventional evaluation index, is high. However, when the evaluation value E _a of the present embodiment is evaluated, it can be seen that, actually, the reduction amount of the operation cost becomes larger in the learning method β. Therefore, when the operation cost is taken into consideration, it is evaluated that the failure prediction using the learning method β is better.

  As described above in detail, according to the present invention, the effectiveness of the prediction result of failure occurrence by the failure prediction device in this business system is also evaluated in consideration of the information related to the actual operation of the business system. Can. That is, it becomes possible to evaluate the practical merits and demerits of failure prediction in the operation of the business system.

  In this case, it is also possible to directly evaluate the effectiveness in failure prediction with respect to the type of machine learning used in the failure prediction device, the parameters in the machine learning, or the format of the teacher data set used. Become. For example, it is also possible to obtain an answer to the task, such as what kind of machine learning algorithm is effective to use in the current business system.

  Furthermore, by evaluating the forecasted results taking into account the operation cost as the operation loss, it becomes possible to calculate the specific operation cost (amount of money) that occurs when the operation countermeasure is performed or left according to the forecasted result. . In this case, an optimal failure prediction apparatus (system) may be determined from the viewpoint of operation cost. In addition, it is also possible to evaluate the failure prediction device (system) in consideration of the degree of importance in terms of operation cost for each prediction result.

  In addition, the method of failure prediction evaluation according to the present invention is not only the communication NW system as described above, but failure prediction in various business systems such as power generation / transmission system, distribution system such as water service, and various production / manufacturing plants. Applicable to

  For the various embodiments of the present invention described above, various modifications, corrections and omissions of the scope of the technical idea and aspect of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The present invention is limited only as defined in the following claims and the equivalents thereto.

1 Failure prediction and evaluation apparatus 101 Communication interface unit 102 Prediction result storage unit 103 Operation cost storage unit 104 Evaluation result storage unit 111 Measurement result acquisition unit 111t Failure type table 112 Operation cost acquisition unit (Operation loss acquisition means)
112a facility replacement cost calculation unit 112b local dispatch cost calculation unit 112t operation cost table 113 evaluation value calculation unit 113a accuracy correspondence evaluation value calculation unit 113b accuracy correspondence evaluation value calculation unit 113c recall correspondence evaluation value calculation unit 121 communication control unit 2 failure prediction device 3 Communication network system

Claims (12)

A failure prediction and evaluation apparatus for evaluating a prediction result by failure prediction means for predicting the occurrence of a failure in a business system, comprising:
For each of a plurality of failure types as prediction results by the failure prediction means, the number of times the failure type is actually predicted as a predicted failure type for each correct failure type that is a failure type that was correct when predicted Forecasting result acquiring means for acquiring such prediction frequency information;
Acquisition of operation loss, which is information related to operation loss including the operation cost that occurs when each correct failure type is dealt with or left according to the content of the predicted failure type for each of the predicted failure types Means,
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type A fault prediction and evaluation apparatus comprising: evaluation value calculation means for calculating an evaluation value as an index of evaluation of a prediction result by the failure prediction means based on a value weighted by operation loss information. The business system comprises at least one facility,
The prediction result acquisition means acquires the predicted number of times information about the failure type indicating that the failure is a failure requiring replacement or repair of the facility and the failure type indicating that the replacement or repair of the facility is unnecessary. ,
The operation loss acquisition means is operation loss information corresponding to the correctness prediction type indicating that the equipment need not be replaced or repaired, and the predicted failure type indicating that the facility needs to be exchanged or repaired. The failure prediction and evaluation apparatus according to claim 1, wherein a value obtained by adding a cost amount related to replacement or repair of the facility is adopted. The business system has at least one site that needs to be dealt with immediately when a failure occurs.
The prediction result acquisition means acquires the prediction frequency information on the failure type to the effect that the local response is required and the failure type to the effect that the local response is not necessary.
The operation loss acquiring unit is used as operation loss information corresponding to the correctness prediction type indicating that the local response is unnecessary and the predicted failure type indicating that the local response is necessary. The failure prediction and evaluation apparatus according to claim 1 or 2, wherein a value to which the dispatch cost for the field is added is adopted.   The business system according to any one of claims 1 to 3, wherein the business system is a communication network system having at least one facility and having at least one site that may need to be dealt with on the spot when a failure occurs. Failure prediction and evaluation device described.   The prediction result for deriving the prediction number information acquired by the prediction result acquisition means is alarm information concerning a network collected by the failure prediction means from the equipment or from monitoring equipment installed in advance in the network The obstacle according to claim 4, wherein the obstacle is the prediction result acquired by inputting alarm information which is an object to be predicted to a learning device constructed by learning the correct answer obstacle type as a teacher data set. Predictive evaluation device.   The plurality of fault types are a fault type indicating that the action does not require any action, a fault type indicating that the action requires a remote reset, and a fault type indicating that the action in the business system requires an on-site action. The method according to any one of claims 1 to 5, including at least one selected from the group consisting of: a failure type indicating that the equipment in the business system needs to be replaced or repaired. Failure prediction and evaluation device described.   The operation loss information is an increase or decrease in the operation cost compared to the operation cost required when there is no failure prediction, the degree of improvement or deterioration of service caused by failure or failure of the failure prediction, and when there is no failure prediction. The failure prediction evaluation according to any one of claims 1 to 6, characterized in that it includes at least one selected from the group consisting of increase and decrease in failure recovery time compared with the time required for failure recovery. apparatus. The operation loss information takes a negative value or zero when the corresponding correct error type and the predicted failure type match, and takes a positive value or zero when they do not match, and the evaluation value takes a negative value. The failure prediction and evaluation apparatus according to any one of claims 1 to 7, wherein the larger the absolute value is, the higher the effectiveness of the prediction result by the failure prediction means is.   The evaluation value calculation means is based on a value obtained by weighting the corresponding prediction frequency information according to the operation loss information for each correct failure type in the predicted failure type to the effect that the equipment needs replacement or repair. The failure prediction and evaluation apparatus according to claim 2, wherein an evaluation value for evaluating the effectiveness of the prediction result that replacement or repair of the facility is necessary is calculated. A failure prediction system comprising: failure prediction means for predicting the occurrence of a failure in a communication network system provided with at least one facility; and a failure prediction evaluation device for evaluating a prediction result by the failure prediction means, the failure prediction The evaluation device is
A plurality of failure types as a prediction result by the failure prediction means, which is a failure indicating that the equipment needs replacement or repair, and a failure indicating that the equipment replacement or repair is unnecessary For each of the plurality of failure types including the type, for each correct failure type that is the failure type that was correct when predicted, the prediction frequency information regarding the number of times the failure type is actually predicted as the predicted failure type is acquired Means for acquiring prediction results
It is an operation cost that occurs when dealing with or leaving each of the correct failure types according to the content of the predicted failure type for each of the correct failure types, that is, a correct answer indicating that replacement or repair of equipment is unnecessary. An operation loss including an operation cost that corresponds to the prediction type and the predicted failure type indicating that the equipment needs replacement or repair, and the cost for replacement or repair of the equipment is added Operation loss acquisition means for acquiring operation loss information which is information related to
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type What is claimed is: 1. A failure prediction system comprising: evaluation value calculation means for calculating an evaluation value as an index of evaluation of a prediction result by the failure prediction means based on a value weighted by operation loss information. A program that causes a computer mounted on a device to evaluate the prediction result by a failure prediction unit that predicts the occurrence of a failure in a business system.
For each of a plurality of failure types as prediction results by the failure prediction means, the number of times the failure type is actually predicted as a predicted failure type for each correct failure type that is a failure type that was correct when predicted Forecasting result acquiring means for acquiring such prediction frequency information;
Acquisition of operation loss, which is information related to operation loss including the operation cost that occurs when each correct failure type is dealt with or left according to the content of the predicted failure type for each of the predicted failure types Means,
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type A failure prediction and evaluation program that causes a computer to function as an evaluation value calculation unit that calculates an evaluation value that is an index of evaluation of a prediction result by the failure prediction unit based on a value weighted by operation loss information. A method of evaluating a prediction result by failure prediction means for predicting the occurrence of a failure in a business system,
For each of a plurality of failure types as prediction results by the failure prediction means, the number of times the failure type is actually predicted as a predicted failure type for each correct failure type that is a failure type that was correct when predicted Acquiring such prediction number information;
Acquiring operation loss information, which is information relating to operation losses including operation costs generated when taking measures or leaving according to the contents of the predicted failure type for each of the correct failure types,
The prediction frequency information corresponding to each pair of the correct failure type and the predicted failure type, for each correct error type in one predicted failure type, or for each predicted failure type in one correct error type And calculating an evaluation value as an index of evaluation of the prediction result by the failure prediction means based on the value weighted by the operation loss information.

Images