KR102533298B1 - Apparatus for evaluating fault diagnosis model based on system availability and method thereof - Google Patents

Abstract

One embodiment may provide a device for evaluating a failure diagnosis model based on system availability. Provided may be a device for evaluating a failure diagnosis model which predicts the occurrence of failures in a system. The device comprises: a control unit which calculates a first availability indicating the degree to which the system operates without interruption based on the uptime and downtime of the system, calculates a second availability by reflecting whether a failure actually occurs in the system and whether a failure is predicted by the failure diagnosis model in the first availability, determines the second availability as the final availability for the system, and determines the performance of the failure diagnosis model based on the second availability; and an input unit which obtains actual data on whether a failure actually occurs and prediction data on whether a failure is predicted and transmits the obtained actual data and prediction data to the control unit. According to the present invention, a more accurate performance evaluation can be achieved by reflecting the malfunction of the failure diagnosis model in evaluating the performance of the failure diagnosis model.

Description

Apparatus and method for evaluating a failure diagnosis model based on system availability

This embodiment relates to a technique for evaluating the performance of a failure diagnosis model for diagnosing a failure occurring in a system.

Many analytical models are being developed for problem prediction and classification. Statistical or machine learning methods or traditional or modern models are being studied and applied in various fields. The performance of the model needs to be evaluated, and through this, the best model among the models considered can be selected and used for problem solving.

In the classification problem, accuracy, precision, reproducibility and F1 score of the model can be used as evaluation indicators. These metrics can often be utilized in condition-baed maintenance (CBM) or predictive health monitoring (PHD) models as well. Predictive safety monitoring can consider continuous monitoring through sensors and automatic classification models through big data analysis. When the model is applied, it is necessary to check how the model affects system availability, maintenance cost, and system criticality. If the effect of a classification model on availability or maintenance cost can be measured, the best model can be selected more realistically.

Accordingly, the inventors of the present invention completed the present invention after long research to develop a technique for evaluating a failure diagnosis model based on system availability.

Against this background, one object of the present embodiment is to calculate the availability of the system by reflecting whether a failure actually occurs in the system and whether the failure is predicted by the failure diagnosis model, and based on the availability of the system, a failure diagnosis model To provide a technique for evaluating the performance of

Meanwhile, other non-specified objects of the present invention will be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

In order to achieve the above object, an embodiment is an apparatus for evaluating a failure diagnosis model for predicting the occurrence of a failure in a system, based on uptime and downtime of the system. A first availability indicating the degree of operation without interruption is calculated, and whether a failure actually occurs in the system and whether a failure is predicted by the failure diagnosis model are reflected in the first availability to calculate a second availability and determining the second availability as the final availability for the system, and determining the performance of the failure diagnosis model based on the second availability, and actual data on whether or not a failure has actually occurred and failure information. Provided is an apparatus for evaluating a failure diagnosis model based on system availability, including an input unit for obtaining prediction data on whether or not to predict and transmitting the obtained prediction data to the control unit.

In the device, the control unit, in order to calculate the second availability, TP (true positive) data indicating the number of cases in which a failure actually occurs and the failure diagnosis model predicts that the failure is actual, FN (false negative) data indicating the number of cases that occur and the failure diagnosis model predicts that it is normal, FP (false positive) data representing the number of cases where failure does not actually occur and the failure diagnosis model predicts that the failure is normal ) data, TN (true negative) data indicating the number of cases in which the failure diagnosis model predicts that a failure does not actually occur, F (fault) data indicating the number of cases where a failure actually occurs, and failure actually occurs It is possible to generate N (normal) data representing the number of cases that are not.

In the device, the control unit reflects a failure interval time indicating an interval between failures in the operation time, and the test time for detecting a failure in the stop time and the system, in order to calculate the second availability. Recovery time for recovery can be reflected.

In the device, the controller determines the second availability from the TP data, the FN data, the FP data, the TN data, the F data, the N data, the failure interval time, the inspection time, and the recovery time. can be calculated.

In the device, the second availability may be calculated using the following equation.

TP: TP data, FN: FN data, FP: FP data, TN: TN data, F: F data, N: N data, MTBF (mean time between failure): failure interval time, Tinspect: test time, MTTR (mean time to restoration)true: 1st recovery time (when a failure actually occurs, if the failure diagnosis model predicts a failure), MTTR (mean time to restoration) miss: 2nd recovery time (if a failure actually occurs, the failure diagnosis if the model predicted normal)

Another embodiment is a method for evaluating a failure diagnosis model predicting the occurrence of a failure in a system, which indicates the degree to which the system operates without interruption based on uptime and downtime of the system. calculating a first availability; calculating a second availability by reflecting whether a failure actually occurs in the system and whether a failure is predicted by the failure diagnosis model in the first availability; determining the second availability as the final availability for the system; and determining the performance of the fault diagnosis model based on the second availability.

In the method, the step of calculating the second availability may include true positive (TP) data indicating the number of cases in which a failure actually occurs and the failure diagnosis model predicts that a failure occurs, and the failure actually occurs and the FN (false negative) data representing the number of cases predicted by the failure diagnosis model to be normal, false positive (FP) data representing the number of cases predicted by the failure diagnosis model to be failure without actually occurring; TN (true negative) data representing the number of cases where a failure does not actually occur and the failure diagnosis model predicts that it is normal, F (fault) data representing the number of cases where a failure actually occurs, and It may include generating N (normal) data representing the number.

In the method, the step of calculating the second availability reflects a failure interval time indicating an interval between failures in the operating time, and a test time for detecting a failure and a recovery for restoring the system in the downtime. The second availability may be calculated by reflecting time.

In the method, the calculating of the second availability may include the TP data, the FN data, the FP data, the TN data, the F data, the N data, the failure interval time, the inspection time, and the recovery The second availability may be calculated from time.

In the above method, the second availability may be calculated using the following equation.

TP: TP data, FN: FN data, FP: FP data, TN: TN data, F: F data, N: N data, MTBF (mean time between failure): failure interval time, Tinspect: test time, MTTR (mean time to restoration)true: 1st recovery time (when a failure actually occurs, if the failure diagnosis model predicts a failure), MTTR (mean time to restoration) miss: 2nd recovery time (if a failure actually occurs, the failure diagnosis if the model predicted normal)

As described above, according to the present embodiment, more accurate performance evaluation can be performed by reflecting malfunctions of the failure diagnosis model in evaluating the performance of the failure diagnosis model.

On the other hand, even if the effects are not explicitly mentioned here, it is added that the effects described in the following specification expected by the technical features of the present invention and their provisional effects are treated as described in the specification of the present invention.

1 is a configuration diagram of an apparatus for evaluating a failure diagnosis model based on system availability according to an exemplary embodiment.
2 is an exemplary diagram of an error matrix for calculating availability according to an embodiment.
3 is an exemplary diagram of an error matrix interpreted in association with maintenance measures according to an embodiment.
4 is a first exemplary view of evaluating a failure diagnosis model based on availability according to an embodiment.
5 is a second exemplary diagram for evaluating a failure diagnosis model based on availability according to an embodiment.
It is revealed that the accompanying drawings are illustrated as references for understanding the technical idea of the present invention, and thereby the scope of the present invention is not limited thereto.

In the description of the present invention, if it is determined that a related known function may unnecessarily obscure the subject matter of the present invention as an obvious matter to those skilled in the art, the detailed description thereof will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals and overlapping descriptions thereof are omitted. do it with

1 is a configuration diagram of an apparatus for evaluating a failure diagnosis model based on system availability according to an exemplary embodiment.

Referring to FIG. 1, an apparatus (100, hereinafter referred to as 'apparatus') for evaluating a fault diagnosis model based on system availability according to an embodiment is configured to evaluate a fault diagnosis model predicting occurrence of a failure in a system. Calculating a first availability indicating the degree to which the system operates without interruption based on the uptime and downtime of the system, whether a failure actually occurs in the system and failure according to the failure diagnosis model calculating the second availability by reflecting the prediction of the first availability to the first availability, determining the second availability as the final availability for the system, and based on the second availability, the fault diagnosis model Steps to judge performance can be performed. To this end, the device 100 may include an input unit 110, a control unit 120, an output unit 130, and a storage unit 140.

The apparatus 100 may evaluate the performance of the failure diagnosis model. The fault diagnosis model can predict whether a fault will occur in one system. If the failure diagnosis model predicts that the system will fail at a certain point in time, it can predict a 'fault' and output it to the user. If the failure diagnosis model predicts that the system will operate normally without failure at a certain point in time, it can predict that the system is 'normal' and output it to the user. The apparatus 100 evaluates the performance of the fault diagnosis model, where the performance may mean how well the fault diagnosis model predicts the system. If the failure diagnosis model predicts a failure when a system failure actually occurs or the failure diagnosis model predicts a normal failure when a system failure does not actually occur, the performance of the failure diagnosis model can be evaluated as excellent.

Here, the input unit 110 may receive data or information necessary for evaluating the performance of the failure diagnosis model and transmit the received data or information to the control unit 120 . The output unit 130 may output the data or information calculated by the control unit 120 so that the user can recognize them with five senses. The storage unit 140 may store data or information input through the input unit 110 or calculated by the controller 120, an algorithm for calculating the first availability and the second availability, and the like.

Meanwhile, the apparatus 100 may refer to the availability of the system in order to evaluate the performance of the failure diagnosis model. Availability of a system can be understood as the ability of the system to properly perform required functions or the ability to maintain performance without interruption of service. The device 100 may calculate the availability of the system through Equation 1 below.

Here, uptime may indicate operation time, downtime may indicate stop time, MTBF (mean time between failure) may indicate mean time between failures, and MTTR (mean time to restoration) may indicate mean time between failures. Uptime can be regarded as MTBF and downtime as MTBF, respectively. MTBF is the average operating time of a system and may correspond to the average interval between one failure and the next. MTTR may correspond to the average time (recovery time) required to restore a system after a failure, regardless of whether the occurrence of an actual failure is correctly predicted.

In detail, the control unit 120 of the device 100 may calculate the initial availability, that is, the first availability, based on the operation time and stop time of the system. The controller 120 may calculate the first availability by using Equation 1, and may use MTBF for the system instead of operating time and MTTR for the system instead of stop time.

The controller 120 of the device 100 may calculate the second availability by reflecting whether a failure actually occurs in the system and whether the failure diagnosis model predicts the failure in the first availability. Then, the controller 120 may determine the second availability as the final availability for the system. The control unit 120 may determine the performance of the failure diagnosis model based on the availability of the system - the second availability.

The control unit 120 of the device 100 according to an embodiment may use system availability to determine the performance of the failure diagnosis model. Specifically, the controller 120 may determine the performance of the fault diagnosis model based on the second availability by newly calculating the second availability by reflecting the prediction result of the fault diagnosis model in the first availability. The reason is that the predicted result of the failure diagnosis model - whether or not a failure actually occurred in the system and whether the failure was predicted by the failure diagnosis model - affects the second availability. Whether a failure diagnosis model predicts failures correctly or incorrectly can affect MTBF and/or MTTR. Therefore, the control unit 120 determines that when a specific failure diagnosis model predicts a fault when a failure actually occurs and when the specific failure diagnosis model predicts that the failure is normal when a failure actually occurs, the failure actually occurs. Values for four cases, such as the case where the specific failure diagnosis model predicted a fault when it did not occur and the case where the specific failure diagnosis model predicted a fault when the failure actually occurred, or The probability may be reflected in Equation 1 for obtaining the first availability. As the second availability is higher, the performance of the system is recognized as excellent, and the performance of the failure diagnosis model for diagnosing the system and predicting failure may be recognized as excellent. Conversely, as the second availability is lower, the performance of the system is recognized as inferior, and the performance of the failure diagnosis model for this system may also be recognized as inferior.

2 is an exemplary diagram of an error matrix for calculating availability according to an embodiment.

Referring to FIG. 2 , an error matrix used by the device according to an embodiment to calculate the final availability-second availability-of the system may be illustrated.

The controller of the device may receive actual data on whether a failure actually occurs and prediction data on whether a failure is predicted, and generate an error matrix from the actual data and the predicted data. The actual data may include information on an event in which a failure actually occurs in the system, and the predicted data may include information on an event in which the failure is predicted. The control unit can classify the row (actual condition) of the error matrix with actual data into a case where a failure actually occurs (fault) and a case where a failure does not actually occur (normal). In addition, the control unit can divide the predicted condition of the error matrix into a case of predicting a failure (failure) and a case of predicting a normal condition (normal) with the prediction data. Then, each region of the error matrix is classified as a case where a failure actually occurred and the failure was predicted (TP, true positive), a failure actually occurred but was predicted as normal (FN, false negative), and a failure did not actually occur but was classified as a failure. This may correspond to a predicted case (FP, false positive) and a case where the failure did not actually occur and was normally predicted (TP, true negative). The control unit can allocate actual data and predicted data to the above four areas. The control unit may identify and allocate the number of cases predicted by the failure diagnosis model to be failure or normal when a failure actually occurs or does not occur in each region of the error matrix for a certain period of time. In this way, the number of cases corresponding to the TP area is TP data, the number of cases corresponding to the FN area is FN data, the number of cases corresponding to the FP area is FP data, and the number of cases corresponding to the TN area is TN data, each of which can be named.

Here, the case where a failure actually occurred but was predicted as normal (FN) is a type 2 error and corresponds to a miss, and the case where a failure did not actually occur but was predicted as a failure (FP) is a type 1 error. (type 1 error) and may correspond to a false alarm. A type 1 error and a type 2 error may respectively describe a malfunction that predicts the occurrence of a system failure differently from reality.

In addition, the control unit may generate F data from TP data and FN data. The F data may correspond to the sum of the number of cases in which the failure diagnosis model predicts a failure when a failure actually occurs and the number of cases in which the failure diagnosis model predicts that a failure is normal when a failure actually occurs. Also, the controller may generate N data from FP data and TN data. The N data may correspond to the sum of the number of cases in which the failure diagnosis model predicts a failure when a failure does not actually occur and the number of cases in which the failure diagnosis model predicts a normal failure when a failure does not actually occur.

Meanwhile, the controller of the device may calculate the second availability by reflecting the TP data, FN data, FP data, TN data, N data, and P data of the error matrix to the first availability. While calculating the first availability through Equation 1, the controller may calculate the second availability through Equation 2 below. Equation 2 may have a form in which whether a failure actually occurs in the system and whether a failure is predicted by a failure diagnosis model are reflected in the first availability.

where TP is TP data, FN is FN data, FP is FP data, TN is TN data, F is F data, N is N data, MTBF (mean time between failures) is the time between failures, Tinspect may mean an examination time, mean time to restoration (MTTR) true may mean a first recovery time, and mean time to restoration (MTTR) miss may mean a second restoration time. Since it is necessary to detect a failure of the system both when a failure actually occurs or when a failure is predicted, the inspection time may mean an average time required for detecting such a failure. The first recovery time may mean an average recovery time when a failure diagnosis model predicts a failure when a failure actually occurs. The second recovery time may refer to an average recovery time when a failure diagnosis model predicts that a failure actually occurs.

The controller may calculate the second availability by applying TP data, FN data, FP data, TN data, N data, and P data for a specific failure diagnosis model to Equation 2. Here, the failure interval time, inspection time, first recovery time, and second recovery time may be fixed for each system that is a target predicted by the specific failure diagnosis model. Accordingly, the control unit may acquire a corresponding value from the user or retrieve a value previously stored in the storage space and apply it to Equation 2. Then, the controller may finally determine the second availability as the availability of the system. Further, the control unit may determine the performance of the failure diagnosis model based on the second availability, that is, system availability. For example, the control unit may determine that the performance of the failure diagnosis model is excellent as the second availability is higher.

3 is an exemplary diagram of an error matrix interpreted in association with maintenance measures according to an embodiment.

Referring to FIG. 3 , the meaning of each region of the error matrix according to an embodiment may be interpreted as follows in relation to maintenance measures.

In a TP area corresponding to a case in which a failure actually occurs and the failure diagnosis model predicts a failure, the system may require inspection and prior maintenance. The system may have to be down. In the FN area corresponding to the case where a failure actually occurs and the failure diagnosis model predicts that it is normal, a hidden failure may exist and post-maintenance may be required. Ex post maintenance can be understood as maintenance applied when an unexpected failure is later discovered. Unexpected failures can cause serious accidents. An FP area corresponding to a case in which a failure does not actually occur and the failure diagnosis model predicts a failure may require a system inspection. The system may have to go down. In an FP area corresponding to a case where a failure does not actually occur and the failure diagnosis model predicts that it is normal, the system may not require any maintenance.

4 is a first exemplary view of evaluating a failure diagnosis model based on availability according to an embodiment.

Referring to FIG. 4 , a first example of evaluating various failure diagnosis models by an apparatus according to an embodiment may be shown. The first example may assume that a failure causes an accident. Then, MTBF, the interval time between failures, is 1000hr, inspection time to detect a failure, Tinspect, is 2hr, MTTRtrue, the recovery time when failure prediction succeeds, is 4hr, and MTTRmiss, the recovery time when failure prediction fails, is 12hr. can

In addition, whether or not a failure actually occurs in one system and whether or not the first to third failure diagnosis models (models 1 to 3) predict a failure for one system may be shown as shown in this figure. Since a perfect model does not incorrectly predict whether a failure actually occurs, both FN data and FP data may have a value of 0 (zero). On the other hand, since the first to third failure diagnosis models (models 1 to 3) are incomplete, it is possible to predict the actual occurrence of a failure incorrectly. In the first to third failure diagnosis models (models 1 to 3), FN data and FP data have constant values.

The controller of the device determines the availability of the first to third failure diagnosis models (models 1 to 3) based on the availability derived when the same system is applied to diagnosis and prediction. performance can be judged. The control unit may create each error matrix using actual data and predicted data for the first to third failure diagnosis models (models 1 to 3). Further, the control unit may calculate the availability, respectively, by applying TP data, FN data, FP data, TN data, N data, and P data constituting the error matrix to Equation 2. The availability of the system calculated from the complete model and the first to third fault diagnosis models (models 1 to 3) may be respectively shown next to the error matrix.

Additionally, accuracy, precision, recall, and F1score can be used as performance indicators of the fault diagnosis model. Accuracy can indicate how accurate the diagnostic model is. Precision can indicate how reliable a diagnostic model is. Reproducibility can indicate how well the failure diagnosis model extracted the region of interest. F1score can be expressed as the product of precision and recall for the sum of precision and recall.

Referring to this figure, in the case of the first example, the availability is highest when the second failure diagnosis model (model 2) diagnoses and predicts the system. The controller of the device may determine that the performance of the second failure diagnosis model (model 2) is the best among the first to third failure diagnosis models (models 1 to 3). The reason is that the FN data is 0 (zero), and the second failure diagnosis model (model 2) generates the least number of misses predicted to be normal even when failures actually occur. If a failure causes an accident, a failure diagnosis model that causes the least number of misses may be advantageous.

5 is a second exemplary diagram for evaluating a failure diagnosis model based on availability according to an embodiment.

Referring to FIG. 5 and FIG. 4 , a second example of evaluating various failure diagnosis models by the apparatus according to an embodiment may be shown. Unlike the first example, the second example may assume that a failure does not cause an accident. Then, the interval time between failures, MTBF, is 1000hr, the inspection time for detecting a failure, Tinspect, is 2hr, the recovery time when failure prediction succeeds, MTTRtrue is 4hr, and the recovery time when failure prediction fails, MTTRmiss is 4hr. can Since failure does not cause an accident, MTTRmiss may be shorter as 4hr, unlike the first example.

Similarly, in the case of the second example, availability is highest when the third failure diagnosis model (model 3) diagnoses and predicts the system. The control unit of the apparatus may determine that the third failure diagnosis model (model 3) has the best performance among the first to third failure diagnosis models (models 1 to 3). The reason is that the FP data is 0 (zero), and the third failure diagnosis model (model 3) generates the least number of false alarms that predicted a failure even though the failure did not actually occur. If the failure did not cause an accident, a fault diagnosis model that produces the fewest false alarms may be advantageous.

The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the protection scope of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention belongs.

Claims (10)

An apparatus for evaluating a failure diagnosis model predicting the occurrence of a failure in a system,
Based on the uptime and downtime of the system, a first availability indicating the degree to which the system operates without interruption is calculated, and whether a failure actually occurs in the system and the failure diagnosis model The failure diagnosis model calculates a second availability by reflecting whether a failure is predicted in the first availability, determines the second availability as a final availability for the system, and based on the second availability A control unit for determining the performance of and
An input unit for acquiring actual data on whether a failure actually occurs and prediction data on whether a failure is predicted and transmitting them to the control unit;
The control unit includes TP (true positive) data indicating the number of cases in which a failure actually occurs and the failure diagnosis model predicts that it is a fault, and a failure actually occurs and the failure diagnosis model predicts that it is normal. FN (false negative) data indicating the number of cases, FP (false positive) data indicating the number of cases where a failure does not actually occur and the failure diagnosis model predicts a failure, and failure does not actually occur and the failure diagnosis model TN (true negative) data representing the number of cases predicted to be normal, F (fault) data representing the number of cases where a failure actually occurred, and N (normal) data representing the number of cases where a failure did not actually occur are generated. ,
The control unit reflects a failure interval time indicating an interval between failures in the operation time, and reflects an inspection time for detecting a failure and a recovery time for restoring the system in the stop time,
The control unit calculates the second availability from the TP data, the FN data, the FP data, the TN data, the F data, the N data, the failure interval time, the inspection time, and the recovery time. A device that evaluates a fault diagnosis model based on availability. delete delete delete According to claim 1,
The control unit,
An apparatus for evaluating a failure diagnosis model based on system availability for calculating the second availability using the following equation.

TP: TP data, FN: FN data, FP: FP data, TN: TN data, F: F data, N: N data, MTBF (mean time between failure): failure interval time, Tinspect: test time, MTTR (mean time to restoration)true: 1st recovery time (when a failure actually occurs, if the failure diagnosis model predicts a failure), MTTR (mean time to restoration) miss: 2nd recovery time (if a failure actually occurs, the failure diagnosis if the model predicted normal), A method for evaluating a failure diagnosis model for predicting the occurrence of a failure in a system,
Calculating a first availability indicating the degree to which the system operates without interruption based on uptime and downtime of the system;
calculating a second availability by reflecting whether a failure actually occurs in the system and whether a failure is predicted by the failure diagnosis model in the first availability;
determining the second availability as the final availability for the system; and
Determining performance of the failure diagnosis model based on the second availability,
Calculating the second availability,
TP (true positive) data indicating the number of cases in which a failure actually occurs and the failure diagnosis model predicts that it is a fault, and the number of cases in which a failure actually occurs and the failure diagnosis model predicts that it is normal FN (false negative) data indicating, FP (false positive) data indicating the number of cases in which a failure does not actually occur and the failure diagnosis model predicts a failure, and failure does not actually occur and the failure diagnosis model predicts that it is normal Generating TN (true negative) data representing the number of cases, F (fault) data representing the number of cases in which a failure actually occurred, and N (normal) data representing the number of cases in which a failure did not actually occur;
reflecting a failure interval time indicating an interval between failures in the operation time and reflecting an inspection time for detecting a failure and a recovery time for restoring the system in the stop time; and
Calculating the second availability from the TP data, the FN data, the FP data, the TN data, the F data, the N data, the failure interval time, the inspection time, and the recovery time
A method for evaluating fault diagnosis models based on system availability. delete delete delete According to claim 6,
A method of evaluating a fault diagnosis model based on system availability, wherein the second availability is calculated using the following equation.

TP: TP data, FN: FN data, FP: FP data, TN: TN data, F: F data, N: N data, MTBF (mean time between failure): failure interval time, Tinspect: test time, MTTR (mean time to restoration)true: 1st recovery time (when a failure actually occurs, if the failure diagnosis model predicts a failure), MTTR (mean time to restoration) miss: 2nd recovery time (if a failure actually occurs, the failure diagnosis if the model predicted normal),

Images