JP7034014B2 - Fault response training equipment, fault response training methods and fault response training programs - Google Patents

Description

The present invention relates to a failure response training device, a failure response training method, and a failure response training program.

Failure response training in system operation is usually conducted before the start of operation. In many systems, regular training for a wide range of disabilities is not carried out in the production environment after the start of operation. Therefore, if a new operator is assigned due to a change in personnel, etc., prior training will be conducted in a test environment or the like. After the actual operation starts, it is not easy to keep the test environment and the production environment the same. Therefore, an operator who has little experience in dealing with failures may not always be able to reliably deal with actual failures in the production environment. Therefore, technology for improving the certainty of troubleshooting in the production environment is important.

Software has been developed that automatically tests the failure recovery capability by intentionally causing a failure in an operating system. A specific example is Chaos Monkey from Netflix. This software tests the ability of the target service to recover from a failure by repeating the operation of randomly selecting virtual machines on the cloud service and forcibly stopping them.
Patent Document 1 discloses a method of performing a failure test after confirming that the service level agreement with the system user, that is, SLA (Service Level Agreement) is not exceeded. SLA is confirmed by calculating the impact on quality, taking into account system conditions such as network performance or usage, before deliberately causing a failure. From the same point of view, software that controls the occurrence of failures according to the system status, specifically, ChAP of Netflix Co., Ltd. has been announced.

Japanese Unexamined Patent Publication No. 2017-135563

The conventional technology does not consider the operator and the operator's system that actually responds to the failure. For this reason, if the failure-generating software causes a failure with a high degree of difficulty during a time period when only an inexperienced operator is available, the SLA cannot be satisfied because it takes a lot of time to recover. In addition, in order to avoid a situation where the SLA cannot be satisfied, if the target of occurrence is limited to the failure that the system can automatically recover by measures such as redundancy, it will not lead to the skill improvement of the operator.

An object of the present invention is to reduce the risk of not being able to meet the SLA in a production environment, and to realize efficient failure response training according to the operator's failure response skill level.

In the failure response training device according to the present invention, in the failure response training device that performs failure response training for the operator by causing a failure in the operation system operated by the operator.
As the operator's failure response training, based on the operator information including the difficulty level of the failure that the operator can handle and the failure information including the failure list and the difficulty level of each failure included in the failure list. A selection unit that selects the failure to be generated as a training failure from the failure list, and
An impact prediction unit that determines whether or not the impact on the operational system when a training failure occurs in the operational system is within an allowable range based on the system information indicating the status of the operational system.
The operation system is provided with a training execution unit that causes the training failure when the influence on the operation system is within an allowable range.

According to the failure response training device according to the present invention, it is possible to execute appropriate failure response training that leads to improvement of the operator's skill while suppressing the influence on the operation system.

The figure which shows the configuration example of the trouble response training system which concerns on Embodiment 1. The figure which shows the detailed configuration example of the training apparatus and the information storage apparatus which concerns on Embodiment 1. FIG. The flow diagram which shows the operation of the trouble response training apparatus which concerns on Embodiment 1. The figure which shows a part of the operator information which concerns on Embodiment 1. The figure which shows a part of the training history which concerns on Embodiment 1. The figure which shows a part of the trouble information which concerns on Embodiment 1. The figure which shows the example of the candidate list which concerns on Embodiment 1. The figure which shows the part of the trouble response skill in the operator information which concerns on Embodiment 1. The figure which shows the part of the skill necessary for the failure response in the failure information which concerns on Embodiment 1. FIG. The figure which shows the example of the training notice displayed on the operator terminal which concerns on Embodiment 1. The figure which shows an example of the trouble correspondence information input screen which concerns on Embodiment 1. The figure which shows the example of the correct answer table of the trouble correspondence information input screen which concerns on Embodiment 1. The figure which shows another example of the trouble correspondence information input screen which concerns on Embodiment 1. The figure which shows the example of the important point list which concerns on Embodiment 1. The figure which shows the correspondence example of the trouble correspondence reference time and the trouble correspondence time which concerns on Embodiment 1. The figure which shows the standard table of the training follow-up necessity determination which concerns on Embodiment 1. The figure which shows the example of the registration correspondence method which concerns on Embodiment 1. The figure which shows the example of the log of the command execution which concerns on Embodiment 1. The figure which shows an example of this trouble correspondence method saved in the training history which concerns on Embodiment 1. The figure which shows the other example of this trouble correspondence method saved in the training history which concerns on Embodiment 1. An example in which a failure handling method is newly registered in the failure information according to the first embodiment. The figure which shows the example of the operator judgment standard table which concerns on Embodiment 1. An example of the hardware configuration of the failure response training device according to the first embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each figure, the same or corresponding parts are designated by the same reference numerals. In the description of the embodiment, the description will be omitted or simplified as appropriate for the same or corresponding parts.

Embodiment 1.
*** Explanation of configuration ***
The configuration of the failure response training system 500 according to the present embodiment will be described with reference to FIG. 1.
The failure response training system 500 includes an operation system 10, a failure response training device 20, an operation monitoring device 50, and an operator terminal 60.
The operation system 10 is a system to be operated by the operator 61.
The failure response training device 20 performs failure response training for the operator by generating a training failure 71 in the operation system 10 operated by the operator 61. The fault handling training device 20 includes a training device 30 and an information storage device 40. The training device 30 determines whether or not a training obstacle, that is, a training obstacle 71, can occur, and generates the training obstacle 71 based on the determination result. The information storage device 40 stores information such as system information, training history, failure information, and operator information.
The operation monitoring device 50 executes an operation process such as monitoring on the operation system 10. The operation monitoring device 50 constantly monitors the operation system 10, and when a failure or a problem is detected, the operation monitoring device 50 displays a message on the operator terminal 60.
The operator terminal 60 is a terminal used by the operator 61. The operator 61 uses the operator terminal 60 to acquire information on the operation of the operation system 10 or perform an operation for the operation of the operation system 10. Further, the operator terminal 60 displays the information obtained from the operation monitoring device 50 or the training device 30 to the operator 61.

A detailed configuration of the training device 30 and the information storage device 40 according to the present embodiment will be described with reference to FIG. 2.
The training device 30 includes a training content selection unit 31, a response availability determination unit 32, an impact prediction unit 33, a training execution unit 34, a training content recording unit 35, a cause acquisition unit 36, a cause evaluation acquisition unit 37, and a training result evaluation unit 38. It includes a failure information update unit 39 and an operator information update unit 391. The training content selection unit 31 and the correspondence availability determination unit 32 are combined to form the selection unit 301. The failure information update unit 39 and the operator information update unit 391 are collectively referred to as an update unit 309.
The information storage device 40 includes system information 41, failure information 42, operator information 43, and training history 44.

Based on the operator information 43 and the failure information 42, the selection unit 301 selects a failure to be generated as the operator's failure response training as a training failure 71 from the failure list.
The operator information 43 includes the difficulty level of the failure that the operator can handle. Further, the operator information 43 includes the types of failure handling skills possessed by the operator.
The failure information 42 includes a failure list and the difficulty level of each failure included in the failure list. Further, the failure information 42 includes the types of skills required for dealing with each failure included in the failure list. Further, the failure information 42 includes a method for dealing with each failure included in the failure list as a registration handling method.

The training content selection unit 31 grasps the currently present operator from the operator information 43. Then, the training content selection unit 31 refers to the past failure response history of the operator from the training history 44, and selects an appropriate training failure for the operator who is present.
The response availability determination unit 32 acquires the failure response skill of the operator who is present from the operator information 43. The response availability determination unit 32 acquires the skills necessary for disability response from the disability information 42 for the training disability 71 selected by the training content selection unit 31. The response availability determination unit 32 collates the disability response skill of the operator who is present with the skill required for disability response of the training disability 71, and determines whether the operator can respond to the training disability 71.

The impact prediction unit 33 determines whether or not the impact on the operation system 10 when the training failure 71 is generated in the operation system 10 is within the allowable range based on the system information 41 indicating the status of the operation system 10. ..
In the operation system 10, the level of service provided to the user of the operation system 10 is set as a service level agreement, that is, SLA. The impact prediction unit 33 determines that the impact on the operation system 10 is within the permissible range when the service provided to the user satisfies the SLA when the training failure 71 is generated in the operation system 10. judge. If it is determined that the SLA is affected, the impact prediction unit 33 does not perform the training and returns the process to the selection of the training content by the training content selection unit 31. When it is determined that the SLA is not affected, the impact prediction unit 33 determines that the training can be performed, and instructs the training execution unit 34 to execute the training.

The training execution unit 34 causes a training failure 71 in the operation system 10 when the influence on the operation system 10 is within the allowable range. As an example at that time, the operator to be trained may perform the training while hiding the training so that he / she can deal with the trouble with the same tension as the actual one. Further, the training execution unit 34 may transmit a training notification notifying the operator that the training is being performed when the training failure 71 is generated in the operation system 10. In addition, a real failure may be generated, or only a pseudo failure warning may be generated. If the failure is not dealt with after the specified reference time, the training may be forcibly terminated and the operator may be notified.
The training content recording unit 35 sets the method for dealing with the training failure 71 in the failure response information as the failure response method, and stores the failure response information in the training history 44. Specifically, the training content recording unit 35 records the failure response information including information such as the failure response method, the failure response time, and the operator's opinion on the training failure 71 in the training history 44.

The cause acquisition unit 36 acquires failure response information including the cause of the training failure 71 and important points in failure response from the operator. The cause acquisition unit 36 acquires failure response information such as an operator's determination result and failure response details regarding the cause of the training failure 71 and important points in the failure response, and stores it in the training history 44. The content acquired by the cause acquisition unit 36 is used by the training result evaluation unit 38 to evaluate the degree of understanding of the operator regarding troubleshooting. As a specific example, the cause acquisition unit 36 may acquire the failure response information by acquiring the failure response information input by the operator, or by acquiring from the operation record recorded in an external incident management system or the like. It may be configured to be used.
The cause evaluation acquisition unit 37 acquires an evaluation of the validity of the failure response by the operator from the operator based on the failure response information acquired by the cause acquisition unit 36. That is, the cause evaluation acquisition unit 37 acquires the content evaluated manually when manually evaluating whether the failure response information acquired by the cause acquisition unit 36 is correct. The cause evaluation acquisition unit 37 outputs the contents evaluated manually to the training result evaluation unit 38. When the training result evaluation unit 38 automatically evaluates the failure response information without human intervention, the cause evaluation acquisition unit 37 may be omitted.

When the training result evaluation unit 38 completes the failure response to the training failure 71 by the operator, the training result evaluation unit 38 outputs the evaluation result 74 for evaluating the failure response. The training result evaluation unit 38 acquires information on the disability response to the training disability 71 from the training history 44 or the cause evaluation acquisition unit 37, and comprehensively evaluates it.

The update unit 309 updates the difficulty level of the failure that the operator can handle for the operator information 43 based on the failure response information indicating the content of the failure response and the evaluation result 74. Further, the update unit 309 updates the registration response method included in the failure information 42 based on the failure response information and the evaluation result 74.
The failure information updating unit 39 updates the failure information 42 based on the evaluation result 74 of the training result evaluation unit 38 and the failure response information stored in the training history 44. Specifically, when it is evaluated by the training result evaluation unit 38 that the failure recovery has been performed without any problem, the registration response method of the failure information 42 and the failure response method in the current training of the training history 44 are compared. .. If these response methods do not match, the failure information updating unit 39 considers that a new failure response method has been found, and newly registers the failure response method in the failure information 42. Alternatively, the failure information updating unit 39 may update the difficulty level of the failure or other information of the failure information 42 based on the time taken for the current failure response in the training history 44.
The operator information update unit 391 has skill information of the operator information 43 based on the evaluation result 74 of the training result evaluation unit 38, the failure response information stored in the training history 44, and the failure information 42 related to the failure generated this time. To update.

The system information 41 is a data store that holds information such as system status information, system configuration information, and system performance information of the operation system 10. The system status information is acquired from the operation monitoring device 50. The system configuration information and the system performance information may be manually input or may be acquired from the operation monitoring device 50.
The failure information 42 is a data store that holds information about failures that occur during training. Specifically, the failure information 42 includes information such as a failure ID (Identifier), failure content, failure cause, registration response method, skill required for failure response, difficulty level of failure, failure response reference time, and training evaluation content. Is retained. The difficulty level of a disability is the difficulty level of dealing with a disability when the disability occurs. Here, the failure response reference time refers to a time within a range that does not interfere with the SLA even if the failure response is performed. The training evaluation content shows the evaluation method of the training result. Specifically, the training evaluation content includes important points in the failure response step for each failure and information on correct response at the important points.

The operator information 43 is a data store that holds information about the operator. Specifically, the operator information 43 includes information such as an operator ID, a name, a trouble handling skill possessed by the operator, and an attendance status. As a specific example, the presence status of the operator may be acquired from the operating status of the operator terminal 60, or may be acquired from a system such as an external schedule management system or an attendance management system. As a specific example, the skill possessed by the operator may be registered by the operator's self-report at the initial stage. The skill information may be updated every time the training is conducted for the operator.
The training history 44 is a data store that holds information recorded in the training such as the operator's failure response method and failure response time at the time of training.

*** Explanation of operation ***
The operation of the failure response training device 20 according to the present embodiment will be described with reference to FIG.

<Step S101: Operator selection>
In step S101, the training content selection unit 31 selects an operator. The training content selection unit 31 acquires the presence status of the operator, the failure handling skill of the operator, and the failure handling history of the operator from the operator information 43 and the training history 44. The training content selection unit 31 selects one or a plurality of operators currently present as training targets. Specifically, the training content selection unit 31 selects an operator based on a selection criterion such as the difficulty level of the obstacles that can be dealt with is low or the training has not been carried out recently.

FIG. 4 is a diagram showing a part of the operator information 43 according to the present embodiment.
Further, FIG. 5 is a diagram showing a part of the training history 44 according to the present embodiment.
In the example of FIG. 4, the operator 10010 is selected based on the lowest possible obstacle difficulty level among the operators 10010 and 10012 that are present. Further, in the example of FIG. 5, when the current date is March 10, 2018, the operator 10010 is selected based on the fact that the training has not been carried out for the past one month. Further, the selection criteria may be combined with the conditions such that both of these plurality of selection criteria are satisfied.

<Step S102: Selection of obstacles caused by training>
In step S102, the training content selection unit 31 selects a candidate for a failure to be generated in the operator selected in step S101. The training content selection unit 31 acquires information from the operator information 43 and the failure information 42, and compares them to create a list of failure candidates to be trained by the operator. The obstacles to be trained are those that the operator has not experienced, those that have been experienced but have failed to respond once, or those that have passed a predetermined time since the last training of the obstacle. It is mentioned as a standard. However, other selection criteria may be used. If there are no obstacles to be listed in the obstacle candidate list, the training may be terminated without implementation, or the selection criteria may be changed and the operator's selection may be restarted.

FIG. 6 is a diagram showing a part of the failure information 42 according to the present embodiment.
FIG. 7 is an example of the candidate list 51 according to the present embodiment.
From the past failure response history in the training history 44 of FIG. 5, it can be seen that the operator 10010 is inexperienced except for the failure ID “1”. Therefore, the training content selection unit 31 refers to the failure list in the failure information 42 of FIG. 6, and uses the failure that the operator 10010 has not experienced as a selection criterion to train the failure ID “2, 3, 4”. Add to candidate list 51.

<Step S103: Judgment of availability of operator>
In step S103, the response availability determination unit 32 selects from the failure list a failure in which the operator has all the necessary skill types as a training failure 71. Specifically, the response availability determination unit 32 selects a failure that can be handled by the operator as a training failure 71 from the failure candidate list 51 created in step S102. The response availability determination unit 32 selects the obstacle that the operator can handle as the training obstacle 71 by comparing the obstacle response skill possessed by the operator selected in step S101 with the skill required for the fault response. Whether or not to respond is determined by checking whether the skill possessed by the operator meets a predetermined standard for the skill required for troubleshooting. If there are no obstacles that can be dealt with, the training should be terminated without implementation, or the selection criteria should be changed and the operator should start over.

FIG. 8 is a diagram showing a portion of the failure handling skill in the operator information 43 according to the present embodiment. Further, FIG. 9 is a diagram showing a part of the skill necessary for dealing with a failure in the failure information 42 according to the present embodiment.
The response availability determination unit 32 confirms the failure response skill possessed by the operator 10010 by the operator information 43 in FIG. 8, and compares it with the skill required for failure response for each failure in the failure information 42 in FIG. The response availability determination unit 32 can determine that the operator 10010 can respond to the failure ID "4". Therefore, the response availability determination unit 32 selects the failure ID “4” as the training failure 71. Here, it is determined that the operator 10010 can deal with the failure ID "4" based on the two determination criteria. The first is that "the two skills required to deal with the failure ID" 4 "are Windows (registered trademark) and network knowledge, but the operator 10010 has both skills." The second is that "there is a problem that can be handled by the operator 10010, and the problem ID is" 4 "or the difficulty level of the problem or higher."

<Step S104: Impact determination on the system>
In step S104, the impact prediction unit 33 determines whether or not the influence on the operation system 10 when the training failure 71 is generated in the operation system 10 is within the allowable range based on the system information 41 indicating the status of the operation system 10. Is determined. In other words, the impact prediction unit 33 acquires the status of the operation system 10 from the system information 41, and determines whether or not there is no problem even if the training failure 71 is generated. Specifically, the presence or absence of an error is used as the degree of impact, and whether or not a real failure has occurred is confirmed by the degree of impact, and if so, the training is stopped because the degree of impact is out of the permissible range. I do. The status of the operation system 10 acquires information from the operation monitoring device 50 and is held in the system information 41 of the information storage device 40. If it is determined that there is an impact on the operation system 10, the training will be terminated without implementation. If it is determined that there is no influence on the operation system 10, the process proceeds to step S105. Further, for example, it is assumed that the influence prediction unit 33 acquires the CPU usage rate information of the operation system 10 as the influence degree, and the CPU usage rate is 90%. In this situation, if a failure occurs, the entire system may go down, so it is judged that there is an impact on the system, and the training is terminated without implementation. That is, the permissible range in this case is less than 90% of the CPU usage rate.

<Step S105: Failure occurred>
In step S105, the training execution unit 34 causes the operator selected in step S101 to generate the training failure 71 selected in step S102 to step S103. At this time, the training execution unit 34 may display the training notification 72 notifying that the training is being performed on the operator terminal 60 of the operator selected in step S101.
FIG. 10 is an example of the training notification 72 displayed on the operator terminal 60 according to the present embodiment.
The operator handles the same failure as a real failure according to the prescribed operation manual. At this time, the training content recording unit 35 collects information such as the work content of the operator and the trouble response time, and stores it in the training history 44.

<Step S106: Failure response reference time determination>
The training execution unit 34 determines whether or not the failure response reference time has been exceeded from the occurrence of the failure in step S105. If the amount is exceeded, the process proceeds to the training follow-up in step S112. If it is within the failure response reference time, the process proceeds to the completion determination in step S107.

<Step S107: Completion determination>
The training execution unit 34 determines whether or not the operator's failure response has been completed. Specific examples of the determination method are as follows. The training execution unit 34 displays a completion button on the operator terminal 60, and determines that the training is completed when the operator presses the completion button. Alternatively, the training execution unit 34 determines that the failure response is completed when the incident management ticket of the operation monitoring device 50 is completed. Alternatively, the training execution unit 34 acquires the status of the operation system 10 from the system information 41, and if there is no error, determines that the training is completed. The method of completion determination by the training execution unit 34 may be other than the method described above.

<Step S108: Evaluation of correspondence validity>
The cause acquisition unit 36 acquires failure response information including the cause of the training failure 71 and important points in failure response from the operator. The training result evaluation unit 38 outputs an evaluation result 74 including an evaluation of the validity of the operator's failure response based on the failure response information acquired by the cause acquisition unit 36. When the validity of the operator's troubleshooting is manually evaluated, the training result evaluation unit 38 outputs the evaluation result 74 including the validity evaluation acquired by the cause evaluation acquisition unit 37.

FIG. 11 is a diagram showing an example of the failure response information input screen 52 according to the present embodiment.
As shown in FIG. 11, the cause acquisition unit 36 presents the failure response information input screen 52 to the operator. The operator inputs the failure response information 73 such as the cause of the training failure 71 and the opinion on the important points in the failure response, and presses the completion button. The cause acquisition unit 36 passes the failure response information 73 acquired by the failure response information input screen 52 to the training result evaluation unit 38. Further, the cause acquisition unit 36 stores the failure response information 73 in the training history 44.

Subsequently, the training result evaluation unit 38 compares important points such as the cause of the failure registered in the failure information 42 with the failure response information 73 based on the judgment of the operator collected by the cause acquisition unit 36, and evaluates the correctness. .. The training result evaluation unit 38 stores the evaluation of the correctness of the failure response information in the training history 44 as an evaluation of the response validity.
FIG. 12 is a diagram showing an example of a correct answer table 53 of the failure response information input screen 52 according to the present embodiment.
As shown in FIG. 11, on the failure response information input screen 52, the answer is selected as in the cause input field and the important point input field. The training result evaluation unit 38 compares the selected answer with the pre-registered correct answer table 53 and determines whether they match. If all match, it is evaluated as positive, and if even one does not match, it is evaluated as negative. In the failure response information 73 input to the failure response information input screen 52 of FIG. 11, since all the answers are correct, it is determined to be correct.

FIG. 13 is a diagram showing an example of the failure response information input screen 52a according to the present embodiment.
FIG. 14 is a diagram showing an example of a list of important points 54 according to the present embodiment.
In the example in which the process of step S108 is manually performed, the failure handling information input screen 52a is set to a free description type as shown in FIG. Then, the failure response information 73 is displayed to the determiner, and the determiner evaluates the correctness of the failure response information 73 via the cause evaluation acquisition unit 37. Specifically, the determiner looks at the answer described in the failure response information input screen 52a, compares it with the important point list 54 in FIG. 14, and evaluates whether the meaning of the description is correct. This time, all the contents are the same, so it is evaluated as positive.

<Step S109: Success / failure evaluation of failure recovery>
The training result evaluation unit 38 evaluates the success or failure of the restoration of the operation system 10, and includes the evaluation of the success or failure of the restoration of the operation system 10 in the evaluation result 74. Specifically, the training result evaluation unit 38 acquires system status information from the system information 41 and evaluates whether or not the failure has been recovered. The training result evaluation unit 38 stores the evaluation of the success or failure of the failure recovery in the training history 44. The method of determining the success or failure of the failure recovery may be a method of confirming the presence or absence of an error, a method of executing a specific command and comparing the result with the normal state, or another method. As a specific example of another method, when the network is interrupted, execute the ping command to the communication destination machine, and if it is confirmed that a response is returned, the success or failure of the failure recovery is set as "success" and the training history 44. Record in.

<Step S110: Evaluation of failure response time>
The training result evaluation unit 38 evaluates the failure response time for troubleshooting, and includes the evaluation of the failure response time in the evaluation result 74. Specifically, the training result evaluation unit 38 compares the current failure response time stored in the training history 44 with the failure response reference time stored in the failure information 42, and evaluates the failure response time. Specifically, the evaluation standard is set to be within 1/2 of the failure response standard time, and if it is shorter than the standard, it is evaluated as ◎, and if it is longer than the standard, it is evaluated as ○. The training result evaluation unit 38 stores the evaluation of the failure response time in the training history 44.
FIG. 15 is a diagram showing an example of correspondence between the failure response reference time and the failure response time according to the present embodiment. In the example of FIG. 15, since the failure response time is within 1/2 of the failure response reference time, the evaluation of the failure response time is set to ⊚ and stored in the training history 44.

<Step S111: Judgment of necessity of training follow>
The training result evaluation unit 38 determines the necessity of training follow-up from the evaluation result 74 stored in the training history 44. The evaluation results stored in the training history 44 are the results of the evaluation of the response validity saved in step S108, the success / failure evaluation of the failure recovery saved in step S109, and the evaluation of the failure response time saved in step S110.
FIG. 16 is a diagram showing a training follow-up standard table 55 according to the present embodiment.
In the training follow-up standard table 55, the evaluation result 74, the necessity of training follow-up for the operator, and the content of the training follow-up are associated with each other.
The training result evaluation unit 38 determines the necessity of training follow according to the training follow standard table 55. If it is determined as necessary as a result of the determination, the process proceeds to the training follow-up in step S112. If it is determined to be negative, the process proceeds to the corresponding method matching degree determination in step S113. Specifically, in FIG. 16, when the evaluation of the response validity is positive, the success / failure evaluation of the failure recovery is successful, and the evaluation of the response time is ⊚, it is determined whether or not the training follow-up is necessary.

<Step S112: Training follow>
The training result evaluation unit 38 determines whether or not the training follow-up for the operator is necessary by using the training follow-up standard table 55, and when it is determined that the training follow-up for the operator is necessary, the training follow-up is performed according to the contents of the training follow-up. I do. Specifically, the training result evaluation unit 38 presents necessary information to the operator when the failure response reference time is exceeded in step S106 and when it is determined in step S111 that training follow-up is required. The content of the information to be presented is as shown in the content of the training follow-up in FIG.
In the example of FIG. 16, when the evaluation of the response validity is negative and the success / failure evaluation of the failure recovery is negative, “failure recovery assist” is extracted as the content of the training follow-up. In this case, we understand the important points, but we have not been able to recover from the disaster. Since the "disaster recovery assist" has been extracted, the training result evaluation unit 38 presents the support information for disaster recovery to the operator. As support information for recovery from the failure, the details of the failure that occurred, the important points of the response, and the correct response method are presented. Further, the training result evaluation unit 38 executes a predetermined assist process such as automatic recovery on the device side of the operation system 10 so that the operator can quickly recover from the failure.

<Step S113: Correspondence method Matching degree determination>
The training result evaluation unit 38 determines whether or not the failure handling method and the registration handling method match. When the failure response method and the registration response method do not match, the update unit 309 registers the failure response method in the failure information 42 as the registration response method. Specifically, the training result evaluation unit 38 compares the registration response method stored in the failure information 42 with the current failure response method stored in the training history 44, and determines the degree of agreement.
FIG. 17 is a diagram showing an example of the registration handling method 56 of the failure information 42 according to the present embodiment.
In the registration response method 56, the failure response pattern is defined by the command execution order and the execution content.
The training result evaluation unit 38 compares and determines the execution order and execution contents of the commands defined in the registration response method 56 with the current failure response method stored in the training history 44.

FIG. 18 is a diagram showing an example of a command execution log according to the present embodiment.
FIG. 19 is a diagram showing an example of the current failure handling method 58 stored in the training history 44 according to the present embodiment.
FIG. 20 is a diagram showing another example of the current failure handling method 58a stored in the training history 44 according to the present embodiment.
The training result evaluation unit 38 acquires a log of command execution as shown in FIG. Then, the training result evaluation unit 38 records the executed command in the training history 44 as shown in FIG. The training result evaluation unit 38 determines the degree of agreement between the corresponding methods by comparing FIGS. 17 and 19. In the examples of FIGS. 17 and 19, the execution order and the execution contents are all the same. On the other hand, when the failure handling method of this time stored in the training history 44 is FIG. 20, when FIG. 17 and FIG. 20 are compared, the execution order is partially different. In this case, the training result evaluation unit 38 determines that there is a partial disagreement. If the execution order and the execution contents are completely different, the training result evaluation unit 38 determines that they are completely inconsistent. If it is determined that all of them match, the process proceeds to step S115. If it is determined that there is a partial disagreement or a complete disagreement, it means that the cause can be identified and the failure recovery can be performed correctly within the failure response reference time, and then a new response method for the failure has been found, and the process proceeds to step S114.

<Step S114: Correspondence method new registration>
The failure information update unit 39 acquires the current failure response method saved in the training history 44, and newly additionally registers the failure information 42 as a registration response method.
FIG. 21 is a diagram showing an example in which the current failure handling method is newly registered as the handling pattern 2 in the failure information 42 according to the present embodiment.

<Step S115: Operator skill update>
The operator information update unit 391 determines the skill of the operator from the results of steps S108, S109, S110, and S111, and updates the operator information 43.
FIG. 22 is a diagram showing an operation determination standard table 57 according to the present embodiment.
The operator information update unit 391 updates the operator information 43 according to the operation determination standard table 57.
In FIG. 22, when the operator 10015 evaluates the response validity as correct, the failure recovery success / failure evaluation is successful, and the necessity of training follow-up is negative, the operator 10015 can raise the level of the responseable failure difficulty level. It has been set. Therefore, when the current manageable obstacle difficulty level of the operator 10015 is low as shown in FIG. 8, the level is raised from the current low level to the middle level. In this way, the operator information update unit 391 updates the available failure difficulty level of the operator skill list of the operator information 43.

It should be noted that the execution contents and the execution order regarding the series of operations from step S101 to step S104 and the series of operations from step S106 to step S115 are merely examples, and are not necessarily limited to the above-described embodiment.

*** Explanation of hardware configuration ***
FIG. 23 is a diagram showing a hardware configuration of the failure response training device 20 according to the present embodiment.
The fault handling training device 20 is a computer. The fault handling training device 20 includes a processor 910 and other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to other hardware via a signal line and controls these other hardware.

The failure response training device 20 has, as functional elements, a selection unit 301, an impact prediction unit 33, a training execution unit 34, a training content recording unit 35, a cause acquisition unit 36, a cause evaluation acquisition unit 37, a training result evaluation unit 38, and an update. A unit 309 is provided. The functions of the selection unit 301, the impact prediction unit 33, the training execution unit 34, the training content recording unit 35, the cause acquisition unit 36, the cause evaluation acquisition unit 37, the training result evaluation unit 38, and the update unit 309 are referred to as the functions of the training device 30. .. Further, the system information 41, the failure information 42, the operator information 43, and the training history 44 are provided in the memory 921. The functions of the system information 41, the failure information 42, the operator information 43, and the training history 44 are referred to as the functions of the information storage device 40.

The function of the training device 30 is realized by software. The information storage device 40 is provided in the memory 921.

The processor 910 is a device that executes a failure response training program. The failure response training program is a program that realizes the functions of the training device 30.
The processor 910 is an IC (Integrated Circuit) that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit).

The memory 921 is a storage device that temporarily stores data. A specific example of the memory 921 is a SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory).
The auxiliary storage device 922 is a storage device for storing data. A specific example of the auxiliary storage device 922 is an HDD. Further, the auxiliary storage device 922 may be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disc, Blu-ray (registered trademark) disk, or DVD. HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash®. DVD is an abbreviation for Digital Versaille Disk.

The input interface 930 is a port connected to an input device such as a mouse, a keyboard, or a touch panel. Specifically, the input interface 930 is a USB (Universal Serial Bus) terminal. The input interface 930 may be a port connected to a LAN (Local Area Network).
The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) (High Definition Multimedia Interface) terminal. Specifically, the display is an LCD (Liquid Crystal Display).

The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, or a telephone line. Specifically, the communication device 950 is a communication chip or a NIC (Network Interface Card).

The fault handling training program is read into the processor 910 and executed by the processor 910. In the memory 921, not only the failure response training program but also the OS (Operating System) is stored. The processor 910 executes a failure response training program while executing the OS. The fault response training program and the OS may be stored in the auxiliary storage device 922. The fault handling training program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. In addition, a part or all of the trouble response training program may be incorporated in the OS.

The fault handling training device 20 may include a plurality of processors that replace the processor 910. These multiple processors share the execution of the fault response training program. Each processor, like the processor 910, is a device that executes a fault handling training program.

The data, information, signal values and variable values used, processed or output by the fault handling training program are stored in the memory 921, the auxiliary storage device 922, or the register or cache memory in the processor 910.

The "part" of each part of the training device 30 may be read as "processing", "procedure" or "process". Further, the "processing" of each part of the training device 30 may be read as "program", "program product", or "computer-readable storage medium on which the program is recorded".
In the trouble response training program, the computer is made to execute each process, each procedure or each process in which the "part" of each of the above-mentioned parts is read as "process", "procedure" or "process". Further, the failure response training method is a method performed by the failure response training device 20 executing a failure response training program.
The fault handling training program may be stored and provided on a computer-readable recording medium. The disability response training program may also be provided as a program product.

*** Other configurations ***
In this embodiment, the function of the training device 30 is realized by software. As a modification, the function of the training device 30 may be realized by hardware.
When the function of the training device 30 is realized by hardware, the failure handling training device 20 includes an electronic circuit instead of the processor.

The electronic circuit is a dedicated electronic circuit that realizes the function of the training device 30.
The electronic circuit is specifically a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.
The function of the training device 30 may be realized by one electronic circuit, or may be distributed and realized by a plurality of electronic circuits.
As another modification, some functions of the training device 30 may be realized by an electronic circuit, and the remaining functions may be realized by software.

Each of the processor and the electronic circuit is also called a processing circuit. That is, in the fault handling training device 20, the function of the training device 30 is realized by the processing circuit.

*** Explanation of the effect of this embodiment ***
The failure response training device according to the present embodiment has an effect that an efficient operator's failure response skill improvement is expected. In order to respond to the recent improvement in business speed, it is necessary to improve the trouble handling skills of operators efficiently in a short period of time. According to the failure response training device according to the present embodiment, the failure is selected according to the failure response skill of the operator, and the training is carried out in a state close to the actual failure response in the production environment, so that the operator is efficient. Can be trained.

Further, according to the failure response training device according to the present embodiment, know-how of the failure response method can be accumulated. According to the failure response training device according to the present embodiment, it is possible to accumulate correspondence patterns between failures and failure response methods. By presenting such information when a real failure occurs, it can be utilized for prompt resolution.

Further, according to the failure response training device according to the present embodiment, the operator information can be used for optimizing the operation shift. By using the operator information related to this embodiment using the data of the failure response skill, it is possible to utilize it for creating the optimum operation shift.

As described above, in the failure response training device according to the present embodiment, the operator's failure response skill can be appropriately evaluated by storing the operator information and the failure information and updating these information as needed. can. In the failure response training device according to the present embodiment, it is possible to evaluate not only the work performed by the predetermined method of routine work but also the atypical work for failure response. Since there is not always one troubleshooting method that is an atypical work, there are cases where a new troubleshooting method is found during training.
According to the failure response training device according to the present embodiment, it is possible to evaluate by combining the evaluation of important points such as whether the failure was recovered and whether the cause was correctly identified and the evaluation of the failure response method. In addition, the newly found troubleshooting method can be updated as a new registration pattern at any time.

In the above-described first embodiment, each part of the failure response training device has been described as an independent functional block. However, the configuration of the failure response training device does not have to be the configuration as in the above-described embodiment. The functional block of the failure response training device may have any configuration as long as it can realize the functions described in the above-described embodiment. Further, the failure response training device may be a system composed of a plurality of devices instead of one device.
Further, in the first embodiment, a plurality of parts may be combined and carried out. Alternatively, one part of this embodiment may be implemented. In addition, this embodiment may be implemented in any combination as a whole or partially.
That is, in the first embodiment, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment.

It should be noted that the embodiments described above are essentially preferred examples and are not intended to limit the scope of the invention, the scope of application of the invention, and the scope of use of the invention. The above-described embodiment can be variously modified as needed.

10 Operation system, 20 Failure response training device, 30 Training device, 301 Selection section, 31 Training content selection section, 32 Response availability judgment section, 33 Impact prediction section, 34 Training execution section, 35 Training content recording section, 36 Cause acquisition section , 37 Cause evaluation acquisition department, 38 Training result evaluation department, 39 Failure information update department, 391 Operator information update department, 309 update department, 41 System information, 42 Failure information, 43 Operator information, 44 Training history, 40 Information storage Equipment, 50 Operation monitoring device, 51 Candidate list, 52, 52a Failure response information input screen, 53 Correct answer table, 54 Important point list, 55 Training follow-up standard table, 56 Registration response method, 57 Operation judgment standard table, 58, 58a Failure Response method, 60 Operator terminal, 61 Operator, 71 Training failure, 72 Training notification, 73 Failure response information, 74 Evaluation result, 500 Failure response training system, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface , 940 output interface, 950 communication device.

Claims (14)

In the failure response training device that performs failure response training for the operator by causing a failure in the operation system operated by the operator.
As the operator's failure response training, based on the operator information including the difficulty level of the failure that the operator can handle and the failure information including the failure list and the difficulty level of each failure included in the failure list. A selection unit that selects the failure to be generated as a training failure from the failure list, and
An impact prediction unit that determines whether or not the impact on the operational system when a training failure occurs in the operational system is within an allowable range based on the system information indicating the status of the operational system.
A failure response training device provided with a training execution unit that causes the training failure in the operation system when the influence on the operation system is within an allowable range. The operator information includes the types of failure handling skills possessed by the operator.
The failure information includes the types of skills required to deal with each failure included in the failure list.
The selection unit is
The failure response training device according to claim 1, wherein a failure in which the operator has all the necessary skill types is selected as the training failure from the failure list. In the operation system, the level of service provided to the user of the operation system is set as a service level agreement.
The impact prediction unit
A claim for determining that the influence on the operation system is within an allowable range when the service provided to the user satisfies the service level agreement when the training failure occurs in the operation system. The fault handling training device according to claim 1 or 2. The failure response training device is
When the operator completes the failure response to the training failure, the training result evaluation unit that outputs the evaluation result for evaluating the failure response, and the training result evaluation unit.
Claims 1 to claim 1 include a failure response information representing the content of the failure response and an update unit for updating the difficulty level of the failure that the operator can handle for the operator information based on the evaluation result. The fault handling training device according to any one of 3. The failure information includes a method for dealing with each failure included in the list of failures as a registration response method.
The update part
The failure response training device according to claim 4, wherein the registration response method included in the failure information is updated based on the failure response information and the evaluation result. The failure response training device is
The training content recording unit is provided, in which the method for dealing with a training failure is set in the failure response information as the failure response method, and the failure response information is stored in the training history.
The training result evaluation unit
It is determined whether or not the failure handling method and the registration handling method match.
The update part
The failure response training device according to claim 5, wherein when the failure response method and the registration response method do not match, the failure response method is registered in the failure information as the registration response method. The failure response training device is
A cause acquisition unit for acquiring the failure response information including the cause of the training failure and important points in the failure response from the operator is provided.
The training result evaluation unit
The item according to any one of claims 4 to 6, which outputs the evaluation result including the evaluation of the validity of the failure response of the operator based on the failure response information acquired by the cause acquisition unit. Failure response training device. The failure response training device is
A cause acquisition unit for acquiring the failure response information including the cause of the training failure and important points in the failure response from the operator.
A cause evaluation acquisition unit for acquiring the validity evaluation of the failure response by the operator based on the failure response information acquired by the cause acquisition unit is provided.
The training result evaluation unit
The failure response training device according to any one of claims 4 to 6, which outputs the evaluation result including the validity evaluation acquired by the cause evaluation acquisition unit. The training result evaluation unit
The failure response training device according to any one of claims 4 to 8, which evaluates the success or failure of the restoration of the operation system and includes the evaluation of the success or failure of the restoration of the operation system in the evaluation result. The training result evaluation unit
The failure response training device according to any one of claims 4 to 9, wherein the failure response time applied to the failure response is evaluated and the evaluation of the failure response time is included in the evaluation result. The failure response training device is
A training follow-up standard table is provided in which the evaluation result, the necessity of training follow-up for the operator, and the content of the training follow-up are associated with each other.
The training result evaluation unit
Using the training follow-up standard table, it is determined whether or not the training follow-up for the operator is necessary, and if it is determined that the training follow-up for the operator is necessary, the request for performing the training follow-up according to the content of the training follow-up. The fault handling training device according to any one of claims 4 to 10. The training execution unit
The failure handling training according to any one of claims 1 to 11, wherein a training notification is transmitted to notify the operator that the training is to be performed when the training failure is generated in the operation system. Device. In the failure response training method of the failure response training device that performs failure response training for the operator by causing a failure in the operation system operated by the operator.
The selection unit of the operator is based on the operator information including the difficulty level of the failure that the operator can handle and the failure information including the failure list and the difficulty level of each failure included in the failure list. Select the disability to be generated as a disability response training as a training disability from the above disability list, and select it.
The impact prediction unit determines whether or not the impact on the operational system when the training failure occurs in the operational system is within the allowable range based on the system information indicating the status of the operational system.
A failure response training method in which the training execution unit causes the training failure in the operation system when the influence on the operation system is within an allowable range. In the failure response training program of the failure response training device that performs failure response training for the operator by causing a failure in the operation system operated by the operator.
As the operator's failure response training, based on the operator information including the difficulty level of the failure that the operator can handle and the failure information including the failure list and the difficulty level of each failure included in the failure list. A selection process that selects the failure to be generated as a training failure from the failure list, and
Based on the system information indicating the status of the operational system, the impact prediction process for determining whether or not the impact on the operational system when the training failure occurs in the operational system is within the allowable range, and
A failure response training program that causes the failure response training device, which is a computer, to execute a training execution process for causing the training failure in the operation system when the influence on the operation system is within an allowable range.

Images