JP7472628B2 - Fault recovery device, fault recovery method and program - Google Patents

Description

The present invention relates to a disaster recovery device, a disaster recovery method, and a program.

As ICT systems become larger and more complex, the types and number of faults occurring are increasing. Responding to faults requires monitoring and analysis of observational data such as logs and metrics, but the volume of data and the complex relationships between the data make fault recovery extremely difficult. As a result, efforts have been made in recent years to automate and improve the accuracy of fault recovery work using machine learning.

Non-Patent Document 1 describes a technology that generates a sequence of recovery commands based on logs at the time of a failure. This technology uses a neural network to learn the relationship between the logs obtained from the target system when a failure occurs and the recovery commands executed to recover from that failure, and when a new failure occurs, it generates plausible recovery commands based on the observed logs.

There is also an approach in which a classifier for identifying the cause is learned by intentionally inserting a fault factor and collecting fault data. In Non-Patent Document 2, a fault factor insertion tool developed for chaos engineering is used to insert various fault factors into the verification environment and collect observation data at that time. By learning a classifier that links the observation data with the fault factor (a combination of the fault type and its occurrence location), it becomes possible to identify the fault factor based on the observation data when a new fault occurs. While Non-Patent Document 1 assumes data obtained during actual operation, Non-Patent Document 2 uses a framework for intentionally inserting a fault factor, so it may be possible to handle faults that have not actually occurred yet or faults that occur infrequently.

On the other hand, there are also approaches that learn recovery strategies that indicate what commands need to be entered to recover the system, based on observational data obtained from the system.

In Non-Patent Document 3, a probabilistic model is created in advance that represents what kind of log is likely to appear when a certain command is executed when a failure occurs. Using this probabilistic model, the most efficient recovery strategy for identifying the cause of the failure is learned using reinforcement learning or other methods.

In Non-Patent Document 4, in a verification environment where a fault factor is inserted, a fault isolation command such as ping is executed, and a recovery policy function is learned by reinforcement learning, which calculates the next command to be executed based on the binary result such as success/failure of the ping obtained as feedback.

Mitsuki Ikeuchi, Satoru Watanabe, Yoichi Matsuo, Takehiro Kawada, "Automatic Generation of Fault Recovery Command Sequences Using Seq2Seq", IEICE General Conference, B-7-25, 2019. Mitsuki Ikeuchi, Yoshifumi Kuzu, Yoichi Matsuo, Keishiro Watanabe, "A study on a method for identifying the cause of faults based on fault data generation," IEICE General Conference, B-7-32, 2020. H. Ikeuchi, A. Watanabe, T. Kawata, and R. Kawahara, "Root-Cause Diagnosis Using Logs Generated by User Actions," in Proc. of IEEE Global Communication Conference (Globecom), pp. 1-7, 2018. M. L. Littman, N. Ravi, E. Fenson, and R. Howard, "An Instance-based State Representation for Network Repair", in Proc. of the 19th National Conference on American Association for Artificial Intelligence (AAAI), pp. 287-292, 2004.

The method described in Non-Patent Document 1 requires that a sufficient number of failure logs and recovery commands be accumulated as learning data, but in reality this is not always possible.

The fault data generation approach in Non-Patent Document 2 partially solves the issues in Non-Patent Document 1, but it is a method that focuses on identifying the cause of the fault and cannot obtain recovery commands, so recovery automation cannot be achieved.

The technology in Non-Patent Document 3 requires the creation of a probabilistic model that represents log occurrences, but this is extremely difficult in a complex system.

The technology in Non-Patent Document 4 does not require learning data or the creation of a prior probability model, but it assumes binary results as input values for the recovery measures, making it difficult to respond to failures that require recovery commands to be determined based on high-dimensional, probabilistically fluctuating, complexly correlated observational data such as logs and metrics.

The present invention was made in consideration of the above points, and aims to make it possible to automatically estimate commands for recovering from a failure.

In order to solve the above problem, a fault recovery device includes a first acquisition unit that acquires first data observed in a first system after an artificial fault factor is inserted, a first execution unit that executes a command estimated by the neural network by inputting the first data into a neural network to the first system, a second acquisition unit that acquires second data observed in the first system after execution of the command, a third acquisition unit that acquires a reward obtained by executing the command, and a learning unit that learns the neural network by deep reinforcement learning based on the first data, the command, the reward, and the second data , and the number of insertions into the first system is changed for each artificial fault factor depending on the learning level of the neural network .

It is possible to automatically estimate the commands required to recover from a failure.

1 is a diagram illustrating an example of a hardware configuration of a failure recovery device 10 according to an embodiment of the present invention. 1 is a diagram illustrating an example of a functional configuration of a failure recovery device 10 according to an embodiment of the present invention. 10 is a flowchart illustrating an example of a processing procedure executed by the failure recovery device 10 in one episode of a learning phase. 10 is a flowchart illustrating an example of a processing procedure executed by the failure recovery device 10 in an automatic recovery phase. FIG. 13 is a diagram showing learning results in an experiment.

In this embodiment, a technology is disclosed in which various faults are artificially generated by injecting fault factors into a target ICT system (hereinafter referred to as the "target system") using a fault factor insertion tool, and deep reinforcement learning is executed in that environment to acquire a recovery policy. That is, a deep reinforcement learning agent repeats a process in which it selects and executes an appropriate recovery command based on observational data such as logs and metrics obtained from the target system, and receives a reward according to changes in the state of the target system. By repeating this process, the agent acquires an optimal recovery policy that maximizes the reward.

In this embodiment, actual observation data is acquired from the target system by inserting artificially generated failure factors and executing recovery commands by an agent, and recovery measures are acquired, so there is no need to accumulate them as learning data in advance, and there is no need to create a probabilistic model in advance. Furthermore, in deep reinforcement learning, neural networks can analyze high-dimensional input values, so it is possible to handle high-dimensional, probabilistically fluctuating, complexly correlated observation data such as logs and metrics. Therefore, it is possible to establish a framework that automatically estimates recovery commands for recovering from failures based on high-dimensional, probabilistically fluctuating, complexly correlated observation data such as logs and metrics, without the need for learning data or the prior creation of a probabilistic model.

First, reinforcement learning, which is the basis of this embodiment, Q-learning, which is one of its algorithms (R. Sutton and A. Barto, (translated in Japanese by Sadayoshi Mikami and Masaaki Minagawa), Reinforcement Learning, Morikita Shuppan, pp. 159-161, 2000.), and deep Q-network, which is one of the algorithms in deep reinforcement learning (V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmillier, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, "Human-level control through deep reinforcement learning," Nature, vol. 518 no. 7540, pp. 529-533, 2015."). However, this embodiment is not limited to these algorithms, and other algorithms may be used.

Reinforcement learning is a method for an agent to acquire a policy for taking optimal actions in a given environment so as to maximize the sum of rewards obtained. Reinforcement learning consists of an agent that is the subject of the action and an environment in which the agent executes the action. An overview of reinforcement learning is as follows.

The agent observes the "state s" of the environment and executes a certain "action a" according to the current "policy π." A policy is a function that returns an action depending on the state, such as π(s) = a. When action a is executed, the state of the environment transitions to state s', and the agent earns a reward r in response to this transition. In the process of repeated interactions between the agent and the environment as described above, the agent improves the policy so as to maximize the sum of the rewards that can be earned over a series of actions.

Q-learning is a type of algorithm for realizing reinforcement learning. In Q-learning, an agent learns a Q-function Q ^π (s, a). Q ^π (s, a) is the expected value of the sum of rewards to be obtained in the future when an action a is executed in state s and then a policy π is followed. The agent repeatedly interacts with the environment and updates the policy π and the Q-function Q ^π (s, a).

When the state space or action space is large or continuous, a method that approximates the Q function with a parameterized function is often used. In particular, the method that uses a neural network as an approximation function is called deep reinforcement learning, and an algorithm called deep Q-network (DQN) is known as the most basic form. In DQN, the weight parameter θ of the neural network is updated to minimize the following loss function L(θ).

ここで、γは割引率と呼ばれるパラメタであり、Ｑ_θ（ｓ'，ａ'）はＱ関数である。

Here, γ is a parameter called the discount rate, and Q _θ (s′, a′) is the Q function.

When updating parameters, a method called experience replay is used, in which experience (s, a, r, s') is stored in memory, and the stored experience is randomly taken out as a mini-batch to update θ. The policy during learning is determined by the ε-greedy method or the like based on the Q function Q _θ (s', a'), and the policy after learning is

で与えられる。

is given by:

This embodiment consists of a learning phase in which deep reinforcement learning is performed on a target system in a verification environment or a target system in a pre-operational production environment (hereinafter, these are referred to as "learning systems") to learn a recovery policy (a neural network that estimates a recovery command), and an automatic recovery phase in which automatic recovery is performed using the learned recovery policy (neural network).

The learning phase will now be explained. First, a fault is artificially or man-made (hereafter referred to as "artificially") generated in the learning system using a fault insertion tool. The fault insertion tool may be a script created by an engineer to reproduce possible faults, or an existing load testing tool, fault insertion tool, etc. may also be used.

Next, various types of observation data are obtained from the learning system in a state where an artificial failure has occurred, i.e., log data obtained from each device constituting the learning system, metrics such as CPU usage and memory usage, and traffic flowing between devices. The observation data must be converted into a numerical vector in some way, and the numerical vector obtained in this way is hereinafter referred to as a "feature vector." A feature vector may be generated, for example, by simply arranging the obtained numerical data and log counts, and performing preprocessing such as appropriate normalization.

In this embodiment, the feature vector obtained in this manner is regarded as the "state s" in reinforcement learning. State s is continuous-valued and high-dimensional, with complex correlations between elements and including probabilistic fluctuations, so in this embodiment, a deep reinforcement learning method such as DQN is suitable.

The agent inputs the feature vector into the DQN neural network (a neural network that estimates recovery commands) and determines action a based on the current policy. The agent executes this action a on the training system and obtains a feature vector s' based on new observation data. The agent also uses a separately placed system status confirmation tool to check how the status of the training system has changed as a result of action a. For example, a check based on the success or failure of a ping to each device that makes up the training system, or an anomaly detection system that calculates the degree of anomaly in the system may be used.

The agent receives a reward r according to the result of the change in the system state. For example, if the learning system can only confirm the binary state of normal or abnormal, a positive reward is given when the learning system returns to the normal state, and a negative or zero reward is given otherwise. On the other hand, if an anomaly detection tool that can calculate the continuous degree of anomaly is used, the degree of decrease in the degree of anomaly (how close the normal state is to the action) may be defined as the reward. If the target system is composed of several components and the normal state of each component can be confirmed, the number of components that return to the normal state may be given as a reward. In addition, the way in which the reward is designed changes depending on the nature of the recovery measure to be acquired. If it is desired to acquire a recovery measure that realizes recovery in the shortest possible time, a reward should be given that takes into account the execution time of each command, and if it is desired to acquire a recovery measure that realizes recovery using only the safest possible recovery commands, a reward should be given that takes into account the system impact of each command (such as the degree of load on the system's CPU).

Meanwhile, the agent that has acquired experience (s, a, r, s') stores this in memory and uses past experience as mini-batches to update the weight parameters of the neural network.

The above process is repeated until the learning system returns to a normal state. The cycle from the insertion of one fault factor into the learning system to the recovery to a normal state is called an episode. If the system does not recover after repeating the above cycle a certain number of times in one episode, the verification state may be returned to a normal state using a backup of the learning system, etc., and the episode may be forcibly terminated. In this embodiment, episodes are repeated many times by changing the fault factors to be inserted (the type of fault that occurs, and the combination of the insertion points of the fault factors). The learning phase ends when a predefined learning convergence condition (for example, the average reward sum for the past few episodes (the average value of the reward sum for each episode for each few episodes) exceeds a certain threshold) is satisfied. From a trained neural network (hereinafter referred to as a "trained neural network"),

に基づいて復旧方策を獲得することができる。

Based on this, a recovery strategy can be obtained.

Next, the automatic recovery phase will be described. In the automatic recovery phase, a target system (hereinafter referred to as the "real system") in operation (real environment) is used. When a failure occurs and observation data is obtained, this is converted into a feature vector, and the feature vector is input to the trained neural network as state s. The trained neural network estimates the action a to be executed based on the learned recovery strategy, and the estimated action a is executed on the target system to obtain new observation data.

The above cycle is repeated until the actual system returns to a normal state (until system status confirmation tools and anomaly detection tools confirm that the system has recovered).

The following describes an embodiment of the present invention with reference to the drawings. FIG. 1 is a diagram showing an example of the hardware configuration of a disaster recovery device 10 in an embodiment of the present invention. The disaster recovery device 10 in FIG. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected by a bus B.

The program that realizes the processing in the disaster recovery device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 via the drive device 100 into the auxiliary storage device 102. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

When an instruction to start a program is received, the memory device 103 reads out the program from the auxiliary storage device 102 and stores it. The processor 104 is a CPU or a GPU (Graphics Processing Unit), etc., and executes functions related to the disaster recovery device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

Figure 2 is a diagram showing an example of the functional configuration of the failure recovery device 10 in an embodiment of the present invention. Figure 2 shows a target system, a failure factor insertion device 20, a system status confirmation device 30, and a failure recovery device 10.

The target system is the learning system in the learning phase and the real system in the automatic recovery phase.

The fault factor insertion device 20 is a computer that has the fault factor insertion tool described above, and in the learning phase, it inserts artificially generated fault factors into the target system.

The system status confirmation device 30 is a computer that confirms the status of the target system and gives (inputs) a reward r based on changes in the status to the agent of the failure recovery device 10.

The target system, the fault factor insertion device 20, and the system status confirmation device 30 constitute the environment for reinforcement learning.

On the other hand, the disaster recovery device 10 has a preprocessing unit 11, a recovery policy learning unit 12, an action execution unit 13, etc. Each of these units is realized by a process in which one or more programs installed in the disaster recovery device 10 are executed by the processor 104. The disaster recovery device 10 also uses an experience history storage unit 14. The experience history storage unit 14 can be realized, for example, using a memory device 103, etc.

The preprocessing unit 11 converts observation data such as logs and metrics observed (acquired) in the target system into feature vectors.

The recovery strategy learning unit 12 updates the parameters of the neural network based on the history of experiences (s, a, r, s') stored in the experience history storage unit 14, and uses the neural network to determine the action to be taken on the target system. Here, the action is the input of a recovery command.

The action execution unit 13 executes the action (recovery command) determined by the recovery strategy learning unit 12 on the target system.

The fault factor insertion device 20 and the system status confirmation device 30 may be included in the fault recovery device 10.

In the learning phase, a process called an episode is repeated many times. For example, if there are five failure factors (combinations of failure type and failure location) and 100 episodes are to be executed, one failure factor is randomly selected from the five failure factors and the episode is executed, and this process is repeated 100 times. One episode consists of the following processing steps. Figure 3 is a flowchart for explaining an example of the processing steps executed by the failure recovery device 10 in one episode of the learning phase. Hereinafter, the failure factor selected for that one episode is referred to as the "target factor."

When the fault factor insertion device 20 injects a target factor into the learning system to artificially cause a fault in the learning system (Yes in S101), the pre-processing unit 11 acquires observation data from the learning system in a state in which the fault occurs (S102). Next, the pre-processing unit 11 converts the observation data into a feature vector and inputs the feature vector to the recovery policy learning unit 12 (S103).

Then, the recovery policy learning unit 12 inputs the feature vector into a neural network to determine (select) a recovery command to be input to the learning system (S104). For example, the neural network outputs the probability of each recovery command. The recovery policy learning unit 12 selects the recovery command with the highest probability. The action execution unit 13 then executes (inputs) the recovery command determined by the recovery policy learning unit 12 to the learning system (S105).

Then, the recovery policy learning unit 12 obtains a reward from the system state confirmation device 30 (S106). For example, the system state confirmation device 30 confirms a state change of the learning system, and provides the recovery policy learning unit 12 with a reward according to the state change. Next, the pre-processing unit 11 obtains observation data from the learning system after the state change (after the execution of the recovery command) (S107). Next, the pre-processing unit 11 converts the observation data into a feature vector, and inputs the feature vector to the recovery policy learning unit 12 (S108).

Then, the recovery policy learning unit 12 stores the experience (s, a, r, s') in the experience history storage unit 14 (S109), in which the feature vector generated in step S103 is set as state s, the recovery command determined in step S104 is set as action a, the reward obtained in step S106 is set as reward r, and the feature vector state s' generated in step S108. Therefore, the experience history storage unit 14 stores the history of experiences.

Then, the recovery policy learning unit 12 randomly extracts the experiences stored in the experience history storage unit 14 as mini-batches and updates the parameters of the neural network (learns the neural network) (S110). Then, the recovery policy learning unit 12 determines whether the learning system is in a normal state (whether it has recovered from the failure) based on, for example, the state acquired from the system state confirmation device 30 (S111).

If the learning system is not in a normal state (No in S111), the recovery policy learning unit 12 determines whether the number of actions (number of times the recovery command is executed) is equal to or greater than a threshold (S112). The number of actions is equivalent to the number of repetitions of step S104 and subsequent steps. If the number of actions is less than the threshold (No in S112), step S104 and subsequent steps are repeated.

On the other hand, if the learning system is in a normal state (Yes in S111) or if the number of actions is equal to or greater than the threshold (Yes in S112), the processing procedure in FIG. 3 ends for one episode. Note that when the processing procedure in FIG. 3 ends based on the condition that the number of actions is equal to or greater than the threshold, the verification state may be returned to a normal state using a backup of the initial state of the learning system, for example. In this way, it is possible to reduce the impact of a failure that occurred in the current episode on the next episode to be executed.

When one episode ends, the next episode begins. For example, each time an episode ends, the recovery policy learning unit 12 determines whether a predefined learning convergence condition (e.g., the sum of the average rewards obtained over the past few episodes exceeds a certain threshold) is satisfied, and if the learning convergence condition is satisfied, the recovery policy learning unit 12 ends the learning phase.

Figure 4 is a flowchart illustrating an example of the processing procedure executed by the failure recovery device 10 during the automatic recovery phase.

When a fault occurs in the real system (Yes in S201), the preprocessing unit 11 acquires observation data from the real system in a state where the fault occurs (S202). Next, the preprocessing unit 11 converts the observation data into a feature vector and inputs the feature vector to the recovery policy learning unit 12 (S203).

Then, the recovery policy learning unit 12 inputs the feature vector into the trained neural network (trained recovery policy) to determine (select) a recovery command to be input to the real system (S204). For example, the trained neural network outputs the probability of each recovery command. The recovery policy learning unit 12 selects the recovery command with the highest probability. The action execution unit 13 then executes (inputs) the recovery command determined by the recovery policy learning unit 12 to the real system (S205).

Steps S202 to S205 are repeated until it is determined that the actual system has returned to a normal state (until it has recovered from the failure), for example, based on the state obtained from the system state confirmation device 30 (S206).

Next, we will discuss some of the issues that may arise when implementing this embodiment and how to solve them.

[Task 1]

In the learning phase of this embodiment, the steps of injecting a fault factor, collecting observational data, and executing an action are repeated many times within each episode. Depending on the target system, completing this single step may take time or be technically difficult. For example, after injecting a fault factor into a server, attempting to recover using a command such as restart or backup may take time on the order of several minutes. In addition, the method of injecting the fault factor may also be difficult. In this embodiment, a volume of data sufficient to perform deep reinforcement learning is required, so the time and difficulty required for one step must be kept as small as possible.

[Solution to problem 1]

If such a problem exists, the learning system may be simulated in a virtual environment such as a container platform, and the learning phase may be performed using the virtual environment. With a container, the time required for restarting is only a few seconds, and an orchestrator such as Kubernetes can also be used, making it easy to operate multiple containers. In addition, since containers are managed as image files, it is possible to reproduce the exact same environment. By preparing multiple identical environments and simultaneously applying the method of this embodiment, distributed learning that efficiently accumulates experience (s, a, r, s') is also possible. Furthermore, various fault factor insertion tools have been developed in recent years as part of chaos engineering efforts (see "C. Rosenthal, L. Hochstein, A. Blohowiak, N. Jones, and A. Basiri, "Chaos Engineering," O'Reilly Media, Incorporated, 2017."), so a wide variety of faults can be easily inserted.

[Task 2]

In this embodiment, deep reinforcement learning is performed by inserting various fault factors, but it is not necessarily a good idea to select the fault factors to be inserted randomly or uniformly. When the same fault is inserted multiple times, there are some faults for which a recovery measure can be easily acquired because similar observation data is obtained each time, and there are other faults for which it takes time to acquire a recovery measure because the observation data fluctuates greatly. It is desirable to focus on inserting fault factors for the latter type of fault and perform learning. The method of selecting the fault factors to be inserted in this way has a significant impact on the accuracy of the recovery measure and the time it takes to acquire the recovery measure.

[Solution to problem 2]

Therefore, it is possible to effectively obtain observation data by adopting the following method of selecting the insertion failure factor. First, the failure factor is inserted randomly to some extent, and deep reinforcement learning is performed to repeat the episode. After that, in the middle of the learning phase (for example, when 50 episodes have been completed if a maximum of 100 episodes are planned to be repeated), the recovery policy learning unit 12 calculates the average of the reward sums of the most recent N times (for example, N = 5, etc.) for each failure factor. If there is a failure factor group whose average is equal to or greater than a threshold set as a criterion for a high reward sum, it may be considered that a recovery policy has been acquired for that failure factor group. On the other hand, for a failure factor group whose average is less than the threshold, since the recovery policy cannot be applied, such a failure factor group is inserted with a focus on the deep learning. For example, in this case, the recovery policy learning unit 12 may send an instruction to the failure factor insertion device 20 to randomly insert a failure factor from the failure factor group. That is, the number of insertions may be changed for each failure factor (for each failure type and for each failure factor insertion location) according to the degree of learning of the recovery policy. This allows for efficient learning of recovery strategies that can deal with any type of failure.

[Challenge 3]

The observed data can change significantly depending not only on the inserted fault factor (type of fault, location of fault factor insertion) but also on conditions such as background traffic. Therefore, if a recovery strategy is learned under certain conditions, a change in the environmental conditions, such as traffic trends, may render the recovery strategy useless.

[How to solve problem 3]

When such a problem exists, for example, a traffic generation device may be used to generate background traffic for the training system, and deep reinforcement learning may be performed while changing the environment to which the training system belongs, such as the size of the background traffic, to learn recovery measures that can handle various situations. That is, the recovery measure learning unit 12 may train the neural network in a situation in which the environment to which the training system belongs changes over time. In this case, the changes in the environment, such as the size of the background traffic, may be changed on an episode-by-episode basis, or may be changed during the process of repeating step S104 for each episode.

Next, we will explain the results of an actual experiment conducted on this embodiment.

A Kubernetes cluster was created using Kubernetes, an orchestrator for container-based virtual environments, and a 3-tier web environment was constructed within it. The 3-tier web environment realized with this container is considered to be the verification environment and the actual environment in this embodiment. The 3-tier web environment consists of two sets of 3 containers, Nginx, Rails, and MySQL, making a total of 6 containers.

As background traffic, HTTP requests were randomly generated using a load test tool. As a fault factor, either 80% packet loss or a delay of 10,000 ± 1,000 ms was inserted into up to two of the six containers.

The inflow and outflow traffic (in bytes and packets) for each container was collected and used as observational data. The feature vector is a 24-dimensional vector obtained by scaling the observational data.

The action that the agent can take is to execute a command to regenerate each container. Since there are six containers in total, the total number of actions is six. A container in which a fault has been injected can be restored by issuing a regeneration command.

The reward was determined as follows: If one fault was inserted, r = 1.0 (recovery completed) or 0.25 (otherwise). If two faults were inserted, r = 0.5 (one fault recovered) or -0.25 (otherwise). In other words, if the agent executed a recovery command that was necessary and sufficient for recovery, it received a reward of +1.0, and if it executed an incorrect command, it received a penalty (negative reward). The algorithm used for deep reinforcement learning was Double DQN, an improved version of DQN (H. Van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double Q-learning," in Proc. of the 30th National Conference on American Association for Artificial Intelligence (AAAI), pp. 2094-2100, 2016.), and learning was considered complete when the average reward obtained over the past 10 episodes exceeded 0.95.

The learning results are shown in Figure 5. The horizontal axis is the number of episodes, and the vertical axis is the average reward obtained in the past 10 episodes. The blue solid line is the learning curve, and the horizontal dashed line is the threshold of reward sum = 0.95 (threshold for completion of learning). It was completed in 732 episodes, taking approximately 58.5 hours. As such, it can be seen that with appropriate learning, recovery measures can be automatically acquired and the system can be automatically restored, demonstrating the effectiveness of this embodiment.

As described above, this embodiment makes it possible to automatically estimate commands to recover from a failure based on high-dimensional, probabilistically fluctuating, complexly correlated observational data such as logs and metrics, without the need for learning data or the creation of a prior probabilistic model.

Since there is no need to create training data or a prior probability model, even in cases where a sufficient amount of training data has not been accumulated or where it is difficult to create a probability model that represents the system behavior, it is possible to acquire a recovery strategy that outputs an appropriate recovery command based on high-dimensional observation data, thereby contributing to the automation and high accuracy of failure recovery.

In this embodiment, the learning system is an example of a first system. The real system is an example of a second system. The preprocessing unit 11 is an example of a first acquisition unit and a second acquisition unit. The observation data and its feature vector are an example of first data and second data. The action execution unit 13 is an example of a first execution unit and a second execution unit. The recovery measure learning unit 12 is an example of a third acquisition unit and a learning unit.

Although the embodiment of the present invention has been described in detail above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims.

REFERENCE SIGNS LIST 10 Fault recovery device 11 Preprocessing unit 12 Recovery policy learning unit 13 Action execution unit 14 Experience history storage unit 20 Fault factor insertion device 30 System state confirmation device 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device B Bus

Claims (8)

a first acquisition unit that acquires first data observed in the first system after an artificial fault factor is inserted;
a first execution unit that inputs the first data into a neural network and executes a command estimated by the neural network on the first system;
a second acquisition unit that acquires second data observed in the first system after execution of the command;
a third acquisition unit that acquires a reward obtained by executing the command;
A learning unit that learns the neural network by deep reinforcement learning based on the first data, the command, the reward, and the second data;
having
The number of times of insertion into the first system is changed for each artificial fault factor according to the learning degree of the neural network.
A disaster recovery device comprising: a second execution unit that inputs data observed in the second system in a state in which a fault occurs in the second system into the trained neural network, and executes a command estimated by the neural network on the second system;
2. The failure recovery device according to claim 1, further comprising: the first system is a virtual environment;
3. The failure recovery device according to claim 1 or 2. The learning unit learns the neural network in a situation in which a situation of an environment to which the first system belongs changes over time.
4. The failure recovery device according to claim 1, wherein the failure recovery device is a data recovery device. a first acquisition procedure for acquiring first data observed in a first system after an artificial disturbance is injected;
a first execution step of inputting the first data into a neural network to execute a command estimated by the neural network on the first system;
a second acquisition step of acquiring second data observed at the first system after execution of the command;
A third acquisition step of acquiring a reward obtained by executing the command;
a learning step of learning the neural network by deep reinforcement learning based on the first data, the command, the reward, and the second data;
The computer executes
The number of times of insertion into the first system is changed for each artificial fault factor according to the learning degree of the neural network.
A disaster recovery method comprising: a second execution procedure for inputting data observed in the second system in a state in which a fault occurs in the second system into the trained neural network, thereby executing a command estimated by the neural network on the second system;
6. The method for recovering from a failure according to claim 5 , wherein the method is executed by a computer. a first acquisition procedure for acquiring first data observed in a first system after an artificial disturbance is injected;
a first execution step of inputting the first data into a neural network to execute a command estimated by the neural network on the first system;
a second acquisition step of acquiring second data observed at the first system after execution of the command;
A third acquisition step of acquiring a reward obtained by executing the command;
a learning step of learning the neural network by deep reinforcement learning based on the first data, the command, the reward, and the second data;
Run the following on your computer :
The number of times of insertion into the first system is changed for each artificial fault factor according to the learning degree of the neural network.
A program characterized by: a second execution procedure for inputting data observed in the second system in a state in which a fault occurs in the second system into the trained neural network, thereby executing a command estimated by the neural network on the second system;
8. The program according to claim 7 , wherein the program is executed by a computer.

Images