JP2020166795A - Reinforced learning method, reinforced learning device, and reinforced learning program for efficient learning - Google Patents

Abstract

To provide a reinforced learning method that learns expression specialized in states where an attention situation such as environment reset occurs so that efficient learning can be performed.SOLUTION: A reinforced learning method that optimizes a behavior policy of an agent from a result learned using a learning device that learns based on a state observed from environmental data, includes determining whether a state where a preset attention situation has occurred is observed during learning of environmental data in one episode. When the state where the attention situation has occurred is observed, a feature extractor (first learning device) learns by using two pieces of environmental data of environmental data of the state where the preset attention situation has occurred and environmental data at time before by one, and performs expression learning. A state classifier (second learning device) learns difference between pieces of feature data, and parameters of the first learning device and the second learning device are updated based on estimated output data and real data.SELECTED DRAWING: Figure 3

Description

The present invention uses a learning device that learns based on the state observed from environmental data, and in reinforcement learning that optimizes the behavioral policy of the agent from the learning result, the data efficiency of the episode can be improved and the cost can be reduced. It relates to methods, devices and programs.

In recent years, reinforcement learning, especially deep reinforcement learning, has been applied to algorithms such as AI (artificial intelligence) used in electronic games, automatic driving control of vehicles such as automobiles, and autonomous control of robots. Reinforcement learning is a method of repeating learning in units of episodes and adapting to the target environment using the rewards of trial and error in the environment, and learning is performed by optimizing the policy. Reinforcement learning uses the information processing power of a convolutional neural network (CNN) to perform reinforcement learning based on high-dimensional input such as image data.

In reinforcement learning and deep reinforcement learning, a problem is set in which an agent in the environment observes the current state and decides an action to be taken, and the agent receives a reward from the environment by selecting an action. While repeating the interaction between the environment and the agent, learning is performed using the three variables of reward, behavior, and state.
In reinforcement learning and deep reinforcement learning, agents act in an environment with a state space and an action space. At each time step, the policy converts the state into a feature vector with the feature extraction learner, and outputs the action probability for the feature vector with the value function calculation learner. After the agent takes action, the environment prints the scalar reward and the next state. The episode ends when you repeat the action until the final state.

In reinforcement learning and deep reinforcement learning, since learning is performed by trial and error, it is necessary to reset many environments, for example, to recover from damage or failure due to game over or collision of a vehicle or robot. Specifically, every time a robot that learns to walk falls, a person needs to stand up the fallen robot. Reinforcement learning algorithms require a lot of experience to cope with such a fall state, the length of the episode increases, and the time required for learning increases accordingly. In other words, resetting the environment not only increases the cost due to physical recovery, but also reduces the learning efficiency due to the increase in learning time. Therefore, a technique that can reduce the number of times the environment is reset, that is, reinforcement learning with good episode efficiency. The method is expected.

Under such circumstances, a technique for reducing the number of times the environment is reset in reinforcement learning has been conventionally known (see, for example, Non-Patent Documents 1 to 3).
Non-Patent Document 1 includes the PILCO framework (see “MP Deisenroth et al.,“ PILCO: A model-based and data-efficient approach to policy search ”, In ICML, pages 465-472, 2011.”). Based on training a Gaussian process model, estimating the probability of violating state-space constraints, implementing only what is considered safe in real-world and during training, and minimizing the risk of failure, or environmental reset. A method of limiting it is disclosed. The method disclosed in Non-Patent Document 1 greatly reduces the reset of the environment, but since the environment is expressed by the mathematical model learned by the PILCO framework, the calculation cost is high and general problems (automatic driving, robot). Control, etc.) has the problem that it is difficult to apply and its versatility is low. Another problem is that it is not clear how to apply the PILCO framework to deep reinforcement learning.

Non-Patent Document 2 discloses an autonomous method for safe and efficient reinforcement learning. In the method disclosed in Non-Patent Document 2, the environment for retry is reset, the forward policy and the reset policy are learned at the same time, and the value function of the policy for resetting the environment is learned. It automatically determines when the forward strategy is about to become irreversible, enables uncertainty-aware safety termination, and reduces the number of manual environment resets. Similar to the method of Non-Patent Document 1, the method disclosed in Non-Patent Document 2 has problems such as high calculation cost and low versatility.

Non-Patent Document 3 is reinforcement learning for learning an environment model using big data. High-dimensional observation data in observation of environmental data is compressed into low-dimensional code, and VAE (Variational) of a deep generative model is described. A method of expression learning using Autoencoder) is disclosed. In the method disclosed in Non-Patent Document 3, in order to learn the environmental model, first, the experience is collected by using a random policy in advance, and the visual module of VAE is trained to extract the visual features from the image data. After a few epochs, all image data is converted to the latent variable space of VAE, the previous state and action are input, and learning is performed to predict the end of the next state and episode. Since VAE is used to acquire spatial features in the method disclosed in Non-Patent Document 3, for example, there are many meaningless expressions in an environment where there is little change in a huge image, and there is a problem that correct learning cannot be performed. There is also a very wide range of searches, and as a result, improvements are needed in terms of data efficiency and reduction of the number of resets of the environment, and there is a problem that learning efficiency is poor. In addition, in the method of Non-Patent Document 3, more experience is required to create an accurate model of the environment than an agent who has no experience in the actual environment directly learns the policy, and efficient reinforcement learning is required. There is a problem that it is hard to say.

Non-Patent Document 4 discloses an expression learning method that can accelerate the search for a place by adjusting the reward in reinforcement learning. Unlike the method of Non-Patent Document 3, the method disclosed in Non-Patent Document 4 does not learn the model of the environment using big data, but the calculation cost is high as in the methods of Non-Patent Documents 1 and 2. There are problems such as high price and low versatility.

K. Polymenakos et al., “Safe Policy Search with Gaussian Process Models”, arXiv: 1712.05556, In NIPS, Dec 2017. B. Eysenbach et al., “Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning”, arXiv: 1711.06782, In NIPS, Nov 2017. D. Ha et al., “World models”, arXiv: 1803.110122, In NIPS, May 2018. D. Pathak et al., “Curiosity-driven Exploration by Self-supervised Prediction”, arXiv: 1705.05363, In NIPS, May 2017.

As described above, agents in reinforcement learning and deep reinforcement learning repeatedly reset the environment due to negative behavior such as failure to learn the optimum behavior, and evaluate it based on the performance and time cost of the finally obtained behavior. Will be done. However, when considering application to the real world, the cost of resetting the environment such as failure (accident in the case of automatic driving, time for recovery of object destruction or robot damage in the case of robot arm) (Physical cost) cannot be ignored.
On the other hand, there are situations in which an agent encounters a situation in which a positive action such as acquisition of a score-up item in an electronic game or a driving operation (so-called eco-driving) for improving fuel efficiency in automatic driving is performed.
In view of this situation, the present invention uses the cost of resetting the environment such as failure as the evaluation axis, builds a high-performance agent with a smaller number of resets, and builds an agent that can actively take positive actions. The purpose is to provide reinforcement learning methods, reinforcement learning devices, and reinforcement learning programs that specialize in expression learning in situations where attention situations such as environmental resets have occurred so that learning can be performed efficiently. To do.

In order to solve the above problem, the enhanced learning method of the present invention is one of the enhanced learning methods for optimizing the behavioral policy of the agent from the result of learning using the learning device that learns based on the state observed from the environmental data. During the learning of the environmental data in the episode, the attention situation occurred when the determination step of determining whether or not the state in which the preset attention situation occurred was observed, and when the state in which the attention situation occurred was observed. It is provided with a learning step of comparing the characteristics of the first environment data of the first state with the characteristics of the second environment data of the second state that goes back to the past from the first state and letting the learner learn the difference between the characteristics. .. The learning step is composed of the following steps a) to c).

a) First learning step In the first learning step, the first environment data of the first state in which the attention situation occurs, the second environment data of the second state that goes back to the past from the first state, the first state, and the first state Each action and reward in the two states are input, and the first learning device, which is the learning device, is made to learn.
b) Second learning step In the second learning step, each feature data of the first environment data and the second environment data output by the first learning device is input, and the difference between the respective feature data is calculated by the second learning device. To learn.
c) Update step The update step is based on the output estimation data and the actual data of the second learner regarding the behavior and reward in the second state and the occurrence or absence of the attention situation in the first state, and the first learner and the first. 2 The parameters of the learner are updated.

According to the reinforcement learning method of the present invention, by learning the probability that an attention situation occurs, it is possible to construct an agent that sensitively avoids or actively faces the attention situation. According to an agent who is sensitive to the attention situation, it is possible to promote the learning of the agent while reducing the frequency of encountering a failure situation (= attention situation) such as an accident in the real space. This makes it possible to build agents of the same or better quality with a small number of episodes. On the other hand, according to agents who actively face attention situations, promoting agent learning while increasing the frequency of encounters with success situations (= attention situations) such as acquisition of objects in real space. Can be done. This makes it possible to build agents of the same or better quality with a small number of episodes.
The attentional situation in the reinforcement learning method of the present invention is a remarkable situation that occurs in the environment of the episode, for example, resetting the environment in the episode, game over in an electronic game, occurrence of a prelude to a disaster or failure, and the like. There are attentions to negative factors that should be avoided, acquisition of score-up items in electronic games, and attention to positive factors such as driving operations that improve fuel efficiency in automatic driving (so-called eco-driving).

In the reinforcement learning method of the present invention, the features of the environment are extracted in the first learning step of a) above, and the state classification of the presence or absence of the occurrence of the attention situation is performed in the second learning step of b) above. Then, the setting parameters of the first learner and the second learner are updated according to the configuration of the update step in c) above. As a result, it is possible to increase or decrease the frequency of encountering attention situations and maximize the efficiency of each episode. Specifically, when a attention situation such as an environment reset occurs during reinforcement learning by a determination step of determining whether or not a state in which a preset attention situation has occurred has been observed, a buffer memory or the like is used. The environment data of the previous time is acquired from the memory unit, and expression learning is executed using the two environment data of the state in which the attention situation has occurred and the environment data of the previous time. As a result, for the attention situation that has occurred once, by extracting each feature from the two environmental data and learning the feature of the difference so that the essential cause of the occurrence can be understood as a feature quantity. After that, it is possible to construct a feature extractor for reinforcement learning that performs processing so that the attention situation does not occur when similar environmental conditions are encountered.

According to the reinforcement learning method of the present invention, it can be applied not only to mathematical models but also to general problems (automatic driving, robot control, etc.) and in an environment with many meaningless expressions (for example, background wall color). Even in an environment where the pattern is complicated, it is possible to learn only the essential cause of the occurrence of the attention situation, so that versatility can be improved.
Further, according to the reinforcement learning method of the present invention, even as a method of using big data for learning, learning specialized only when a attention situation occurs can be performed, so that the data efficiency of the episode is also excellent.

In the reinforcement learning method of the present invention, the second state that goes back to the past from the first state in which the attention state occurred is the state observed at the time t-1 immediately before the time t when the first state was observed. Further, in the determination step, when the state in which the attention state is generated is observed, the second environmental data of the state observed at time t-1 and the memory unit in which the action and the reward are stored are stored at time t-1. Obtain second environmental data, actions and rewards. Further, in the first learning step, the characteristic data h and h'of the first environment data and the second environment data are extracted from the first environment data at time t and the second environment data at time t-1. Further, in the second learning step, the behavior and reward between the time t-1 and the time t and the presence / absence of the attention state at the time t are estimated from the feature data h, h'.
Then, in the update step, the estimated result is compared with the actual action and reward between the time t-1 and the time t and the attention state at the time t using the loss function, and the value of the loss function becomes smaller. As described above, in each step of the first learning step, the second learning step, and the update step, the parameters are repeatedly updated until the performance of reinforcement learning becomes equal to or more than the threshold value or the number of repetitions becomes equal to or more than a predetermined number of times. It should be noted that the condition that the value of the loss function is equal to or less than a predetermined threshold value may be used for the repeated condition determination for updating the parameter.
According to the reinforcement learning method of the present invention, when the attention situation that occurs once encounters the same environmental data, behavior, and reward environmental conditions as when it occurred after the occurrence, the attention situation is less likely to occur. A first learning device is constructed that facilitates the positive generation of attention.

Here, the environmental data includes image data of electronic games and in-vehicle cameras, data of various sensors used for automatic driving control of vehicles, and the like.
In particular, when the environment data is image data, the first learning step causes the first learning device to learn the image data to be input by using the convolutional neural network. Further, in the second learning step, the second learning device is made to learn the image feature data to be input by using the neural network. Then, in the update step, the output estimation data of the second learner and the actual data are compared using the loss function, and the optimization function is used so that the value of the loss function becomes smaller, and the first learner and the second learner are used. Update the learner weight parameter.

Further, the attention situation in the reinforcement learning method of the present invention is specifically a situation corresponding to an environment reset or an environment reset at the end of an episode. Here, the environment reset is the reset of the environment of the episode, and the situation corresponding to the environment reset is the game over in the electronic game, the accident or the vehicle failure in the automatic driving, the destruction of the object in the case of the robot arm, or the robot. Damage, emergency stop button operation, etc.

Next, the reinforcement learning device of the present invention will be described.
The enhanced learning device of the present invention is an enhanced learning device that optimizes the behavioral policy of an agent from the result of learning using a learning device that learns based on the state observed from the environmental data, and is learning the environmental data in one episode. In addition, when a state in which a preset attention situation occurs is observed, the characteristics of the first environmental data of the first state in which the attention situation occurs and the second state of the second state that goes back to the past from the first state. It is equipped with a learning device that compares the characteristics of environmental data and learns the differences between the characteristics. The learner includes the following A) to C) that function when a state in which an attention situation occurs is observed.

A) First learner The first learner is the first environmental data of the first state in which the attention situation occurs, the second environmental data of the second state that goes back to the past from the first state, and the first state and the first. It is the above-mentioned learning device that inputs and learns each action and reward in two states.
B) Second learner The second learner is a learner that inputs the feature data of the first environment data and the second environment data output by the first learner and learns the difference between the feature data. is there.
C) Update unit The update unit uses the first learner and the first learner based on the output estimation data and the actual data of the second learner regarding the behavior and reward in the second state and whether or not the attention situation occurs in the first state. 2 The parameters of the learner are updated.

In the above-mentioned reinforcement learning device of the present invention, the second state that goes back to the past from the first state in which the attention situation occurred is the state observed at the time t-1 immediately before the time t when the first state was observed. is there. When the state in which the attention situation is generated is observed, the determination unit is the second environment data of the state observed at time t-1 and the memory unit in which the action and the reward are stored. 2 Acquire environmental data, actions and rewards. The first learner extracts the characteristic data h, h'of the first environment data and the second environment data from the first environment data at time t and the second environment data at time t-1. The second learner estimates the behavior and reward between the time t-1 and the time t from the feature data h, h', and whether or not the attention state at the time t occurs. The update unit compares the estimated result with the actual behavior between time t-1 and time t, the reward, and the attention state at time t, using the loss function. Then, the parameters are repeatedly updated so that the value of the loss function becomes smaller until the performance of reinforcement learning becomes equal to or more than the threshold value or the number of repetitions becomes equal to or more than a predetermined number of times. As a result, when the environmental data, behavior, and reward environmental conditions similar to those at the time of occurrence are encountered for the attention situation that occurred once, the attention situation is less likely to occur or the attention situation is positively generated. A first learner is constructed to facilitate.

Here, specifically, the environmental data is image data. Further, the first learner is configured by using a convolutional neural network, and causes the first learner to learn the image data to be input. The second learner is configured by using a neural network, and causes the second learner to learn the image feature data to be input. Then, in the update step, the output estimation data of the second learner and the actual data are compared using the loss function, and the optimization function is used so that the value of the loss function becomes smaller, and the first learner and the second learner are used. Update the learner weight parameter.

The attention situation in the reinforcement learning device of the present invention is a situation corresponding to an environment reset or an environment reset at the end of an episode. Similar to the above-mentioned reinforcement learning method, the situations corresponding to the environment reset are game over, accidents and vehicle failures in automatic driving, destruction of objects and robots in the case of robot arms, and emergency stop button operation.

The reinforcement learning program of the present invention is a program for causing a computer to execute each of the determination step, the first learning step, the second learning step, and the update step in the above-mentioned reinforcement learning method of the present invention. Further, the reinforcement learning program of the present invention is a program for operating a computer as a determination unit, a first learning device, a second learning device, and an update unit in the above-mentioned reinforcement learning device of the present invention.

According to the present invention, it is possible to construct a high-performance agent with a smaller number of resets, and to construct an agent capable of positively acting positively, and to efficiently perform reinforcement learning.

Reinforcement learning method and processing explanatory diagram of reinforcement learning device Input / output data flow diagram of the learner in the state where the attention situation does not occur Input / output data flow diagram of the learner when the attention situation occurs Functional block diagram of the learner of the reinforcement learning device Processing flow diagram of reinforcement learning method Reinforcement learning processing flow diagram in the processing flow of the reinforcement learning method Processing flow diagram of the reinforcement learning method of Example 1 Explanatory diagram of neural network of image feature extractor and state classifier Input / output data flow diagram of the learner after reinforcement learning Graph showing the efficiency of episodes in the reinforcement learning method of Example 1 Input / output data flow diagram of the reinforcement learning method of Example 2 Processing flow diagram of the reinforcement learning method of Example 2

Hereinafter, an example of the embodiment of the present invention will be described in detail with reference to the drawings. The scope of the present invention is not limited to the following examples and illustrated examples, and many modifications and modifications can be made.

The reinforcement learning method and the processing of the reinforcement learning device of the present invention will be described with reference to FIGS. 1 to 3.
In the processing of the reinforcement learning method and the reinforcement learning device of the present invention, as shown in FIG. 1, an agent (not shown) in the environment 2 should observe and take the current state as in the case of general reinforcement learning. A policy 3 for determining an action is set, and an action is selected from the policy 3. The agent gets a reward from the environment 2 by selecting an action from the measure 3 so that the value of the value function becomes large. We will learn such interaction between environment 2 and agents while feeding back using three variables of reward, behavior, and state.

The environment 2 is, for example, a set of image data of the screen of an electronic game, a time series of image data of a moving image taken by a camera mounted on a robot or a vehicle, and the like, and a state is a set of image data. It is individual image data obtained by observing a certain environment 2. Further, the action is, for example, an input device operation of a game in the case of an electronic game, a steering wheel operation in the case of automatic driving of an automobile, or the like.

In the reinforcement learning method and the processing of the reinforcement learning device of the present invention, when the attention situation does not occur, that is, when the attention situation does not occur in the observed image data, the learner 1 observes as shown in FIG. capture image data x _t (10), through the determination unit 11 whether or not to generate the attention status, feature extractor (first learning device) 13 by extracting the feature of the image data. In the feature extractor (first learner) 13, the extracted features are output as vector data h (14), and in the value function computer 17, vector data h (15) is input (16), and the value function 19 Is output (18).

The attention situation is a preset notable situation such as a game over in an electronic game, and the information on whether or not the attention situation has occurred is included in the [actual data] linked to the image data captured in FIG. It is provided as flag information (1/0) at the end of the episode t _t . With reference to the flag information of the episode end t _t, the determination unit 11 determines whether or not the attention state has occurred. Flag information for this episode termination t _t is generally reinforcement learning is a state variable that is not used.

Then, a state in which attention condition occurs, i.e., when the target situation in the observed image data is generated, learning device 1, as shown in FIG. 3, takes in the observed image data x _t (10), the image associated data referring to the flag information of the episode termination t _t in [actual data] is checked by the determination unit 11 that the attention condition occurs. Then, when it is determined that the attention situation has occurred, the observed image data x _t is read from the memory unit 4 in which the observed state is stored immediately before the time t, that is, at the time t-1 immediately before. (22), and the image data x _{t-1 at} time t-1 and the action a _t-1 and the reward r _t-1 are acquired (23).

In the feature extractor (first learner) 13, the image data x _{t from} the image data x _t as the state observed at the time t and the image data x _t-1 as the state observed at the time t-1 Vector data h, h'showing the characteristics of _t and the image data x _t-1 are extracted and output (14, 24).
In the state classifier (second learner) 23, the time t-1 and the time t are obtained from the vector data h (15) and h'(25) indicating the characteristics of the image data x _t and the image data x _t-1. action _{a t-1} between, to estimate the flag information of the reward _{r t} and the episode end _{t t} of the time t.

The updating unit 28, an action a _t-1 between time t-1 and time t, and the result of the flag information estimated reward r _t and episode termination t _t at time t, the actual time t-1 and time action a _t-1 between _t, and a flag information of the reward r _t and the episode end _t t of the time t, compared with a loss function. Then, the parameters of the feature extractor (first learner) 13 and the state classifier (second learner) 23 are updated so that the value of the loss function becomes smaller. The parameter update is repeated until the value of the loss function is less than the predetermined threshold value or the number of repetitions becomes the predetermined number or more.
In the update unit 28, the result of estimating the action a _t-1 and the reward r _t-1 between the time t _-1 and the time t, the flag information of the episode end t _t of the time t, and the actual time t. The action a _t-1 and reward r _t-1 between -1 and time t may be compared with the flag information of the episode end t _t at time t using a loss function.

In the processing of the reinforcement learning method and the reinforcement learning device of the present invention, the same environmental data, actions, and rewards as those at the time of occurrence after the occurrence of the attention situation that occurred once by the above processing performed when the attention situation occurs When the environmental conditions of are encountered, the learning device will be constructed so that the attention situation is unlikely to occur.

FIG. 4 shows a functional block of the learning device of the reinforcement learning device of the present invention. The determination unit 11, the feature extractor (first learner) 13, the value function calculator 17, the state classifier (second learner) 23, and the update unit 28 in the functional block perform the above-described processing. The environmental data acquisition unit 9 captures environmental data as a state observed in the environment from the environment 2 and the memory unit 4 in FIG. 1, for example, image data as described above.

The processing flow of the reinforcement learning method of the present invention will be described with reference to FIGS. 5 and 6. In the reinforcement learning method of the present invention, as shown in FIG. 1, normal reinforcement learning is performed in which the interaction between the environment 2 and the agent is learned while feeding back using three variables of reward, behavior, and state (S10). It is determined whether or not the attention situation occurs (S11), and if the attention situation does not occur, the reinforcement learning (S10) is repeated. On the other hand, if there is a state in which attention status occurs while repeating reinforcement learning (S10), environmental data, rewards, and actions a certain time before the occurrence time of the attention status are acquired from the memory unit 4 (S12), and attention is paid. Each vector data showing each feature is extracted from the environmental data at the time when the situation occurs and the environmental data before a certain time (S13). Then, based on the difference between the vector data of the extracted features, the occurrence status of the behavior, reward, and attention status is estimated (S14), and based on these estimation results and the actual data of the actual behavior, reward, and attention status. , Update the parameters of the learner (S15). The parameters of steps S13 to S15 are updated repeatedly until the learning performance exceeds a predetermined threshold value and other conditions are satisfied (S16).

Here, the processing flow of the above-mentioned normal reinforcement learning (S10) is shown in FIG. The environmental data is observed, the environmental data and the reward are acquired (S101), and the characteristics of the environmental data are extracted (S102). The vector data indicating the extracted features is input to the value function calculator (S103), and the value function calculator outputs the total sum (value function) of the rewards that can be obtained in the future from the state and the reward (S104). .. The action is selected according to the value function and the policy (S105), and the parameters of the value function calculator are updated based on the reward obtained from the environment (S106). It is determined whether or not the end of the episode has been reached (S107), and if the end of the episode has not been reached, the time is advanced by one (S108), and the environmental data and reward from the next observation are obtained (S101). S101 to S106 are repeated. On the other hand, if the end of the episode is reached, the episode is switched (S109), and the reinforcement learning for optimizing the action policy of the agent is terminated from the result of learning using one episode.

FIG. 7 shows a processing flow diagram of the reinforcement learning method of the first embodiment. The reinforcement learning method of FIG. 7 shows an example of a processing flow for avoiding resetting the environment of attention, for example, a game over of an electronic game or an accident of driving a vehicle. The environmental data is an image of an electronic game or an image data of an in-vehicle camera or the like.
Reinforcement learning method of FIG. 7, normal performs reinforcement learning (S10) of the process flow of FIG. 6, the state flags of one episode termination t _t is determined whether no reset (0) or reset (1) (S21) If there is no reset, the reinforcement learning (S10) is repeated.
On the other hand, if there is a state in which a reset occurs while repeating reinforcement learning (S10), the time t when the episode end t _t is “reset” is set as the time t, and the image at the time t-1 that goes back to the past from the reset time. The data x _t-1 , the reward r _t-1, and the action a _t-1 are acquired from the memory unit (S22), and each image is taken from the image data x _t at time t and the image data x _{t-1 at} time t-1. Each vector data h, h'showing the characteristics is extracted (S23). Then, the extracted h, than h', action _{a t-1} between time t and time t-1, estimates the reward _{r t} and episode termination _{t t} at time t (output estimation situation results) to (S24), with loss functions, action _{a t-1,} reward _{r t,} compares the estimated status results and real data episode termination _{t t} (S25). Then, the parameters of the image feature extractor (first learner) and the state classifier (second learner) are changed so that the value of the loss function becomes smaller (S26). The parameters are updated in steps S23 to S26 until the learning performance exceeds a predetermined threshold value or the number of repetitions exceeds a predetermined number (S27).

The first learning device and the second learning device in the reinforcement learning method of FIG. 7 will be described with reference to FIG. In the reinforcement learning method of FIG. 7, since image data is treated as environmental data, a convolutional neural network (CNN) can be used for both the first learning device and the second learning device. The first learner is an image feature extractor that extracts each vector data h, h'indicating each image feature from the image data x _t at time _t and the image data x _{t-1 at} time t-1. 2 learner may act a _t-1, a state classifier for comparing the estimated status results and real data of the reward r _t and episode termination t _t.

FIG. 9 shows an input / output data flow diagram of the learner after reinforcement learning.
After reinforcement learning is not able to get past environmental data (such as image data), captures the image data x _t as the current state from the environment 2 (12), and inputs to the feature extractor 13. The determination unit 11 (not shown) is not used and is unnecessary. The feature extractor 13 outputs vector data h (15) indicating the features of the environmental data (image data) (14), inputs the feature data h to the value function computer 17 (16), and changes the state and reward into the future. It outputs a value function that represents the sum of the rewards that can be obtained over time (18). The agent selects the optimal action according to the value function and policy 3.

FIG. 10 is a graph showing the efficiency of episodes when the reinforcement learning method of Example 1 is used. The environment used for the evaluation was HERO of the Atari game domain. It is a display image of an electronic game. HERO. In order to successfully perform these tasks, it is necessary to balance risk and reward.
HERO. The goal is for the agent to manipulate the human character and rescue the miner trapped underground. The player character's equipment is a helicopter (which can fly but cannot jump), a laser attached to the helmet, and a limited number of explosives. There are three ways agents can get points (rewards): kill enemies with a laser, destroy a cave-in with explosives, and rescue a miner. Also, when the miner is rescued, the remaining time limit and explosives at the current level are converted into points, and the game moves to the next level.

OpenAI Baselines (http://github.com/openai/baselines) was used as the open source for deep learning, A2C was used as the reinforcement learning implementation of OpenAI, and the hyperparameters were basically the same. In addition to A2C, one linear hidden layer was added to each of the goals of expression learning.
Experiments were conducted with different weights to evaluate the effects of expression learning schemes on learning. Negative rewards were not set and binary classification was used instead of reward classifier ternary. The value function calculator, a behavior classifier, was optimized with softmax cross-entropy loss. The feature extractor (first learner) and the state classifier (second learner) were optimized with mean-square loss. Reward and end-of-episode state sampling were balanced.
At the beginning of the learning, as a data collection period, we decided to learn only the measures during 16 episodes, and then all the goals were optimized. Since memory usage can be very high, we decided to run the experiment until the first of 5000 episodes or 10 million (post-skipping) frames. All experiments were performed up to 4 times to assess the stability of the method.

FIG. 10 shows the results of an experiment of expression learning performed with different weights (Weight: ratio of learning image features) (reinforcement learning ratio is fixed at 1). For each condition, the mean and standard deviation of the scores of the four experiments are represented by straight lines and grayscale colored areas, respectively. The combination of learning goals in the present invention improves the learning speed and score of A2C. In particular, when the weight is 2, the average score at the end of 5000 episodes is close to 10,000 (10,000 episodes were required when the original A2C scored this level). In addition, it was confirmed that as the weight increases, the improvement from the existing method (Control: Weight = 0) also increases.
From the above experiments, it is shown that the learning goal of the present invention can improve the learning speed and score of the deep reinforcement learning algorithm. In addition, this result once again shows the importance of a good feature extractor for deep reinforcement learning.

The reinforcement learning method and device of Example 1 were related to reinforcement learning in a dynamically changing real environment, but in the reinforcement learning method and device of this example, reinforcement learning using past environmental data is used. This is a method, which will be described below.
In the reinforcement learning method and device of this embodiment, data stored in the past such as behavior, reward, and environmental data when trained in reinforcement learning in a real environment is used. For example, by parallel processing, the process of collecting learning data (environmental data, etc.) at the time of reinforcement learning in the real environment and saving it in the memory, and the process of reading the collected data (environmental data, etc.) saved in the memory, and environmental data The learner is trained by expression learning based on the extracted features. As an example of parallel processing, the process of playing an electronic game and collecting data and the process of training a learner by reinforcement learning and expression learning based on the collected data may operate completely independently. (Multiprocessing). In this case, the parameters learned in reinforcement learning and expression learning are periodically copied to the parameters of the process to be collected. The reinforcement learning method and device of this embodiment randomly acquire a large amount of data, and perform reinforcement learning and expression learning using all the data regardless of the presence or absence of reset.

FIG. 11 shows an input / output data flow diagram of the reinforcement learning method of the second embodiment, and FIG. 12 shows a processing flow diagram of the reinforcement learning method of the second embodiment.
As shown in FIGS. 11 and 12, in the reinforcement learning method of the second embodiment, instead of acquiring and learning the environmental data from the actual environment, the past actions, rewards and states stored in the memory unit 4 are acquired and learned. To do. The configuration of the learner 1 is the same as that of the first embodiment.
As shown in FIG. 12, in the processing flow of the reinforcement learning method of this embodiment, first, data is randomly acquired from the memory unit (S31). Here, the environment data (image data group) saved in each episode is randomly acquired, but the data is not limited to this, and may be data of a plurality of episodes or data of a certain time length.

After randomly acquiring data from the memory unit (S31), normal reinforcement learning of the processing flow of FIG. 6 is performed (S10).
Then, after reinforcement learning (S10), the corresponding image data _xt is acquired from the memory unit (S32). Subsequently, the image data x _t -1 at time _t-1 , the reward r _t-1, and the action a _t-1 related to the image data x _t at time t are acquired from the memory unit (S33).
Then, the time t image data _{x t} and time t-1 of the image data _{x t-1} from each respective vector data h indicating the image feature of the extracts h'(S34), the extracted h, than h', action _{a t-1} between time t and time t-1, estimates the reward _{r t} and episode termination _{t t} at time t (estimation conditions result output) to (S35), using a loss function, action _{a t- 1,} reward _{r t,} compares the estimated status results and real data episode termination _{t t} (S36). Then, the parameters of the image feature extractor (first learner) and the state classifier (second learner) are changed so that the value of the loss function becomes smaller (S37). The parameters of steps S34 to S37 are updated repeatedly until the learning performance exceeds a predetermined threshold value or the number of repetitions exceeds a predetermined number (S38).
As described above, a large amount of data is randomly acquired from the memory unit, and reinforcement learning and expression learning are performed using all the data regardless of whether or not there is a reset. The weights of reinforcement learning and expression learning are adjusted as appropriate for learning.

The present invention is useful for reinforcement learning in a wide range of fields such as automatic driving of automobiles and automatic control of industrial robot arms.

1 Learner 2 Environment 3 Measures 4 Memory section 11 Judgment section 13 Feature extractor (1st learner)
15, 25 Feature vector 17 Value function calculator 23 State classifier (second learner)
28 Update Department

Claims (12)

In the reinforcement learning method that optimizes the behavior policy of the agent from the result of learning using the learning device that learns based on the state observed from the environmental data
A determination step for determining whether or not a preset attention situation has occurred during the learning of environmental data in one episode, and a determination step.
When the state in which the attention situation occurs is observed, the characteristics of the first environmental data of the first state in which the attention situation occurs and the characteristics of the second environmental data of the second state that goes back to the past from the first state. A reinforcement learning method comprising a learning step of making the learner learn the difference between the features. The reinforcement learning method according to claim 1, wherein the learning step includes the following steps a) to c):
a) The first environmental data of the first state in which the attention situation occurred, the second environmental data of the second state retroactive from the first state, and the actions and rewards in the first state and the second state, respectively. The first learning step of inputting and making the first learning device which is the learning device learn,
b) A second learning step in which each feature data of the first environment data and the second environment data output by the first learner is input and the difference between the feature data is learned by the second learner.
c) Parameters of the first learner and the second learner based on the behavior and reward in the second state and the output estimation data and the actual data of the second learner regarding the occurrence or absence of the attention situation in the first state. Update step to update. The second state, which goes back to the past from the first state in which the attention state occurred, is a state observed at time t-1 immediately before the time t when the first state was observed.
In the determination step
When the state in which the attention situation occurs is observed, the second environmental data of the state observed at time t-1 and the second environmental data of the time t-1 from the memory unit in which the action and the reward are stored are stored. And get actions and rewards,
In the first learning step,
From the first environmental data at time t and the second environmental data at time t-1, the characteristic data h and h'of the first environment data and the second environment data are extracted.
In the second learning step,
From the feature data h, h', the behavior and reward between the time t-1 and the time t, and the presence or absence of the attention situation at the time t are estimated.
In the update step
The estimated result is compared with the actual action and reward between the time t-1 and the time t and the attention state at the time t using the loss function, and the value of the loss function is reduced. Each step of the first learning step, the second learning step, and the update step is repeatedly updated with the parameters until the performance of reinforcement learning is equal to or more than the threshold value or the number of repetitions is equal to or more than a predetermined number of times.
When the environmental data, behavior, and reward environmental conditions similar to those at the time of occurrence are encountered for the attention situation that occurred once, the attention situation is made difficult to occur or the attention situation is positively generated. The reinforcement learning method according to claim 2, wherein the first learning device is constructed to facilitate facilitation. The environmental data is image data and
In the first learning step, the convolutional neural network is used to make the first learning device learn the image data to be input.
In the second learning step, the second learning device is made to learn the image feature data to be input by using the neural network.
In the update step, the output estimation data of the second learner and the actual data are compared using the loss function, and the optimization function is used so that the value of the loss function becomes smaller, and the first learner and the second learner are used. Update the vessel weights parameter,
The reinforcement learning method according to claim 2 or 3, characterized in that. The reinforcement learning method according to any one of claims 1 to 4, wherein the attention situation is a situation corresponding to an environment reset or an environment reset at the end of an episode. In a reinforcement learning device that optimizes agent behavior strategies from the results of learning using a learning device that learns based on the state observed from environmental data.
A determination unit that determines whether or not a preset attention situation has occurred during learning of environmental data in one episode.
When the state in which the attention situation occurs is observed, the characteristics of the first environmental data of the first state in which the attention situation occurs and the characteristics of the second environmental data of the second state that goes back to the past from the first state. An enhanced learning device provided with the learning device for comparing and learning the difference between the features. The reinforcement learning device according to claim 6, wherein the learning device includes the following A) to C) that function when the state in which the attention situation occurs is observed.
A) The first environmental data of the first state in which the attention situation occurred, the second environmental data of the second state retroactive from the first state, and the actions and rewards in the first state and the second state, respectively. The first learning device, which is the learning device for inputting and learning, and
B) A second learner that inputs the feature data of the first environment data and the second environment data output by the first learner and learns the difference between the feature data.
C) Parameters of the first learner and the second learner based on the behavior and reward in the second state and the output estimation data and the actual data of the second learner regarding the occurrence or absence of the attention situation in the first state. Update department to update. The second state, which goes back to the past from the first state in which the attention state occurred, is a state observed at time t-1 immediately before the time t when the first state was observed.
The determination unit
When the state in which the attention situation occurs is observed, the second environmental data of the state observed at time t-1 and the second environmental data of the time t-1 from the memory unit in which the action and the reward are stored are stored. And get actions and rewards,
The first learner is
From the first environmental data at time t and the second environmental data at time t-1, the characteristic data h and h'of the first environment data and the second environment data are extracted.
The second learner is
From the feature data h, h', the behavior and reward between the time t-1 and the time t, and the presence or absence of the attention situation at the time t are estimated.
The update part
The estimated result is compared with the actual action and reward between the time t-1 and the time t and the attention state at the time t using the loss function, and the value of the loss function is strengthened so as to be small. The parameters are repeatedly updated until the learning performance exceeds the threshold value or the number of repetitions exceeds a predetermined number.
When the environmental data, behavior, and reward environmental conditions similar to those at the time of occurrence are encountered for the attention situation that occurred once, the attention situation is made difficult to occur or the attention situation is positively generated. The reinforcement learning device according to claim 7, wherein the first learning device is constructed to facilitate facilitation. The environmental data is image data and
The first learner is configured by using a convolutional neural network, and learns with input image data.
The second learner is configured by using a neural network, learns with input image feature data, and then learns.
The update unit compares the output estimation data of the second learner with the actual data using a loss function, and uses an optimization function so that the value of the loss function becomes smaller, and uses the first learner and the first learner. 2. The reinforcement learning device according to claim 7 or 8, wherein the weight parameter of the learner is updated. The reinforcement learning device according to any one of claims 6 to 9, wherein the attention situation is a situation corresponding to an environment reset or an environment reset at the end of an episode. A reinforcement learning program for causing a computer to execute the determination step, the first learning step, the second learning step, and the update step in the reinforcement learning method according to any one of claims 2 to 5. A reinforcement learning program for operating a computer as the determination unit, the first learning device, the second learning device, and the updating unit in any of the reinforcement learning devices of claims 7 to 10.