JP2018005739A - Method for learning reinforcement of neural network and reinforcement learning device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for learning neural network reinforcement with which it is possible to effectively prevent the neural network from being overfitted when learning neural network reinforcement.SOLUTION: A data deletion unit 15 calculates a uniqueness parameter pertaining to experience data stored in an experience data storage unit 13 that indicates a degree of difference of each experience data from the other. The data deletion unit 15 deletes experience data similar to the other experience data on the basis of the calculated uniqueness parameter. Thus, it is possible to prevent the experience data stored in the experience data storage unit 13 from being one sided to experience data of high similarity. Therefore, it is possible to effectively prevent the overfitting of a DNN 12 by learning enforcement of the DNN 12 using a variety of experience data stored in the experience data storage unit 13.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a reinforcement learning method and reinforcement learning apparatus for reinforcement learning of a neural network when a neural network is used as a learning device for learning an optimal policy for determining an action according to a state of a control target.

  For example, Non-Patent Document 1 describes a method of reinforcement learning when a neural network having a multilayer structure is used as a function approximator of an optimal action value function.

  Reinforcement learning is a machine learning technique that enables an agent placed in a certain environment to obtain an optimal policy through interaction with the environment. Specifically, the agent observes the current state of the environment and determines an action to be taken based on the policy. The state of the environment changes according to the action determined by the agent. Rewards are determined according to how the state of the environment has changed. In reinforcement learning, an agent's policy is learned so as to determine an action that can obtain the most reward through a series of actions. TD learning and Q learning are known as typical methods of reinforcement learning.

  In Non-Patent Document 1, reinforcement learning is performed by a method based on Q-learning. In the method based on Q-learning, an optimal policy is learned by approximating a function called an action value function. In other words, an approximate function of an action value function that maximizes the sum of cumulative rewards in the future is learned as an optimal policy. In Non-Patent Document 1, a multi-layered neural network is used as a learning device for this optimal policy.

To briefly explain the reinforcement learning method of the neural network having a multilayer structure in Non-Patent Document 1, first, the state of the environment described above, the action for the state, the reward obtained by the action, and the state of the environment transitioned by the action are described. Collect them and store them as experience data in a predetermined memory. In reinforcement learning, experience data is sampled from the memory, and a teacher signal is created according to the following Equation 1.

In Equation 1, r represents a reward, γ represents a parameter of reinforcement learning called a discount rate (0 <γ <1), and Q _θ (s, a) is an action expressed using the parameter θ of the neural network. An approximate function of the value function is shown, s ′ represents the next state when the action a is taken in the state s, and a ′ represents the next action to be taken in the next state s ′.

By using this teacher signal target, the error function can be determined as in the following Equation 2.

Then, the back propagation method is applied to the neural network to update the weight of each neuron. As a result, when it is determined that the error function L _θ (s, a) has become sufficiently small, the learning ends.

  Here, there is a problem that if the neural network is learned so as to be excessively adapted to certain kinds of experience data by reinforcement learning, the neural network cannot be adapted well to new data. . That is, there is a problem that the neural network cannot determine an action for obtaining the maximum reward for data having a tendency different from that of the experience data used for learning. One of the causes of such overfitting is the correlation of continuously observed experience data. Continuously observed empirical data usually does not change significantly and has a certain correlation. Therefore, when reinforcement learning is performed using experience data having such correlation, the learning result is biased by the correlation.

  In order to deal with such a problem, Non-Patent Document 1 uses a technique called “Experience Replay”. “Experience Replay” stores experience data in various situations experienced by an agent in a memory, and randomly samples the experience data from the memory during reinforcement learning. Thereby, the correlation of experience data is reduced and it can suppress that a learning result receives a bias.

“Human-level control through deep reinforcement learning”, Volodymyr Mnih, et al., Nature, vol. 518, no.7540, pp.529-533, 2015

  As described above, in order to execute “Experience Replay”, it is necessary to store the experience data experienced by the agent in a memory. However, the memory cannot store the experience data indefinitely. When the storage amount of the experience data reaches the upper limit value of the storage capacity of the memory, it is necessary to delete any experience data. At this time, the oldest experience data is generally deleted by the “First In First Out (FIFO)” method.

  However, when the experience data is deleted by the FIFO method, as the learning progresses, the variation of the situation faced by the agent decreases. Therefore, the experience data with low similarity is deleted while the experience data with high similarity is newly added. The possibility of being preserved increases. As a result, the ratio of highly similar experience data increases as the entire experience data stored in the memory. For this reason, even if “Experience Reply” is executed, there is a high possibility that experience data with high similarity will be sampled as experience data for learning. There is a tendency to overfit.

  The present invention has been made in view of the above-described points, and provides a reinforcement learning method and reinforcement learning device for a neural network that can effectively prevent excessive adaptation of the neural network when performing reinforcement learning on the neural network. The purpose is to provide.

In order to achieve the above object, the reinforcement learning method of the neural network (12) according to the present invention uses the neural network (12) as a learning device for learning an optimal policy for determining an action according to the state of the controlled object. In some cases, reinforcement learning of the neural network (12),
The computer (10) collects experience data including the state of the controlled object, the action for the controlled object, the reward obtained by the action, and the state of the controlled object changed by the action, and the experience data having a finite storage capacity Store in the storage unit (13),
The computer (10) calculates a uniqueness parameter indicating how different each experience data stored in the experience data storage unit (13) from other experience data,
Based on the calculated uniqueness parameter, the computer (10) deletes experience data similar to other experience data from the experience data storage unit (13),
The computer (10) performs reinforcement learning of the neural network (12) using the experience data stored in the experience data storage unit (13).

Further, the reinforcement learning device of the neural network (12) according to the present invention is:
Each time empirical data is collected that includes the state of the controlled object, the action for the controlled object, the reward obtained by the action, and the state of the controlled object that has been transitioned by the action, a finite storage capacity is stored to store the experience data. Having an experience data storage unit (13);
For each experience data stored in the experience data storage unit, a calculation unit (S200) that calculates a uniqueness parameter indicating how different from other experience data;
A deletion unit (S210) that deletes experience data similar to other experience data from the experience data storage unit based on the uniqueness parameter calculated by the calculation unit;
A reinforcement learning unit (11) that performs reinforcement learning of the neural network using the experience data stored in the experience data storage unit.

  As described above, in the neural network reinforcement learning method and reinforcement learning apparatus according to the present invention, the uniqueness parameter indicating how different each experience data stored in the experience data storage unit is from other experience data. calculate. Based on the calculated uniqueness parameter, experience data similar to other experience data is deleted from the experience data storage unit. Thereby, it is possible to prevent the experience data stored in the experience data storage unit from being biased toward highly similar experience data. In other words, experience data having a low similarity with other experience data, that is, highly unique experience data is left in the experience data storage unit without being deleted. Therefore, when the experience data stored in the experience data storage unit is plotted in a space centered on the elements of the experience data, the experience data is distributed over a wide range and the occurrence of extreme differences in distribution density is also suppressed. Is done. Therefore, by performing reinforcement learning of the neural network using such widely distributed experience data, it is possible to effectively prevent overfitting of the neural network.

  The reference numerals in the parentheses merely show an example of a correspondence relationship with a specific configuration in an embodiment described later in order to facilitate understanding of the present invention, and are intended to limit the scope of the present invention. Not intended.

  Further, the technical features described in the claims of the claims other than the features described above will become apparent from the description of embodiments and the accompanying drawings described later.

It is the figure which showed notionally the structure of the reinforcement learning apparatus of the neural network which concerns on embodiment. It is the flowchart which showed the process for an agent to collect experience data and to preserve | save in an experience data memory | storage part. It is a flowchart which shows the detail of the data deletion process performed by a data deletion part, when an experience data storage part becomes full. It is explanatory drawing for demonstrating an example of the calculation method of a uniqueness parameter. (A) is a figure which shows the mode of dispersion | distribution of experience data when the experience data storage part is almost full in a FIFO system, but has not deleted data yet, (b) is a figure which shows FIFO It is a figure which shows the mode of dispersion | distribution of the experience data preserve | saved in the experience data storage part, after repeating an episode 7000 times, deleting experience data by a system. (A) is a figure which shows the mode of dispersion | distribution of the experience data of the initial stage when the experience data storage part is almost full when performing data deletion by the data deletion process of embodiment, (b) It is a figure which shows the mode of distribution of the experience data preserve | saved in the experience data memory | storage part, after repeating an episode 7000 times, performing data deletion by the data deletion process by embodiment. If you delete data using the FIFO method and repeat DNN reinforcement learning using experience data stored in the experience data storage unit, it will be obtained during the execution of those episodes every time 5 episodes are completed. It is a figure which shows the result of having counted the accumulated value of the received reward. When DNN reinforcement learning is repeated using experience data stored in the experience data storage unit while data is deleted by the data deletion processing according to the present embodiment, the episodes are completed every time five episodes are completed. It is a figure which shows the result of having counted the cumulative value of the reward obtained during execution.

  A neural network reinforcement learning method and learning device according to an embodiment of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a diagram conceptually illustrating the configuration of a reinforcement learning device for a neural network according to the present embodiment. In the present embodiment, a reinforcement learning device for a neural network is implemented by executing an application having a learning function for a neural network on the computer 10. In FIG. 1, various functions realized by a computer by executing an application are shown as blocks.

  As shown in FIG. 1, the neural network reinforcement learning apparatus includes an agent 11 and an environment 14.

  The agent 11 includes a neural network (Deep Neural Network: DNN) 12 having a multilayer structure as a learning device, and an experience data storage unit 13. The agent 11 collects experience data for reinforcement learning and stores it in the experience data storage unit 13.

In the collection of empirical data, agent 11, at a point in time t, to observe the state s _t environment 14 as a control object, select an action a _t based on the state s _t. At this time, the action a based on the state s is selected at a certain ratio using an action selection method such as the ε-greedy method or the Boltzmann selection method so that various pairs of the state s and the action a are selected. The For example, immediately after learning is started, the action a is determined by the above-described action selection method at a ratio of 95%, and the action a is determined at the remaining 5% according to the policy that the DNN 12 has at that time. To do. When the state s is input, the DNN 12 is configured to output the highest evaluation value with respect to the action a most suitable for the state s based on the policy possessed.

  Then, as reinforcement learning proceeds, the agent 11 gradually decreases the ratio of determining the action a by the above-described action selection method, and increases the ratio of selecting the action a by the DNN 12 policy. For example, every time learning is performed, the ratio of action selection by the action selection method is reduced by 99%. When the action selection ratio according to the action selection method reaches a predetermined minimum ratio (for example, 5%), the further decrease is stopped.

Agent 11 gives the action _{a t} selected according to the state _{s t} at a certain point of time t in the environment 14. Environment 14, by current state _{s t} and selected the action _{a t,} then determines the state _{s t + 1} transition. For example, the environment 14, the current state _{s t} and action _{a t,} the probability of transition to the next state _{s t + 1,} reward r (expected value at the time of transition from the current state _{s t} to the next state _{s t + 1} ) Is calculated. The environment 14 gives the agent 11 the next state s _{t + 1} with the highest transition probability and the reward r expected at that time.

Agent 11, when the environment 14 receives the next state s _{t + 1} and rewards r, creating an empirical data in the state s _t and the action a _t the set of current time, stores the empirical data storage unit 13. In other words, each experience data e _i includes a current state s _t , an action a _t , a reward r _t , and a next state s _{t + 1} , and e _i = (s _t , a _t , r _t , s _{t + 1} ).

When the experience data e _i is accumulated in the experience data storage unit 13, the agent 11 randomly samples a predetermined number of experience data from the experience data storage unit 13 and executes reinforcement learning. For this reinforcement learning, a known method such as TD learning or Q learning is applied as in the prior art. Then, by using the empirical data of TD and Q learning, the error function is determined using the sampled experience data, the error back propagation method is applied to the DNN 12, and the weight of each neuron is updated. Perform reinforcement learning.

  Here, also in the present embodiment, experience data is randomly sampled from the experience data storage unit 13 for the reinforcement learning of the DNN 12, and the conventional “Experience Replay” is employed. Thereby, the correlation of experience data is reduced to some extent, and it is possible to suppress the learning result from being biased.

  However, the storage capacity of the experience data storage unit 13 is finite. Therefore, when the amount of experience data stored reaches the upper limit of the storage capacity of the experience data storage unit 13, in order to store new experience data, it is necessary to delete the already stored experience data. In this case, when the experience data is deleted by the so-called FIFO method, the variation of the situation that the agent 11 faces decreases as the learning progresses. Therefore, the experience data with low similarity is deleted while the experience data with low similarity is deleted. There is a high possibility that only data will be newly stored. As a result, the ratio of experience data with high similarity increases as the entire experience data stored in the experience data storage unit 13. For this reason, even if “Experience Reply” is executed, the neural network tends to be excessively adapted to the highly similar experience data.

  Therefore, in the reinforcement learning apparatus of the neural network according to the present embodiment, as shown in FIG. 1, the dissimilarity between each experience data and other experience data is evaluated instead of a simple FIFO method, and the dissimilarity is evaluated. Based on the above, a data deleting unit 15 for selecting and deleting the experience data is provided. More specifically, the data deleting unit 15 deletes the experience data having low dissimilarity with other experience data (that is, approximating other experience data), while having high dissimilarity (that is, Leave highly unique) experience data.

  Thereby, it is possible to prevent the experience data stored in the experience data storage unit 13 from being biased to highly similar experience data. In other words, experience data having a low similarity to other experience data, that is, a highly unique experience data is left in the experience data storage unit 13 without being deleted. Therefore, when the experience data stored in the experience data storage unit 13 is plotted in a multidimensional space with the elements of the experience data as an axis, the experience data is distributed over a wide range and an extreme difference occurs in the distribution density. This is also suppressed. Therefore, by performing reinforcement learning of the DNN 12 using such widely distributed experience data, excessive adaptation of the DNN 12 can be effectively prevented.

  Hereinafter, a method for storing and deleting experience data in the reinforcement learning apparatus for a neural network according to the present embodiment will be described in detail with reference to the flowcharts of FIGS. The flowchart of FIG. 2 shows a process for the agent 11 to collect experience data and store it in the experience data storage unit 13. In addition, the flowchart of FIG. 3 illustrates a data deletion process executed by the data deletion unit 15 when the experience data storage unit 13 is full.

First, the experience data storage process will be described with reference to the flowchart of FIG. In step S100 of the flowchart of FIG. 2, the agent 11 selects the action a for the observed state s and gives it to the environment 14. In step S110, the agent 11 from the environment 14 receives the next state _{s t + 1} and reward r, a state _{s t} and the action _{a t} and a set of current. Thereby, experience data is collected.

  Next, in step S120, it is determined whether the storage amount of the experience data has reached the upper limit of the storage capacity of the experience data storage unit 13 and the experience data storage unit 13 is full. If it is determined in step S120 that the experience data storage unit 13 is full, the process proceeds to step S130. In step S130, the deletion process of the experience data by the data deletion part 15 is performed. This experience data deletion process will be described later.

  Then, when the experience data is deleted by the data deletion process of step S130 and a free space for storing new experience data is secured in the experience data storage unit 13, the process of step S140 is executed. In step S140, the collected experience data is stored in the experience data storage unit 13. On the other hand, in the determination process of step S120, if it is determined that the experience data storage unit 13 is not full, the process directly proceeds to the process of step S140, and the agent 11 transfers the collected experience data to the experience data storage unit 13. Save to.

  Then, the agent 11 performs reinforcement learning of the DNN 12 using the experience data stored in the experience data storage unit 13. In this reinforcement learning, for example, when a predetermined number of experience data is stored in the experience data storage unit 13 and the control end condition for the control target (environment 14) is determined, the control start to the control end are performed. One episode is repeated at a predetermined timing, for example, when a predetermined number of episodes have been completed, or when a predetermined time has elapsed since the last reinforcement learning.

  Next, the experience data deletion process will be described with reference to the flowchart of FIG. In step S200 of the flowchart of FIG. 3, the data deletion unit 15 calculates, for each experience data, a uniqueness parameter for evaluating dissimilarity between each experience data and other experience data.

  For example, as shown in FIG. 4, when each experience data is plotted in a multi-dimensional space with each element of the experience data as an axis, the reciprocal of the number of experience data belonging to the range of the Euclidean distance k from a certain experience data x is obtained. It can be defined as a uniqueness parameter u. This is because as the number of experience data belonging to the range of the Euclidean distance k from a certain experience data x increases, the experience data x can be regarded as data having higher similarity to other experience data.

  By defining the uniqueness parameter u in this way, a lower value is calculated as u as the number of surrounding experience data increases, and conversely, a higher value is calculated as u as the number of surrounding experience data decreases. It becomes like this. Note that when the number of surrounding experience data is zero, it may be determined that a predetermined maximum value is calculated as u.

  The distance for obtaining the similarity between the experience data is not limited to the above-mentioned Euclidean distance, and other known distances (for example, Mahalanobis distance) may be used. Furthermore, each experience data may be regarded as a vector, and the similarity as a vector may be evaluated using a cosine similarity or the like. For example, the reciprocal of the number of experience data having a predetermined similarity vector or more with respect to a certain experience data vector may be defined as the uniqueness parameter.

Further, when calculating the uniqueness parameter of each experience data, it is not always necessary to use all elements included in the experience data. Specifically, among the elements included in the experience data (current state s _t , action a _t , reward r _t , next state s _{t + 1} ), the uniqueness parameter is obtained from three elements excluding the next state s _{t + 1.} It may be calculated. This is because the next state s _{t + 1} is highly correlated with the current state s _t, and even if both elements are used, the information is only redundant.

  Furthermore, when at least one of the state s and the action a included in the experience data is composed of high-dimensional data, the uniqueness parameter of each experience data is calculated after reducing the high-dimensional data. Also good. For example, when the state s is stored as an image, a reduced feature quantity is extracted using a dimension compression algorithm such as an auto encoder, and a uniqueness parameter is calculated from the extracted feature quantity. good. Thereby, the calculation load for calculating the uniqueness parameter can be reduced. Note that principal component analysis or the like may be used as the dimension compression algorithm.

  In step S210 of the flowchart of FIG. 3, experience data having a low uniqueness parameter u is deleted from the experience data storage unit 13. At this time, one piece of experience data having the lowest uniqueness parameter u may be deleted. However, each time new experience data is collected, each piece of experience data stored in the experience data storage unit 13 is deleted. Since the uniqueness parameter u must be calculated, the calculation load increases. Therefore, in the present embodiment, a plurality of pieces of experience data having uniqueness parameters u that are equal to or less than a predetermined deletion reference value are deleted together. Thereby, the calculation frequency of the uniqueness parameter u can be reduced, and an increase in calculation load due to the calculation of the uniqueness parameter u can be suppressed.

Regarding the deletion of the experience data, the uniqueness parameter u may be directly compared with the deletion reference value, and the experience data having the uniqueness parameter u less than or equal to the deletion reference value may be deleted deterministically. By calculating the probability of deleting experience data based on the uniqueness parameter u, the experience data to be deleted may be selected stochastically. For example, as shown in Equation 3 below, the deletion probability P is defined by normalizing the uniqueness parameter u for each experience data using the uniqueness parameter of all experience data, and the deletion probability P is deleted according to the deletion probability P. You may make it determine the experience data which should be. As a specific implementation method, for example, a random number generator for generating a uniform random value of 0 to 1 is prepared, and the random value is compared with the random number generated by the random number generator and the deletion probability P. A method of deleting the experience data when the deletion probability P is smaller is conceivable.

  By determining the experience data to be deleted probabilistically, for example, when the experience data held in the experience storage unit is dense, an effect of thinning out the data in the dense part is expected. When the uniqueness parameter u itself is compared with the deletion reference value and a plurality of pieces of experience data having the uniqueness parameter u equal to or less than the deletion reference value are deleted at once, depending on the density of the experience data, the dense experience Most of the data can be deleted. On the other hand, by using the probabilistic method described above, it is possible to sparsely delete the experience data even when the experience data is dense.

  Next, the effect of the above-described data deletion process on the experience data stored in the experience data storage unit 13 and, as a result, how much it contributes to the stability of learning will be described. The case where the experience data is deleted by the data deletion processing according to the embodiment and the case where the experience data is deleted by the FIFO method will be described in comparison.

Note that, in the case of comparison, the agent 11 is assumed to learn the DNN 12 so that the vehicle is a control target and the vehicle travels along the center line of the straight road. Agent 11, as a state s, using a lateral distance l _C, and the traveling direction O _C of the vehicle with respect to the direction of the center line of the road between the center line of the center position and the road vehicle, the action a, straight, right One of three types of actions, steering and left steering, is selected.

The vehicle control is started from the vehicle travel start point, and a period from when a predetermined end condition is satisfied to when the vehicle travels is stopped is defined as one episode. The ending conditions were three conditions: the vehicle reached a goal that was separated by a predetermined distance, the center position of the vehicle was more than a predetermined distance away from the center line of the road (departed from the road), or a predetermined time passed. The reward function r _i is defined as the following Equation 4.

In Equation 4, w _l , w _o , and w _offload are weighting elements for giving a negative reward (penalty) for the lateral distance l _{C of} the vehicle, the traveling direction O _{C of the} vehicle, and the road deviation.

  The DNN 12 has a four-phase structure, the number of neurons in the first phase that is the input phase is 2, the number of neurons in the second phase is 50, the number of neurons in the third phase is 20, and the number of neurons in the fourth phase that is the output phase Is 3. All the weights of DNN12 were randomly initialized so as to be evenly distributed in the range of -0.05 to 0.05. The learning rate has an initial value of 0.001, and is gradually reduced to 0.00003 at a discount rate of 99% per learning.

  FIG. 5 and FIG. 6 show how experience data is distributed when data is deleted by the FIFO method under the above-described conditions and when data is deleted by the data deletion processing according to the present embodiment. In FIG. 5 and FIG. 6, the empirical data is two-dimensionally compressed by principal component analysis.

  FIG. 5A shows how the experience data is distributed when the experience data storage unit 13 is almost full in the FIFO method, that is, when the data has not yet been deleted. FIG. 6A similarly shows how the experience data is distributed when the experience data storage unit 13 is almost full in the present embodiment. In both FIG. 5A and FIG. 6A, since various combinations of the state s and the action a are selected by the action selection method described above, the experience data is widely distributed in the initial stage almost similarly. I understand that.

  On the other hand, FIG. 5B shows how the experience data stored in the experience data storage unit 13 is distributed after repeating the episode 7000 times while deleting the experience data by the FIFO method. FIG. 6B shows how the experience data stored in the experience data storage unit 13 is distributed after the episode has been repeated 7000 times while performing data deletion by the data deletion processing according to the present embodiment. Yes.

  In FIG. 5A, it can be confirmed that the experience data is concentrated near the center, and the experience data around the center is sparse. This is presumed to be because when old experience data is deleted each time new experience data is saved, the tendency to select a similar action a for a similar state s increases as learning progresses. The On the other hand, when data deletion is performed by the data deletion processing according to the present embodiment, the state in which the experience data is distributed over a wide range is maintained without largely changing from the initial stage shown in FIG. I can confirm that. This is because, as described above, in the present embodiment, in the experience data deletion process, similar experience data is deleted to leave highly unique experience data.

  Next, when the reinforcement learning of the DNN 12 is repeated using experience data stored in the experience data storage unit 13 when a predetermined learning execution condition is satisfied while performing data deletion by the FIFO method, 5 times FIG. 7 shows the result of counting the accumulated value of the reward r obtained during the execution of each episode. Similarly, while performing data deletion by the data deletion process according to the present embodiment, when predetermined learning execution conditions are satisfied, DNN 12 reinforcement learning is repeated using experience data stored in the experience data storage unit 13. FIG. 8 shows the result of counting the cumulative value of the reward r obtained during execution of the episodes every time five episodes are completed.

  From FIG. 7, when the experience data is deleted by the FIFO method, it can be confirmed that even if the number of times of learning increases, the fluctuation of the accumulated value of the reward does not stop, and learning that always obtains a stable reward cannot be performed. On the other hand, when the experience data is deleted by the data deletion process according to the present embodiment, it can be confirmed that the fluctuation of the accumulated value of the reward r is clearly reduced as learning progresses. As described above, the experience data stored in the experience data storage unit 13 can maintain a state distributed over a wide range. In other words, various experience data can be stored in the experience data storage unit 13. It is because it has been. As a result, learning using empirical data can suppress over-fitting to highly similar data, and even if the input state s is rare, an action that provides a good reward r It becomes possible to select a.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. .

  For example, in the above-described embodiment, the example in which the agent 11 has one DNN 12 has been described. For example, the agent 11 includes a neural network used as an Actor and a neural network used as Critic separately. May be.

  Further, in the above-described embodiment, the example in which the agent 11 learns to drive the automobile as the control target along the center line of the road has been described. However, for example, more complicated control learning such as driving an automobile automatically while avoiding an obstacle may be performed, and the control target is not limited to the automobile, and the image (recognition), voice (recognition), robot Any object that can be controlled or processed by the incoming network is acceptable.

  Furthermore, in the above-described embodiment, an example in which one computer realizes the functions of the agent 11, the environment 14, the data deletion unit 15, and the like has been described. However, each of the functions is configured to be realized by a plurality of computers. Also good.

10 Computer 11 Agent 12 Deep Neural Network
13 Experience data storage unit 14 Environment 15 Data deletion unit

Claims (14)

In the case of using a neural network (12) as a learning device for learning an optimal policy for determining an action according to a state of a controlled object, a reinforcement learning method for reinforcement learning of the neural network,
The computer (10) collects empirical data including the state of the controlled object, the action on the controlled object, the reward obtained by the action, and the state of the controlled object transitioned by the action, and has a finite storage capacity Store it in the experience data storage unit (13),
The computer (10) calculates a uniqueness parameter indicating how different each experience data stored in the experience data storage unit from other experience data,
Based on the calculated uniqueness parameter, the computer (10) deletes experience data similar to other experience data from the experience data storage unit,
A reinforcement learning method in which a computer (10) performs reinforcement learning of the neural network using experience data stored in the experience data storage unit.   The computer according to claim 1, wherein the computer calculates a uniqueness parameter of each experience data based on the state, action, and reward among the state, action, reward, and transition state included in the experience data. Reinforcement learning method.   The computer calculates a uniqueness parameter of each experience data after lowering the high-dimensional data when at least one of a state and an action included in the experience data is composed of high-dimensional data. Or the reinforcement learning method of 2.   The said computer determines the experience data which has the lowest uniqueness parameter, or the experience data which has a uniqueness parameter below a predetermined deletion reference value as experience data which should be deleted. Reinforcement learning method.   The reinforcement learning method according to claim 1, wherein the computer evaluates the uniqueness parameter by a probabilistic method to determine experience data to be deleted.   The enhancement according to claim 5, wherein the computer normalizes the uniqueness parameter for each experience data using the uniqueness parameter of all the experience data, and determines the experience data to be deleted based on the normalized result. Learning method.   The computer calculates uniqueness parameters of all the experience data stored in the experience data storage unit when the storage amount of the experience data reaches the upper limit of the storage capacity of the experience data storage unit. The reinforcement learning method according to any one of claims 1 to 6, wherein a plurality of pieces of experience data are collectively deleted based on the uniqueness parameter. In the case of using a neural network (12) as a learning device for learning an optimal policy for determining an action according to a state of a control target, a reinforcement learning device for reinforcement learning of the neural network,
A finite memory that stores experience data each time it collects experience data that includes the state of the control object, the action on the control object, the reward obtained by the action, and the state of the control object that has been transitioned by the action An experience data storage unit (13) having a capacity;
For each experience data stored in the experience data storage unit, a calculation unit (S200) that calculates a uniqueness parameter indicating how different from other experience data;
Based on the uniqueness parameter calculated by the calculation unit, a deletion unit (S210) that deletes experience data similar to other experience data from the experience data storage unit,
A reinforcement learning device comprising: a reinforcement learning unit that performs reinforcement learning of the neural network using experience data stored in the experience data storage unit.   The said calculation part calculates the uniqueness parameter of each experience data based on a state, an action, and reward among the state, action, reward, and the transition state which are contained in experience data. Reinforcement learning device.   The calculation unit, when at least one of a state and an action included in experience data is composed of high-dimensional data, lowers the high-dimensional data and then calculates a uniqueness parameter of each experience data. The reinforcement learning apparatus according to 8 or 9.   The said deletion part determines the experience data which has the lowest uniqueness parameter, or the experience data which has a uniqueness parameter below a predetermined deletion reference value as experience data which should be deleted. Reinforcement learning device.   The reinforcement learning device according to claim 8, wherein the deletion unit evaluates the uniqueness parameter by a probabilistic method and determines experience data to be deleted.   13. The deletion unit according to claim 12, wherein the deletion unit normalizes a uniqueness parameter for each experience data using a uniqueness parameter of all experience data, and determines experience data to be deleted based on the normalized result. Reinforcement learning device.   When the amount of experience data stored reaches the upper limit of the storage capacity of the experience data storage unit, the calculation unit calculates uniqueness parameters of all experience data stored in the experience data storage unit, and The reinforcement learning device according to claim 8, wherein the deletion unit deletes a plurality of pieces of experience data at a time based on the calculated uniqueness parameter.

Images