KR102209917B1 - Data processing apparatus and method for deep reinforcement learning - Google Patents

Abstract

It relates to a data processing method for deep reinforcement learning, and the data processing method for deep reinforcement learning includes (a) storing human demonstration data among target data in a human replay buffer, and storing actor data among the target data in an actor replay buffer. Storing; And (b) performing data processing by applying online frame skipping to the human replay buffer for sampling the human demonstration data stored in the human replay buffer, wherein the online frame skipping comprises: Two types of frame skipping may be included in consideration of whether or not the skipping interval is variable.

Description

Data processing device and method for deep reinforcement learning {DATA PROCESSING APPARATUS AND METHOD FOR DEEP REINFORCEMENT LEARNING}

The present application relates to a data processing apparatus and method for deep reinforcement learning. In particular, for accelerating deep reinforcement learning, we have Human Demonstration Data based on Dual Replay Buffer Management and Online Frame Skipping. It relates to a data processing apparatus and method for deep reinforcement learning using ).

Deep reinforcement learning is a method of merging a deep neural network and a reinforcement learning algorithm.

AlphaGo and Deep Q-Network (DQN) demonstrate the effectiveness of deep learning on complex decision-making problems. Here, DQN is an example in document 1 [V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller, "Playing atari with deep reinforcement learning," CoRR, vol. abs/1312.5602, 2013. [Online]. Available: http://arxiv.org/abs/1312.5602].

Recent advances in deep reinforcement learning have led to breakthroughs in robotics, game AI and self-driving cars. Deep reinforcement learning algorithms generally require a huge amount of interactions with the environment in order to achieve good performance. Therefore, the learning procedure of deep reinforcement learning is computationally expensive compared to other deep learning techniques.

Two categories of approaches exist to accelerate the learning process of deep reinforcement learning.

The first category of approaches uses actor-critic agents, which include Asynchronous Advantage Actor-Critic (A3C). Here, A3C is an example in document 2 [V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning," in International conference on machine learning, 2016, pp. 1928-1937.]. This agent trains transitions collected by multiple actors running in parallel, and this architecture greatly shortens the overall training time.

The second category of approaches uses human demonstration data for deep reinforcement learning, including Deep Q-Learning from Demonstration (DQfD) and Normalized Actor-Critic (NAC) models.

Here, DQfD is described in Document 3 [T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold et al., "Deep qlearning from demonstrations," arXiv preprint arXiv:1704.03732, 2017.]. In addition, NAC is described in document 4 [Y. Gao, H. Xu, J. Lin, F. Yu, S. Levine, and T. Darrell, "Reinforcement learning from imperfect demonstrations," CoRR, vol. abs/1802.05313, 2018. [Online]. Available: http://arxiv.org/abs/1802.05313].

DQfD specially applied supervised loss in addition to the loss model of the existing DQN, and showed excellent learning performance not only in the pre-training stage but also in the reinforcement learning stage.

However, DQfD over-applies the model to expert data due to the loss of supervision. In particular, when the amount of expert dataset is limited, the overfitting of DQfD is worse. NAC tried to solve the overfitting problem of DQfD with a new loss model, but it shows lower performance than DQfD with a small number of human demonstrations.

It is well known that using a sufficient amount of experts' demonstration data can help accelerate the learning process of deep reinforcement learning and achieve good results. However, in most cases, it is difficult to sufficiently secure (or generate) expert demonstration data. To aggravate this problem, most recently published documents do not share human demonstration data regarding their performance evaluation for privacy and/or security reasons. Therefore, it is difficult to reproduce the results because high quality human demonstration data is not sufficiently contained in the available and/or open datasets.

For example, although the human demonstration dataset in the Atari 2600 Games (ALE Environments) environment is publicly available, the dataset does not contain high-scoring episodes. Here, ALE Environments is described in document 5 [M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, "The arcade learning environment: An evaluation platform for general agents," CoRR, vol. abs/1207.4708, 2012. [Online]. Available: http://arxiv.org/abs/1207.4708]. However, the dataset used in conventional DQfD studies contains episodes with very high scores.

Therefore, it is relatively difficult to collect high-quality human demonstration data from practical cases (ie, real-world cases), and therefore, it is necessary to more effectively utilize the available human demonstration data.

An object of the present application is to provide a data processing apparatus and method for deep reinforcement learning that more effectively utilizes human demonstration data to accelerate deep reinforcement learning to solve the problems of the prior art described above.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a data processing method for deep reinforcement learning according to an embodiment of the present application includes (a) storing human demonstration data among target data in a human replay buffer, and among the target data Storing actor data in an actor replay buffer; And (b) performing data processing by applying online frame skipping to the human replay buffer for sampling the human demonstration data stored in the human replay buffer, wherein the online frame skipping comprises: Two types of frame skipping may be included in consideration of whether or not the skipping interval is variable.

In addition, the online frame skipping, as a first type of frame skipping, includes FS-ER skipping in which skipping is performed by keeping the frame skipping interval constant without varying, and the second type of frame The skipping may include DFS-ER skipping, which dynamically performs skipping by varying the frame skipping interval.

In addition, the step (b), prior to the application of the online frame skipping, generates a frame state for the human demonstration data stored in the human replay buffer, and the frame states are a number corresponding to each frame skipping coefficient. It can be created by stacking frames.

In addition, in step (b), when the first type of frame skipping is applied, a number of frames corresponding to the frame skipping coefficient may be stacked at intervals corresponding to a preset number of frames.

In addition, in step (b), the second type of frame skipping is applied in consideration of an action repetition value representing an action duration of each frame of the human demonstration data, and the action repetition value is two skipping values. Can include.

In addition, in step (b), when the second type of frame skipping is applied, when generating the frame state, a skip value for each frame is checked, and the frame state is generated in consideration of the checked skip value. can do.

In addition, the data processing method for deep reinforcement learning according to an embodiment of the present application includes (c) the human demonstration data sampled by the data processing in step (b) and the neural network using the actor data as inputs. The method further includes redefining a return value required to calculate a loss value to be used, and the step (c) may redefine the return value for each of the two types of frame skipping applications.

Meanwhile, a data processing apparatus for deep reinforcement learning according to an embodiment of the present disclosure includes: a replay manager storing human demonstration data among target data in a human replay buffer and storing actor data among the target data in an actor replay buffer; And a data processing unit that performs data processing by applying online frame skipping to the human replay buffer for sampling the human demonstration data stored in the human replay buffer, wherein the online frame skipping comprises: frame skipping. It may include two types of frame skipping considering whether or not the interval of is variable.

In addition, the online frame skipping, as a first type of frame skipping, includes FS-ER skipping in which skipping is performed by keeping the frame skipping interval constant without varying, and the second type of frame The skipping may include DFS-ER skipping, which dynamically performs skipping by varying the frame skipping interval.

In addition, the data processing unit generates a frame state for the human demonstration data stored in the human replay buffer before the application of the on-line frame skipping, and the frame states each generate a number of frames corresponding to the frame skipping coefficient. It can be created by laminating.

Meanwhile, an apparatus for deep reinforcement learning of a neural network according to an embodiment of the present application includes: a data processing apparatus that processes data for deep reinforcement learning on target data; And a learning control unit for in-depth reinforcement learning of the neural network by using human demonstration data and actor data sampled by data processing of the data processing device as inputs of the neural network, wherein the data processing device includes human demonstration data among target data A replay manager that stores the data in a human replay buffer and stores actor data among the target data in the actor replay buffer; And a data processing unit that performs data processing by applying online frame skipping to the human replay buffer for sampling the human demonstration data stored in the human replay buffer, wherein the online frame skipping comprises: frame skipping. It may include two types of frame skipping considering whether or not the interval of is variable.

In addition, the online frame skipping, as a first type of frame skipping, includes FS-ER skipping in which skipping is performed by keeping the frame skipping interval constant without varying, and the second type of frame The skipping may include DFS-ER skipping, which dynamically performs skipping by varying the frame skipping interval.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, by providing a data processing apparatus and method for deep reinforcement learning, it is possible to accelerate deep reinforcement learning by effectively utilizing human demonstration data.

According to the above-described problem solving means of the present application, by performing online frame skipping based on a dual replay buffer including a human replay buffer and an actor replay buffer, all human demonstration data is not discarded without discarding good quality human demonstration data. In-depth reinforcement learning can be achieved by using the Through this, the present application can effectively improve the results and performance of deep reinforcement learning.

According to the above-described problem solving means of the present application, instead of storing both human demonstration data and actor data in a single replay buffer, they are separately stored in each of the two replay buffers, and based on this, the humans stored in the human replay buffer Online frame skipping can be applied independently for demonstration data. In this case, all human demonstration data can be used as learning data for neural network learning.

However, the effect obtainable in the present application is not limited to the effects as described above, and other effects may exist.

1 is a diagram showing the architecture of a conventional DQfD.
2 is a diagram schematically showing the architecture (a) of the conventional DFDQN and the architecture (b) of the FiGAR.
3 is a diagram showing an example of a screen-shot sample of the Atari Grand Challenge dataset.
4 is a block diagram showing a schematic configuration of a data processing apparatus for deep reinforcement learning according to an embodiment of the present application.
5 is a diagram schematically illustrating an architecture of a data processing apparatus for deep reinforcement learning according to an embodiment of the present application.
6 is a diagram illustrating a sampling approach of human demonstration data by a data processing device for deep reinforcement learning according to an embodiment of the present application.
7 is a diagram illustrating an algorithm for an operation of DFS-ER skipping by a data processing device for deep reinforcement learning according to an embodiment of the present application.
8 is a block diagram showing a schematic configuration of an apparatus for deep reinforcement learning of a neural network according to an embodiment of the present application.
9 is an experiment result of the present application, showing the number of human data in a mini-batch.
10 is a diagram showing an average episode score of three games as an experiment result of the present application.
11 is a view showing a summary of the average score achieved in four games as an experiment result of the present application.
12 is an operation flowchart of a data processing method for deep reinforcement learning according to an embodiment of the present application.

Hereinafter, exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be "connected" with another part, it is not only "directly connected", but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including the case.

Throughout this specification, when a member is positioned "on", "upper", "upper", "under", "lower", and "lower" of another member, this means that a member is located on another member. It includes not only the case where they are in contact but also the case where another member exists between the two members.

Throughout the specification of the present application, when a certain part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

Human Demonstration Data accelerates the training process at the early stage of deep reinforcement learning and guides the reinforcement learning agent to create a complex policy. ) Plays an important role in learning.

However, most of the conventional reinforcement learning methods with human demonstration data and reward assume that there is a sufficient amount of high-quality human demonstration data, and a sufficient amount of expert demonstration data is always available. This is not the case for most limited real-world learning cases.

In order to overcome this limitation, the present application proposes a new deep reinforcement learning method through Dual Replay Buffer Management and Online Frame Skipping.

In other words, in order to overcome the conventional limited human demonstration data problem, the present application proposes a new deep reinforcement learning method in human demonstration using dual replay buffer management and online frame skipping for human demonstration sampling. Hereinafter, the technology proposed by the present application (online frame skipping technology based on a dual replay buffer, that is, a data processing method for deep reinforcement learning by the apparatus 100 to be described later) will be referred to as the proposed technology for convenience of explanation. To

Among the proposed technologies, the dual replay buffer will consist of a human replay buffer (human replay memory), an actor replay buffer (an actor replay memory), and a replay manager (replay manager). I can. In the present application, a buffer may be differently expressed as a memory. In addition, according to the present application, two replay buffers can be managed with independent sampling policies.

The dual replay buffer management considered in this application consists of a human replay buffer (memory), an actor replay buffer (memory), and a replay manager, whereas the conventional replay buffer contains a human in the same replay buffer. Both human demonstration data and actor data are included.

In addition, because the conventional static frame skipping method skips (skips) the human demonstration data stored in the memory buffer for each frame, the remaining dropped frames are reinforced. There is a problem that cannot be used during learning. In addition, the conventional frame skipping method has a problem in that the reinforcement learning algorithm prevents capturing (capturing) features of action repeat (action repeat) of human demonstration data.

On the other hand, the architecture of this proposed technology has two distinct replay buffers (i.e., a human replay buffer and an actor replay buffer, in other words, a human replay memory and an actor replay memory), so the replay manager has two replay buffers. Each can be processed with independent sampling policies. The dual replay buffer architecture of this proposed technology can help to process human demonstration data more sophistically.

In addition, the present application proposes online frame skipping in order to fully (fully, completely, all) utilize the available human data. In the present application, human data may be expressed differently as human demonstration data. During the training period, frame skipping herein may be dynamically performed in a human replay buffer in which all human demonstration data is stored. Herein, in order to sample data from the human replay buffer, two types of on-line frame skipping can be used as Frame Skipping-Experience Replay (FS-ER) and Dynamic Frame Skipping-Experience Replay (DFS-ER).

In other words, the present application proposes a Frame Skipping Experience Replay (FS-ER) technology in order to use the entire human demonstration data with frame skipping. With FS-ER, the entire human demonstration data is stored in the human replay buffer, and frame skipping can be performed online during the training period. By using FS-ER, the model can be trained with a full set of human demonstration data available.

In addition, the present application proposes a DFS-ER (Dynamic Frame Skipping Experience Replay) technology by adopting dynamic frame skipping of DFDQN (Dynamic Frame-Skipping Deep Q-Network) in FS-ER. Here, DFDQN is described in Document 6 [A. S. Lakshminarayanan, S. Sharma, and B. Ravindran, "Dynamic frame skip deep Q network," CoRR, vol. abs/1605.05365, 2016. [Online]. Available: http://arxiv.org/abs/1605.05365].

The DFS-ER of this proposed technology can train action repeat (action repeat) of human demonstration data, and because it dynamically skips (skips) frames, conventional static frame skipping You can explore the environment faster than the (conventional static frame skipping) method.

Hereinafter, prior to a detailed description of the proposed technology, a technology that is a background of the proposed technology will be described in more detail.

The description of Deep Reinforcement Learning From Demonstration is as follows.

Human demonstration data can be used to improve the initial stage of deep reinforcement learning, and can be helpful for reinforcement learning in a sparse reward environment.

Currently, the most advanced models using human demonstration data are DQfD and NAC.

DQfD adds supervised loss when training with human demonstration data. Because supervised loss increases the overall loss value when the agent's action is different from the expert's action, it is supervised learning when the model is trained with human demonstration data. It can work like a supervised learning algorithm.

1 is a diagram showing the architecture of a conventional DQfD.

Referring to FIG. 1, it can be seen that in the architecture of DQfD, both human demonstration data (human data) and actor data are stored and managed in a single replay buffer. In particular, human demonstration data is permanently stored in the replay buffer, while older actor data is replaced with incoming actor data.

Also, both human demonstration data and actor data are already frame-skipped before being stored in the replay buffer. Therefore, in DQfD, when data is sampled during the learning process, Pre-State, Post-State, Reward, Action, and n-level states (n -Step-State) and experience data including n Step-Reward are sampled without frame-skipping. The minibatch in Fig. 1 represents this sampling process.

In the mini-batch (Minibatch) of FIG. 1, a blue-colored part (P1) represents a pre-state and a post state (Post-State), and a green-colored part (P2) is It represents the n-Step-State and n Step-Reward. Minibatch may be inputted to a neural network (Network) after sampling. The neural network can be back-propagated by the sum of the four losses.

The loss equation (that is, the equation of the loss function) can be expressed as Equations 1 and 2 below.

[Equation 1]

[Equation 2]

Equation 1 shows an equation for calculating the overall loss used to update the DQfD network. The total loss of DQfD consists of Q loss, n-step Q loss, supervised loss, and L2 regularization loss, and each loss is weighted by a λ value. Weighted.

Equation 2 shows the equation for calculating the supervision loss.

는 종래의(original) DQfD에서 0.8을 가질 수 있다. 또한

는 샘플 데이터(sampled data)가 휴먼 데몬스트레이션 데이터인 경우에만 존재한다.In Equation 2, a _expert denotes a performed action by a human (a human, an expert among humans) in the state s. If, as a _expert is not equal to a value (same value) like (a data value of each action contained in the human data),

May have 0.8 in the original DQfD. Also

Is present only when sampled data is human demonstration data.

DQfD shows better performance than DQN through prioritized experience replay in arcade learning environment (ALE). It can train games like Montezuma's Revenge on a small but high-quality human demonstration dataset. Montezuma's Revenge is a game that is difficult to train with models based on conventional deep reinforcement learning algorithms such as A3C and DQN because of sparse rewards.

Because DQfD operates based on supervised loss and training is performed using a small amount of human demonstration data, it often presents an overfitting problem. Therefore, DQfD can easily calculate the highest score. However, after reaching a high score during pre-training, the score hardly increases or rarely gets the highest score again. The L2 regularization loss can be adopted to alleviate the overfitting problem of DQfD. In the performance evaluation between DQfD and NAC, a supervised loss decay can be introduced as the iteration progresses. However, DQfD's overfitting problem has not been completely resolved.

As a model for solving the overfitting problem of DQfD, NAC can use imperfect human demonstration data in deep reinforcement learning. Unlike DQfD, which uses a separate loss model for each actor (actor) and human demonstration data, the NAC model uses a unified loss model. The NAC model shows the cause of the problem that the DQN model learns from human demonstration data. The authors of Document 4, which discloses an example of the NAC model, point out that the main cause of this problem is the overestimation of the Q value.

To solve this problem, in Document 4, the existing soft Q learning model is modified, and the modified NAC is made to subtract the gradient of the value function of the state from the gradient of the Q value, and then the Q value is soft Q A technique of normalizing according to the learning loss model is disclosed. However, this NAC model has better performance than the DQfD model when the amount of human demonstration data is 300,000 transitions, but performance is lower than that of DQfD when the amount of human demonstration data is 100,000 transitions. That is, the conventional NAC model shows that the performance is lower than that of DQfD when the amount of human demonstration data is small.

Meanwhile, a description of dynamic frame skipping is as follows.

Frame skipping is a technique for faster exploration of the environment in DQN. With frame skipping, the agent of reinforcement learning repeats a predicted action (action) until a frame skipping coefficient is met. This consequently reduces the actual training time as the number of action decisions decreases.

Moreover, the agent to which the improved frame skipping method or dynamic frame skipping is applied sets the additional output value of the neural network as the frame skipping coefficient, You can learn repeated actions in each state. The next two architectures (ie, DFDQN and FiGAR) are architectures that apply dynamic frame skipping.

2 is a diagram schematically showing the architecture (a) of the conventional DFDQN and the architecture (b) of the FiGAR.

DFDQN (Dynamic Frame Skipping DQN) can be understood with reference to the above-described document 6, briefly described as follows. DFDQN is designed to learn both actions (actions) and repetition of actions in each state. This model is the action iteration value fs ₁ And fs ₂ are set in two cases. In addition, the model doubled the dimensions of the output in order to represent the iteration of the action (repeated action) as the output of the neural network. Therefore, if the predicted action value is lower than the action dimension, the agent sets the action (action) to fs ₁ Repeated once, otherwise an action (operation) ₂ times fs.

DFDQN also doubled the last full connected layer to handle the double action dimension. As training progresses, an action value larger than the action dimension may be selected in a specific state. By doing so, the exploration of each episode can be accelerated.

As a result, DFDQNs learn faster and get more rewards than DQNs in the same time steps when evaluating the performance of their models in the Atari gaming environment.

In Figure 2 (a) is fs ₁ 4 in, fs ₂ DFDQNs with 20 in In the neural network model shown in FIG. 2A, if the action value is lower than the action dimension, the action is repeated 4 times, otherwise, the action is repeated 20 times.

On the other hand, FiGAR (Fine Grained Action Repetition) is an example of document 7 [[S. Sharma, A. S. Lakshminarayanan, and B. Ravindran, "Learning to repeat: Fine grained action repetition for deep reinforcement learning," arXiv preprint arXiv:1702.06054, 2017.], and briefly described as follows.

FiGAR (Fine Grained Action Repetition) is a model that extends the previous (previous) dynamic frame skipping of DFDQN. This model can select not only two cases of action repeat value, but also a certain range of action repeat. FiGAR can adopt a variety of deep reinforcement learning algorithms.

In FIG. 2, (b) shows the structure of FiGAR in which two networks (neural networks) exist. One of the two networks selects an action, and the other selects an action repetition (repeated action). However, when applying this model to various deep reinforcement models, it is necessary to modify the target (target) deep reinforcement model.

Document 7 discloses a case where FiGAR is applied to three models such as A3C, TRPO, and DDPG. In addition, FiGAR defines an estimate of the n-step return of A3C as shown in Equation 4 below to match the number of skipped frames.

[Equation 3]

[Equation 4]

의 지수(exponent)를 나타낸다. 즉, 액션의 반복 횟수가 많고 y의 값이 커질수록, r _k 의 값(각 프레임 k의 보상값(reward))은

에 의해 더 감소(decayed)될 수 있다.In Equation 3, x _k is the state (state) in the attenuation values of s _k mean repeat value of the action a _k (repeat value), and y is from expression 4 (decay value)

Represents the exponent of That is, as the number of repetitions of the action increases and the value of y increases, the value of r _k (reward value of each frame k) becomes

Can be further decayed by

Meanwhile, the description of the Atari Game Human Demonstration Dataset is as follows.

3 is a diagram showing an example of a screen-shot sample of the Atari Grand Challenge dataset. In particular, FIG. 3 shows, in the Atari Grand Challenge dataset, Q*bert (a), Video Pinball (b), Ms. Pacman (c), Space Invaders (d) and Monte The following is an example of a screen shot for Montezuma's Revenge (e).

3, the Atari Grand Challenge dataset is an open dataset of five Atari games for deep reinforcement learning. Each dataset contains different episodes with varying score distribution ranging from novice to expert. Each episode of the game has several image files and one trajectory file. The image data set contains raw color images from Atari games. The trajectory dataset contains information such as reward, frame ID, score, terminal state, and action.

From a frame skipping point of view, it has full 60Hz human demonstration data without dropping any frames in the game. This is different from the dataset of Document 3 which discloses for a conventional DQfD using 15Hz dormant demonstration data that has already been frame skipped. In terms of scores, the score of most episodes in the dataset is lower than the dataset of Document 3, which discloses for conventional DQfD. In addition, the data set lacks information on life loss that makes it difficult to give negative rewards to each frame.

Hereinafter, an Online Frame Skipping based on Dual Replay Buffer, which is the proposed technology (a technology proposed herein), will be described in more detail.

4 is a block diagram showing a schematic configuration of a data processing apparatus 100 for deep reinforcement learning according to an embodiment of the present application. 5 is a diagram schematically showing an architecture of a data processing apparatus 100 for deep reinforcement learning according to an embodiment of the present application. That is, FIG. 5 shows a system architecture for an online frame skipping technology based on a dual replay buffer by the data processing apparatus 100 for deep reinforcement learning according to an embodiment of the present application.

Hereinafter, the data processing apparatus 100 for deep reinforcement learning according to an embodiment of the present application will be referred to as the present apparatus 100 for convenience of description.

Referring to FIGS. 4 and 5, the apparatus 100 relates to a data processing device for deep reinforcement learning, in particular, to a data processing device for accelerating deep reinforcement learning.

The device 100 may perform the proposed technology described above. That is, the device 100 is a new deep reinforcement learning based on dual replay buffer management and online frame skipping technology to accelerate the training process using human demonstration data 111'. Data processing for (deep reinforcement learning) can be performed.

In other words, the apparatus 100 may perform data processing for deep reinforcement learning (especially acceleration of deep reinforcement learning) based on dual replay buffer management and online frame skipping technology.

The device 100 may include a replay manager 110, a data processing unit 120, and a redefinition unit 130. In addition, the device 100 may include a human replay buffer 111 and an actor replay buffer 112 managed by the replay manager 110. Herein, the human replay buffer 111 and the actor replay buffer 112 may be referred to as dual replay buffers 111 and 112.

The replay manager 110 may store human demonstration data 111 ′ among target data in the human replay buffer 111 and may store actor data 112 ′ among target data in the actor replay buffer 112. In other words, the human replay buffer 111 may be a buffer that stores human demonstration data 111 ′ among target data, and the actor replay buffer 112 may be a buffer that stores actor data 112 ′ among target data. . Herein, the buffer may be expressed differently as a memory.

Here, the target data may mean target (target) data on which learning is performed, for example. The target data may mean image data as an example. As a specific example, when the target data is data related to a game, the target data may refer to image data related to the game.

In addition, human demonstration data (human data, 111') may mean data played (demonstrated) by humans (persons, humans). That is, when a person (human) plays a game, the human demonstration data 111 ′ may mean data generated (generated) by the human (person) playing (demonstration) the game. The actor data 112 ′ may refer to data performed by an agent (or system, device). That is, the actor data 112 ′ is data generated (generated) by an agent's action (action), and in other words, may mean data related to an agent's action (action).

The data processing unit 120 may perform data processing by applying online frame skipping to the human replay buffer 111 in order to sample the human demonstration data 111 ′ stored in the human replay buffer 111.

Here, online frame skipping may include two types of frame skipping (121, 122) in consideration of whether or not the frame skipping interval is variable. In particular, the two types of frame skipping (121, 122) are online frame skipping (Online Frame Skipping, Frame-skipping Experience Replay, FS-ER skipping, 121) and dynamic online frame skipping (Dynamic Online Frame Skipping, Dynamic Frame-skipping Experience Replay, DFS-ER skipping, 122) may be included.

In other words, online frame skipping is the first type of frame skipping among the two types of frame skipping (121, 122). In order to use the entire human demonstration data (111'), the frame skipping interval is not changed and On-line frame skipping (FS-ER skipping, 121) for performing skipping by maintaining the same level may be included. Here, online frame skipping (FS-ER skipping, 121) may be expressed differently as static online frame skipping or fixed (constant) frame skipping. In addition, online frame skipping is the second type of frame skipping among the two types of frame skipping (121, 122). In order to train the action repetition of the human demonstration data 111 ′, the interval of frame skipping is varied. It may include dynamic online frame skipping (DFS-ER skipping, 122) that dynamically performs skipping.

As online frame skipping, the data processing unit 120 applies these two types of frame skipping (121, 122) in consideration of whether or not the frame skipping interval is variable to the human replay buffer 111, so that the human demonstration data ( 111') can be processed.

A more detailed description is as follows. Compared to the DQfD of FIG. 1, the device 100 has three main characteristics different from the prior art, and the three main characteristics include dual replay buffer management and online frame skipping. (Online Frame Skipping) and dynamic online frame skipping (Dynamic Online Frame Skipping) may be included.

The dual replay buffer management may include a human replay buffer 111, an actor replay buffer 112, and a replay manager 110. The replay manager 110 may process (manage) the human replay buffer 111 and the actor replay buffer 112 by separating them into separate management policies.

In the case of DQfD, both human demonstration data and actor data are stored in a single replay buffer. Human demonstration data stored in a single replay buffer has a relatively small amount of data compared to actor data, whereas actor data has a relatively large amount of data compared to human demonstration data. Therefore, when frame skipping is applied to the corresponding buffer, some human demonstration data among the small amount of human demonstration data included in the buffer is discarded due to the application of frame skipping. There is a problem that the learning performance (eg, accuracy, etc.) of the neural network is deteriorated because the amount of available training data is reduced (that is, the amount of human demonstration data that can be used when training a neural network is insufficient).

On the other hand, the present apparatus 100 uses the on-line frame skipping (121, 122) technology based on the dual replay buffer, so that the human demonstration data is FS-ER or DFS-regardless of the sampling method of the actor replay buffer. ER can be applied independently.

As the present apparatus 100 has conventionally stored both human demonstration data and actor data in a single replay buffer, human demonstration data is discarded when frame skipping is applied to the single replay buffer, thereby providing sufficient data for neural network learning. Problems that could not be secured can be solved. That is, the device 100 does not store both human demonstration data and actor data in a single replay buffer, but separates and stores them separately in each of the two replay buffers 111 and 112, and based on this, the human replay buffer among the target data Online frame skipping (121, 122) can be independently applied to the human demonstration data (111') stored in (111). Accordingly, the apparatus 100 may be used as training data for neural network training without discarding the human demonstration data 111 ′ through online frame skipping based on the dual replay buffer.

A description of the FS-ER (Frame Skipping-Experience Replay) 121, which is one of the two online frame skipping (121, 122) considered in the present apparatus 100, is as follows.

The FS-ER stores all human demonstration data 111' in the human replay buffer 111, and frame skipping may be performed online during a training period.

In other words, the replay manager 110 may store the human demonstration data 111 ′ in the human replay buffer 111. Thereafter, the data processing unit 120 may perform data processing by applying the FS-ER skipping 121 to the human replay buffer 111 and the human demonstration data 111 ′ stored therein. In this case, the data processing unit 120 may perform the FS-ER skipping 121 online during the training period.

Conventional DQfD skips (skips) frames to create them before storing both human demonstration data and actor data in a single replay buffer, so that only selected frames can be used during the learning process. . On the other hand, when the training procedure (training procedure) requires input of the human demonstration data 111 ′, the data processing unit 122 uses FS- for skipping frames on demand (on-demand). ER skipping 121 may be performed. The device 100 may utilize the entire frame of each episode for model training by using the FS-ER skipping 121.

Referring to FIG. 5, the mini-batch considered in the present apparatus 100 is each sampled frame (MiniBatch) to describe frame skipping for sampling human data 111'. Each sampled frame) has spacing (spacing). In other words, in the mini-batch considered by the present apparatus 100, spacing (spacing, spacing, spacing, spacing) is placed in each sampled frame to describe frame skipping for sampling human data.

A description of another one of the two online frame skipping techniques considered in the present application, Dynamic Frame Skipping Experience Replay (DFS-ER, 122) is as follows.

In the present application, in order to process the action repetition of the human demonstration data 111', a DFS-ER technology, which is a variant of the FS-ER, is proposed. That is, in order to process the action repetition of the human demonstration data 111 ′, the data processing unit 120 skips DFS-ER, which is a variant of the FS-ER, with respect to the human demonstration data 111 ′. Data processing can be performed by applying (122). The apparatus 100 adopts a dynamic frame skipping method of DFDQN, and thus, DFS-ER skipping 122 as a dynamic frame skipping method (scheme, structure) for processing all 60Hz human demonstration data. Technology can be implemented.

In FIG. 5, an output dimension of a neural network 50 is a size of twice the action dimension in order to represent two repeat values. In the MiniBatch, frames are not sampled at uniform intervals due to dynamic frame skipping (DES-ER, 122). The data processing unit 120 dynamically defines the skip intervals when the DES-ER skipping 122 is applied by the skip value of the action in the human demonstration data 111'. I can.

Accordingly, the dual replay buffer management proposed in the present application can be said to be an architecture that enables independent management between the human demonstration data 111 ′ and the actor data 112 ′. The data processing unit 120 of the device 100 can apply online frame skipping called FS-ER 121 to the human replay buffer 111 regardless of the actor data 112 ′ through this management architecture. . In addition, the data processing unit 120 of the apparatus 100 may apply dynamic online skipping called DFS-ER as online frame skipping in order to process the action repetition of the human demonstration data 111'.

A more detailed description of the device 100 is as follows.

Regarding the dual replay buffer (111, 112) management technology, in the conventional DQfD, human demonstration data and actor data are stored and managed in one memory (single memory, single memory, single buffer), so human data (human demonstration data) ) And actor data, as well as applying a different sampling method to each other (separately) is difficult to apply a different frame skipping method.

On the other hand, in the present apparatus 100, the replay manager 110 manages the human data 111 ′ and the actor data 112 ′ in a separate memory (buffer) space through the dual replay buffer 111 and 112 management technology. This problem can be solved by managing The data processing unit 120 can easily apply the online frame skipping (121, 122) scheme (scheme, structure, and technology) proposed herein based on the dual replay buffers 111 and 112.

The replay manager 110 manages the dual replay buffers 111 and 112, and samples training data (training data, target data including human demonstration data and actor data), or replays new applications. , 112), it is possible to process requests from agents.

The replay manager 110 may directly control a sampling ratio of the human demonstration data 111 ′. In addition, in the present apparatus 100, unlike the conventional DQfD, there may be no limitation on the proportion of human demonstration data in the replay buffers 111 and 112. Accordingly, the replay manager 110 may flexibly manage the sampling ratio of the human demonstration data 111 ′ through the dual replay buffer 111 and 112 management technology.

A detailed description of the application of the FS-ER skipping (FS-ER frame skipping, 121) technique by the data processing unit 120 is as follows. This can be more easily understood with reference to FIG. 6.

6 is a diagram for explaining sampling approaches of human demonstration data 111 ′ by the data processing apparatus 100 for deep reinforcement learning according to an embodiment of the present disclosure. That is, FIG. 6 illustrates a sampling technique based on FS-ER skipping 121 and a sampling technique based on DFS-ER skipping 122 as a sampling approach for human demonstration data 111 ′ by the data processing unit 120 It is a drawing to do.

In particular, (a) in FIG. 6 is a diagram for explaining a sampling scheme of human demonstration data for a DQfD architecture (Original). In FIG. 6 (b) is an FS-ER 121 considered in the present apparatus 100, that is, an FS-ER skipping 121 technology for skipping an on-demand (on-demand) frame during the sampling process. It is a figure for explaining. In FIG. 6 (c) shows the DFS-ER 122 considered in the present apparatus 100, that is, the DFS-ER skipping 122 technology for processing the repetition of the action of the human demonstration data 111 ′ during the sampling process. It is a drawing for explanation.

Referring to FIG. 6, in order to prepare a mini-batch for reinforcement learning with respect to the demonstration data 111', multiple experiences may be sampled. Here, the various experiences may refer to human demonstration data 111 ′ stored in the human replay buffer 111.

Data processing unit 120 is S _pre, S _post and S _n as the three states in order to sample a respective experience _- may generate a _step, each state (each state) is the number of frames corresponding to the frame skipping coefficient It can be created by stacking. In this case, the frame skipping coefficient may be set in advance by a user input, and may be 4 as an example. Accordingly, the number of frames corresponding to the frame skipping coefficient may be four (four) frames by way of example.

In other words, the data processing unit 120 may generate a frame state for the human demonstration data 111 ′ stored in the human replay buffer 111 prior to application of the online frame skipping (FS-ER skipping, 121). have. In particular, the data processing unit 120 may generate S _pre , S _post and S _{n -step} as a plurality of types of frame states for the human demonstration data 111 ′. Here, the frame state may be generated by stacking a number of frames (for example, four frames) corresponding to each frame skipping coefficient (for example, 4).

That is, before applying the FS-ER skipping 121 to the human demonstration data 111' in the human replay buffer 111, the data processing unit 120 first, as shown in FIG. 6A. Each state can be created (configured) by grouping four frames.

As mentioned above, (a) in FIG. 6 shows a sampling scheme of human demonstration data for a conventional DQfD architecture (Original). For example, suppose you train a reinforcement learning model using 60Hz human demonstration data from the Atari Grand Challenge dataset. In this case, in relation to the Atari game, an interval of frame skipping may be generally set to 4. That is, the conventional DQfD method skips (skips, omits) frames every four intervals to sample human demonstration data.

In FIG. 6 (b) shows the FS-ER of the proposed technology proposed for skipping an on-demand (on-demand) frame during a sampling process. When the first type of frame skipping (FS-ER skipping, 121) is applied, the data processing unit 120 may stack a number of frames corresponding to the frame skipping coefficient at intervals corresponding to a preset number of frames. have. Here, the preset number may be set by user input. For example, the preset number may be 4. Therefore, when the FS-ER skipping 121 is applied, the data processing unit 120 generates a frame state with respect to the human demonstration data 111 ′, at each interval corresponding to a preset number of frames (for example, A number of frames (for example, four frames) corresponding to the frame skipping coefficient may be stacked for every interval corresponding to four frames, every four intervals.

In other words, when the frame skipping coefficient is 4, for example, in order to look similar to conventional frame stacking for deep reinforcement learning, the data processing unit 120 performs FS-ER skipping. When applying (121), it is possible to create a frame state by stacking four frames every four intervals. That is, the device 100 stacks 4 frames every 4 intervals for the application of the FS-ER skipping 121, whereas the conventional DQfD stacks 4 consecutive frames without spaces between frames. There is a difference in that.

Accordingly, during the FS-ER skipping 121, the frame skipping interval may not be varied and may be kept constant. In this case, the frame skipping interval maintained constant is a preset number of frames (for example, 4 ) May mean an interval corresponding to. That is, when applying the FS-ER skipping 121, the data processing unit 121 uniformly sets the frame skipping interval to an interval corresponding to a preset number of frames (for example, an interval corresponding to 4 frames). I can.

On the other hand, the redefinition unit 130 inputs human demonstration data 111 ′ sampled by data processing by the data processing unit 120 and previously sampled actor data 112 ′ stored in the actor replay buffer 112. It is possible to redefine a return value required to calculate a loss value (loss function) used when updating the neural network 50. That is, the actor data 112 ′ may be pre-sampled and stored in the actor replay buffer 112.

The redefinition unit 130 may redefine a return value (return value) for each of the application of the two types of frame skipping 121 and 122. That is, the redefinition unit 130 is the return value (return value) of the neural network 50 required to obtain a loss function (loss value) applicable to the FS-ER skipping 121 technology and the DFS-ER skipping 122 technology. ) Can be redefined. In other words, the redefinition unit 130 may redefine the method of obtaining a return value (return value) for obtaining a loss function for each application of the FS-ER skipping 121 and the application of the DFS-ER skipping 122. I can.

As shown in Equation 1 above, the redefinition unit 130 calculates the loss of DQfD: 1-step Q loss, n-step Q loss, and supervised loss. To construct, it is possible to calculate the reward and action of the selected frame as well as n-step return. The equation of n-step return can be expressed as Equation 5 below.

[Equation 5]

In addition, the redefinition unit 130, in order to redefine the applicable return value of the FS-ER skipping 121, the compensation value ( R _t ) of each frame corresponding to the frame skipping coefficient as shown in Equation 6 below. It can be calculated by the sum of the compensation of the number of frames (4 frames) series. Equation 6 means the equation for the compensation value of each frame. This is because the deep reinforcement learning environment generally returns the sum of all rewards between frames skipped as rewards.

[Equation 6]

[Equation 7]

The redefinition unit 120 may redefine the n-step return of FS-ER as a return value applicable to the FS-ER skipping 121 proposed in the present application to satisfy Equation 7. In other words, the n-step return of the FS-ER redefined by the redefinition unit 120 may be expressed as in Equation 7 above. That is, Equation 7 means the equation for the n-step return value of FS-ER.

를 곱한

를 최대 Q 값(maximum Q value)으로 더하여 산출될 수 있다. 이에 따르면, FS-ER의 n-단계 반환(n-step return) 값은 상기 식 6 및 상기 식 7을 이용하여 계산될 수 있다.The n-step return of FS-ER is a decayed R _t calculated by Equation 6 below for each interval corresponding to a preset number of frames (for example, 4 frames). It can be calculated by summing. In addition, the return of the n-step of FS-ER is the decay value after the n-step

Multiplied by

It can be calculated by adding to the maximum Q value. According to this, the n-step return value of FS-ER can be calculated using Equations 6 and 7 above.

Through the application of the FS-ER skipping 121, the device 100 does not drop any data on the human demonstration data 111' (that is, without being wasted, without being deleted). ), the entire 60Hz human demonstration data can be fully used (utilized). By using (adopting) FS-ER, the present application guarantees higher diversity of sampled training data compared to the conventional frame skipping method using the same number of episodes. can do. That is, the present apparatus 100 can utilize all of the human demonstration data 111 ′ as learning data without at least partially discarding the human demonstration data 111 ′ through the application of the FS-ER skipping 121. Yes, it can guarantee the diversity of training data. Due to the diversity of training data, the apparatus 100 may increase training efficiency.

A detailed description of the application of the DFS-ER skipping (DFS-ER frame skipping, 122) technique by the data processing unit 120 is as follows.

DFS-ER is the addition of dynamic frame skipping to FS-ER in order to process action repetition of human demonstration data 111' through online frame skipping.

The data processing unit 120 applies a second type of frame skipping (DFS-ER, 122) in consideration of an action repetition value indicating the action duration of each frame of the human demonstration data 111 ′. Data processing can be performed. Here, the action repetition value is two skip values (i.e. fs ₁ And fs ₂ ).

At this time, when the DFS-ER skipping 122 is applied, the data processing unit 120 checks the skip value for each frame when generating the frame state in generating the frame state for the human demonstration data 111 ′. , It is possible to generate a frame state in consideration of the checked skip value.

Through the generation of the frame state in consideration of the skip value checked for each frame as described above, the apparatus 100 may dynamically apply frame skipping when the DFS-ER skipping 122 is applied.

That is, when the DFS-ER skipping 122 is applied by the data processing unit 120, the frame skipping interval may be changed to dynamically perform skipping. In this case, the frame skipping interval that is dynamically skipped (In other words, the frame skipping interval during dynamic skipping) may be dynamically set by a skip value of an action (ie, a skip value related to an action repetition value) of the human demonstration data 111'. In other words, when the DFS-ER skipping 121 is applied, the data processing unit 120 sets the frame skipping interval to a skip value of the action repetition value for each frame of the human demonstration data 111' (i.e., fs ₁ And fs ₂ ) can be dynamically set (defined).

In order to apply the DFS-ER skipping 122, the data processing unit 120 includes the number of frames corresponding to the frame skipping coefficient at each interval dynamically set by the skipping value of the action repetition value (for example, 4 frames). ) Can be stacked.

In other words, the DFS-ER skipping (dynamic frame skipping, 122) operation by the data processing unit 120 is human demonstration data 111 ′ defined according to how long the current action of each frame lasts. ) Can be operated (working) based on the action repetition value of each frame.

For example, when the action repetition value of the current frame is 7, the action is maintained for 7 frames after this frame (current frame). Can be. The action repetition value is fs ₁ , which is the hyper-parameters for each of the small and big skip values. And fs ₂ can be abstracted into two skip values. A description of the abstraction of these two skip values may be more easily understood with reference to the conventional document 6 described above as an example, and a detailed description thereof will be omitted herein.

These, two skip values, fs ₁ And fs ₂ are the main roles (adapting, adapting, and adjusting) dynamic frame skipping to frame stacking, reward, and n-step return. role) can be performed.

If the DFS-ER stacks frames at fixed spacing intervals as shown in FIG. 6(b), spacing intervals between frames of human demonstration data 111' ) Is different from the interval steps of actor data, and thus may be stored in the actor replay buffer 112 after dynamic frame skipping (DFS-ER, 122).

In addition, the data processing unit 120 calibrates the difference between the behaviors of the human (human) and the actor (current frame) of the human demonstration data 111 ′. ) repeat of the action values (action repetition value) to Figure 7 the algorithm shown in 1 ₍₁ fs in accordance with algorithm 1) fs _{2 _condition} the (condition) condition of fs _2, as shown in And fs ₂ can be classified. Description of Algorithm 1 will be described later in more detail.

As can be seen from the difference between (b) of Figure 6 and (c) of Figure 6, the data processing unit 120, for the application of the DFS-ER skipping 122, for example, four frames When generating (configuring) a plurality of frame states (i.e., S _pre , S _post and S _{n -step} respectively) by _stacking , checks the skip value of every frame. )can do.

For example, a description of the process of generating S _post for application of the DFS-ER skipping 122 is as follows. As an example, assuming that the index of the selected frame is t , the data processing unit 120 stacks a frame x _t and three frames before it to generate an S _post . ). In order to stack these four frames, the data processing unit 120 may check (backwards) skip values of each of the four frames.

의 스킵 값(skip values)을 확인할 수 있다. 만약,

의 스킵 값이 fs ₁ 이면, 이(

)는 S _post 에 추가될 수 있다. 그렇지 않고, 만약

의 스킵 값이 fs ₂ 이면, 이(

)는 S _post 에 추가될 수 없다. 그 이유는, y _t 와 y _{t +1} 사이의 차이(difference)(즉, fs ₁ )가

의 스킵 값(즉, fs ₂ )과 동일하지 않기 때문이라 할 수 있다. 이러한 과정은 S _post 의 네 프레임(4개의 프레임, four frames)이 채워질 때까지 반복될 수 있다. Referring to Figure 6 (c), for example, the data processing unit 120

You can check the skip values of. if,

The skip value of fs ₁ If, if (

) Can be added to S _post . Otherwise, if

The skip value of fs ₂ If, if (

) Cannot be added to S _post . The reason is the difference between y _t and y _{t +1} (i.e. fs ₁ )end

This is because it is not the same as the skip value of (ie, fs ₂ ). This process can be repeated until four frames (four frames) of S _post are filled.

의 스킵 값(skip values)을 확인할 수 있다. 만약,

의 스킵 값이 fs ₁ 이면, 데이터 처리부(120)는

가 S _post 에 추가(반영)되도록 프레임 상태 S _post 를 생성할 수 있다. 그렇지 않고, 만약

의 스킵 값이 fs ₂ 이면, 데이터 처리부(120)는

가 S _post 에 추가(반영)되지 않도록 하여 프레임 상태 S _post 를생성할 수 있다. 이러한 과정은 S _post 의 네 프레임(4개의 프레임, four frames)이 채워질 때까지 반복될 수 있다. 즉, 데이터 처리부(120)는 이러한 과정을 S _post 에 4개의 프레임이 포함될 때까지 반복 수행할 수 있다.In other words, the data processing unit 120 applies the DFS-ER skipping 122,

You can check the skip values of. if,

The skip value of fs ₁ If the back, the data processing unit 120

That can generate the frame state S to _post addition (reflected) in the S _post. Otherwise, if

The skip value of fs ₂ If the back, the data processing unit 120

A frame status to prevent more (reflected) to the _post S S _post Can be created. This process can be repeated until four frames (four frames) of S _post are filled. That is, the data processing unit 120 may repeat this process until four frames are included in S _post .

On the other hand, _pre S and S _{n -} generating a _step may be understood in the same or similar as the procedure for generating a _post S, thus less detailed description will be omitted.

The data processing unit 120 may perform DFS-ER skipping 122 based on Algorithm 1 shown in FIG. 7.

7 is a diagram illustrating an algorithm for an operation of the DFS-ER skipping 122 by the data processing apparatus 100 for deep reinforcement learning according to an embodiment of the present application.

Referring to FIG. 7, the following parameters may be given as input values in Algorithm 1. In other words, as input to the algorithm for DFS-ER, M _human is a human replay buffer, M _actor is an actor replay buffer, and the number of pre-train steps ) of k, human replay buffer size (human replay buffer size) size of _human, the starting conditions of the pre-training defined by size _human (the pretrain start condition), the skipping condition _{start pretrain, fs 2 (skip condition} ) the fs _{2 _condition,} action repeat value of the selected frame x _t number of actions in which a _expert, and the environment (environment) (action repetition value) of repeat _t, action index (action index) of x _t (e.g., ALE ), action_dim may be considered.

In order to obtain a reward for human data 111' having a dynamic time interval, such as a reward of the actor data 112', the redefinition unit ( 130) may modify an equation of a reward in the human demonstration data. That is, the redefinition unit 130 obtains a reward for the human demonstration data 111 ′ having a dynamic time interval, such as a reward for the actor data 112 ′, in the human demonstration data 111 ′. You can modify the equation of compensation.

In order to adapt the dynamic frame skipping of DFDQN in DQfD, the redefinition unit 130 performs dynamic frame skipping (DSR-ER, 122) because n-step return is not considered in DFDQN. In order to reflect the considered correct n-step return, the n-step return of DQfD can be modified and redefined. That is, the redefinition unit 130 may modify the n-step return of DQfD to redefine the n-step return (n-step return value) of DFS-ER to satisfy Equation 11 described later.

As shown in line 3 and line 4 of Algorithm 1 in FIG. 7, a dynamic frame skipping method (DFS-ER skipping, 122) is used for reward and n-step return. In order to apply, the redefinition unit 130 is a reward R _t In order to obtain candidates for (that is, R _t , which is a reward of the selected frame x _t ), partial rewards of a series of fs ₁ or fs ₂ may be summed. That is, the compensation of the series of fs ₁ or fs ₂ can be partially summed.

In addition, for redefinition of n-step return, the redefinition unit 130 not only finds both R _t and y , but also skips the value of the action to obtain these two variables ( R _t and y ). You can (skip, omitted). Here, y may mean an index of a frame after dynamically skipping the frame.

In the human demonstration data 111 ′, a skip value of an action may be defined as an action value according to line 5 of Algorithm 1 in FIG. 7. If, action value is action_dim less than (that is, the action_dim if repeat _t is fs _{2 _} smaller than the _condition), a skip value of the action (skip value associated with the action repeat value) is fs _1, otherwise be fs ₂ have.

In Algorithm 1, M _human , a human replay buffer, can satisfy Equation 8 below. Reward R _t considered when applying DFS-ER(122) May be selected as one of R _r1 and R _r2 as shown in Equation 9 below (that is, by checking the action value a _i of the current frame). In addition, when the DFS-ER 122 is applied, the apparatus 100 uses the following equation 10 (i.e., by checking the skip value of the current frame x _t ) after the skipped frame index y You can find the next dynamic frame.

[Equation 8]

[Equation 9]

[Equation 10]

The device 100 uses R _i in Equation 9 and y _{i in} Equation 10 After calculating, it is possible to obtain an n-step return value of DFS-ER through Equation 11 below.

That is, the redefinition unit 30 can redefine the n-step return of DFS-ER as a return value applicable to the DFS-ER skipping 122 proposed by the present application to satisfy Equation 11 below. have. In other words, the n-step return of the DFS-ER redefined by the redefinition unit 120 may be expressed as Equation 11 below. Equation 11 refers to the equation for the n-step return value of DFS-ER.

[Equation 11]

The apparatus 100 can decay the reward by using Equation 4 in Equation 11.

The redefinition unit 130 may redefine a return value (return value) for obtaining a loss function applicable to FS-ER skipping as shown in Equation 7 above, and to obtain a loss function applicable to DFS-ER skipping. The return value (return value) can be redefined as in Equation 11 above. Based on the return value redefined by the redefinition unit 130, the value of the loss function that can be used when applying the online frame skipping technique considered in the device 100 using Equation 1 (i.e., A loss value used when updating a neural network) can be calculated.

There are two differences between the conventional FiGAR shown in FIG. 2B and the DFS-ER skipping 122 proposed by the present application.

The first difference is that FiGAR has a skip value of a specific range (a certain range), whereas the DFS-ER skipping 122 of the present proposed technology is a case of two frame skipping (ie, online There is a difference in that it has frame skipping (FS-ER) and dynamic online frame skipping (DFS-ER).

을 감마(gamma,

)의 지수(exponent)로 사용한다는 점에서 차이가 있다. 그 이유는 fs ₁ 스킵 값이 n 단계 Q 손실(n-step Q loss)로 에이전트(agent)를 훈련할 때 하나의 감쇄 단위(one decay unit)로 사용되기 때문이라 할 수 있다.The second difference is that the DFS-ER 122 of the proposed technology is yi Instead of using

Gamma,

There is a difference in that it is used as an exponent of ). The reason is fs ₁ This is because the skip value is used as one decay unit when training an agent with n-step Q loss.

According to this, the apparatus 100 can solve a problem in which high quality human demonstration data is limited to most real-world cases. That is, conventional reinforcement learning generally performs learning using high-quality learning data in order to calculate higher accuracy (for effective performance improvement). For this reason, there is a limit to improving the performance of reinforcement learning in a situation where it is difficult to sufficiently secure high-quality learning data.

In a situation where the use (use) of high-quality learning data is difficult, in order to increase the performance of reinforcement learning, valuable data (this is high-quality data, which can mean human demonstration data for example) is not wasted. It needs to be used effectively. However, conventionally, as frame skipping is performed on a single replay buffer in which both human demonstration data and actor data are stored, there is a problem in that human demonstration data is discarded. That is, according to this prior art, as valuable data (human demonstration data) is discarded by the application of frame skipping, the amount of learning data that can be utilized (used) when learning a neural network decreases (that is, There was a problem that the learning performance of the neural network was degraded because the amount of human demonstration data was insufficient.

Therefore, in order to effectively improve the performance of reinforcement learning (that is, to accelerate reinforcement learning) in situations where it is difficult to secure enough high-quality learning data, we use all human demonstration data as training data for neural network learning without being discarded. As a possible technology, we propose a dual replay buffer-based online frame skipping technology.

That is, the present application proposes a new deep reinforcement learning technique using dual replay buffer management and online frame skipping 121 and 122 for sampling the human demonstration data 111 ′.

According to the dual replay buffer (121, 122) management technology proposed by the present application, the replay manager 110 uses an independent sampling policy to provide a human replay buffer and an actor replay buffer. buffer) can be processed separately (separately, separately).

In addition, the present apparatus 100 is to enable all of the high-quality human demonstration data discarded due to the conventional frame skipping technology to be used without being discarded (that is, in order to enable the entire human demonstration data to be used,) FS -ER skipping 121 technology is proposed. In addition, the present apparatus 100 proposes a DFS-ER skipping 122 technique, which is a dynamic online skipping technique, in order to train action repeat of the human demonstration data 111'.

That is, in an environment where reinforcement learning cannot be performed with high-quality learning data (that is, even in an environment where high-quality learning data cannot be sufficiently secured), the human demonstration data, which is high-quality learning data, is discarded. It is possible to effectively improve the performance (speed and accuracy) of reinforcement learning by making it fully utilized as learning data for reinforcement learning (all human demonstration data is utilized without waste).

In other words, even in an environment where reinforcement learning cannot be performed with high-quality learning data, the apparatus 100 may provide reinforcement learning with high performance through sufficient utilization of human demonstrations.

This application proposes a technology for processing data used in deep reinforcement learning. The present application utilizes all high-quality human data discarded due to conventional frame skipping without discarding data, thereby improving the results and performance of deep reinforcement learning.

In addition, unlike the prior art, in the present application, by separately constructing a human replay buffer and an actor replay buffer, online frame skipping can be easily applied only to human data. In addition, the present application is a FS-ER skipping technology that performs skipping by maintaining a constant frame skipping interval, and DFS-ER skipping that performs skipping by varying the frame skipping intervals. By applying the technology, all human data can be used (utilized) without wasting it.

In addition, in this application, a method of calculating the return (return) value of a neural network for obtaining a loss function applicable to the FS-ER skipping technology and the DFS-ER skipping technology may be redefined. .

8 is a block diagram showing a schematic configuration of an apparatus 1 for deep reinforcement learning of a neural network according to an embodiment of the present application.

Referring to FIG. 8, the apparatus 1 for deep reinforcement learning of a neural network according to an exemplary embodiment of the present disclosure may include a data processing apparatus 100 and a learning control unit 200.

Here, the data processing apparatus 100 may refer to a data processing apparatus 100 (the present apparatus) for deep reinforcement learning according to an exemplary embodiment of the present disclosure described above. Accordingly, even if the contents are omitted below, the contents described above with respect to the apparatus 100 may be equally applied to the description of the data processing apparatus 100 illustrated in FIG. 8.

The data processing apparatus 100 may perform data processing for deep reinforcement learning on target data.

The data processing apparatus 100 may store human demonstration data among target data in a human replay buffer, and store actor data among target data in an actor replay buffer. In addition, the data processing apparatus 100 may perform data processing by applying online frame skipping to the human replay buffer in order to sample the human demonstration data stored in the human replay buffer.

Here, online frame skipping may include two types of frame skipping in consideration of whether or not the interval of frame skipping is variable.

Online frame skipping is a first type of frame skipping, and may include FS-ER skipping in which skipping is performed by keeping the frame skipping interval constant without changing. In addition, online frame skipping is a second type of frame skipping, and may include DFS-ER skipping, which dynamically performs skipping by varying an interval of frame skipping.

In addition, the data processing device 100 calculates a loss value (loss function) used when updating the neural network 50 using the human demonstration data sampled by data processing and the actor data previously sampled and stored in the actor replay buffer as inputs. You can override the return value required to do so.

The learning control unit 200 may perform deep reinforcement learning of the neural network 50 by using human demonstration data and actor data sampled by data processing of the data processing apparatus 100 as inputs of the neural network 50.

The deep reinforcement learning device 1 of a neural network according to an embodiment of the present application effectively improves the performance (speed and accuracy) of reinforcement learning by using (using) all human demonstration data to deeply reinforce the neural network 50 I can make it.

Hereinafter, an experimental example of the present application for evaluating (proving) the performance of the data processing apparatus 100 for deep reinforcement learning according to an exemplary embodiment of the present application will be described.

In one experiment of the present application, in order to evaluate the performance of the device 100 (that is, to evaluate the performance of the proposed technology), empirical experiments on four popular Atari games. ) Was performed. The results show that the two online frame skipping techniques that use dual replay memory, which is the proposed technique, are outperforms the existing baselines.

In particular, DFS-ER was shown to show the fastest score increment during the reinforcement learning process in 3 of the 4 experiments. In addition, FS-ER has been shown to perform best in other environments where it is difficult to train the model due to sparse rewards.

In other words, in one experiment of the present application, empirical experiments were performed using the DQfD model of Document 3 and four Atari Grand Challenge Datasets. Here, four Atari Grand Challenge Datasets are described in Document 8 [V. Kurin, S. Nowozin, K. Hofmann, L. Beyer, and B. Leibe, "The atari grand challenge dataset," CoRR, vol. abs/1705.10998, 2017. [Online]. Available: http://arxiv.org/abs/1705.10998].

According to the experimental results, the FS-ER was found to be superior to other conventional methods in Montezuma's Revenge, one of the four evaluation environments. Montezuma's Revenge is a difficult environment to train reinforcement learning models due to sparse rewards. In addition, the DFS-ER of the proposed technology was found to be superior to other conventional methods in three of the four evaluation environments. In particular, in the pre-training stage, the average score of DFS-ER was improved compared to the conventional method. In addition, the averaged score increased rapidly in the reinforcement learning stage.

A more detailed description of one experiment of the present application (hereinafter referred to as this experiment for convenience of description) is as follows.

First, the description of the evaluation settings in this experiment is as follows.

In this experiment, an empirical experiment was conducted on the Arcade Learning Environment (ALE) to evaluate the effectiveness of the proposed technology. In this experiment, a test program was implemented using Tensorflow 1.8 and the Atari Grand Challenge v1 dataset was used for human demonstration data. The replay buffer class considered in this experiment may be an extended version of the replay buffer.

In addition, in this experiment, the DQfD model shown in Document 3 was used as a test neural network model and a baseline model for performance comparison. In addition, in order to use the DQfD model with an OpenAI baseline replay buffer, the replay buffer class was extended by adding the methods of DFSER and FS-ER.

In addition, with regard to the use of the Atari Grand Challenge dataset, only 4 out of 5 games were used. The reason for this is that the video pinball (Fig. 3(b)) of the five games can be said to be very easy to obtain a high score even if the action is performed according to a random uniform policy in this experiment. Therefore, in this experiment, video pinball was excluded from evaluation environments.

In addition, in this experiment, the size of the human replay buffer 111 may be 50,000 when frame skipping is not performed on the human demonstration data. However, in order to evaluate the conventional method of skipping frames before storing human demonstration data in the human replay buffer, the size of the human replay buffer 111 considered in this experiment is 12,500 (i.e., 5 It can be set to ten thousandths)

Also, in this experiment, human demonstration data can be stored so that only the human demonstration data that received high-scored (high-scored) after sorting the episode scores of each game in descending order can be used. In addition, the size of the actor replay buffer 112 may be set equal to the size of the human replay buffer 111.

In addition, the parameters (parameters) used in this experiment may be set as follows.

The size of the mini-batch can be set to 32. The total pre-training steps may be set to 500,000 steps, and the total reinforcement learning steps may be set to 1,000,000 steps. The learning rate may be set to 0.0001. Also, fs ₁ Skip value is 4, fs ₂ The skip value of is 12, fs ₂ The condition value of may be set to 70. In addition, the maximum episode length of the environment may be set to 50,000 steps. Values of λ ₁ , λ ₂ , λ ₃ and λ ₄ of DQfD may be set to 1.0, 1.0, 1.0, and 0.0001, respectively.

This experiment for evaluating the proposed technology may focus on comparing a conventional frame skipping technology through single replay buffer management and an online frame skipping technology through dual replay buffer management, which is the present technology.

In this experiment, three approaches were evaluated, and each of the four experiments was averaged using random seeds.

In other words, the three approaches mean single, FS-ER and DFS-ER. Single (SINGLE) refers to a frame skipping technology based on the conventional single replay buffer management. FS-ER is the first type of frame skipping technology proposed in the present application, and refers to an FS-ER skipping technology based on dual replay buffer management. DFS-ER is the second type of frame skipping technology proposed in the present application, and refers to a DFS-ER skipping technology based on dual replay buffer management.

In order to keep the human sampling rate of each reinforcement learning step the same, in this experiment, a sampling ratio policy was set in which the dual replay buffer management mimics the single replay buffer management.

9 shows the number of human data in a mini-batch as an experiment result of the present application. That is, FIG. 9 shows an example of sampling human demonstration data for dual replay buffer management and single (single) replay buffer management techniques.

Referring to FIG. 9, the reason that the difference between the dual replay buffer management and the single (single) replay buffer management technology is small is that the number of human demonstration data sampled from the human replay buffer is converted into an integer.

Hereinafter, the evaluation result by this experiment will be described.

10 is a diagram showing an average episode score of three games as an experiment result of the present application. In particular, FIG. 10 shows Q*bert (a), Ms. For each of the three games including Pacman (b) and Montezuma's Revenge (c), showing the average episode score when applying the three approaches described above, SINGLE, FS-ER and DFS-ER It is a drawing.

Figure 10 (a) shows a comparison of the evaluation results of the Q * bert game.

Referring to (a) of FIG. 10, it can be seen that the FS-ER shows excellent performance of other methods in a pre-training step. This can be said to be a meaningful result because the FS-ER technology proposed in the present application obtained a higher score than the result reported in the above-described document 7 which discloses the conventional DQfD using human demonstration data having a high score.

In addition, it can be seen that the proposed technology, DFS-ER, shows a better score than the conventional method, even though the output dimension of the neural network is doubled. This means that the proposed technique, DFS-ER, has a positive effect in the pre-training stage.

In addition, in the Q*bert results, FS-ER and single (SINGLE) were shown to get closer as they train. In addition, it can be seen that the score of DFS-ER is better than other methods at the end of the graph (the right part of the graph on the drawing) through rapid episode detection of dynamic frame skipping.

Figure 10 (b) shows a comparison of the evaluation results of the Pacman (Ms. Pacman) game.

Referring to FIG. 10B, the pattern of the evaluation result for the Pac-Man game was different from the pattern of the Q*bert game shown in FIG. 10A. DFS-ER was found to have a slightly higher score than other methods in the pre-training stage of the Pac-Man game, and showed excellent performance in the reinforcement learning stage. This shows that learning of action repetition of human demonstration data using DFS-ER is effective.

On the other hand, the evaluation result of the Space Invaders game appeared almost the same as the pattern of the evaluation result of Pac-Man shown in FIG. 10B.

FIG. 10C shows a comparison of evaluation results of Montezuma's Revenge game.

Referring to Figure 10 (c), unlike Figures 10 (a) and (b), in the case of a multiple game of Montezuma, since the length of the episode is longer than that of the others, an epoch (one execution cycle of neural network learning) ) The length (i.e., the average score interval) was increased to 100,000.

In the case of Montezuma's revenge game, FS-ER was found to outperform other methods at the end of the graph (right side of the graph on the drawing). This result means that FS-ER is using human demonstration data more efficiently than the existing method by observing that the entire human demonstration data does not drop in any frame.

11 is a view showing a summary of the average score achieved in four games, as an experiment result of the present application. Specifically, FIG. 11 is a diagram showing the average score shown when applying the above-described three approaches, SINGLE, FS-ER, and DFS-ER for four games. That is, FIG. 11 is a view summarizing the average score of all environments considered in this experiment.

In FIG. 11, the Pretrain column denotes the average score of four experiments with different random seeds at the end of a pre-training step. In addition, the RL column means the average score at the end of the reinforcement learning step. In addition, Figure 11 shows, in the evaluation of the three approaches (single, FS-ER, DFS-ER) in each game, the entire training epoch (i.e., the entire training process including the pre-training step and the reinforcement learning step). A column for the maximum average score is shown.

Referring to FIG. 11, for all cases (i.e., for each of the Pretrain column, RL column, and maximum average score column), the FS-ER skipping technology or the DFS-ER skipping technology proposed herein is a conventional single ( SINGLE) technology (ie, skipping technology based on a single replay buffer) showed higher performance.

That is, according to the results of this experiment, the dual replay buffer (memory) architecture with the experimental results of the four Atari games described above and the online skipping technology (FS-ER, DFS-ER) It has been confirmed that the proposed technology providing a better performance than existing baselines (existing baselines).

DFS-ER showed the fastest score increment during pre-training in 3 out of 4 environments, showing superior performance than other conventional methods. In addition, FS-ER has been shown to perform best in other environments where it is difficult to train the model due to sparse rewards (eg, Montezuma's Revenge).

The experimental results according to the experiment of the present application show that the effect of the proposed technology on the use of limited human demonstration data to reinforce the training problem is demonstrated.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly described.

12 is a flowchart of a data processing method for deep reinforcement learning according to an embodiment of the present application.

The data processing method for deep reinforcement learning shown in FIG. 12 may be performed by the apparatus 100 described above. Accordingly, even if the contents are omitted below, the contents described with respect to the apparatus 100 may be equally applied to the description of the data processing method for deep reinforcement learning.

Referring to FIG. 12, in step S110, human demonstration data among target data may be stored in a human replay buffer, and actor data among target data may be stored in an actor replay buffer.

Next, in step S120, in order to sample the human demonstration data stored in the human replay buffer in step S110, online frame skipping may be applied to the human replay buffer to perform data processing.

Here, online frame skipping may include two types of frame skipping in consideration of whether or not the interval of frame skipping is variable.

In addition, the online frame skipping, as the first type of frame skipping, may include FS-ER skipping in which skipping is performed by maintaining a constant frame skipping interval without varying. In addition, online frame skipping, as a second type of frame skipping, may include DFS-ER skipping, which dynamically performs skipping by varying an interval of frame skipping.

In addition, in step S120, a frame state for the human demonstration data stored in the human replay buffer may be generated prior to the application of the online frame skipping. In this case, the frame state may be generated by stacking a number of frames corresponding to each frame skipping coefficient.

Further, in step S120, when the first type of frame skipping is applied, a number of frames corresponding to the frame skipping coefficient may be stacked at intervals corresponding to a preset number of frames.

Also, in step S120, the second type of frame skipping may be applied in consideration of an action repetition value indicating an action duration of each frame of the human demonstration data. Here, the action repetition value may include two skip values.

In addition, in step S120, when the second type of frame skipping is applied, when generating a frame state, a skip value for each frame may be checked, and a frame state may be generated in consideration of the checked skip value.

In addition, although not shown in the drawing, the data processing method for deep reinforcement learning according to an embodiment of the present application is used when updating a neural network that inputs human demonstration data and actor data sampled by the data processing in step S120. It may include redefining the required return value to calculate the lost value.

Here, in the redefining step, the return value may be redefined for each of the two types of frame skipping applications.

In the above description, steps S110 and S120 may be further divided into additional steps or may be combined into fewer steps, depending on the embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The data processing method for deep reinforcement learning according to an exemplary embodiment of the present disclosure may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

Further, the above-described data processing method for deep reinforcement learning may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

100: Data processing unit for deep reinforcement learning
110: Replay Manager
111: human replay buffer
112: Actor replay buffer
120: data processing unit
130: Ministry of Finance

Claims (12)

In the data processing method for deep reinforcement learning in which each step is performed by a data processing device implemented as a computer,
(a) storing human demonstration data among target data in a human replay buffer, and storing actor data among the target data in an actor replay buffer; And
(b) for sampling the human demonstration data stored in the human replay buffer, performing data processing by applying online frame skipping to the human replay buffer,
The online frame skipping includes two types of frame skipping in consideration of whether or not an interval of frame skipping is variable. The method of claim 1,
The online frame skipping,
As a first type of frame skipping, it includes FS-ER skipping, which performs skipping by maintaining a constant frame skipping interval without changing,
The second type of frame skipping, which includes DFS-ER skipping, which dynamically performs skipping by varying an interval of frame skipping. The method of claim 2,
The step (b),
Before the application of the online frame skipping, a frame state for human demonstration data stored in the human replay buffer is generated,
The frame state is generated by stacking a number of frames corresponding to each frame skipping coefficient. The method of claim 3,
The step (b),
When the first type of frame skipping is applied, a number of frames corresponding to the frame skipping coefficients are stacked at intervals corresponding to a preset number of frames. The method of claim 3,
The step (b),
Applying the second type of frame skipping in consideration of an action repetition value indicating an action duration of each frame of the human demonstration data,
The action repetition value includes two skip values. A data processing method for deep reinforcement learning. The method of claim 5,
The step (b),
When the second type of frame skipping is applied, a skip value for each frame is checked when generating the frame state, and the frame state is generated in consideration of the checked skip value, for deep reinforcement learning Data processing method. The method of claim 1,
(c) redefining a return value required to calculate a loss value used when updating a neural network using the human demonstration data sampled by the data processing in step (b) and the actor data as inputs,
Including more,
The step (c) is to redefine a return value for each of the two types of frame skipping application, the data processing method for deep reinforcement learning. In the data processing device for deep reinforcement learning,
A replay manager that stores human demonstration data among target data in a human replay buffer, and stores actor data among the target data in an actor replay buffer; And
A data processing unit that performs data processing by applying online frame skipping to the human replay buffer for sampling the human demonstration data stored in the human replay buffer,
The online frame skipping includes two types of frame skipping in consideration of whether or not an interval of frame skipping is variable. The method of claim 8,
The online frame skipping,
As a first type of frame skipping, it includes FS-ER skipping, which performs skipping by maintaining a constant frame skipping interval without changing,
As the second type of frame skipping, the data processing apparatus for deep reinforcement learning includes DFS-ER skipping, which dynamically performs skipping by varying an interval of frame skipping. The method of claim 9,
The data processing unit,
Before the application of the online frame skipping, a frame state for human demonstration data stored in the human replay buffer is generated,
The frame state is generated by stacking a number of frames corresponding to each frame skipping coefficient. In the deep reinforcement learning device of a neural network,
A data processing device that performs data processing for deep reinforcement learning on target data; And
And a learning control unit for deep reinforcement learning of the neural network by using human demonstration data and actor data sampled by data processing by the data processing device as inputs of the neural network,
The data processing device,
A replay manager that stores human demonstration data among target data in a human replay buffer, and stores actor data among the target data in an actor replay buffer; And
A data processing unit that performs data processing by applying online frame skipping to the human replay buffer for sampling the human demonstration data stored in the human replay buffer,
The online frame skipping includes two types of frame skipping in consideration of whether or not the frame skipping interval is variable. The method of claim 11,
The online frame skipping,
As a first type of frame skipping, it includes FS-ER skipping, which performs skipping by maintaining a constant frame skipping interval without changing,
The second type of frame skipping, which includes DFS-ER skipping, which dynamically performs skipping by varying the frame skipping interval.

Images