KR20220162096A - Deep Neural Network Structure for Inducing Rational Reinforcement Learning Agent Behavior - Google Patents

Abstract

In a step of executing the reinforcement learning agent of a deep neural network, a behavior induced by a user is provided as input to the reinforcement learning agent. The reinforcement learning agent calculates, as a final value function value, a value obtained by subtracting a parameter value for reflecting the forcibleness of the induced behavior from the output value of a value function of all the rest behaviors excluding the induced behavior. A final behavior to be performed is determined using the calculated final value function value. The parameter value can be changed to adjust the degree of forcibleness of the induced behavior. The present invention can effectively respond to a situation in which a reinforcement learning agent user desires to induce or command an agent to conduct a specific behavior after violating a greedy behavior decision method for selecting a behavior with the highest expected cumulative reward among behaviors that can be conducted in the current state.

Description

Method for Inducing Rational Behavioral Decisions of Reinforcement Learning Agents of Deep Neural Networks {Deep Neural Network Structure for Inducing Rational Reinforcement Learning Agent Behavior}

The present invention relates to a reinforcement learning agent in a deep neural network, and more particularly, to a method for inducing action decisions in a reinforcement learning agent.

Various studies on deep reinforcement learning combining reinforcement learning and deep learning are applied not only to games such as Atari games, Go, and Dota 2, but also to fields such as computer vision, intelligent robots, and natural language processing, and are continuing to develop very successfully. The detailed structure of reinforcement learning models differs depending on what each model aims, but they all have in common sequential decision-making in which the expected value of cumulative reward is maximized in an environment that can be expressed as a Markov Decision Process (MDP). learn to get off

1 is a conceptual diagram of a general reinforcement learning model that proceeds with reward-based learning. As shown, the agent of reinforcement learning determines and performs the next action (A _t ) using a policy, which is a function that determines the action to be performed when information on the state (S _t ) is given as an input . The agent's environment returns a reward (R _t+1 ) based on the agent's current state (S _t+1 ) and the action taken. Through this reward signal, the agent performs self-evaluation on the previously performed actions, and then proceeds with learning by modifying the policy little by little. Therefore, a well-learned agent will have a policy that returns an action that yields a high cumulative reward for all possible states.

Even after learning is complete, the agent determines the action to be performed in a specific state based on the policy, similar to the learning phase. At this time, the behavior decision algorithm of the learned agent has the following differences from the learning stage. That is, in the learning phase, an action decision algorithm (e.g. Epsilon Greedy Algorithm), such as performing a random action instead of an action with the highest expected value of the current reward with a certain probability, is often used to allow trial and error in as many situations as possible. Adopted. On the other hand, there is a difference that trial and error is no longer necessary after learning is over, so the action with the highest expected value of reward is always greedily determined. Therefore, after learning is completed, in general, the reinforcement learning agent selects and performs the action with the highest expected cumulative reward among the actions that can be performed in the current state.

However, sometimes users may encounter a situation where they want to induce or command the reinforcement learning agent to perform a specific action, violating this greedy action decision method. Examples include actions such as changing the driving direction of an autonomous vehicle learned through reinforcement learning or adjusting the play method or behavior of a game agent.

1. "Playing Atari with Deep Reinforcement Learning," in NIPS Deep Learning Workshop, Mnih et al., 2013.

An object of the present invention is to provide a method for inducing a reasonable action decision of a reinforcement learning agent of a deep neural network that can reasonably induce an action induced by a user in an execution step of a reinforcement learning agent of a deep neural network.

A method of inducing a reasonable action decision of an agent includes providing an action induced by a user as an input to a reinforcement learning agent in a step of executing a reinforcement learning agent of a deep neural network; Calculating, in the reinforcement learning agent, a value obtained by subtracting a parameter value for reflecting the coercion of the induced action from an output value of the value function of all other actions except the induced action, as a final value function value; and determining, in the reinforcement learning agent, a final action to be performed using the calculated final value function value.

In an exemplary embodiment, the reinforcement learning agent uses the following equation as a modified value function q' to be used in determining the final action, and simultaneously considers the derived action and the current observational information and situation to determine the final action. can decide

Here, a and s represent the action and the current situation, respectively, λ represents the above parameter, i and j are the indices of action a, and q is the expected reward for performing an action in the current situation represents a value function that approximates the expected value of

In an exemplary embodiment, the method for inducing a rational action decision of the reinforcement learning agent of the deep neural network may further include changing the parameter value to adjust the degree of coercion of the induced action.

In an exemplary embodiment, the step of changing the parameter value reduces the value function output values of all other behaviors except for the specific induced behavior (a _i ) so that the reinforcement learning agent performs the induced behavior (a _i ). A step of changing the value of the parameter λ to a large value in order to increase the possibility of selecting the final action may be included.

In an exemplary embodiment, the deep neural network may be configured such that the same number of nodes as the number of actions that the reinforcement learning agent can perform exists in the last layer, and each node can be obtained when a specific action is performed in the current situation. We can learn a value function q that approximates the expected value of the expected reward.

In an example embodiment, the induced action may include a plurality of actions.

The present invention is effective in a situation in which a reinforcement learning agent user wants to induce or command a specific action to be performed, violating the greedy action decision method in which the agent selects an action with the highest expected cumulative reward among actions that can be performed in the current state. can respond That is, in the present invention, when a user induces a specific action to a reinforcement learning agent in the execution phase of a reinforcement learning agent, the final action can be reasonably determined by considering the agent's induced action and the current observational information and situation at the same time. .

Since the present invention does not modify the existing value function in the implementation stage, it has the advantage of not affecting the behavior decision after inducing behavior for a certain period of time. In addition, it can be easily applied even when there are many actions to be induced.

The present invention also makes it possible to reflect the degree of coercion of an action to be induced through an additional parameter in the agent's decision step.

Through this methodology, agent users will be able to use agents more flexibly.

1 is a conceptual diagram of a general reinforcement learning model that proceeds with reward-based learning.
Figure 2 shows the input/output structure of a deep neural network when it is assumed that the number of possible actions in deep reinforcement learning is N.
3 is a flowchart illustrating an algorithm of a method for inducing a rational action decision in a reinforcement learning agent of a deep neural network according to an exemplary embodiment of the present invention.
4 shows a new value function derivation process for making action decisions in consideration of observation information and induced actions.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. The same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

In most deep reinforcement learning with discrete action space, the action of the agent is determined in a similar way to the process of solving multi-classification problems using deep neural networks. The last layer of the deep neural network has the same number of nodes as the number of actions that the agent can perform, and each node has a value that approximates the expected value of the reward that is expected to be obtained when a specific action a _i is performed in the current situation s Learn function q.

In the process of selecting an action that makes the value of the value function q the largest by using this structure of deep reinforcement learning, the problem can be solved by adding a new structure to induce an action.

2 illustrates an input/output structure of a deep neural network according to an exemplary embodiment of the present invention.

Referring to FIG. 2 , the input/output structure of the illustrated deep neural network 100 assumes that the number of possible actions is N. In most deep reinforcement learning with discrete action space, the action of the agent is determined in a similar way to the process of solving multi-classification problems using deep neural networks. In an exemplary embodiment, the same number of nodes as the number of actions (N) that the reinforcement learning agent 110 can perform exists in the last layer of the deep neural network 100 . Each node can learn a value function q that approximates the expected value of the reward expected to be obtained when a specific action _ai is performed in the current situation s.

The reinforcement learning agent 110 may determine an action to be performed in a manner such as selecting an action that makes the value of the value function q the largest.

An exemplary embodiment of the present invention relates to a method for constraining the behavior of a reinforcement learning agent in the special case mentioned in the last paragraph of the background description above. That is, the exemplary embodiment presents the requirements necessary to constrain the behavior of the reinforcement learning agent 110 from the viewpoint of the model structure, and then the rational behavior of the reinforcement learning agent of the deep neural network that can reasonably induce a specific behavior We present a method for inducing decisions. Through the method presented in this embodiment, the reinforcement learning agent 110 will determine the final action by simultaneously considering the induced action and the current observation information and situation.

3 is a flowchart illustrating an algorithm of a method for inducing a method for inducing a rational action decision of a reinforcement learning agent according to an exemplary embodiment of the present invention.

Referring to FIG. 3, consider the case where the user sometimes wants to induce or command the agent to perform a specific action, violating the greedy action decision method of selecting an action with the highest expected cumulative reward among actions that can be performed in the current state. do. In this case, the action a _i induced or commanded by the user in the execution step of the reinforcement learning agent 110 of the deep neural network 100 may be provided as an input to the reinforcement learning agent 110 (S10).

The first thing to consider for inducing a specific behavior of a reinforcement learning agent is that induction of behavior for a certain period of time should not affect future behavior decisions. Therefore, it is necessary to use a method of inducing a specific action without modifying the parameters in the deep neural network 100 .

The second thing to consider is that the reinforcement learning agent 110 should select an appropriate action by simultaneously considering the induced action and the current observation information. That is, the reinforcement learning agent 110 should also be able to make a decision to perform another action instead of the induced action under appropriate judgment.

Then, in order to satisfy both the above two points, in the reinforcement learning agent 110, the value obtained by subtracting the parameter λ from the output value of the value function of all other actions except for the action a _i induced by the user is the final value. It can be calculated as a function value (S20). In addition, the reinforcement learning agent 110 may determine and perform a final action to be performed according to the calculated final value function (S30).

Figure 4 shows this schematically. That is, FIG. 4 shows a new value function derivation process for determining an action considering observation information and induced action. At this time, the parameter λ is a parameter to reflect the coerciveness of action a _i . The modified value function q' to be used in determining the final action can be defined as in Equation 1.

Here, a and s represent the action and the current situation, respectively, λ represents the above parameter, i and j are the indices of action a, and q is the expected reward for performing an action in the current situation represents a value function that approximates the expected value of

Specifically, q'(s, a _j , a _i ) means the output value of the value function of action a _j when a _i is given as an induced action. If a parameter λ having a large value is given, all actions except the induced action a _i will have a small value as the output value of the value function, and the probability that the reinforcement learning agent 110 will select the action a _i becomes very high . This means that high coercion can be given to the action a _i by using a parameter λ with a large value.

On the other hand, when a parameter λ having a small value is given, the action a _i for which i≠j can also have a large output value of the value function, so the reinforcement learning agent 110 considers the observed value and considers the action a _i You are left with a choice of action.

In this way, the parameter λ used to reflect the degree of coercion of the behavior is input into the model at the same time as the subscript i corresponding to the behavior desired to be induced, and the reinforcement that determines the final behavior by considering the observed information and the induced behavior at the same time A learning agent 110 may be utilized.

Additionally, the methodology can be easily applied even when there are multiple behaviors to be induced.

In an exemplary embodiment, the method for inducing a rational action decision of a reinforcement learning agent may further include changing the value of the parameter λ to adjust the degree of coercion of the induced action. In an exemplary embodiment, it is determined whether to terminate the use of the reinforcement learning agent (S40), and if the reinforcement learning agent is to be used more, the value of the parameter λ is modified to a desired value or the current value is maintained. It can (S50). In changing the parameter value, the output value of the value function of all other actions except for the specific induced action a _i is reduced so that the reinforcement learning agent 110 is more likely to select the induced action a _i as the final action. To do this, the value of the parameter λ can be changed to a large value.

In order to evaluate the performance of the reinforcement learning agent's rational action decision induction method according to an embodiment of the present invention, an experiment is conducted in the 'CartPole' environment, which is one of the 'OpenAI Gym' environments commonly used for performance evaluation of reinforcement learning agents. proceeded. The environment aims to keep a pole propped up on a cart resting on a frictionless surface from tipping over. The reinforcement learning agent can balance the bar by moving the cart left or right. The reinforcement learning agent receives a reward of +1 for every unit time in which the bar does not fall, and can proceed with learning to maximize it. Due to the nature of the environment in which the bar needs to be balanced, it can be predicted that a reinforcement learning agent that maintains a similar number of times of moving the cart to the left and to the right will receive a high reward.

For the 'CartPole' agent, which has been trained using the deep Q-network (DQN) method, evaluation is performed for 1,000 episodes without applying the method for inducing rational action decisions of the reinforcement learning agent according to the embodiment of the present invention. The results are shown in Table 1 below. Table 1 summarizes the performance of the reference model.

average reward move left move right reference model 500 250,001 249,999

Since the 'CartPole' environment includes cases where the unit time of an episode exceeds 500 as an episode end condition, an average reward of 500 means that the reference model has completed an episode without knocking over the bar in every episode. It can be seen in Table 1 that the number of times the cart was moved to the left and the number of times that the cart was moved to the right during 1,000 episodes were very similar.

In order to evaluate the performance after applying the rational action decision induction method of the reinforcement learning agent of the exemplary embodiment to the reference model, it is assumed that 'moving the cart to the right' is the desired action. In addition, in order to investigate the performance change according to the degree of coercion of the behavior desired to be induced, an experiment was conducted on two comparative models with the parameter λ set to 0.007 and 0.01. The induction of action is made only when the remainder of dividing the unit time by 10 is less than 5, and it is intended to provide the agent with an opportunity to escape from the situation where the bar tends to fall to the left. Table 2 shows the results of performance evaluation for 1,000 episodes for the two comparative models.

average reward move left move right Comparison model 1
(λ=0.007) 493.738 245,112
(49.6%) 248,626
(50.4%) Comparison model 2
(λ=0.01) 168.379 80,034
(47.5%) 88,345
(52.5%)

In the case of Comparative Model 1, which has a lower degree of coercion, the average reward was 493.738, which is close to 500, and showed high performance. In addition, 'move right', which is an action that induces the agent, accounted for 50.4% of the total actions, showing a higher rate than 'move left'. Through the results of this comparison model 1, it can be confirmed that the maximum action was induced to the agent without significantly degrading the performance.

Comparative model 2, which had a high degree of coercion, had the highest rate of 'move right', an induced action, than the other two models (52.5%), and obtained the lowest average reward with an average reward of 168.379.

In the case of the 'CartPole' environment where the experiment was conducted, there were only two types of behaviors that could be performed, and due to the nature of the environment, high performance could be expected only when the number of times the two behaviors were performed was similar, so performance degradation due to behavior induction was inevitable. If the number of actions that can be performed is greater and the number of actions performed is not similar, in an environment where high performance can be expected, when the reinforcement learning agent action induction method of the exemplary embodiment is applied, the agent performance is maintained and at the same time a more distinct agent You will be able to see changes in the pattern of behavior.

In the above, when a specific action is induced to a reinforcement learning agent according to an exemplary embodiment, a method of using a deep neural network that allows the agent to reasonably determine the final action by considering the induced action and the current observational information and situation at the same time has been suggested. . Through the proposed method, agent users will be able to use the agent more flexibly.

An exemplary embodiment relates to deep reinforcement learning with a discrete action space, but even when an agent has a continuous action space, it is possible to obtain a method of applying constraints on actions.

The method for inducing a rational action decision of a reinforcement learning agent according to embodiments of the present invention described above may be implemented in the form of a computer program that can be executed through various computer means. The implemented computer program may be recorded in a storage unit of a computer or a separate computer-readable recording medium. The method of the present invention can be performed by reading and executing the implemented computer program stored in a recording medium by an arithmetic processing unit in a computer device.

The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Features, structures, effects, etc. described in the above embodiments are included in one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, features, structures, effects, etc. given as examples in each embodiment can be combined or modified in other embodiments by a person having ordinary knowledge in the field to which the embodiments belong. Therefore, contents related to these combinations and variations should be construed as being included in the scope of the present invention.

In addition, although the embodiments have been described above, this is only an example and does not limit the present invention. Those skilled in the art to which the present invention pertains will know that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present embodiment. And differences related to these modifications and applications should be construed as being included in the scope of the present invention as defined in the appended claims.

The present invention can be used in the field of reinforcement learning of deep neural networks.

100: deep neural network
110: reinforcement learning agent

Claims (8)

Providing an action induced by a user as an input to the reinforcement learning agent in the execution step of the reinforcement learning agent of the deep neural network;
Calculating, in the reinforcement learning agent, a value obtained by subtracting a parameter value for reflecting the coercion of the induced action from an output value of the value function of all other actions except the induced action, as a final value function value; and
A method for inducing a rational action decision of a reinforcement learning agent of a deep neural network, comprising determining, in the reinforcement learning agent, a final action to be performed using the calculated final value function value. The method of claim 1, wherein the reinforcement learning agent uses the following equation as a modified value function q' to be used to determine the final action, and simultaneously considers the derived action and current observation information and situations to determine the final action. decide,

Here, a and s represent the action and the current situation, respectively, λ represents the above parameter, i and j are the indices of action a, and q is the expected reward for performing an action in the current situation A method for inducing a rational action decision of a reinforcement learning agent of a deep neural network, characterized in that it represents a value function approximating the expected value of. The method of claim 1, further comprising the step of changing the parameter value to adjust the degree of coercion of the induced action. The method of claim 3, wherein the step of changing the parameter value reduces the output value of the value function of all other behaviors except for the specific induced behavior (a _i ) so that the reinforcement learning agent performs the induced behavior (a _i ). A method for inducing a rational action decision of a reinforcement learning agent of a deep neural network, comprising the step of changing the value of the parameter (λ) to a large value in order to increase the possibility of selecting the final action. The method of claim 1, wherein the deep neural network is configured so that the same number of nodes as the number of actions that can be performed by the reinforcement learning agent exists in the last layer, and each node is expected to be obtained when a specific action is performed in the current situation. A method for inducing rational action decisions of a reinforcement learning agent of a deep neural network, characterized by learning a value function approximating the expected value of an expected reward. The method of claim 1, wherein the induced behavior includes a plurality of behaviors. A computer executable program stored in a computer readable recording medium to perform the method for inducing rational behavior decision of a reinforcement learning agent of a deep neural network according to any one of claims 1 to 6. A computer-readable recording medium on which a computer executable program is recorded for performing the method for inducing a rational action decision of a reinforcement learning agent of a deep neural network according to any one of claims 1 to 6.

Images