JP2006309519A - Reinforcement learning system and reinforcement learning program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a reinforcement learning system capable of significantly improving learning speed of an agent in an early stage of learning. <P>SOLUTION: The reinforcement learning computer 2 used for a reinforcement learning system 1 is provided with: an undefined state setting means 12 for setting an initial value of Q value in an undefined state, a state observation means 14 for observing the state of the agent 13, a behavior output means 15 for outputting behavior, a next state observation means 16 for observing the next state of the agent 13, a reward providing means 17 for providing the agent 13 with a reward r, a determination means 20 for determining the reward r and the next Q value according to a criterion 19, an unlearning means 21 for setting the Q value in an unlearned state according to the criterion 19, a second determination means 23 for performing determination according to a second criterion 23a, a learning means 24 for updating the Q value, an initialization means 25 for initializing the Q value, and a state update means 26 for updating the next state. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a reinforcement learning system and a reinforcement learning program. In particular, the present invention experiences changes in the surrounding environment and various events that occur, such as a biped robot and a search device used for planetary exploration. The present invention also relates to a reinforcement learning system and a reinforcement learning program for performing reinforcement learning that can be experienced and a robot or the like can determine an action autonomously by learning.

  Conventionally, a learning method called “reinforcement learning” is sometimes used to enable autonomous control of a robot or the like (see, for example, Non-Patent Document 1). Here, “reinforcement learning” is a method in which a learning object (agent) generally obtains a reward only when a user randomly acts in a certain environment and reaches a target as a result. In the next episode, if the agent encounters the same situation as the previous episode, the possibility of selecting an action with a high possibility of obtaining a reward based on the previous experience increases. Then, by repeating episodes in which the agent obtains rewards, it is possible to learn to finally select an optimal action that can obtain rewards in all states (scenes).

  Here, in reinforcement learning, a learning method called “Q-learning” is known as the most typical example. Q-learning is a direct approximation of the Q-value indicating an optimal action value function that is given an initial value in advance, and is multiplied by the total number of states and the total number of possible actions in the environment in which reinforcement learning is performed. It is indicated by the product.

  At this time, learning of the agent (in other words, updating of the Q value) is performed by the following equation (5). Here, Q (s, a) (corresponding to the Q value) indicates the value of the action a in the state s, α is the learning step size, γ is the discount rate, r is the reward, and T is the target value. Yes. Specifically, as shown in FIG. 7, the Q learning system initializes all the Q values indicating the optimum action value function to predetermined values (for example, assigns Q = 0 to all the tables, etc.) (Step T1), and then the state s of the agent to be learned is observed (Step T2). Furthermore, the action a from the state s of the agent is output (step T3). Then, the next state s ′ based on the result of the action a is observed (step T4), and the reward r is acquired (step T5). Thereafter, learning (update of Q value) is performed using the following equation (5) (step T6). Then, a process of replacing the next state s' with the state s is performed (step T7). And it returns to the process of step T3 again and continues learning of an agent. As a result, for each action a, the Q value is updated, and the process until the agent reaches the target and obtains the reward r (corresponding to an episode) is repeated a plurality of times. Based on this, the optimum Q value is gradually approximated. Thereby, by updating the Q value, the time required to reach the target, the number of required actions, and the like can be shortened as compared with the initial stage of learning.

Richard S. Sutton Andrew G. Barto, Sadayoshi Mikami and Masaaki Minagawa, “Reinforcement Learning”, 1st edition, Morikita Publishing, December 20, 2000

  However, in the case of reinforcement learning using Q-learning described above, there is a problem that the learning speed is slow, in particular, the learning efficiency is low in the initial learning stage. That is, according to the basic equation of Q learning by the above equation (5), the step size α (0 <α <1) is a parameter for determining the learning speed, and determines the update efficiency of the Q value. To do. However, the step size α is a numerical value given in the range of 0 <α <1, and the value of the second term on the right side with respect to the entire Q value is extremely small compared to the first term on the right side of the first equation. Therefore, in the entire Q value, the first term on the right side of the first expression is dominant, and the obtained reward is very little reflected in the entire Q value. As a result, when trying to obtain an approximated Q value, the agent had to repeat a great number of experiences (episodes). In particular, when an initial value that is significantly different from the Q value is set, the time required for learning increases. In particular, the required learning time increases exponentially as the number of agent states increases. As a result.

  In the second equation for calculating the target value T in equation (5), the value of the second term on the right side (excluding the step size α) may be given an arbitrary initial value at the start of learning. It was. For this reason, there is no guarantee that the initial value given is accurate, and there is a case where there is no meaning in causing the agent to repeatedly act in order to reach the target value T.

  Therefore, in view of the above circumstances, an object of the present invention is to provide a reinforcement learning system and a reinforcement learning program capable of dramatically improving the learning speed of an agent in an early stage of learning.

（α：ステップサイズ、γ：割引率）
に基づいて、前記価値Ｖを更新し、学習する学習手段と、前記第二判定手段の前記第二判定基準に従って、前記価値Ｖが未定義であると判定されると、次式：

に基づいて、前記価値Ｖを初期化する初期化手段と、前記判定手段、前記初期化手段、及び前記学習手段のいずれか一つの処理が行われた前記次状態を前記状態に更新する状態更新手段と」を主に具備して構成されている。 In order to solve the above-mentioned problem, the reinforcement learning system according to the present invention includes: “undefined setting means for setting an initial value of a value V indicating a value function including an action value function or a state value function to be undefined; State observation means for observing the state of the learning target agent, behavior output means for outputting the agent's action in the state, and next state observation for observing the next state of the agent transitioned by the outputted action Means, reward providing means for providing reward r to the agent that has transitioned to the next state, and the next value V ′ indicating the reward and the value function in the next state is determined according to a predetermined criterion. When it is determined that the reward is zero and the next value V ′ is undefined according to the determination means and the determination criterion of the determination means, An unlearned means for canceling the process or the initialization process, a second determination means for determining the value V in the state according to a second determination criterion, and the value V according to the second determination criterion of the second determination means. If it is determined that it is already defined, the following formula:

(Α: Step size, γ: Discount rate)
If it is determined that the value V is undefined according to learning means for updating and learning the value V and the second determination criterion of the second determination means based on

On the basis of the state update, the state update unit updates the state after the initialization unit that initializes the value V and the next state after any one of the determination unit, the initialization unit, and the learning unit is performed. It is mainly provided with “means”.

When the reinforcement learning system of the present invention is applied to “TD learning”, the above equation (6) is replaced with the following equation (8), and the equation (7) is replaced with the equation (9). Can be used. Further, when applied to “Sarsa”, it is possible to use the expression (6) by replacing the expression (6) with the following expression (10) and replacing the expression (7) with the expression (11). Furthermore, when applied to “Q learning”, it is possible to replace equation (6) with the following equation (12) and replace equation (7) with equation (13). That is, the present invention can be applied to reinforcement learning that employs a learning method in which a value function (including an action value function and a state value function) is asymptotic to an optimal value function. In the case of TD learning, the state value function V (s) corresponds to a value function, and the next value corresponds to a “state value”. In the case of Sarsa, the next value corresponds to “actual action value actually acted”, and in the case of Q learning, the next value corresponds to “maximum action value”. In the claims and the formulas (6) and (7), for convenience, the Q value indicating the action value function is expressed. However, the above formulas (6) and (7) are expressed by the state value function V (s). It may be expressed (see Formula (8) and Formula (9)). In order to simplify the description, the following description is given for a case where the present invention is applied to Q learning.

  Therefore, according to the reinforcement learning system of the present invention, the initial value of the Q value is not set to a predetermined value (for example, Q = 0, etc.) but is set to an undefined state. Based on the set condition, the current state of the agent is observed and the action is output. Furthermore, the next state of the agent that has transitioned as a result of the action is observed, and a reward for the action is provided to the agent. Note that a series of processing from observation of the current state to provision of reward is the same as that performed in conventional Q-learning. Thereafter, the processing so far is determined according to the determination criteria of the determination means.

  Here, in the determination means, it is determined whether the acquired reward has a value other than zero and whether the next Q value is already defined or not defined. At this time, in the conventional Q-learning, the reward is often set to be provided only when the agent reaches the goal (goal). That is, when the reward is zero, the agent indicates a state where the goal has not been reached, and when the next Q value of the next state is undefined, the second term on the right side in Equation (6) is “target The value does not contain information about the solution to the problem. Therefore, when such a situation is determined by the determination unit, a process of not learning (unlearned) is performed. In other words, either the post-learning process or the initialization process is performed by clearing the condition of either one of the two criteria.

  Then, according to the determination criteria of the determination means, if any one of the conditions “reward has a value other than zero” and / or “next Q value is already defined” is satisfied, the second determination means The determination for the Q value at is performed. At this time, when it is determined that the value (Q value) of the current state is undefined, the Q value is initialized based on Expression (7). That is, the first term on the right side in Equation (6) is meaningless because the Q value is undefined. Therefore, as shown in Expression (7), initialization is achieved using the value of the acquired reward and the value that maximizes the Q value. As a result, the Q value is defined. On the other hand, when the Q value has been defined, both the first term on the right side and the second term on the right side in Equation (6) have meaning, in other words, “the target value includes information about the solution of the problem”. It will be.

  For this reason, a process of updating (= learning) the Q value is performed using Expression (6). After that, the transitioned next state is updated to a new state, state observation → action output → next state observation → reward acquisition → determination (determination means and / or second determination means) → unlearned / initialized / The learning process is repeated for each episode. Thereby, learning of the agent proceeds. At this time, in the case where there is no meaning in updating the Q value (reward = 0 and the next Q value is undefined), the learning is canceled. For this reason, when gradually approximating the optimum Q value, the number of learnings is not wasted by an arbitrarily set initial value. Furthermore, since the reward has a value other than zero and the initial value is given only to the state where the Q value is undefined, the given initial value is a significant value compared to the conventional one. As a result, the possibility of approximating the optimum Q value and convergence is increased, and learning efficiency is improved. In particular, since the setting of the initial value at the initial stage of learning is omitted when it does not make sense, there is no significant difference from the approximated Q value.

  Further, the reinforcement learning system according to the present invention includes, in addition to the above-described configuration, “the determination unit determines a reward according to a determination criterion for determining whether the reward has a value other than zero, and the next And “next value determination means for determining according to a next value determination criterion for determining whether or not the value V ′ has been defined”.

  Therefore, according to the reinforcement learning system of the present invention, the reward determination means and the next Q value determination means are individually provided. This makes it possible to accurately classify the reinforcement learning algorithm into unlearned, initialized, and learned processes based on the determination based on both criteria, and in particular, leap in learning efficiency at the initial stage of learning. Can be improved.

（α：ステップサイズ、γ：割引率）
に基づいて、前記価値Ｖを更新し、学習する学習手段、前記第二判定手段の前記第二判定基準に従って、前記価値Ｖが未定義であると判定されると、次式：

に基づいて、前記価値Ｖを初期化する初期化手段、及び前記判定手段、前記初期化手段、及び前記学習手段のいずれか一つの処理が行われた前記次状態を前記状態に更新する状態更新手段として、強化学習コンピュータを機能させる」ものから主に構成されている。 On the other hand, the reinforcement learning program according to the present invention is “undefined setting means for setting an initial value of a value V indicating a value function including an action value function or a state value function to be undefined, an agent of a learning target agent that performs reinforcement learning. State observation means for observing the state, action output means for outputting the action of the agent in the state, next state observation means for observing the next state of the agent that is transitioned by the outputted action, and the state that has transitioned to the next state Remuneration providing means for providing a reward r to the agent, determination means for determining the reward and the next value V ′ indicating the value function in the next state according to a predetermined determination criterion, according to the determination criterion of the determination means If it is determined that the reward is zero and the next value V ′ is undefined, the learning process or the initialization process is canceled. The value V in the state is determined according to the second determination criterion, and the value V is determined to be defined according to the second determination criterion of the second determination unit. And the following formula:

(Α: Step size, γ: Discount rate)
If the value V is determined to be undefined according to the second determination criterion of the learning means that updates and learns the value V, and the second determination means based on:

On the basis of the state update, the state update for updating the next state after the process of any one of the initialization unit for initializing the value V, the determination unit, the initialization unit, and the learning unit is performed. As a means, it is mainly comprised from what makes a reinforcement learning computer function.

  Furthermore, the reinforcement learning program according to the present invention includes, in addition to the above configuration, “a reward determination means for determining according to a reward determination criterion for determining whether or not the reward has a value other than zero, and the next value V ′. The reinforcement learning computer may be further functioned as the determination means having the next value determination means for determining according to the next value determination criterion for determining whether or not it has been defined.

  Therefore, according to the reinforcement learning program of the present invention, by executing the program, the reinforcement learning computer can exhibit an excellent action in the above-described reinforcement learning system.

  As an effect of the present invention, by initially setting the Q value indicating the value function to be undefined, it is possible to prevent a significant deviation of the initial value in the initial stage of learning from the value approximated as in the past. As a result, the learning time at the initial learning stage can be shortened and the learning efficiency of the agent can be greatly increased. Furthermore, according to each state (situation), the determination unit and the second determination unit can perform the three types of processing of unlearning, initialization, and learning according to each state (situation). It will be performed more efficiently than As a result, the learning efficiency is improved, and the time for approximating the Q value to the optimum value and converging is greatly reduced as compared with the learning method in which the value function such as conventional Q learning is asymptotic to the optimum value function. be able to.

  Hereinafter, a reinforcement learning system 1 according to an embodiment of the present invention will be described with reference to FIGS. 1 to 7. Here, FIG. 1 is a block diagram showing a functional configuration of the reinforcement learning computer 2 used in the reinforcement learning system 1 of the present embodiment, and FIG. 2 represents a learning procedure 3 (learning algorithm) in the reinforcement learning system 1. FIG. 3 is an explanatory diagram in which the processes performed based on the determination of the determination means 20 and the second determination means 23 are classified into a list form. FIG. 4 is a process flow of the reinforcement learning computer 2 5 is an explanatory diagram showing an example of (a) a 100 × 100 grid world 4 and (b) Q value data 29. FIG. 6 is a diagram of the reinforcement learning system 1 and the Q learning system 5. It is the graph which compared the simulation result.

  Here, the reinforcement learning system 1 of this embodiment has illustrated what was applied based on the conventional Q learning. The reinforcement learning computer 2 is constructed to be able to execute and function the reinforcement learning program 6 stored in advance in a storage medium such as a hard disk (storage means 32 or the like). The reinforcement learning program 6 is programmed to cause the reinforcement learning system 1 to function according to the learning procedure 3 shown in FIG. 2 and the determination criteria 19 and 23a shown in FIG. In addition, in part of FIGS. 1 to 6, the Q value 11 is indicated as Q (s, a), and the next Q value 18 is indicated as Q (s', a) for convenience.

  More specifically, as shown in FIG. 1, the reinforcement learning system 1 of the present embodiment is configured by a reinforcement learning computer 2 that can execute various arithmetic processes and storage processes. Here, the reinforcement learning computer 2 is connected to an agent 13 that can be controlled to observe the state s of the surrounding environment E and output a predetermined action a according to the observation result. In the present embodiment, the agent 13 exists as a virtual body that can move in the grid world 4 virtually constructed on the computer. Here, the agent 13 includes, for example, a plurality of sensors (for example, visual sensors) and uses an entity such as an autonomous mobile robot that can move autonomously by a driving travel unit, and the surrounding environment E You may use what outputs the action with respect to appropriately.

  Further, the reinforcement learning computer 2 has, as other functional configurations, as shown mainly in FIG. 1, an undefined setting means 12 for setting the initial value of the optimum action value function Q value 11 undefined, and reinforcement learning. State observation means 14 for observing (recognizing) a state s (here, a position in a grid world 4 to be described later) with respect to the environment E around the agent 13 to be performed, and a plurality of predefined actions of the agent 13 in the state s. The action output means 15 for selecting from the action criteria and outputting the action a, the next state observing means 16 for observing the next state s ′ of the agent 13 transitioning from the state s by the action a, and the next state s ′. According to reward providing means 17 for providing reward r to the transitioned agent 13, determination means 20 for determining according to reward r and next state s', and according to determination criterion 19 If the determination means 20 determines that the value of the reward r is zero and the next Q value 18 is in an undefined state, both the learning process by updating the Q value 11 and the initialization process of the Q value 11 are canceled. The unlearned means 21 that performs the unlearned process of “putting into a state that does not learn”; the second determination means 23 that determines the Q value 11 in the state s according to the second determination criterion 23 a; and the second determination of the second determination means 23 If it is determined that the Q value 11 has been defined according to the standard 23a, the learning means 24 performs a process of updating the Q value based on the formula A (see FIG. 4 and the formula (1), etc.) and performs learning. If the Q value is determined to be undefined, the initialization means 25 for initializing the Q value 11 based on the formula B (see FIG. 4 and the formula (2), etc.), unlearned, learned, and initialized After any one of the processes is executed, the previous state s ′ And and a state update means 26 for updating the s is mainly composed.

  Note that the determination unit 20 determines whether the reward r has a value other than zero according to the reward determination criterion 27a and whether the next Q value 18 has been defined according to the next Q value determination criterion 28a. And a next Q value judging means 28 for judging whether or not. In addition, the reinforcement learning computer 2 stores the Q value 11 and the next Q value 18 that are defined, undefined, initialized, and updated as other functional configurations, and holds the Q value in a tabulated state. Data 29 (see FIG. 5B), the observed state s and the next state s ′ are stored, the state s and action history of the agent 13 are accumulated, the state data 30 to be retained, and the reward r are stored. And storage means 32 for storing the reward data 31 to be held together. Here, the storage means 32 also stores a reinforcement learning program 6 for causing the reinforcement learning computer 2 to function as the reinforcement learning system 1, and can be executed based on the program execution means 33. Here, the next Q value determination standard 28a corresponds to the next value determination standard of the present invention, and the next Q value determination means 28 corresponds to the next value determination means of the present invention.

  Here, as the reinforcement learning computer 2 used in the reinforcement learning system 1 of the present embodiment, a commercially available general-purpose computer is used in the present embodiment, and each of the means described above is an arithmetic process mainly comprising each CPU. Based on the circuit, it is formed to be able to exhibit such a function. The storage means 32 can use a fixed storage medium such as a hard disk or a non-volatile storage medium such as a semiconductor memory, and can sequentially store various information such as the action a of the agent 13. . In the case of the above-described autonomous mobile robot, the configuration of the reinforcement learning computer 2 may be constructed in a control circuit inside the autonomous mobile robot.

  Next, an example of the reinforcement learning system 1 simulated by the reinforcement learning computer 2 will be described mainly based on FIGS. Here, as shown in FIG. 5 (a), for the reinforcement learning system 1 of the present embodiment, a virtual space (corresponding to the grid world 4) in which “100 × 100” is vertically and horizontally divided is assumed. To do. That is, in the grid world 4, there are 10,000 grids from the grid position M1 (corresponding to the start point S) to the grid position M10000 (corresponding to the goal point G). At this time, the shortest number of steps required for the agent 13 to reach the goal point G in the upper right corner from the start point S in the lower left corner is 99 steps in the <upward> direction and 99 steps in the <rightward> direction. 198 steps. Further, the agent 13 can obtain a real value reward r other than “0” for the first time when the agent 13 starts from the start point S and reaches the goal point G. In other cases, the agent 13 can obtain “0” as the reward r. In the present embodiment. Then, based on the action a taken by the agent 13 from the start point S to the goal point G, it repeats the process of updating and initializing the Q value so as to converge to the shortest number of 198 steps described above. Can learn to. In the present embodiment, one episode is from the start point S to the goal point G. Further, the processing from step S1 to step S10 in FIG. 4 corresponds to the reinforcement learning program of the present invention.

  First, the reinforcement learning program 6 stored in the storage means 32 is executed by the program execution means 33 to cause the reinforcement learning computer 2 to function and to construct the reinforcement learning system 1 on the grid world 4. First, the initial value of the Q value 11 stored as a table in the Q value data 29 of the storage means 32 is set to an undefined state (step S1). As a result, the state of the optimal action value function indicating the value in the state s in which the agent 13 is placed is not defined. Thereafter, the state s of the agent 13 is observed (step S2). Here, in the reinforcement learning system 1 of the present embodiment, the position on the grid world 4 where the agent 13 exists is observed as the state s. Further, the agent 13 outputs an action a that makes a transition from the state s (step S3). At this time, as shown in FIG. 5A, in the virtually constructed grid world 4, the agent 13 is one of four directions, up, down, left, and right from the grid indicating the position of the current state s. It is defined to be able to go in the direction.

  That is, in the state of FIG. 5A, the agent 13 is located in the grid M303 (corresponding to the state s), and the upward direction (grid M403), the downward direction (grid M203), the left direction (grid M302), and the right direction. It is possible to move to (grid M304) (action a). At this time, since the Q value 11 is set to be undefined in the initial state, the Q value 11 having a value indicating which direction can be reached the fastest to reach the goal point G is not included. . In such a case, it is possible to arbitrarily move in one direction from the four directions (here, “upward direction: corresponding to the grid M403 direction) by the action a. The next state s ′ in M403) is observed (step S4), and then the agent 13 obtains the reward r by the transition to the next state s ′ for the action a (step S5). Stored in data 31.

  Further, the reinforcement learning computer 2 performs an appropriate determination process using the observed state s, next state s', and reward r (step S6 or step S7). Here, when the determination means 20 has either the reward r has a value other than zero or the next Q value 18 in the transitioned next state s ′ is already defined (YES in step S6), the second determination The determination based on the means 23 is performed (step S71). On the other hand, if the condition for both the reward r is zero and the next Q value 18 is undefined (NO in step S6), steps S7 to S7 are performed without performing learning processing or initialization processing described later. The process of step S8 is cancelled, and the process proceeds to step S10. That is, “NO processing in step S6” corresponds to the unlearned means 21 in the present invention.

  Further, the reinforcement learning computer 2 determines that the state s when the reward r is a value other than zero or the next Q value 18 in the transitioned next state s ′ satisfies one of the defined conditions (YES in step S6). The determination of the Q value 11 is determined according to the second determination criterion 23a. If the Q value 11 is already defined (YES in step S7), the Q value 11 is updated according to the equation A in FIG. 4 (step S8). In this case, either Q (s, a) indicating the Q value 11 in the current state s of the first term on the right side, or Q (s ′, a) indicating the reward r or the next Q value 18 on the second term on the right side. One will be significant and will have a solution to the problem. As a result, the Q value is updated and learning is performed. On the other hand, when the Q value 11 is undefined (NO in step S7), since the first term on the right side in the equation A has no significance, the Q value 11 is initialized according to the equation B (step S9). Thereby, a value having significance is set as an initial value of the Q value. Then, after the learning process (step S8), the initialization process (step S9), or the unlearned process (NO in step S6), a process of updating the next state s' to the state s is performed (step S10). Thereafter, the process returns to step S3, and the process of outputting the action a (step S3), observing the next state s' (step S4), and acquiring the reward r (step S5) is repeated.

  Thereby, the agent 13 gradually updates the Q value 11 corresponding to each grid (grid M1 to grid M10000) set to an undefined state by experiencing a plurality of episodes, and is tabulated. The Q value data 29 (see FIG. 5B) can be stored sequentially. As a result, the agent 13 can determine the optimum action a based on the Q value 11 and transition to the state s ′ suitable for reaching the goal point G from the start point S.

  Here, what compares the effect in the reinforcement learning system 1 of this embodiment with the conventional Q learning system 5 is shown. FIG. 6 is a graph comparing the results of simulations from the start point S to the goal point G using the 100 × 100 grid world 4 described above. Here, the vertical axis of the graph indicates the number of steps for each episode required to reach the goal point G from the start point S, and the horizontal axis of the graph indicates the number of episodes. This graph shows that when the reinforcement learning system 1 of the present embodiment is employed, the value converges to approximately 198 steps, which is the shortest number of steps, when about 500 episodes are exceeded. On the other hand, in the case of the conventional Q learning system 5, although the number of steps tends to decrease so that it gradually converges to 198 steps, even if it exceeds 1000 episodes, 198 steps as in the reinforcement learning system 1 of the present invention. Never converge. In particular, at the initial stage of learning, the speed of the learning efficiency is remarkable. In the case of about 100 episodes, the present system 1 has about 3000 steps or less, whereas the Q learning system 5 requires about 15000 steps. For this reason, the usefulness of the reinforcement learning system 1 of this invention can be shown.

  The present invention has been described with reference to preferred embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the spirit of the present invention as described below. And design changes are possible.

  That is, in this embodiment, in order to confirm the effect of the reinforcement learning system 1, what was used the grid world 4 constructed virtually was shown, but it is not limited to this. The reinforcement learning system 1 may be applied to a robot. As a result, the autonomous mobile robot that outputs each action a according to the situation of the surrounding environment E learns quickly at the initial stage, and is optimal with a shorter number of episodes than the conventional Q learning system 5. Control that takes action a can be performed.

It is a block diagram which shows the functional structure of the reinforcement learning computer used for a reinforcement learning system. It is explanatory drawing expressing the learning procedure in a reinforcement learning system. It is explanatory drawing which classified into the list form the process implemented based on the determination of a determination means and a 2nd determination means. It is a flowchart which shows the flow of a process of reinforcement learning computer. It is explanatory drawing which shows an example of (a) 100 * 100 grid world and (b) Q value data. It is the graph which compared the simulation result of the reinforcement learning system of this embodiment, and the Q learning system. It is a flowchart which shows the flow of a process of the conventional Q learning system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Reinforcement learning system 2 Reinforcement learning computer 6 Reinforcement learning program 11 Q value (Q (s, a), value)
12 undefined setting means 13 agent 14 state observation means 15 action output means 16th order state observation means 17 reward providing means 18th order Q value (Q (s ′, a), next value)
DESCRIPTION OF SYMBOLS 19 Determination criteria 20 Determination means 21 Unlearned means 23 Second determination means 23a Second determination criteria 24 Learning means 25 Initialization means 26 State update means 27 Reward determination means 27a Reward determination criteria 28 Next Q value determination means (next value determination means )
28a Next Q value criteria (next value criteria)
a Action E Environment r Reward s State s' Next State

Claims (4)

Undefined setting means for setting an initial value of value V indicating a value function including an action value function or a state value function to be undefined;
A state observing means for observing the state of the learning target agent for reinforcement learning;
Action output means for outputting the action of the agent in the state;
A next state observing means for observing a next state of the agent that is transited by the output action;
Reward providing means for providing reward r to the agent that has transitioned to the next state;
Determination means for determining the reward and the next value V ′ indicating the value function in the next state according to a predetermined criterion;
In accordance with the determination criteria of the determination means, if it is determined that the reward is zero and the next value V ′ is undefined, unlearned means for canceling the learning process or the initialization process;
Second determination means for determining the value V in the state according to a second determination criterion;
When it is determined that the value V has been defined according to the second determination criterion of the second determination means, the following formula:

(Α: Step size, γ: Discount rate)
Learning means for updating and learning the value V based on
If it is determined that the value V is undefined according to the second determination criterion of the second determination means, the following formula:

Based on the initialization means for initializing the value V;
A reinforcement learning system comprising: a state update unit that updates the next state in which any one of the determination unit, the initialization unit, and the learning unit is performed to the state. The determination means includes
Reward determination means for determining according to a reward determination criterion for determining whether or not the reward has a value other than zero;
The reinforcement learning system according to claim 1, further comprising: a next value determining unit that determines in accordance with a next value determination criterion that determines whether or not the next value V ′ is already defined. An undefined setting means for setting an initial value of value V indicating a value function including an action value function or a state value function to be undefined, a state observing means for observing the state of a learning target agent that performs reinforcement learning, Action output means for outputting the action of the agent, next state observation means for observing the next state of the agent transitioned by the outputted action, reward providing means for providing a reward r to the agent transitioned to the next state, A determination means for determining a reward and a next value V ′ indicating the value function in the next state according to a predetermined criterion, and according to the determination criterion of the determination means, the reward is zero and the next value V ′ is If determined to be undefined, unlearned means for canceling the learning process or initialization process, the value in the state Second determination means for determining in accordance with the second criterion, was prepared in accordance with the second criterion of the second determination means, when the value V is determined to be defined by the following formula:

(Α: Step size, γ: Discount rate)
If the value V is determined to be undefined according to the second determination criterion of the learning means that updates and learns the value V, and the second determination means based on:

On the basis of the state update, the state update for updating the next state after the process of any one of the initialization unit for initializing the value V, the determination unit, the initialization unit, and the learning unit is performed. A reinforcement learning program characterized by causing a reinforcement learning computer to function as a means.   Reward determination means for determining according to a reward determination criterion for determining whether or not the reward has a value other than zero, and a next value determined according to a next value determination criterion for determining whether or not the next value V ′ is already defined The reinforcement learning program according to claim 3, wherein the reinforcement learning computer is further caused to function as the determination means having a determination means.

Images