JP5398414B2 - Learning system and learning method - Google Patents

Description

  The present invention relates to a learning system and learning method based on reinforcement learning.

  Reinforcement learning is known as a learning method in which a machine such as a robot is improved by learning so that its own control rules meet a purpose (for example, Non-Patent Document 1). Furthermore, the possibility that the brain is performing reinforcement learning with an environmental model explicitly in biological research is shown (for example, Non-Patent Document 2). Reinforcement learning with an explicit environment model can respond to changes in the environment that traditional reinforcement learning without an environment model is not good at, and manage acquired action sequences as a group There are advantages such as being able to.

  On the other hand, reinforcement learning with an explicit environment model requires a search for a tree structure representing the environment model, and the calculation cost is very high.

  Thus, the reinforcement learning system and reinforcement learning method of the low calculation cost which have an environmental model explicitly are not developed.

N. D. Daw & K. Doya, “The computational neurobiology of learning and reward”, Current Opinion in Neurobiology, 2006, 16, pp199-204 N. D. Daw, Y. Niv & P. Dayan, “Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control”, Nature Neuroscience, 2005, 8, pp1704-1711

  Accordingly, there is a need for a low learning cost reinforcement learning system and reinforcement learning method that can explicitly have an environment model, can respond to changes in the environment, and can manage acquired action sequences as a whole.

  The learning system according to the present invention includes an event list database that holds a plurality of event lists, with a series of state / action pairs reaching an immediately preceding state / action pair as an event list. An event list management unit that classifies action pairs into the plurality of event lists and stores them, and a learning control unit that updates a reward expected value of a state / action pair that is an element of each event list. ing.

  The learning method according to the present invention includes an event list database that holds a plurality of event lists, with an event list as a set of state / action pairs that reach a state / action pair immediately before a reward is obtained, and an event A learning method using a learning system including a list management unit and a learning control unit. In the learning method according to the present invention, the event list management unit classifies and stores state / action pairs into the plurality of event lists, and the learning control unit is an element of each event list. Updating the expected reward value of the state / action pair.

  According to the learning system and the learning method of the present invention, a set of a state / action pair that reaches a state / action pair immediately before a reward is obtained is an event list, and the state / action pair is converted into a plurality of event lists. Sort and memorize. As a result, an environmental model is created for each state / action pair immediately before the reward is obtained. Therefore, the learning system and the learning method of the present invention can cope with changes in the environment, and can manage the acquired action series as a group, that is, as an event list.

  According to the embodiment of the present invention, the event / list management unit temporarily holds a state / action pair each time an action is selected, and temporarily holds the state / action each time a reward is obtained. Of the set of pairs, state / action pairs not stored in the event list database are stored in the event list database as elements of the event list of the state / action pair immediately before the reward is obtained. .

  According to the present embodiment, state / action pairs can be efficiently classified and stored in a plurality of event lists.

  According to the embodiment of the present invention, every time the learning control unit obtains a reward, the reward expectation value of the state / action pair which is an element of the event / list of state / action pair immediately before the reward is obtained, Update using the value of the reward, and the reward value of the state / action pair that is an element of the event list other than the event list of the state / action pair immediately before obtaining the reward is zero. Update as

  According to the present embodiment, for each event list, the expected reward value of the state / action pair that is an element of the event list can be efficiently updated.

It is a figure which shows the structure of the apparatus containing the learning system by one Embodiment of this invention. It is a figure for demonstrating the data structure of an event list database. It is a flowchart for demonstrating operation | movement of an event list management part. It is a flowchart for demonstrating operation | movement of a learning system. It is a flowchart for demonstrating operation | movement of an action selection part. It is a figure for demonstrating the Markov decision process (MDP: Markov Decision Process) which is a 1st simulation environment. It is a figure for demonstrating the high order Markov decision process (HOMDDP: High Order Markov Decision Process) which is a 2nd simulation environment. It is a figure which shows the simulation result of the learning system by embodiment of this invention, and the conventional learning system.

  FIG. 1 is a diagram illustrating a configuration of an apparatus 200 including a learning system 100 according to an embodiment of the present invention. Device 200 may be, for example, a robot. The apparatus 200 includes an information acquisition unit 201, an acquisition information processing unit 203, an action selection unit 205, an action output unit 207, a supervisor 209, and a learning system 100.

  The information acquisition unit 201 acquires input information from the environment 300 and acquires state information of the device 200 itself. When the apparatus 200 is a robot, the information acquisition unit 201 may include a camera and acquire information on the environment 300 based on an image of the environment 300 captured by the camera. Further, the information acquisition unit 201 may acquire state information of the device 200 including the position and orientation of the robot. The information acquisition unit 201 sends the acquired information to the acquisition information processing unit 203.

  The acquired information processing unit 203 classifies the state in which the device 200 is placed into one of a plurality of predetermined states based on the environment and its own state information.

  The learning system 100 stores the action selected by the apparatus 200 as a state / action pair in the state in which the apparatus 200 is placed, and learns the expected reward value of the state / action pair according to the reward of the result. Here, the reward is determined by the acquisition information processing unit 203 based on the information acquired by the information acquisition unit 201. The learning system 100 includes an event list management unit 101, a temporary list storage unit 103, an event list database 105, and a learning control unit 107. The event list management unit 101 stores state actions / pairs in the temporary list storage unit 103 and the event list database 105. The learning control unit 107 learns an expected reward value for each state / action pair according to the reward, and stores it in the event list database 105 in association with the state / action pair. Details of the learning system 100 will be described later.

  The behavior selection unit 205 receives the state of the device 200 from the acquired information processing unit 203, and the expected reward value stored in the event list database 105 in association with the state / behavior pair is maximum for the state. Select the action with the maximum probability.

  The behavior output unit 207 outputs the behavior selected by the behavior selection unit 205. The change in the environment 300 as a result of the action is acquired as information by the information acquisition unit 201.

  The supervisor 209 teaches a series of actions that can be rewarded at the fastest rate for the state of the device 200. It is used to assist learning of the learning system 100 in the early stages of learning.

  The learning system 100 according to the embodiment of the present invention classifies state / action pairs into sets for each state / action pair immediately before obtaining a reward, stores the state / action pairs for each set, and stores the states / action pairs. It is characterized by learning the expected reward value of the pair. Memorizing the state / action pair as a set of each state / action pair immediately before receiving the reward and learning the expected value of reward for that state / action pair is the environment for each state / action pair immediately before receiving the reward Corresponds to creating a model. Therefore, the learning system 100 according to the present embodiment can cope with changes in the environment, and can manage the acquired action series as a group. This will be described in detail below.

式(１)において、Ｅ［｜］は条件付期待値を表す。 Here, the expected reward value R can be expressed by the following equation.

In equation (1), E [|] represents a conditional expected value.

s _t represents the observed state at time t. There are a plurality of observed states, for example, s0, s1,..., Si,.
It can be expressed. At time t, to observe one of these actually represent to as s _t.

a _t represents the action selected at time t. There are a plurality of actions to be selected. For example, a0, a1,..., Ai,.
It can be expressed. At time t, selecting one of these actually, this is expressed as a _t.

rt _{+ k} is a reward obtained at time t + k.

  γ is a parameter called a discount rate.

Equation (1) can be modified as follows.

  p (k |...) is the probability that the episode will reach the end after k steps from the current time. Here, “episode” refers to one state in a series of states that are sequentially generated as a result of selecting an action in a certain state, and “end” refers to the last state in the series of states. The state reaches the last state when a reward is obtained or the action selection is interrupted.

(S, A) represents (s _{t + k−1} , a _{t + k−1} ), and is a state / action pair immediately before obtaining the reward r _{t + k} . To observe the state _{s t} at time t, representing the state-action pair if you took the action _{a t} and _{(s t, a} t).

  E (S, A) [• | ••] represents a portion of the expected reward value divided by the state / action pair (S, A) immediately before the reward is obtained, and is referred to as a partial expected value.

  Expression (2) indicates that the reward expected value can be expressed by the sum of the partial expected values. It also indicates that all state / action pairs (si, aj) can be classified into a plurality of (S, A) groups. Therefore, as described above, the state / action pair is classified into a set for each state / action pair immediately before the reward is obtained, and the state / action pair is stored for each set, and the reward expectation of the state / action pair is stored. The value can be learned.

  FIG. 2 is a diagram for explaining the data structure of the event list database 105. In FIG. 2, (S, A) n indicates the state / action pair immediately before the reward is obtained. S and A indicate the state and action immediately before the reward is obtained, and n indicates the number of the state / action pair immediately before the reward is obtained. (S, A) n forms a set with a series of state / action pairs up to (S, A) n. This set is called an event list. (Si, aj) indicates a state / action pair included in the event list. s and a indicate the state and action, respectively. i and j indicate the numbers of the state s and the action a, respectively. The reward expected value E [r] p of the state / action pair is stored in the event list in association with (si, aj). r indicates the reward, and p indicates the number of the expected reward value.

  As described above, the event list database 105 is classified for each state / action pair 1051 immediately before the reward is obtained. The event list includes a state / action pair 1051 immediately before the reward is obtained, a series of state / action pairs 1053 reaching the state / action pair 1051, and a state / action pair (si, aj) which is an element of the event list. The associated reward expectation value E [r] p is included. The expected reward value E [r] p corresponds to the “partial expected value”.

  Here, a certain state / action pair (si, aj) may be included in a plurality of event lists of the state / action pair (S, A) immediately before the reward is obtained. In this case, the reward expectation value of the state / action pair (si, aj) is the reward expectation of the state / action pair (si, aj) included in the event list of the plurality of state / action pairs (S, A). It is the sum of values.

  FIG. 3 is a flowchart for explaining the operation of the event list management unit 101.

  In step S <b> 105 of FIG. 3, the event / list management unit 101 determines whether a state / action pair (si, aj) has been received from the action selection unit 205. Here, the action selection unit 205 sends a state / action pair (si, aj) to the event / list management unit 101 each time an action is selected. If the state / action pair (si, aj) has been received, the process proceeds to step S110. If the state / action pair (si, aj) has not been received, the process proceeds to step S115.

  In step S <b> 110 of FIG. 3, the event / list management unit 101 stores the state / action pair (si, aj) in the temporary list storage unit 103.

  In step S <b> 115 of FIG. 3, the event list management unit 101 determines whether a reward has been received from the acquired information processing unit 203. Here, the acquired information processing unit 203 determines a reward based on the information acquired by the information acquiring unit 201 after a predetermined time has elapsed since the behavior output unit 207 outputs the behavior, and sends the reward to the event / list management unit 101. If a reward has been received, the process proceeds to step S120. If no reward has been received, the process returns to step S105 after a predetermined time has elapsed.

  In step S120 of FIG. 3, the event / list management unit 101 uses the state / action pair (si, aj) stored last in the temporary list storage unit 103 as the state / action pair (S , A).

  In step S125 of FIG. 3, the event list management unit 101 determines whether (S, A) exists in the event list database 105. If (S, A) exists, the process proceeds to step S135. If (S, A) does not exist, the process proceeds to step S130.

  In step S <b> 130 of FIG. 3, the event list management unit 101 stores (S, A) in the event list database 105.

In step S135 of FIG. 3, the event list manager 101, stored in the temporary list storing unit 103 the state-action pair (si, aj) each have, in the event list database 105 (S, A) To be included in the event list. If it is included in the event list of (S, A), the process proceeds to step S145. If it is not included in the event list of (S, A), the process proceeds to step S140.

  In step S140 of FIG. 3, the event list management unit 101 adds the state / action pair (si, aj) not included in the event list of (S, A) to the event list of (S, A). To do. At this time, the number of state / action pairs to be added is limited to a predetermined number.

  In step S145 of FIG. 3, the event / list management unit 101 determines whether or not the processing of step S135 has been performed for all the state / action pairs (si, aj) stored in the temporary list storage unit 103. If the process of step S135 is performed for all the state / action pairs (si, aj), the process proceeds to step S150. If the process of step S135 is not performed for all the state / action pairs (si, aj), the process returns to step S135.

  In step S150 of FIG. 3, the event / list management unit 101 clears (deletes) all the state / action pairs (si, aj) stored in the temporary list storage unit 103.

  FIG. 4 is a flowchart for explaining the operation of the learning control unit 107 of the learning system 100.

  In step S <b> 205 of FIG. 4, the learning control unit 107 determines whether or not a reward or end of episode notification has been received from the acquired information processing unit 203. Here, the acquisition information processing unit 203 determines a reward based on the information acquired by the information acquisition unit 201 after a predetermined time has elapsed after the behavior output unit 207 outputs the behavior, and sends the reward to the learning control unit 107. Also, the acquired information processing unit 203 sends a notification of the end of the episode to the learning control unit 107 when the episode reaches the end for some reason. If a reward or an end of episode notification is received, the process proceeds to step S210. If no reward or end of episode notification is received, the process returns to step S205 after a predetermined time has elapsed.

In step S210 of FIG. 4, the learning control unit 107 calculates the expected reward value of the state / action pair (si, aj) in the event list of the state / action pair (S, A) immediately before receiving the most recent reward below. Update with the formula of

  Here, α is a parameter called a learning constant and is a constant between 0 and 1.

Tv is given by the following equation.

  Here, τ is the time when the action aj is selected in the state si and the state / action pair (si, aj) actually occurs.

  In step S215 of FIG. 4, the learning control unit 107 expects a reward value of the state / action pair (si, aj) in the event list of (S, A) other than the state / action pair immediately before the most recently obtained reward. Is updated according to equation (3). In this case, the target value Tv is set to zero. When the end of episode notification is received, the target value Tv is set to zero, and the expected reward value of the state / action pair (si, aj) in all (S, A) event lists is updated according to the equation (3). To do.

  In this way, the expected reward value is updated for each event list grouped according to the state / action pair immediately before the reward is obtained.

  FIG. 5 is a flowchart for explaining the operation of the action selection unit 205.

  In step S <b> 305 of FIG. 5, the behavior selection unit 205 receives the current state from the acquired information processing unit 203. Here, the acquisition information processing unit 203 may send a reward to the learning control unit 107 and send a state to the action selection unit 205 after confirming that the learning control unit 107 has updated the expected reward value.

  In step S310 of FIG. 5, the action selection unit 205 selects a state / action pair having the current state from the event list database 105, and further selects a state / action pair having the maximum reward expectation value for the state / action pair. Select. As described above, when a state / action pair having the current state is included in a plurality of event lists, the sum of the expected reward values of the state / action pairs of the plurality of event lists is calculated as the state / action. The expected value of the pair's reward.

  In step S315 of FIG. 5, the action selection unit 205 sends the action of the selected state / action pair to the action output unit 207, and sends the selected state / action pair to the event / list management unit 101.

  Next, a simulation experiment for confirming the function of the learning system 100 according to the embodiment of the present invention will be described. A first simulation environment and a second simulation environment are prepared for the simulation experiment.

FIG. 6 is a diagram for explaining a Markov decision process (MDP) which is a first simulation environment. There are ten actions a0, a1,... A9 that can be selected. If the behavior is selected in the order of a0 to a9 after observing the state of s0, a reward r = 1 is given. However, each transition is probabilistic. Transition from s0 to s1 has a probability of 0.3. In other cases, transition is made with a probability of 0.9. Furthermore, there are two signals that can be observed for each state. For example, for s0, O ₀₀ and O ₀₁ signals exist, and either one is observed with a probability of 0.5. Therefore, there are 2 ¹⁰ = 1024 combinations of appearance of the observation signal until the reward is obtained.

FIG. 7 is a diagram for explaining a high order Markov decision process (HOMDP) which is a second simulation environment. There are ten actions a0, a1,..., A9, and six actions a0, a1,. This process includes process A and process B. In order to obtain the next reward when the reward is obtained in the process A, it is necessary to select the process B. To obtain the next reward when the reward is obtained in the process B, it is necessary to select the process A. is there. That is, it is necessary to select different actions in the process A and the process B for the same observation signal. Again, each transition is probabilistic. In the process A, a transition is made from s0 to s2 with a probability of 0.3. In other cases, transition is made with a probability of 0.9. In the process B, the transition from s1 to s2 is performed with a probability of 0.3. In other cases, transition is made with a probability of 0.9. Furthermore, there are two signals that can be observed for each state. For example, for s0, O ₀₀ and O ₀₁ signals exist, and either one is observed with a probability of 0.5.

  The procedure of the simulation experiment using the above simulation environment will be described below. First, using the environment as a HOMDP simulation environment, in the first 10 trials, the supervisor 209 teaches the action selection unit 107 a series of actions that can be rewarded at the fastest speed. During this time, the learning system 100 performs learning. However, not all behavior patterns can be learned during this time.

  Next, from the 251st trial, the environment is set as an MDP simulation environment, and until the 260th trial, the supervisor 209 teaches the action selection unit 107 a series of actions that can be rewarded at the fastest speed. During this time, the learning system 100 performs learning. However, not all behavior patterns can be learned during this time.

  Next, from the 501st trial, the environment is again set as a HOMDP simulation environment. The supervisor 209 does not teach. Therefore, the learning system 100 needs to cope with a suddenly changed environment.

  Next, from the 751st trial, the environment is again set as the MDP simulation environment. The supervisor 209 does not teach. Therefore, the learning system 100 needs to cope with a suddenly changed environment.

FIG. 8 is a diagram illustrating simulation results of the learning system according to the embodiment of the present invention and the conventional learning system. The horizontal axis of the graph in FIG. 8 indicates the number of trials. The number of trials is 1000. As described above, the trial is performed 250 times while changing the simulation environment in the order of HOMDP, MDP, HOMDP, and MDP. The vertical axis of the graph in FIG. 8 indicates the average number of steps leading to a reward. The average is the average 2000 set of the 1000 trials of above as a set. Here, the step indicates selection of an action. That is, the number of steps is the number of selected actions. Each trial starts from the final state of the previous trial and ends when the learning system gets a reward or if the number of steps reaches 100 without getting a reward.

  In the graph of FIG. 8, a thick line shows the learning system of this invention. In FIG. 8, it described as the present invention. The thin line shows a learning system using a conventional SARSA (State-Action-Reward-State-Action) learning rule with a qualification trace. In FIG. 8, it was described as a conventional example. The qualification trace parameter λ is 0.7. A straight line indicates the fastest number of steps. In FIG. 8, it was expressed as an ideal value.

  In the learning system of the present invention, the learning constant α in equation (3) is 0.05, and the discount rate γ in equation (1) is 0.95. In the learning system of the conventional example, when the same numerical value as above was used, the performance deteriorated. Therefore, the learning constant α was set to 0.1 and the discount rate γ was set to 0.9.

  As shown in the graph of FIG. 8, in the conventional example, the final average number of the second HOMDP is about 45 times, which is larger than the final average number of the first HOMDP (about 35 times). . Further, the final average number of the second MDP is about 40 times, which is larger than the final average number of the first MDP (about 35 times). In contrast, in the present invention, the final average number of the first HOMDP, the first MDP, the second HOMDP, and the second MDP is all about 30 times.

  In the conventional example, the learning result of the environment before the change affects the learning of the environment after the change, and the learning speed is reduced. However, in the present invention, the learning speed does not decrease even if the environment changes. The average number in each environment of the present invention is also smaller than the average number in the conventional example, and learning in each environment in the present invention is also superior to the conventional example.

  Thus, the learning system of the present invention is superior to a learning system using a conventional SARSA (State-Action-Reward-State-Action) learning rule in learning corresponding to environmental changes and learning in the same environment. Yes. In addition, the learning system of the present invention does not use an environment model having a complicated structure, so that the calculation cost can be reduced.

DESCRIPTION OF SYMBOLS 100 ... Learning system, 101 ... Event list management part, 103 ... Temporary list memory, 105 ... Event list database, 107 ... Learning control part

Claims (6)

An event list database that holds a plurality of event lists, with a set of state / action pairs reaching the state / action pairs immediately before rewarding as an event list,
An event list management unit for classifying and storing state / action pairs into the plurality of event lists;
A learning control unit that updates the expected reward value R (S _t , a _t ) of the state / action pair, which is an element of each event list , as shown in the following Equation 1 :
Learning system with

Here, _St is the state observed at time t. a _t is the action that you selected in time t. E [|] is a conditional expected value. rt _{+ k} is the reward obtained at time t + k. γ is a discount rate. A temporary list storage unit;
The event list management unit stores a state / action pair in the temporary list storage unit each time an action is selected, and stores the state / action stored in the temporary list storage unit every time a reward is obtained. Of the set of pairs, state / action pairs not stored in the event list database are stored in the event list database as elements of the event list of the state / action pair immediately before the reward is obtained. The learning system according to claim 1.   The learning control unit updates the reward expectation value of the state / action pair, which is an element of the event / list of the state / action pair immediately before the reward is obtained, using the reward value every time the reward is obtained. And updating the expected reward value of the state / action pair, which is an element of the event list other than the event list of the state / action pair immediately before obtaining the reward, assuming that the reward value is zero. 2. The learning system according to 2. Event list database that holds multiple event lists as a list of state / action pairs that reach the state / action pairs immediately before the reward is obtained, and an event list management unit, and learning A learning method using a learning system comprising a control unit,
The event list management unit classifying and storing state / action pairs into the plurality of event lists;
The learning control unit includes a step of updating the expected reward value R (S _t , a _t ) of the state / action pair, which is an element of each event list, as shown in Equation 1 below .

Here, _St is the state observed at time t. a _t is the action that you selected in time t. E [|] is a conditional expected value. rt _{+ k} is the reward obtained at time t + k. γ is a discount rate.   The event list management unit temporarily holds a state / action pair each time an action is selected, and each time a reward is obtained, the event list management unit 5. The learning method according to claim 4, wherein a state / action pair that is not stored in the list database is stored in the event list database as an element of an event list of the state / action pair immediately before the reward is obtained.   The learning control unit updates the reward expectation value of the state / action pair, which is an element of the event / list of the state / action pair immediately before the reward is obtained, using the reward value every time the reward is obtained. And updating the expected reward value of the state / action pair, which is an element of the event list other than the event list of the state / action pair immediately before obtaining the reward, assuming that the reward value is zero. 5. The learning method according to 5.

Images