JP5528214B2 - Learning control system and learning control method - Google Patents

Description

  The present invention relates to a learning system and a learning method using reinforcement learning.

  Reinforcement learning is known as a learning method in which a machine such as a robot improves its own control rules by learning (for example, Non-Patent Document 1). On the other hand, there is a method called supervised learning in which learning is performed by explicit teaching from others. By combining the two, for example, it can be expected that the robot will be taught by a person and can be used skillfully according to the situation by trial and error. . However, a learning control system and a learning control method that can efficiently learn the teachings from others and can perform learning while combining the taught contents by trial and error have not been developed.

N.D.Daw & K. Doya, “The computational neurobiology of learning and reward”, Current Opinion in Neurobiology, 2006, 16, pp199-204

  Therefore, there is a need for development of a learning control system and a learning control method that can efficiently learn the teachings from others and that can perform learning while combining the taught contents by trial and error.

  The learning control system according to one aspect of the present invention provides a plurality of events by using a set of state / action pairs as a list of states / action pairs immediately before a reward is obtained and a state when the reward is obtained. An event list database that holds a list, an event list management unit that classifies state / action pairs into the plurality of event lists and stores them in the event list database, and elements of each event list An event list learning control unit that updates an expected reward value of a state / action pair, an action plan unit that obtains a first action value function using the event list of the event list database, and reinforcement learning A reinforcement learning unit for obtaining a second action value function based on the first action value function received from the action plan unit and the reinforcement learning part And a, a behavior selection unit that selects an action based on the second action value function.

  According to the learning control system of this aspect, the event / list learning control unit sets the state / action pair for each event list classified according to the state / action pair immediately before the reward is obtained and the state when the reward is obtained. Since the reward expectation value is updated, teachings from others can be efficiently learned. Further, according to the learning control system according to this aspect, in addition to the teachings from the above, the result of trial and error learning performed by the reinforcement learning unit is also reflected in the event list, so that the taught content is tried. You can learn while combining with mistakes.

  In the learning control system according to one embodiment of the present invention, the action planning unit uses the event list of the event list database to set the target from the current state when the target state is given. The route to the state is searched, and when the route search is successful, an action value function is obtained based on the result of the route search.

  In the present embodiment, the behavior planning unit performs a route search using the event list of the event list database when a target state is given, so that the behavior value function is more efficiently obtained. Can be sought. Therefore, the taught contents can be combined more efficiently by trial and error.

  In the learning control system according to one embodiment of the present invention, the event list learning control unit sets a reward expected value of a state / action pair as an element of each event list to a partial reward that is an expected value for the reward value. Expressed as the sum of the product of the expected value and the expected partial distance that is the expected value of the distance until the reward is obtained, and configured to update the expected partial reward and the expected partial distance separately. Yes.

  According to the present embodiment, since the partial reward expected value and the partial distance expected value are updated separately, learning can be performed more efficiently.

  In the learning control system according to one embodiment of the present invention, the simple moving average value for obtaining the partial reward expectation value and the simple moving average value for obtaining the partial distance expectation value are stored in the event list database. It is configured.

  According to the present embodiment, since the simple moving average value is used to obtain the partial reward expectation value and the partial distance expectation value, it is possible to efficiently learn at a low calculation cost.

  The learning control method according to one aspect of the present invention provides a plurality of events by using a set of state / action pairs as a list of states / action pairs immediately before the reward is obtained and the state when the reward is obtained.・ Learning and action by a learning control system that includes an event list database that holds lists, an event list management unit, an event list learning control unit, an action planning unit, and a reinforcement learning unit Is a learning control method for selecting. In this method, the event list management unit classifies the state / action pairs into the plurality of event lists and stores them in the event list database, and the event list learning control unit Updating an expected reward value of the state / action pair which is an element of the event list. The method includes a step in which the action plan unit obtains a first action value function using an event list in the event list database, and the reinforcement learning unit performs a second action based on reinforcement learning. Obtaining a value function; and selecting the action based on the first action value function received from the action planning unit and the second action value function received from the reinforcement learning unit; Further included.

  According to the learning control method according to this aspect, the event / list learning control unit sets the state / action pair for each event list classified according to the state / action pair immediately before the reward is obtained and the state when the reward is obtained. Since the reward expectation value is updated, teachings from others can be efficiently learned. Further, according to the learning control method according to this aspect, in addition to the teaching from the above, the result of trial and error learning performed by the reinforcement learning unit is also reflected in the event list. You can learn while combining with mistakes.

It is a figure which shows the structure of the apparatus containing the learning control system by one Embodiment of this invention. It is a figure for demonstrating the data structure of an event list database. It is a flowchart for demonstrating operation | movement of an event list management part. It is a flowchart for demonstrating operation | movement of an event list learning control part. It is a figure for demonstrating the detailed operation | movement of step S2030 of FIG. It is a figure for demonstrating the detailed operation | movement of step S2035 of FIG. It is a flowchart for demonstrating operation | movement of an action plan part. It is a figure for demonstrating the method of searching the path | route from an initial state to a target state using the information of an event list database. It is a flowchart for demonstrating operation | movement of an action selection part. It is a figure for demonstrating the procedure of a simulation experiment. It is a figure which shows the result of a 1st simulation. It is a figure which shows the result of a 2nd simulation. It is a figure which shows the result of a 3rd simulation. It is a figure which shows the result of a 4th simulation. It is a figure which shows the result of a 5th simulation. It is a figure for demonstrating the high order Markov decision process (HOMDDP: High Order Markov Decision Process) which is a 5th simulation environment.

  FIG. 1 is a diagram illustrating a configuration of an apparatus 200 including a learning control system 150 according to an embodiment of the present invention. Device 200 may be, for example, a robot. The apparatus 200 includes an information acquisition unit 201, an action output unit 203, a target acquisition unit 205, a teaching acquisition unit 207, and a learning system 150.

  The information acquisition unit 201 acquires input information from the environment 300 and acquires state information of the device 200 itself. When the apparatus 200 is a robot, the information acquisition unit 201 may include a camera and acquire information on the environment 300 based on an image of the environment 300 captured by the camera. Further, the information acquisition unit 201 may acquire state information of the device 200 including the position and orientation of the robot. The information acquisition unit 201 sends the acquired information to the learning control system 150.

  The behavior output unit 203 outputs the behavior selected by the learning control system 150. The change in the environment 300 as a result of the action is acquired as information by the information acquisition unit 201.

  The teaching acquisition unit 205 acquires a teaching of a series of actions from a user or the like so that a reward described later can be obtained at the fastest speed with respect to the state of the apparatus 200, and sends the teaching to the learning control system 150. . The teachings are used to assist learning of the learning system 150 in the early stages of learning.

  The goal acquisition unit 207 receives a goal that the device 200 should achieve from a user or the like, and sends the goal to the learning control system 150.

  The learning control system 150 includes an acquisition information processing unit 109, a reinforcement learning unit 111, an action selection unit 113, and an event / list learning control system 100.

  The acquisition information processing unit 109 processes the information received from the information acquisition unit 201 and determines the “state” of the device 200. Further, “reward” that is an evaluation of the result of “behavior” of the apparatus 200 is determined.

  The event list learning control system 100 and the reinforcement learning unit 111 determine an action value function for evaluating the value of action based on a series of state, action, and reward information. Details of the event list learning control system 100 will be described later. The reinforcement learning unit 111 is a conventional reinforcement learning system, and may be, for example, a system using a SARSA (State-Action-Reward-State-Action) algorithm. The SARSA algorithm is described in detail in, for example, literature (R. S. Sutton, A. G. Barto, Reinforcement Learning: Introduction, MIT Press).

  The behavior selection unit 113 selects the behavior of the device based on the first behavior value function received from the event / list learning control system 100 and the second behavior value function received from the reinforcement learning unit 111. Further, when the behavior selection unit 113 is instructed by the teaching acquisition unit 205, the behavior selection unit 113 selects the taught behavior.

  Here, the basic concept of the event list learning control system 100 will be described.

ここで、γは割引率と呼ばれる定数であり、E[]は期待値を意味する。 First, the behavior value function will be described. The observed state space (state space) is S, and the action option space (action space) is A. Let | S | be the number of elements in the state space, and | A | be the number of elements in the action space. The element of the state space is represented by s, and the element of the action space is represented by a. At this time s _t is an element of the observed state space at time t, is the same a _t. The action value function is an expected value of the reward r that is obtained from the current t in the future when the state _st is observed at the current time t and the action a _t is taken, and is expressed as follows.

Here, γ is a constant called a discount rate, and E [] means an expected value.

ここで、

を報酬期待値と呼称する。 Equation (1) can be expanded as follows.

here,

Is called a reward expectation value.

ここで、Pr(|)は、条件付確率を示す。Pr(k|)は、一連の状態が現在時刻からkステップ後に終端に達する確率である。ここで、「終端」とは、上記一連の状態の最後の状態（この場合は、報酬を得た状態）を指す。Pr(r_t+k|)は、t+kで報酬rを得る確率である。 Equation (3) can be modified as follows under the general assumption of a Markov Decision Process.

Here, Pr (|) represents a conditional probability. Pr (k |) is the probability that a series of states will reach the end after k steps from the current time. Here, the “termination” refers to the last state of the series of states (in this case, a state where a reward is obtained). Pr (r _{t + k} |) is the probability of obtaining reward r at t + k.

(S, A, S ′) represents (s _{t + k−1} , a _{t + k−1} , s _{t + k} ), and the state / action pair (s _{t + k} ) immediately before the reward r _{t + k} is obtained. _-1 , a _{t + k-1} ) and the state s _{t + k} when the reward r _{t + k} is obtained. Here, the state / action pair refers to a pair of the state and the action when the device 200 selects the action in a certain state.

式（５）の左辺を部分距離期待値と呼称する。「部分」とは、(S,A,S’)の場合に限定された期待値であることを意味する。「距離」とは、現在の状態から報酬を得る状態までの距離の期待値であることを意味する。 From the definition, the following equation holds.

The left side of Expression (5) is referred to as a partial distance expected value. “Part” means an expected value limited to the case of (S, A, S ′). “Distance” means an expected value of the distance from the current state to the state where the reward is obtained.

式（６）の左辺を部分報酬期待値と呼称する。「部分」とは、(S,A,S’)の場合に限定された期待値であることを意味する。「報酬」とは、報酬の値の期待値であることを意味する。 Also, from the definition, the following formula is established.

The left side of Equation (6) is referred to as a partial reward expected value. “Part” means an expected value limited to the case of (S, A, S ′). “Reward” means an expected value of a reward value.

  Equation (4) indicates that the expected reward value can be expressed as the sum of products of the partial distance expectation value represented by Equation (5) and the partial distance expectation value represented by Equation (6). In addition, a series of state-action pairs ending with a state in which a reward is obtained indicates that they can be classified into “parts”, that is, groups (sets) for each (S, A, S ′). Therefore, a series of state / action pairs are classified into sets for each (S, A, S '), the state / action pairs are stored for each set, and the reward expectation value of each state / action pair is set for each partial distance. It is possible to learn for each expected value and each partial reward expected value.

ここで、mの値は、環境３００の変化にすばやく追随するために、１０などの比較的小さい値が望ましい。また初期状態でデータの個数がm以下の場合は通常の平均値を計算する。 In order to calculate the partial distance expectation value represented by equation (5) and the partial distance expectation value represented by equation (6), the simple moving average, which is the average of the most recent m data, May be used. The simple moving average can be expressed by the following formula.

Here, the value of m is preferably a relatively small value such as 10 in order to quickly follow changes in the environment 300. In the initial state, when the number of data is less than or equal to m, a normal average value is calculated.

の場合に、

である。それ以外の場合には、

である。 For the partial distance expectation represented by Equation (5),

In the case of

It is. Otherwise,

It is.

の場合に、

である。それ以外の場合には、

である。 For the partial reward expectation value expressed by Equation (6),

In the case of

It is. Otherwise,

It is.

The simple moving average for the partial distance expected value is expressed as ma _{SAS '} [γ | s, a], and the simple moving average for the partial reward expected value is expressed as ma [r | SAS'].
Next, the configuration of the event list learning control system 100 will be described. As shown in FIG. 1, the event list learning control system 100 includes an event list management unit 101, a temporary list storage unit 103, an event list database 105, an event list learning control unit 107, and an action planning unit 108. Including.

The event list management unit 101 stores a series of state / action pairs in the temporary storage unit 103, and when receiving a reward, the series of state / action pairs is stored for each (S, A, S ′). The data is classified into sets, and each set is stored in the event list database 105. The event list database 105 includes a simple moving average ma [r | SAS '] for the partial reward expectation value and a simple moving average ma _SAS' [for the partial distance expectation value of the set for each (S, A, S ′). γ | s, a] is also stored.

  Every time a new state / action pair is received, the event list learning control system 100 calculates ma [r | SAS '] and a simple moving average of the set for each (S, A, S ′) with respect to the minute reward expectation value. Update (learn) the simple moving average for the partial distance expected value.

  When the target plan is given from the target acquisition unit 207, the action plan unit 108 uses the data of the event list database 105 to search for a route from the current state to the target state, If the search is successful, an action value function is obtained based on the result of the route search. In other cases, the action value function is obtained using the data of the event list database 105.

  Details of each component of the event list learning control system 100 will be described below.

FIG. 2 is a diagram for explaining the data structure of the event list database 105. In FIG. 2, (S, A, S ′) n indicates a state / action pair immediately before obtaining a reward and a state when the reward is obtained. (S, A, S ′) n forms a set with a series of state / action pairs up to (S, A, S ′) n. This set is called an event list. (s _i , a _j ) indicates a state / action pair included in the event list. Note that (S, A) is also included in this set. In association with (s _i , a _j ), the simple moving average ma _{SAS ′} [γ | s, a] of the partial distance expectation value is also held in the event list. Furthermore, auxiliary variables e [s _i , a _j ] for updating the simple moving average of the partial distance expectation values are also held. This auxiliary variable will be described later. In addition, a simple moving average ma [r | (S, A, S ′) _n ] of a corresponding partial reward expectation value is held in association with each (S, A, S ′) _n .

  As described above, the data in the event list database 105 is classified into data for each state (S, A, S ′) when the reward is obtained, that is, the state / action pair immediately before the reward is obtained, that is, the event list.

  FIG. 3 is a flowchart for explaining the operation of the event list management unit 101.

  In step S <b> 1005 of FIG. 3, the event / list management unit 101 determines whether or not a state / action pair (s, a) has been received from the action selection unit 113. Here, the action selection unit 113 sends a state / action pair (s, a) to the event / list management unit 101 each time an action is selected. If the state / action pair (s, a) has been received, the process proceeds to step S1010. If the state / action pair (s, a) has not been received, wait.

  In step S <b> 1010 of FIG. 3, the event / list management unit 101 stores the state / action pair (s, a) in the temporary list storage unit 103.

  In step S1015 of FIG. 3, the event list management unit 101 determines whether or not the reward and the state s ′ when the reward is obtained from the acquired information processing unit 109. Here, the acquired information processing unit 109 determines a reward based on the information acquired by the information acquisition unit 201 after a predetermined time has elapsed since the behavior output unit 207 outputs the behavior, and sends the reward to the event / list management unit 101. If the reward has been received, the process proceeds to step S1020. If no reward has been received, the process returns to step S1005 after a predetermined time has elapsed.

  In step S1020 of FIG. 3, the event / list management unit 101 uses the state / action pair (s, a) stored last in the temporary list storage unit 103 as the state / action pair (S , A), the state s ′ when the reward is obtained is set as S ′, and the state / action pair immediately before the reward is obtained and the state (S, A, S ′) when the reward is obtained are generated.

  In step S1025 of FIG. 3, the event list management unit 101 determines whether (S, A, S ′) exists in the event list database 105. If (S, A, S ') exists, the process proceeds to step S1035. If (S, A, S ') does not exist, the process proceeds to step S1030.

  In step S 1030 of FIG. 3, the event list management unit 101 stores (S, A, S ′) in the event list database 105.

  In step S1035 of FIG. 3, the event list management unit 101 determines that each of the state / action pairs (s, a) stored in the temporary list storage unit 103 is (S, A, It is determined whether it is included in the event list of S '). If it is included in the event list of (S, A, S '), the process proceeds to step S1045. If it is not included in the event list of (S, A, S '), the process proceeds to step S1040.

  In step S1040 of FIG. 3, the event list management unit 101 selects a state / action pair (s, a) that is not included in the event list of (S, A, S ′) as (S, A, S ′). Add to your event list. At this time, the number of state / action pairs to be added is limited to a predetermined number.

  In step S <b> 1045 of FIG. 3, the event / list management unit 101 determines whether or not the processing of step S <b> 1035 has been performed for all the state / action pairs (s, a) stored in the temporary list storage unit 103. If the process of step S1035 is performed for all state / action pairs (s, a), the process proceeds to step S1050. If the process of step S1035 is not performed for all the state / action pairs (s, a), the process returns to step S1035.

  In step S1050 of FIG. 3, the event / list management unit 101 clears (deletes) all the state / action pairs (s, a) stored in the temporary list storage unit 103.

  FIG. 4 is a flowchart for explaining the operation of the event list learning control unit 107.

  In step S2005 of FIG. 4, the event / list management unit 101 determines whether a state / action pair (s, a) is received from the action selection unit 113. If the state / action pair (s, a) has been received, the process proceeds to step S2010. If the state / action pair (s, a) has not been received, wait.

  In step S2010 of FIG. 4, the event list management unit 101 determines whether or not the next state s ′ has been received from the acquired information processing unit 109. If the next state s' has been received, the process proceeds to step S2015. If the next state s' has not been received, the process waits.

  In step S2015 in FIG. 4, the event list management unit 101 determines whether or not the reward at that time has been received. If a reward has been received, the process proceeds to step S2020. If no reward has been received, the process proceeds to step S2025.

  In step S2020 of FIG. 4, the event list management unit 101 substitutes the value of reward into r.

  In step S2025 of FIG. 4, the event list management unit 101 substitutes zero for r.

  In step S2030 of FIG. 4, the event list management unit 101 updates the auxiliary variable e [s, a] of each event list in the event list database 105.

  In step S2035 of FIG. 4, the event list management unit 101 updates the value of the simple moving average of each event list in the event list database 105.

  FIG. 5 is a diagram for explaining the detailed operation of step S2030 of FIG.

  In step S3005 of FIG. 5, the event list management unit 101 extracts one event list (S, A, S ′) n from the event list database 105.

  In step S3010 of FIG. 5, the event list management unit 101 extracts one state / action pair (s ″, a ″) from the extracted event list (S, A, S ′) n.

  In step S3015 of FIG. 5, the event / list management unit 101 determines whether or not the extracted state / action pair (s ″, a ″) is the same as the received state / action pair (s, a). If they are the same, the process proceeds to step S3020. If not, the process proceeds to step S3025.

In step S3020 of FIG. 5, the event / list management unit 101 updates the auxiliary variable e [s ″, a ″] of the state / action pair (s ″, a ″) according to the following equation. Here, the initial values of the auxiliary variables are all zero.

ここで、γは割引率と呼ばれる定数である。 In step S3025 of FIG. 5, the event / list management unit 101 updates the auxiliary variable e [s ″, a ″] of the state / action pair (s ″, a ″) according to the following equation.

Here, γ is a constant called a discount rate.

  In step S3030 of FIG. 5, the event list management unit 101 determines whether or not all state / action pairs (s ″, a ″) in the extracted event list (S, A, S ′) n have been checked. . If all state / action pairs (s ″, a ″) have been checked, the process proceeds to step S3035. If all the state / action pairs (s ″, a ″) are not checked, the process returns to step S3010.

  In step S3035 of FIG. 5, the event list management unit 101 determines whether all event lists (S, A, S ′) n in the event list database 105 have been checked. If all event lists (S, A, S ') n have been checked, the process ends. If all event lists (S, A, S ') n are not checked, the process returns to step S3005.

  FIG. 6 is a diagram for explaining the detailed operation of step S2035 of FIG.

  In step S4005 of FIG. 6, the event list management unit 101 selects only the event list (S, A, S ′) n in the event list database 105 that includes s in its state / action pair. Take out.

  In step S4010 of FIG. 6, the event list management unit 101 has the same (S, A, S ′) as (s, a, s ′) in the extracted event list (S, A, S ′) n. It is determined whether it is. If they are the same, the process proceeds to step S4015. If not, the process proceeds to step S4020.

ただし、上述にように

である。 In step S4015 of FIG. 6, the event list management unit 101 calculates the simple moving average ma [r | SAS ′] for the partial reward expectation value of the extracted event list (S, A, S ′) n as follows: Update according to

However, as mentioned above

It is.

  In step S4020 of FIG. 6, the event list management unit 101 extracts one state / action pair (s ″, a ″) from the extracted event list (S, A, S ′) n.

  In step S4025 of FIG. 6, the event / list management unit 101 determines whether the auxiliary variable e [s ″, a ″] of the state / action pair (s ″, a ″) is positive. If the auxiliary variable e [s ″, a ″] is positive, the process proceeds to step S4030. If the auxiliary variable e [s ″, a ″] is not positive, the process proceeds to step S4050.

  In step S4030 of FIG. 6, the event list management unit 101 has (S, A, S ′) of the extracted event list (S, A, S ′) n the same as (s, a, s ′). It is determined whether it is. If they are the same, the process proceeds to step S4035. If not, the process proceeds to step S4040.

ただし、上述にように

である。 In step S4035 of FIG. 6, the event list management unit 101 performs simple moving average ma _{SAS ′} [γ | s, a with respect to the partial distance expectation value of the extracted event list (S, A, S ′) n. ] Is updated according to the following formula.

However, as mentioned above

It is.

ただし、上述にように

である。 In step S4040 of FIG. 6, the event list management unit 101 performs the simple moving average ma _{SAS ′} [γ | s, a for the partial distance expected value of the extracted event list (S, A, S ′) n. ] Is updated according to the following formula.

However, as mentioned above

It is.

In step S4045 in FIG. 6, the event / list management unit 101 updates the auxiliary variable e [s ″, a ″] of the state / action pair (s ″, a ″) according to the following formula.

  In step S4050 of FIG. 6, the event list management unit 101 determines whether or not all state / action pairs (s ″, a ″) of the extracted event list (S, A, S ′) n have been checked. . If all state / action pairs (s ″, a ″) are checked, the process proceeds to step S4055. If all the state / action pairs (s ″, a ″) are not checked, the process returns to step S4020.

  In step S4055 of FIG. 6, the event list management unit 101 determines whether all event lists (S, A, S ′) n in the event list database 105 have been checked. If all event lists (S, A, S ') n have been checked, the process ends. If all event lists (S, A, S ') n are not checked, the process returns to step S4005.

  FIG. 7A is a flowchart for explaining the operation of the action plan unit 108.

  In step S5005 of FIG. 7A, the action planning unit 108 determines whether or not the state s ′ is received from the acquired information processing unit 109. If the state s' has been received, the process proceeds to step S5010. If the state s' has not been received, the process waits.

  In step S5010 of FIG. 7A, the action planning unit 108 determines whether or not the target S ′ has been received from the target acquisition unit 207. If the target S ′ has been received, the process proceeds to step S5015. If the target S 'has not been received, the process proceeds to step S5030.

  In step S5015 of FIG. 7A, the behavior planning unit 108 uses the information in the event list database 105 to search for a route from the initial state s ′ to the target state S ′.

FIG. 7B is a diagram for explaining a method for searching for a route from the initial state to the target state using information in the event list database 105. FIG. 7B (a) is a diagram showing a state transition from S ₀ to S ₃ . FIG. 7B (b) is a diagram showing an event list corresponding to the state transition of FIG. 7B (a). In the event list of FIG. 7B (b), (S, A, S ′) is (S ₂ , a ₃ , S ₃ ). FIG. 7B (c) is a diagram showing combinations of event lists. As shown in FIG. 7B (c), a route from the initial state to the target state is searched by combining a plurality of event lists. The route search method may be, for example, a best-first search method (for example, the Japanese Society for Artificial Intelligence, Artificial Intelligence Dictionary, 2006, Kyoritsu Publishing).

  In step S5020 of FIG. 7A, the action planning unit 108 determines whether the route search has been successful. If the route search is successful, the process proceeds to step S5025. If the route search is not successful, the process proceeds to step S5030.

式（８）は、図７Ｂ（ｃ）のイベント・リストの組合せに対応し、初期状態は、sである。 In step S5025 of FIG. 7A, the action planning unit 108 uses the value of the corresponding simple moving average in the event list database 105 as the action value function corresponding to the recommended action a based on the result of the route search. Is obtained according to the following equation and output.

Equation (8) corresponds to the event list combination of FIG. 7B (c), and the initial state is s.

In step S5030 of FIG. 7A, the action planning unit 108 uses the information in the event list database 105 to obtain and output an action value function corresponding to the state s ′. Specifically, the action planning unit 108 calculates the partial remuneration expectation value and the partial distance expectation value of the part corresponding to the state s ′ in the event list stored in the event list database 105 according to the following formula. An expected reward value for the state s ′ represented by the equation (4) is obtained.

  FIG. 8 is a flowchart for explaining the operation of the action selection unit 113.

  In step S6005 of FIG. 8, the action selection unit 113 determines whether or not the state s ′ is received from the acquired information processing unit 109. If the state s ′ has been received, the process proceeds to step S6010. If the state s' has not been received, the process waits.

  In step S6010 of FIG. 8, the action selection unit 113 determines whether or not a teaching is received from the teaching acquisition unit 205. If the instruction has been received, the process proceeds to step S6015. If the instruction has not been received, the process proceeds to step S6020.

  In step S6015 of FIG. 8, the action selecting unit 113 selects and outputs the taught action a, and ends the process.

  In step S6020 of FIG. 8, the behavior selection unit 113 determines whether or not the behavior value function Q is received from the reinforcement learning unit 111. If the behavior value function Q has been received, the process proceeds to step S6025. If the action value function Q has not been received, the process returns to step S6005.

  In step S6025 of FIG. 8, the action selection unit 113 determines whether or not the action value function tQ is received from the action plan unit. If the action value function tQ has been received, the process proceeds to step S6030. If the behavior value function tQ has not been received, the process proceeds to step S6035.

  In step S6030 of FIG. 8, the action selection unit 113 sets the sum of tQ and Q as tQ.

  In step S6035 of FIG. 8, the action selection unit 113 sets Q to tQ.

  In step S6040 of FIG. 8, the action selection unit 113 selects the action a probabilistically based on tQ and outputs it.

  Hereinafter, a simulation experiment of the learning control system 150 according to the present embodiment will be described.

FIG. 9 is a diagram for explaining the procedure of the simulation experiment. There are 8 observed states from s ₀ to s ₇ . There are 8 actions from a ₀ to a ₇ . “Teached episode” indicates an episode taught to the apparatus 200 including the learning control system 150 via, for example, the teaching acquisition unit 205. Here, an episode refers to a series of states and actions that occur continuously. Hereinafter, the apparatus 200 is referred to as an agent.

For example, in episode A, s ₀ is first observed as the observation state. At this time they are instructed to take action a _1. When the agent selects action a ₁ , the observation state is changed to s _{1 as a} result. Similarly, when reaching the observation state s ₃ in the same manner, a positive reward value is given to the agent. Similarly, episode B, episode C, and episode D are taught once each. A positive reward value is given at the end of episode B and episode C. However, at the end of episode D, a negative reward value is given, and episode D is taught as undesirable.

Next, a problem is given to the agent. In the case of Problem 1 in FIG. 9, the agent is placed in the observation state s ₀ and the target state is presented as s ₆ . The agent is required to transition the state from the state s ₀ to the state s ₆ in the shortest step. Here, a step is the number of actions taken for a state. In the case of Problem 2 in FIG. 9, the agent is placed in the observation state s ₀ and the target state is presented as s ₇ . The agent is required to transition the state from the state s ₀ to the state s ₇ in the shortest step.

  In the actual simulation, a simulation consisting of 20 trials was performed. Here, the trial is a series of actions that the agent performs until reaching the terminal depending on the state. However, the maximum number of trial steps is 50. In other words, if the agent's action does not reach the end even after 50 steps, the trial ends. The first four trials, that is, the first to fourth trials are episode teaching periods and are taught. Specifically, in the first to fourth trials, episodes A to D are taught as described above. The fifth to twentieth trials are problem-handling periods. During the problem handling period, problems 1 and 2 are alternately given to the agent. Specifically, in the fifth trial, problem 1 is given, problem 6 is given in the sixth trial, problem 1 is given in the seventh trial, and problem 2 is given in the eighth trial. In this way, problem 1 and problem 2 are alternately given until the 20th trial.

  FIG. 10 is a diagram illustrating a result of the first simulation. Here, the horizontal axis of the graphs of FIGS. 10 to 13 indicates the number of trials, and the vertical axis indicates the number of steps of each trial. The number of steps in each trial is the average of 1000 repeated results. FIGS. 10 to 13 show the results of the SARSA (0.1) algorithm and the SARSA (0.5) algorithm in addition to the learning control system 150 according to this embodiment of the present invention. 0.1 and 0.5 are parameters of the SARSA algorithm represented by λ (R. S. Sutton, A. G. Barto, Reinforcement Learning: Introduction, MIT Press). Note that the reinforcement learning unit 111 of the learning control system 150 according to the present embodiment uses the SARSA (0.5) algorithm.

  In FIG. 10, the result of the SARSA (0.1) algorithm indicates that learning is not performed at all because the number of steps does not decrease in the problem coping period. The result of the SARSA (0.5) algorithm shows that learning is performed only for problem 1 because the number of steps is reduced only for problem 1. The result of the learning control system 150 according to the present embodiment indicates that learning is correctly performed for the problems 1 and 2.

  FIG. 11 is a diagram illustrating a second simulation result. In this simulation, the problem 1 and the problem 2 are alternately given to the agent during the problem handling period, as in the case of the first simulation. However, in the second simulation, the trial of the problem coping period is probabilistic. Specifically, even if the agent's chosen action is correct, it will only correctly transition to the next state with a probability of 0.8. According to FIG. 11, even in this case, the learning control system 150 according to the present embodiment performs learning correctly.

FIG. 12 is a diagram illustrating a third simulation result. In this simulation, the problem handling period environment has changed from the episode teaching period environment. Specifically, in episode A corresponding to problem 1 in the episode teaching period, if the agent selects action a ₂ in state s ₁ , it becomes state s ₂ , but in the trial of problem 1 it becomes s _7. Suppose this happens. The transition at this time is not probabilistic but “deterministic”. In FIG. 12, “1 ′” indicates that the problem 1 has changed. Problem 2 is the same as the episode teaching period. According to FIG. 12, even in this case, the learning control system 150 according to the present embodiment effectively solves the problem 1 that is not taught by combining learning by the event list learning control system 100 with learning by the reinforcement learning unit 111. Also learn correctly for '.

  FIG. 13 is a diagram illustrating a fourth simulation result. In this simulation, as in the second simulation, the trial of the problem coping period is probabilistic, and the problem 1 has changed as in the third simulation. Even in this case, the learning control system 150 according to the present embodiment correctly learns the problem 1 ′ and the problem 2 by effectively combining the learning by the event / list learning control system 100 with the learning by the reinforcement learning unit 111. Do.

  FIG. 14 is a diagram illustrating a fifth simulation result. In the fifth simulation, there is no teaching and no target is given.

FIG. 15 is a diagram for explaining a high order Markov decision process (HOMDDP) which is a fifth simulation environment. The selected may behavior, a _0, a _1, a ten ... a _9, of which the related to the reward is six of _{a 0, a 1, ... a} 5. This process includes process A and process B. In order to obtain the next reward when the reward is obtained in the process A, it is necessary to select the process B. To obtain the next reward when the reward is obtained in the process B, it is necessary to select the process A. is there. That is, it is necessary to select different actions in the process A and the process B for the same observation signal. Each transition is probabilistic. In the process A, a transition is made from s ₀ to s ₂ with a probability of 0.3. In other cases, transition is made with a probability of 0.9. In the process B, a transition is made from s ₁ to s ₂ with a probability of 0.3. In other cases, transition is made with a probability of 0.9. Furthermore, there are two signals that can be observed for each state. For example, for s _0, there are O ₀₀ and O ₀₁ signals, one of which is observed with a probability of 0.5.

  The horizontal axis of the graph in FIG. 14 indicates the number of trials, and the vertical axis indicates the number of steps in each trial. The number of steps in each trial is the average of 1000 repeated results. FIG. 14 shows the result of the SARSA (0.5) algorithm in addition to the learning control system 150 according to the present embodiment of the present invention. In FIG. 14, the solid line indicates the average value of the number of steps by the learning control system 150, and the alternate long and short dash line indicates the average value of the number of steps by the SARSA (0.5) algorithm. A dotted line indicates the standard deviation of the number of steps by the learning control system 150, and a two-dot chain line indicates the standard deviation of the number of steps by the SARSA (0.5) algorithm. According to FIG. 14, the learning control system 150 according to the present embodiment of the present invention converges with fewer steps than the SARSA (0.5) algorithm. As a result, it can be seen that the event list learning control system 100 assists the learning of the reinforcement learning unit 111 even when there is no teaching or presentation of the goal.

DESCRIPTION OF SYMBOLS 100 ... Event list learning control system, 101 ... Event list management part, 103 ... Temporary list storage, 105 ... Event list database, 107 ... Event list learning control part, 108 ... Action plan part

Claims (2)

An event list database that holds a plurality of event lists, with a set of state / action pairs as a list of states / action pairs immediately before the reward is obtained and the state when the reward is obtained;
An event list management unit for classifying state / action pairs into the plurality of event lists and storing them in the event list database;
An event list learning control unit that updates a reward expected value of a state / action pair that is an element of each event list;
An action planning unit for obtaining a first action value function using an event list of the event list database;
A reinforcement learning unit for obtaining a second action value function based on reinforcement learning;
An action selection unit that selects an action based on the first action value function received from the action plan unit and the second action value function received from the reinforcement learning unit;
With
The event list management unit has the observed state, action, and reward, respectively, the state at the time immediately before obtaining one reward, the action taken for the state, and the result of the action And a state-action chain represented by a set composed of the state when the reward is obtained, and a list configured by associating a set of state / action pairs up to the state-action chain. The event list is associated with the one reward and stored in the event list database.
The first and second action value functions represent expected values of rewards that will be obtained from the present to the future if the actions determined by the action planning unit and the reinforcement learning unit are executed, respectively. The
The first behavior value function includes a plurality of the expected reward values that are expected values of rewards obtained at other times after the behavior taken at the one time with respect to the state at the one time. Defined as a weighted addition over other times,
The expected reward value is a probability that a reward is obtained through one state-behavior chain at another time after the action taken at the one time with respect to the state at the one time. The partial distance expectation value, which is a value obtained by weighting over the time, and the action taken at the one time with respect to the state at the one time, through the one state action chain at another time thereafter The product of the expected value from which one reward is obtained and the partial reward expectation value, which is a value obtained by weighting and adding all the rewards obtained through the one state-action chain, to the given target state. Calculated as an added value for one set of the state-action chain constituting the route,
Learning control system. An event list database that holds a plurality of event lists, with a set of state / action pairs as a list of states / action pairs immediately before the reward is obtained and the state when the reward is obtained; A learning control method for performing learning and selecting an action by a learning control system including an event list management unit, an event list learning control unit, an action planning unit, a reinforcement learning unit, and an action selection unit. There,
The event list management unit classifying state / action pairs into the plurality of event lists and storing them in the event list database;
The event list learning control unit updates a reward expectation value of a state / action pair which is an element of each event list;
The action planning unit using the event list of the event list database to obtain a first action value function;
The reinforcement learning unit obtaining a second action value function based on reinforcement learning;
The action selecting unit selecting an action based on the first action value function received from the action plan unit and the second action value function received from the reinforcement learning unit;
Including
In the storing step, the observed state, action, and reward are respectively the state at the time immediately before obtaining one reward, the action taken for the state, and the result of the action, A list consisting of a set of state / action pairs up to the state-action chain is associated with the state-action chain represented by the set consisting of the state when the reward is obtained, and one event Classify it as a list, associate the one event list with the one reward, and store it in the event list database;
The first and second action value functions represent expected values of rewards that will be obtained from the present to the future if the actions determined by the action planning unit and the reinforcement learning unit are executed, respectively. The
The first behavior value function includes a plurality of the expected reward values that are expected values of rewards obtained at other times after the behavior taken at the one time with respect to the state at the one time. Defined as a weighted addition over other times,
The expected reward value is a probability that a reward is obtained through one state-behavior chain at another time after the action taken at the one time with respect to the state at the one time. The partial distance expectation value, which is a value obtained by weighting over the time, and the action taken at the one time with respect to the state at the one time, through the one state action chain at another time thereafter A route to the target state is formed by multiplying the expected value from which one reward is obtained and the partial reward expected value, which is a value obtained by weighting and adding all rewards obtained through the one state-action chain. Calculated as an added value for the set of state-action chains.
Learning control method.

Images