JPH11175493A - Experience enhancement type enhanced learning method using behavior selection network and record medium recorded with experience enhancement type enhanced learning program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform learning that has flexibility and to perform learning in the shortest path by changing the attenuation factor of enhancement value with the existence of experience that attenuates an episode and finishing active propagation when the enhancement value falls below prescribed threshold. SOLUTION: When an enhancement value is propagated from an adjacent storage module, a storage module voluntarily propagates the enhancement value to other storage modules it adjoins. A propagation method is to propagate storage modules which have experience that configures an episode and enhancement values R due to different attenuation factors except them after attenuation is performs as a whole as shown in a diagram. Active propagation is finished when enhancement value to be propagated falls below certain threshold. Learning in an L-ANA is to store how many enhancement values are propagated to each storage module. This active propagation can easily operate its characteristic with two parameters which are the dimension of enhancement value that is used at the time of performing active propagation and an attenuation factor that is used at the time of propagating.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an action selection network devised as an action planning method that enables an autonomous action subject operating in a complex and dynamically changing environment to effectively adapt to changes. A new experience-based reinforcement learning method using a behavior selection network that adapts the framework to the conventional experience-based reinforcement learning method, profit-sharing, specifically, an autonomous mobile robot that interacts with humans in the real world, In order to enable autonomous actors, such as interface agents that perform autonomous interaction between the Internet and users, to not only act according to conventional action planning modules, but also to efficiently adapt to individual situations in the environment. Experience-based reinforcement learning method and experience-based reinforcement learning program using behavior selection network for learning On recording the recording medium.

[0002]

[Prior Art] Behavior selection network (P.Maes: The Ag
ent Network Architecture (ANA), SIGART Bulletin, Vol.
2, No. 4, pp. 115-120, 1991) is a framework in which a set of relatively single-function modules cooperate by propagating activity values to each other to plan a suitable action as a whole module. is there. Since each module behaves autonomously without the need for centralized control, it is particularly characterized by its robustness and scalability, and can perform an appropriate action plan while flexibly and immediately responding to dynamic environmental changes. Therefore, this is an effective means when constructing an action plan module for an autonomous mobile robot operating in an environment such as the real world or the Internet, or a software agent (electronic secretary) that mediates between the user and the Internet.

However, in order for the autonomous action subject to adapt to the environment more, a "learning function" is indispensable for individually coping with various situations encountered in the environment. Therefore, an approach that incorporates a learning function into the framework of the action selection network without damaging the features of the action selection network is desired. However, at present, no framework that incorporates the learning function has been proposed.

[0004] profit-sharing (JJGrefenstette: Cred
it Assignment in Rule DiscoverySystems Based on Ge
neric Algorithms, Machine Learning, Vol. 3, pp. 225-245
(1988)) is an experience-based reinforcement learning method, which collectively reinforces the action sequence up to that point when a reward is obtained.
The action sequence at this time is called an “episode”. This prof
it-sharing requires a small number of trials for learning, and Q-learning (CJC Watkins and P. Dayan: Techni
cal Note: Q-Learning, Machine Learning, Vol.8, pp.5
5-68 (1992)).

As a reinforcement learning method, Q-learning has recently attracted attention. Q-learning is an environment identification type reinforcement learning method, and it has been proved that an optimum learning effect can be obtained if an environment state for obtaining a Q value is accurately identified. However, problems were pointed out, such as the fact that it requires a much larger number of trials than profit-sharing, and that if the environment changes dynamically, the overall learning results obtained so far will be affected. ing.

Accordingly, Q-learning is suitable in a situation in which detailed knowledge of the entire environment of the action subject can be acquired, but in an environment that changes dynamically, such as the autonomous action subject we are interested in, this time. In situations where it runs within and therefore always has only incomplete knowledge of the environment, learning methods like profit-sharing are more appropriate.
However, even in profit-sharing, the learning of the part affected by the change has to be invalidated, and there is a limit that the learning effect largely depends on the scale of the dynamic environment change.

[0007]

In order for an autonomous action subject to learn, it is necessary to collect environmental information. However, it is impossible to obtain all the information of the real world and the Internet in detail in advance. Therefore, these actors build an environmental model while collecting local information using sensors and the like. However, the model obtained is always incomplete because the environment changes dynamically. In such a situation, it is necessary to consider an experience-reinforcement-type reinforcement learning method that is particularly robust to dynamic environmental changes.

[0008] The present invention has been made in view of the above,
The objective is to enhance experience using a behavior selection network that is more robust to dynamic environmental changes than conventional profit-sharing and can efficiently adapt to individual environmental situations. It is an object of the present invention to provide a recording medium on which a reinforcement learning method and an experience reinforcement type reinforcement learning program are recorded.

[0009]

SUMMARY OF THE INVENTION To achieve the above object, the present invention according to the first aspect provides an autonomous action subject operating in a complex and dynamically changing environment, effectively adapting to the change. To be able to profit action selection network framework
-This is an experience-based reinforcement learning method adapted to -sharing,
A storage agent, which is an autonomous subject, is assigned to each state constituting an episode, which is a transition sequence of state elements.A storage agent is also assigned to states other than the episode in view when the internal state moves. When the enhancement value is propagated, the enhancement value is spontaneously propagated to the adjacent memory agent, and after the attenuation is performed as a whole, the attenuation rate of the enhancement value is changed depending on whether or not the user has experienced episode attenuation. The point is that the active propagation is realized as a cooperative operation of the storage agent group, such that the active propagation is terminated when the threshold value becomes equal to or less than the threshold value.

According to the first aspect of the present invention, it is possible to perform flexible learning by performing activity propagation even on storage agents that are not directly related to episodes but are adjacent to each other. In addition, efficient learning can be performed with the shortest path.

Further, the present invention according to claim 2 proposes a framework of an action selection network that enables an autonomous action subject operating in a complex and dynamically changing environment to effectively adapt to changes. A recording medium that records an experience-reinforcement-type reinforcement learning program adapted to sharing, in which a storage agent, which is an autonomous subject, is assigned to each state constituting an episode that is a transition sequence of state elements, and when an internal state moves. When a storage agent is also assigned to a state other than the episode that has entered the field of view, when the reinforcement value from the adjacent storage agent is propagated, the reinforcement value spontaneously propagates to the adjacent storage agent,
Active propagation is realized as a cooperative operation of a group of storage agents, such as changing the decay rate of the reinforcement value depending on the experience of attenuating the episode and terminating the activation propagation when the reinforcement value falls below a predetermined threshold. Is the gist.

According to the present invention, a storage agent is assigned to each state constituting an episode, and a storage agent is also assigned to a state other than the episode in view when the internal state moves. When the enhancement value from the adjacent storage agent is propagated, the enhancement value is propagated to the adjacent storage agent, and after the attenuation is performed as a whole, the decay rate of the enhancement value is changed depending on whether or not the episode has been attenuated. Since the experience-enhancement-type reinforcement learning program for terminating the activity propagation when is less than or equal to a predetermined threshold value is recorded as a recording medium, it is possible to enhance the circulation by using the recording medium.

[0013]

BEST MODE FOR CARRYING OUT THE INVENTION First, in order to explain an experience-reinforcement type reinforcement learning method L-ANA of the present invention, consider an autonomous action subject A moving in a grid-like state space S as shown in FIG. The individual states constituting the state space S are S1
a,..., S7f. Autonomous action subject A
Can move one block at a time up, down, left, and right in the state space S, and use a virtual battery mounted on the action subject A when moving. The battery decreases as it moves, but can be replenished at charging point B. There are several charging points B in the state space S, and the amount of energy that can be supplied is also different. Also, there are obstacles in the state space S, and the subject A cannot pass through the obstacles. The position and number of the charging point B do not change, but the position and number of the obstacle dynamically change. The reward is given a value proportional to the amount of replenishment energy.

In the initial state, the action subject A has no information on the position of the charging point B or the obstacle except that the state space S can move up, down, left and right in a grid-like environment. However, a virtual sensor is mounted on the action subject A, and it is possible to obtain the environmental status regarding the shaded area in FIG.

The action subject A has an internal state space A _S that changes depending on the state space S. In this example, A _S =
{A _s1 : charge required, A _s2 : no charge required} 2
One can think of a variety of internal states.

The learning required for the action subject A is to learn a charging point B capable of supplying more energy and a moving route to the charging point B. Also,
Although charging energy is small, nearby charging point B
It is hoped that it is possible to learn that replenishing is more efficient. And since the obstacle disappears / appears dynamically, it is most important to be able to respond flexibly when detecting this, and not to reduce the learning efficiency.

The L-ANA performs experience-based reinforcement learning in the same manner as profit-sharing. In profit-sharing, learning is performed in units of state transition series corresponding to episodes,
An episode and a learning value assigned to the episode are stored. For example, as shown in FIG. 2A, if the action subject A accidentally arrives at the charging point B by the state transition sequence of p, a reward 100 is first given to the state where the charging point B exists, and the episode p is added. An enhancement value based on an enhancement function is sequentially assigned to each of the constituent states. In this way, remembering one episode is pr
It is one learning in ofit-sharing.

[0018] In L-ANA, internal state A _si is subject to learning is changed by obtaining the reward, it satisfied the internal state A _si since episode when this is the initial state or the internal state A _si is satisfied It is a transition sequence of the state element until becomes. For example, if charging can be performed in a state where the internal state _Asi is established, the internal state _As1 is not established, and the internal state _As2 is established instead. In this case, the reinforcement learning for the internal state _As1 is performed.

On the other hand, in the L-ANA, as shown in FIG. 2B, one storage module is assigned to each state constituting an episode. Reinforcement learning performed when a reward is obtained is similar to profit-sharing. If a storage agent assigned to a state in which a reward is obtained is s _ij , “s _ij (t + 1) = s _ij (t) −bs _ij ( t) + b
The reinforcement value is updated according to "p (t), b: learning rate, p: reward". Profit-sharing assigns an episode an enhancement value based on an enhancement function that returns an enhancement value from the reward based on whether it is more past. As described above, profit-sharing assigns enhancement values only to the state series that make up an episode, but this is an effective means in situations where uncertain factors other than experience are eliminated and only experience can be trusted. However, given the situation in which the actor has a view and can collect local information, it is more flexible and robust for learning to assign reinforcement values at a smaller percentage than the reinforcement value assigned to episodes within the range visible to the actor. May be given. LA
As with profit-sharing, NA is a basic method that enhances experience, so it is necessary to make a difference between states other than episodes that have not been directly experienced.

Therefore, in the L-ANA, a storage module is allocated to a state other than an episode that is in view when the action subject A moves. For example, FIG.
If such an episode is obtained, the storage module to which the action subject A can assign is the part shown in FIG. When each storage module is assigned,
It remembers which storage agent you are adjacent to and whether you are making up an episode. Even if the episode is not composed when assigned, it may become part of the episode in a subsequent trial. Note that the storage agents located in the adjacent relationship are located at the top, bottom, left and right.

The method of using the learned reinforcement value is pr
Similar to ofit-sharing, the policy is to move to a state where the reinforcement value is higher.

When an enhancement value is propagated from an adjacent storage module, the storage module spontaneously propagates the enhancement value to another adjacent storage module. As shown in FIG. 3, after a certain attenuation is performed as a whole, an enhancement value R with a different attenuation rate is propagated to a storage module having experience in forming an episode and to other storage modules. The activation propagation ends when the propagation enhancement value falls below a certain threshold. Learning in L-ANA means that each storage module stores how much reinforcement value it propagates to which adjacent storage module.

The activity propagation is performed as follows: 1. the magnitude of the reinforcement value used when performing active propagation; The characteristics can be easily manipulated by the two parameters of the attenuation rate used when propagating, and the two learning characteristics can be selectively used as described below depending on how these two parameters are set.

(1) If it is desired to perform learning so that the learning effect is exhibited only when the action subject A comes near a state where a reward can be obtained, the reinforcement value is increased and the attenuation rate is increased.
That is, considering the peak of the reinforcement value with the rewarded state at the top, its altitude is high and steep. For example, consider a predator for a subject A in a state space S,
It is only necessary to escape from the predator when the predator approaches the vicinity of the subject A. To perform such learning, the setting of (1) is effective.

(2) Conversely, in order for the learning effect to be exhibited even when the action subject A is away from the state in which the reward is obtained, the reinforcement value is reduced and the attenuation rate is reduced. In other words, when considering the peak of the reinforcement value with the rewarded state at the top, the altitude is low and the gradient is gentle. For example, when considering learning to charge energy as in this case, it is better that the learning effect is widespread as in (2). The reason why the enhancement value must be set smaller than that of (1) is to prevent (1) from being completely included in the distribution of the enhancement value of (2).
This is because the learning effect of (1) must be exhibited when the predator approaches, even while moving to the charging point B according to the learning result of (2).

By performing activity propagation to a storage module other than an episode, learning with more flexibility and robustness can be performed. For example, FIG.
In profit-sharing as shown in Fig. 4 (b), there is no choice but to move randomly until one of the episodes is encountered.
As shown in (1), in the L-ANA, if the action subject A is already in the state of active propagation, it can be efficiently moved to the charging point B because it is drawn into the neighboring episode by the shortest route.

Since each storage module functions independently, even if the function of a certain storage module is impaired, activation propagation is performed in a state where the storage module is missing, and the function is impaired. A route that bypasses the part is automatically selected. In profit-sharing, missing one of the states that make up an episode affects the entire episode. Therefore, L-ANA has more robustness and is suitable as a learning method for an autonomous action subject operating in a dynamic environment such as the real world.

In profit-sharing, there is a problem in suppressing invalid rules at the time of strengthening value assignment. In L-ANA, invalid rules are also actively reused as a route to a state where a reward is obtained.

[0029] The study is carried out in A _si unit, own of the active propagation figure in A _si unit is learning Sarukoto. Therefore, for example, in a situation where certain _As1 and _As2 are both established, the action subject A may perform the action selection using a distribution map in which both the reinforcement value distributions are superimposed.

Also, considering a framework in which a plurality of actors A exist and cooperate with the actors A, it is easy to use the learning results of each other by sharing the reinforcement value distributions learned by different actors. Can be realized.

Although positive reinforcement learning has been described above, negative reinforcement learning can be easily realized in L-ANA by performing reverse activity propagation that absorbs the activity value. The action subject A always uses the learning result in a method of moving to a state where the reinforcement value is large. Here, a setting of a pit is added in the state space S. In this case, in order to prevent the pit from approaching the pit, it is sufficient to perform activity propagation that absorbs the activity value around the pit. And by superimposing the reinforcement value distribution for positive reinforcement learning for charging energy, both learning effects can be easily integrated,
The route to the charging point can be optimally selected while avoiding pitfalls.

FIGS. 5 and 6 show the results of comparative evaluation of profit-sharing and L-ANA using the state space S. FIG.

FIGS. 5 (a) and 5 (b) show the accuracy of the moving route learned in profit-sharing and L-ANA when a certain environment s is used. Specifically, it shows how shortly the autonomous action subject A was able to move to the charging point B, that is, the ratio of the route traveled by the autonomous action subject A to the calculated shortest route. For example, ten times indicates that the learned path surface is ten times the shortest path. Profit-shari shown in Fig. 5 (a)
As can be seen by comparing ng with L-ANA shown in FIG. 5 (b), L-ANA can always move along the route closest to the shortest route, but considerable variability is seen in profit-sharing. -It can be confirmed that ANA learns the shortest path more.

Further, in FIGS. 5A and 5B, 25
When a dynamic change occurs in the environment at the 0th step, that is, when an obstacle appears dynamically, profit-shari
It can be confirmed that the performance temporarily deteriorates with ng, but does not deteriorate with L-ANA. Accordingly, it was confirmed that L-ANA is more robust to dynamic changes in the environment than profit-sharing.

FIGS. 6A and 6B respectively show profit-s
FIG. 9 is a diagram showing where each state in the environment for haring and L-ANA is learned as a path to a charging point; In the profit-sharing shown in FIG. 6A, the charging point B is notwithstanding the charging point B1.
A situation arises where the route to 3 has been learned,
Although the efficiency is low, in the L-ANA shown in FIG. 6B, the route to the charging point B1 is learned near the charging point B1, and the travel distance to the charging point B1 and the charged energy supply amount are calculated. It is possible to confirm that the considered route has been learned.

Next, the operation of the experience-based reinforcement learning method using the action selection network according to one embodiment of the present invention will be described with reference to the flowcharts shown in FIGS. FIG. 7 is a flowchart showing an overall flow of L-ANA, specifically, a flow of learning regarding the internal condition _Asi . FIG. 8 is a flowchart showing an algorithm for activity propagation in step S21 in FIG.

Referring to FIG. 7, the overall flow of L-ANA will be described. The processing shown in the figure is an example of the reinforcement learning regarding the internal condition _Asi . First, it is checked whether or not the internal condition _Asi is satisfied (step S11). If the condition is not satisfied, it is not necessary to perform reinforcement learning, so that the robot randomly moves to a movable state and returns to the first step (step S13).

If the internal condition A _si is satisfied, the process moves to a state having a larger enhancement value. If there are a plurality of candidates, the candidates are randomly selected (step S1).
5). The enhancement value is set to 0 for all states as an initial value. The moved state is registered in the episode registration table (step S17). Then, it is checked whether or not a reward has been obtained (step S19). If the reward has not been obtained, the process returns to step S15, moves to a state having a larger reinforcement value, and repeats the same processing. In step S2, activity propagation is performed.
1). The active propagation will be described in detail with reference to the flowchart shown in FIG.

After the activation propagation, the episode registration table is initialized (step S23), the internal condition _Asi is not satisfied, and the process returns to the first step (step S25).

Next, the activation propagation shown in FIG. 8 will be described. In FIG. 8, when the activation propagation starts, the reinforcement value R is first given to the state _Asi that has obtained the reward, and the reinforcement value R is substituted into the activation propagation reference reinforcement value s (step S33,
S35). Then, one of the states A _sj adjacent to the state A _si
The following processing is performed for each one (step S37).

First, it is checked whether the state _Asj is registered in the episode registration table (step S).
39). If it has been registered, the enhancement value of the state A _sj is the reference enhancement value s and the attenuation rate α (the attenuation rate at the time of active propagation to the state constituting the episode, and 0 <α <1). It is checked whether or not it is smaller than the enhancement value which is the product (step S41). If the enhancement value of A _sj is not small, the process returns to step S37 and repeats the same process. If the enhancement value is small, that is, the activation value has already been propagated to the state where the enhancement value is to be propagated. Only when the value is smaller than the reinforcement value to be propagated this time, the reinforcement value is propagated again. Therefore, when the reinforcement value is small, the reinforcement value which is the product of the attenuation rate α and the reference reinforcement value s becomes It is checked whether or not the condition is smaller than the minimum reinforcement value (min) (step S43). That is, the minimum enhancement value to be propagated is set to min.

If the enhancement value to be propagated is smaller than the preset minimum value (min), the active propagation of this part is terminated, and the process returns to step S37 to perform active propagation in another state. If the fortification value is not smaller than the minimum value (min), the fortification value αs is given to the state A _sj (step S45).

Then, it is checked whether or not the state _Asj is registered in the activity propagation table (step S4).
7) If not registered, register (step S
49) If it is registered, if it is checked that the state A _sj has already been activated, the state is canceled (step S51).

Next, all the states A adjacent to the state A _si
Check whether or not _sj has been completed (step S
53). That is, activity propagation ends for all storage modules registered in the activity propagation table, and a series of activity propagation for the state _Asi ends. If all the states have not been completed, the process returns to step S37, and the active propagation is repeatedly performed for another state. However, if all the states have been completed, it is determined that the state _Asj has completed the active propagation. A check is made (step S55). It is checked whether or not any unchecked state remains in the active propagation table. If not, the process is terminated.
If remaining is regarded a state that is oldest registered among those not checked in the status registered in the active propagation table as the new state A _si repeats the following processing (step S59 ). That is,
It substitutes its own enhancement value for the active propagation reference enhancement value s, returns to step S37, and repeats the same processing (step S61).

On the other hand, in the check of step S39, when the state A _sj is not registered in the episode registration table, the attenuation factor strengthening value with reference reinforcing value s state A _sj beta (the state constituting the episode It is checked whether or not this is an attenuation rate at the time of active propagation and is smaller than an enhancement value which is a product of 0 <β <1 (step S63). A _sj
If the fortification value is not small, the process returns to step S37, and the same processing is repeated.
It is checked whether or not the strengthening value, which is the product of the attenuation rate β and the reference strengthening value s, is smaller than the minimum strengthening value (min) (step S65). If the enhancement value to be propagated becomes smaller than the preset minimum enhancement value (min), the active propagation of this portion is terminated, and step S
Returning to 37, the activation propagation of another state is performed. If the fortification value is not smaller than the minimum value (min), the fortification value βs is given to the state A _sj (step S67), and step S67 is performed.
Proceeding to 47, the above-described processing is performed.

[0046]

As described above, according to the present invention,
Active propagation is also performed on adjacent storage agents that are not directly related to the episode, enabling flexible learning.Also, efficient learning can be performed with the shortest path, and the conventional profit- It is more robust than sharing, and is most suitable for autonomous mobile robots and software agents operating in complex and dynamically changing environments such as the real world and the Internet, and controls the characteristics of activity propagation. By doing so, the characteristics of the learning can be easily manipulated, and it can be easily combined with the conventional real-time reactive planning or the like.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an autonomous action subject moving in a grid-like state space constituting an example (grid world) for explaining an experience-reinforcement type reinforcement learning method L-ANA using an action selection network according to the present invention. FIG.

FIG. 2 is an explanatory diagram showing the relationship between episodes and storage modules.

FIG. 3 is an explanatory diagram showing a method of propagating an activity.

FIG. 4 is an explanatory diagram showing the effect of activity propagation for conventional profit-sharing and L-ANA of the present invention.

FIG. 5: Conventional profit-sharing when a certain environment s is used
FIG. 7 is a diagram showing the accuracy of a moving route learned by the L-ANA of the present invention.

FIG. 6 shows conventional profit-sharing and L-AN of the present invention.
FIG. 6 is a diagram showing where each state in the environment of A is learned as a path to a charging point;

FIG. 7 is a flowchart showing an overall flow of L-ANA according to an embodiment of the present invention.

FIG. 8 is a flowchart showing an algorithm for activity propagation in step S21 of FIG. 7;

[Explanation of symbols]

 A Autonomous action subject B Charging point p Episode R Strengthening value S State space

Claims (2)

[Claims] An experience-enhanced reinforcement learning adapting a framework of an action selection network to profit-sharing so that an autonomous action subject operating in a complex and dynamically changing environment can effectively adapt to the change. A method, in which a storage agent, which is an autonomous subject, is assigned to each state constituting an episode, which is a transition sequence of state elements, and a storage agent is also assigned to states other than the episode in view when the internal state moves. When the reinforcement value from the neighboring memory agent is propagated, the reinforcement value is spontaneously propagated to the neighboring memory agent, and after attenuating as a whole, the decay rate of the reinforcement value is changed depending on whether or not the episode has been attenuated. Active propagation is realized as a cooperative operation of a group of storage agents, such as terminating active propagation when the enhancement value falls below a predetermined threshold. Experience enhanced reinforcement learning method using the action selection network characterized by Rukoto. 2. An experience-enhanced reinforcement learning adapting a framework of an action selection network to profit-sharing so that an autonomous action subject operating in a complex and dynamically changing environment can effectively adapt to the change. A storage medium on which a program is recorded, where a storage agent, which is an autonomous subject, is assigned to each state constituting an episode, which is a transition sequence of state elements, and a state other than an episode that is in view when the internal state moves. Also assigns a storage agent, and when the reinforcement value from the neighboring storage agent is propagated, the reinforcement value is spontaneously propagated to the neighboring storage agent, and when the whole is attenuated, it is strengthened based on the experience of episode attenuation Active propagation is a storage agent, such as changing the decay rate of the value and terminating the active propagation when the reinforcement value falls below a predetermined threshold. Recording medium recorded with experience enhanced reinforcement learning program using a behavior selection network, characterized in that it is implemented as a cooperative operation.