JP5150371B2 - Controller, control method and control program - Google Patents

Description

  The present invention relates to the configuration of a controller, a control method, and a control program for controlling a controlled object by a policy gradient method.

  The control problem formulated as a “Markov decision process” is an important technology that has a wide range of applications as an autonomous control problem for robots, plants, mobile machines (trains, cars, etc.).

  There is so-called “reinforcement learning” as a conventional technique related to optimal control for a Markov decision process.

  “Reinforcement learning” is a behavioral rule called “policy” that maximizes the cumulative amount of reward that is obtained through trial and error through interaction with the environment. It is a theoretical framework for learning. This learning method attracts attention from various fields in that it requires little a priori knowledge about the environment and the agent itself.

  Reinforcement learning can be roughly classified into two. The value function is used to express the policy indirectly, and the value function is updated to update the policy, and the value function is updated, and the policy is updated according to the objective function gradient. “Policy update method (policy gradient method)”.

  The policy gradient method is particularly attracting attention because it can acquire a stochastic policy by including a parameter for controlling the randomness of the action in the policy parameter, and has high applicability to a continuous system. However, in general, when applied to a real task, the time to acquire an appropriate behavioral rule may be unrealistic. Therefore, active research has been conducted to shorten the learning time by using auxiliary mechanisms such as simultaneous use of multiple learners, use of models, use of teaching signals, etc., and results have been remarkable.

Here, policy gradient reinforcement learning method (PGRL), by using the partial differential of the average compensation for measures parameters, reinforcement learning to maximize the average reward to improve measures parameters (RL: Reinforcem e nt Learning ) Is a general algorithm. Here, the partial differential of the average reward is called a policy gradient (PG).

  In other words, the policy gradient reinforcement learning method uses a time average value of reward obtained when an agent interacts with the environment as an objective function, and searches for a policy aiming at obtaining a policy (action law) that maximizes this objective function locally. This is achieved by sequentially updating the policy parameters with the gradient of the objective function.

  Even if the measures are appropriately parameterized, they can be implemented in the Markov Decision Process (MDP) without requiring knowledge about the agent or the environment. In addition, it is possible to acquire a probabilistic policy by including a parameter that controls the randomness of the action in the policy parameter. Even for the decision process (POMDP: Partially Observable MDP), the parameterized policy can be optimized within the range that can be expressed [Non-Patent Document 3] to [Non-Patent Document 7].

  On the other hand, depending on the problem, there is a problem that the time until obtaining a good policy becomes enormous. Therefore, research aimed at shortening the learning time by using auxiliary mechanisms such as the introduction of supplementary rewards [Non-patent Document 8], the use of models [Non-Patent Document 9] and the use of teaching signals [Non-Patent Document 10] has been conducted However, in general, a specific problem is assumed, and since prior knowledge about the problem is required, the versatility is poor. Therefore, it is desirable to improve the policy gradient algorithm that does not change the standard reinforcement learning framework, that is, does not depend on the task.

Hereinafter, prior art documents related to the policy gradient learning method to be cited in the text are listed below.
S. Amari: “Natural gradient works efficiently in learning”, Neural Computation, 10, 2, pp. 251-276 (1998). S. Kakade: “A natural policy gradient”, Advances in Neural Information Processing Systems, Vol. 14, MIT Press (2002). RJ Williams: “Simple statistical gradient-following algorithms for connectionist reinforcement learning”, Machine Learning, 8, pp. 229-256 (1992). DP Bertsekas and JN Tsitsiklis: “Neuro-Dynamic Programming”, Athena Scientific (1996). H. Kimura and S. Kobayashi: “An analysis of actor / critic algorithms using eligibility traces: Reinforcement learning with imperfect value function”, International Conference on Machine Learning, pp.278-286 (1998). J. Baxter and P. Bartlett: “Infinite-horizon policygradientestimation”, Journal of Artificial Intelligence Research, 15, pp. 319-350 (2001). J. Baxter, P. Bartlett and L. Weaver: “Experiments with infinite-horizon policy-gradient estimation”, Journal of Artificial Intelligence Research, 15, pp. 351-381 (2001). AY Ng, D. Harada and S. Russell: “Policy invariance under reward transformations: theory and application to reward shaping”, International Conference on Machine Learning, pp. 278-287 (1999). D. Bagnell, S. Kakade, A. Ng and J. Schneider: “Policy search by dynamic programming”, Advances of Neural Information Processing Systems (2004). MT Ronsenstein and AG Barto: “Supervised actor-critic reinforcement learning”, Learning and Approximate Dynamic Programming: Scaling Up to the Real World, John Wiley & Sons, Inc., pp. 359-380 (2004). KFS Amari: “Local minima and plateaus in hierarchical structures of multilayer perceptrons”, Neural Networks, 13, 3, pp. 317-327 (2000). S. Amari, A. Cichocki and HH Yang: “A new learning algorithm for blind signal separation”, Advances in Neural Information Processing Systems, Vol. 8, MIT Press (1996). D. MacKay: “Maximum likelihood and covariant algorithms for independent component analysis”, Technical report, University of Cambridge (1999). S. Amari, H. Park and K. Fukumizu: “Adaptive method of realizing natural gradient learning for multilayer perceptrons”, Neural Computation, 12, 6, pp. 1399-1409 (2000). J. Peters, S. Vijayakumar and S. Schaal: “Reinforcement learning for humanoid robotics”, IEEE-RAS International Conference on Humanoid Robots (2003). D. Bagnell and J. Schneider: “Covariant policy search”, Proceedings of the International Joint Conference on Artificial Intelligence (2003). J. Peters, S. Vijayakumar and S. Schaal: “Natural actor-critic”, European Conference on Machine Learning (2005). Y. Nakamura, T. Mori and S. Ishii: “Natural policy gradient reinforcement learning for a CPG control of a biped robot”, International conference on parallel problem solving from nature, pp. 972-981 (2004). T. Morimura, E. Uchibe and K. Doya: “Utilizing natural gradient in temporal difference reinforcement learning with eligibility traces”, International Symposium on Information Geometry and its Applications (2005). S. Richter, D. Aberdeen and J. Yu: “Natural actorcritic for road traffic optimisation”, Advances in Neural Information Processing Systems, MIT Press (2007). DP Bertsekas: “Dynamic Programming and Optimal Control, Volumes 1 and 2”, Athena Scientific (1995). RS Sutton and AG Barto: “Reinforcement Learning”, MIT Press (1998). RS Sutton, D. McAllester, S. Singh and Y. Mansour: “Policy gradient methods for reinforcement learning with function approximation”, Advances in Neural Information Processing Systems, Vol. 12, MIT Press (2000). S. Amari and H. Nagaoka: “Method of Information Geometry”, Oxford University Press (2000). R. Fletcher: “Practical Methods of Optimization”, Wiley (1987). T. Morimura, E. Uchibe, J. Yoshimoto and K. Doya: “Reinforcement learning with log stationary distribution gradient”, Technical report, Nara Institute of Science and Technology (2007). CM Bishop: “Neural Networks for Pattern Recognition”, Oxford University Press (1995). Fukumizu, Kuriki, Takeuchi, Akahira: “Statistics of Singular Models”, Iwanami Shoten M. Rattray and D. Saad: “Analysis of natural gradient descent for multilayer neural networks”, Physical Review E, 59, 4, pp. 4523-4532 (1999). SJ Bradtke and AG Barto.Linear least-squares algorithms for temporal difference learning.Machine Learning, 22 (1-3): 33-57, 1996. JA Boyan. Least-squares temporal difference learning.Machine Learning, 49 (2-3): 233-246, 2002.

  However, when considering the reason for slowing down the convergence to the optimal policy from the nature of the structure of the parameter space to be learned, it has not been clear how to improve this.

  So what makes learning time enormous? Of course, if the problem becomes complicated, the policy parameter search space will generally become enormous and learning will be difficult, but other than this, the differences in the sensitivity of each policy parameter to the MDP probability distribution and its correlation are ignored. There's a problem. In other words, the set of probability distributions that can be expressed in MDP with parameters as the coordinate system is not a Euclidean space, but a Riemannian manifold. It is a problem that learning is stagnant. It has been reported that even a very simple model such as two-state MDP actually falls into a serious plateau (learning stagnation period) [Non-Patent Document 2]. This is because there is a portion where the geometric structure of the objective function whose parameter is the coordinate system is locally flat, and the gradient is extremely small there [Non-Patent Document 11].

  As a solution to this problem, the “natural gradient method”, which is the steepest gradient method in Riemann space, is known [Non-Patent Document 1]. This is well studied in other machine learning fields [Non-patent document 1], [Non-patent document 12] to [Non-patent document 14], but theoretical research on reinforcement learning has not progressed much [ Patent Document 2], [Non-Patent Document 15] to [Non-Patent Document 17].

  In particular, since the gradient direction differs depending on which Riemann metric matrix is used in the natural gradient method, the design of the Riemann metric is an important issue.

  However, the application of natural gradient to reinforcement learning [Non-Patent Document 15] to [Non-Patent Document 20] all use the Riemann metric on the probability distribution of behavior proposed by Kakade [Non-Patent Document 2]. There was no discussion or suggestion about weighing.

  Therefore, in the present invention, a new Riemann metric is proposed for the simultaneous distribution of states and actions related to policy parameters, and the natural stationary policy gradient method (NSG: National Stationary policy Gradient) is a new natural policy gradient method based on the natural gradient of the average reward in the Riemann metric. ) Is proposed.

  Therefore, the present invention has been made to solve the above-described problems, and its object is to avoid the occurrence of a learning plateau and reduce the number of trials (learning time). It is to provide a controller, a control method, and a control program based on reinforcement learning using a natural gradient.

As will be apparent from the following explanation, the Riemann metric we propose is reasonable compared to Kakade's Riemann metric and the Hessian matrix of average rewards. Furthermore, the natural gradient we propose agrees with the linear combination coefficient when the immediate reward is approximated by least squares by the linear combination of basis functions defined by the policy parameters.

  In order to achieve such an object, the controller of the present invention observes a measure which is a control law for the system state when the time evolution of the target system is described as a Markov process. Is a controller that learns based on a policy, and is specified by a control signal generating means for generating a control signal for controlling the system, a state quantity detecting means for observing the state quantity of the system, and the state quantity Estimated results by natural steady policy gradient estimation means that estimates the natural steady policy gradient, which is the natural gradient of the average reward, using the Fisher information matrix of the simultaneous distribution of state and control signal values, and the natural steady policy gradient estimation means And a policy update means for updating the policy by updating the policy parameter that defines the policy based on the above.

  Preferably, the natural steady policy gradient estimation unit is based on a reward value acquisition unit that acquires a reward value that depends on a predetermined relationship between the state and the control signal, and the state quantity and the control signal at each time step. Estimating the partial derivative of the logarithm of the steady distribution and regressing the reward value with a linear function approximator that defines the basis function between the state specified by the estimated partial derivative and the control signal, the natural steady policy gradient is Estimating means for estimating.

  According to another aspect of the present invention, there is provided a control method for learning a policy, which is a control law for a system state, by observing a state quantity of the system when the time evolution of the target system is described as a Markov process. A control signal generation step for generating a control signal for controlling the system based on the strategy, a state quantity detection step for observing the state quantity of the system, and a simultaneous distribution of the state specified by the state quantity and the control signal value The policy is defined based on the natural steady policy gradient estimation step that estimates the natural steady policy gradient, which is the natural gradient of the average reward, and the estimation result from the natural steady policy gradient estimation step, using the Riemann metric matrix as the Fisher information matrix A policy update step of updating the policy by updating the policy parameter to be updated.

  According to still another aspect of the present invention, when a time evolution of a target system is described as a Markov process, a control method for learning a policy that is a control law for the state of the system by observing a state quantity of the system A control signal generation step for generating a control signal for controlling the system, a state amount detection step for observing the state amount of the system, and a state amount. Estimating the natural steady policy gradient estimation step to estimate the natural steady policy gradient, which is the natural gradient of the average reward, and the natural steady policy gradient estimation step, using the Fisher information matrix of simultaneous distribution of control state and control signal value as the Riemann metric matrix Based on the results, update the policy by updating the policy parameters that define the policy. And a step to execute the control process computer.

(Outline of the contents of the present invention)
In general, a parameter space of a statistical model or machine learning has a property as a Riemann space instead of Euclidean with respect to changes in its output, and its steepest gradient direction does not always coincide with the partial differential of the output which is a conventional gradient.

  Amari [Non-Patent Document 1] proposes a natural gradient method for this problem, and Kakade [Non-Patent Document 2] applies a natural gradient to the policy gradient reinforcement learning method, which is one of the optimization methods of the Markov decision process. did.

  Since the natural gradient direction is determined based on the Riemann metric that defines the Riemann structure, its selection is an important issue. However, the Riemann metric matrix used by Kakade is a metric matrix that takes into account only the change in the probability distribution of the action due to the parameter perturbation of the policy, and similarly ignores the change in the probability distribution of the state that should be affected by the policy. Therefore, the present invention proposes a new Riemannian metric that also considers the probability distribution of states, and derives a new natural policy gradient and a natural steady policy gradient based on the metric. Furthermore, it is proved that this natural steady policy gradient matches the parameter learned when the immediate reward is approximated by a linear function approximator having a basis function defined by the policy parameter. In addition, numerical experiments applied to Markov decision problems with various numbers of states show that the proposed method works more effectively than the conventional method, especially when the number of states is large.

  The outline of the composition of the following explanation is explained. In (1. Outline of the present invention), the overall structure of the present invention is explained, and the policy gradient method which is the gradient method in reinforcement learning is explained in Section 2.1. To do. Section 2.2 explains the natural gradient method proposed by Amari [Non-Patent Document 1], introduces the natural policy gradient method that is applied to reinforcement learning, and discusses the question in Section 2.3. Point out.

(1. Overview of the present invention)
As will be described later, in the present invention, the “natural steady policy gradient” is derived, the “policy” is updated, and the control target is controlled. Here, in the derivation of the natural steady policy gradient, as will be described later, an LSDG (logarithmic partial derivative of the steady distribution) is used as a gradient of the steady distribution by a backward Markov chain method and a TD (temporal difference) learning algorithm. -LSDG ( LogStationary Distribution Gradients)) is derived and can be used.

Embodiments of the present invention will be described below with reference to the drawings.
As will be apparent from the following description, the present invention has a wide range of applications as control problems for robots, plants, mobile machines (trains, automobiles), and the like.

  However, in the following, as a specific application example of the present invention, a description will be given on the assumption that a particularly simple automatic robot control problem is targeted. The numerical calculation results show a comparison with a simpler model. However, the present invention is not limited to such an application, and more generally can be applied to control of the target system when the time development of the target system is complicated. Examples of such are giant plants (blast furnaces, nuclear power plants), multi-link robots (humanoid robots), non-holonomic systems (space stations), and the flow of people in subway platforms. All of these are difficult to control by the classical control method and are important control objects.

( 2. System configuration of the present invention)
FIG. 1 is a conceptual diagram showing an example of a system 1000 using a controller to which a control method and a control program of the present invention are applied.

  Referring to FIG. 1, system 1000 includes controlled device 200 to be controlled and a computer 100 for giving a control signal to controlled device 200.

  Referring to FIG. 1, this computer 100 reads / writes information from / to a CD-ROM drive 108 and a flexible disk (FD) 116 for reading information on a CD-ROM (Compact Disc Read-Only Memory). A computer main body 102 having the FD drive 106, a display 104 as a display device connected to the computer main body 102, and a keyboard 110 and a mouse 112 as input devices also connected to the computer main body 102.

FIG. 2 is a block diagram showing the configuration of the computer 100. As shown in FIG.
As shown in FIG. 2, in addition to the CD-ROM drive 108 and the FD drive 106, the computer main body 102 constituting the computer 100 includes a CPU (Central Processing Unit) 120 connected to the bus BS, and a ROM ( It includes a memory 122 including a read only memory (RAM) and a random access memory (RAM), a direct access memory device, for example, a hard disk 124, and a communication interface 128 for exchanging data with the controlled device 200. A CD-ROM 118 is attached to the CD-ROM drive 108. An FD 116 is attached to the FD drive 106.

  From the controlled device 200, information on parameters (state quantities) indicating the state of the controlled device 200 to the computer 100, for example, information on the position, velocity, acceleration, angle, angular velocity, etc. of the movable part of the controlled device 200 Is given. On the other hand, from the computer 100, control information for controlling these state quantities is given to the controlled device 200 as a control signal (corresponding to “behavior” described below).

  The CD-ROM 118 may be another medium, such as a DVD-ROM (Digital Versatile Disc) or a memory card, as long as it can record information such as a program installed in the computer main body. In this case, the computer main body 102 is provided with a drive device that can read these media.

  The main part of the controller of the present invention is composed of computer hardware and software executed by the CPU 120. Generally, such software is stored and distributed in a storage medium such as a CD-ROM 118 or FD 116, read from the storage medium by the CD-ROM drive 108 or FD drive 106, and temporarily stored in the hard disk 124. Alternatively, when the device is connected to the network, it is temporarily copied from the server on the network to the hard disk 124. Then, the data is further read from the hard disk 124 to the RAM in the memory 122 and executed by the CPU 120. In the case of network connection, the program may be directly loaded into the RAM and executed without being stored in the hard disk 124.

  The computer hardware itself and its operating principle shown in FIGS. 1 and 2 are general. Therefore, the most essential part of the present invention is software stored in a storage medium such as the FD 116, the CD-ROM 118, and the hard disk 124.

  As a general tendency, various program modules are prepared as a part of a computer operating system, and an application program generally calls a module in a predetermined arrangement and advances the processing when necessary. In such a case, the software itself for realizing the controller does not include such a module, and the controller is realized only in cooperation with the operating system on the computer. However, as long as a general platform is used, it is not necessary to distribute software including such modules, and the software itself not including these modules and the recording medium storing the software (and the software distributes on the network). Data signal) can be considered to constitute the embodiment.

[General description of control method]
Hereinafter, the theoretical configuration of the configuration of the present invention will be described. Note that in the formulas, the fine characters are scalars, and the variables represented by bold characters represent vectors (or matrices). However, it should be noted in advance that the text does not distinguish between bold and thin.
(2.1 Policy gradient reinforcement learning)
As a reinforcement learning problem, a discrete-time Markov decision process (MDP) defined on a finite state set S∋s and a finite action set A∋a is considered [Non-patent document 21], [Non-patent document 22].

At each time step t 1, the agent selects an action a _t according to the following probabilistic policy defined by the policy parameter θ∈R ^d .

After that, the transition to the new state s _{t + 1} according to the following state transition probability, and the immediate reward r _{t + 1} determined by the bounded predetermined reward function r (s _t , a _t , s _{t + 1} ) To win. Here, it is assumed that the policy π (a | s; θ) is always smooth with respect to the parameter θ, and the following Markov chain defined by the state transition probability and the policy satisfies the ergodic property.

Therefore, if the policy parameter is defined, there is only one (state) steady distribution d (subscript θ) that does not depend on the initial state s ₀ as shown below, and further, as defined below, Matches:

The purpose of reinforcement learning (policy search) is to identify a policy parameter θ ^* that maximizes the average reward represented by the following equation (3), which is the time average of the immediate reward.

Here E {...} represents the expected value, the right side from the assumption of ergodicity is noted that a value which does not depend on the initial state s _0. Moreover, since the time average and the space average coincide (formula (2)), the average reward can be rewritten as the following formula (4) [Non-patent document 21]

  Therefore, the slope of the average reward related to the policy parameter can be expressed as the following equation (5).

It becomes. Where a ^T represents the transpose of a vector a.
Therefore, if the policy parameter is updated from the following equation, the average reward R (θ) that is the objective function increases:

Here, the symbol connecting the left side and the right side in the above expression represents an operator that substitutes the value on the right side for the variable on the left side, and η is called a learning rate and is a sufficiently small constant. The above framework is generally called a policy gradient (reinforcement learning) method [Non-patent document 6], [Non-patent document 23].
(2.2 Natural gradient [Non-Patent Document 1])
A certain parameter space is a Riemann space means that the parameter θ∈R ^d is on the Riemannian manifold defined by the Riemann metric matrix G (θ) ^{∈R dxd} (positive definite matrix). That is, it means that the square distance between the parameter θ and the vector (θ + dθ) that can be obtained by slightly changing θ is defined by the following equation.

Here, g _{i, j} is the (i, j) -th element of the matrix G.
The direction of dθ that maximizes the function R (θ) under the constraint of the movement distance | dθ | = ε, where ε is a sufficiently small positive constant, that is, the steepest gradient direction in the Riemann space is expressed by the following equation (6). It is called “natural gradient”.

  In particular, in reinforcement learning, that is, when θ is a policy parameter and R (θ) is an average reward, it is called a natural policy gradient [Non-Patent Document 2].

  In the natural (policy) gradient method, the objective function is (locally) maximized by sequentially updating parameters according to the following equation (7).

  When considering a statistical model Pr (x | θ) defined by a parameter θ with a certain probability of occurrence of a variable x, generally, as a Riemannian metric matrix G (θ), a Fisher information matrix Fx ( θ) is often used:

  This is because the Fisher information matrix is associated with a local approximation of Kullback-Leibler (KL) divergence representing the pseudorange between probability distributions. That is, the KL divergence of the two probability distributions Pr (x | θ) and Pr (x | θ + Δθ) can be expressed as follows [Non-patent Document 24].

(2.3 Questions about the natural policy gradient method)
The policy gradient reinforcement learning method optimizes the policy parameter θ on the statistical model defined by the stochastic policy π (a | s; θ) and the state transition probability P (s' | s, a). Since the Riemannian metric matrix G (θ) is designed based on the Fisher information matrix of the statistical model, the natural policy gradient is naturally derived from the equation (6). The natural policy gradient method is not a parameter space of a policy that is arbitrarily parameterized, but a gradient method in Riemann space defined by the Riemann metric matrix G (θ). Therefore, if the Riemann metric in the reinforcement learning problem can be designed, the natural policy Gradient is a very effective technique.

However, as pointed out by Kakade [Non-Patent Document 2], the Riemannian metric in reinforcement learning is not necessarily limited to one, and various ones are considered, and the natural policy gradient direction is clearly used from Equation (6). Depends on the Riemann metric matrix G (θ).

Therefore, the question of what is the appropriate Riemannian metric should be discussed. However, the research on the natural policy gradient [Non-Patent Document 15] to [Non-Patent Document 20] does not touch on this point, and the Riemannian metric matrix for the Fisher information of action distribution proposed by Kakade [Non-Patent Document 2]. Was used. In other words, there has not been much discussion about what statistical models can be considered for policy gradient reinforcement learning methods and which Riemannian metrics are effective in solving reinforcement learning problems. Therefore, the present invention discusses statistical models and Riemann metrics in reinforcement learning, and proposes new Riemann metrics.
(3. Riemannian metric matrix for policy gradients)
3． In section 1, we propose a new Riemann metric matrix for reinforcement learning. 2 and 3. Section 3 compares this Riemann metric with the Riemann metric proposed by Kakade [Non-Patent Document 2] and the Hessian matrix of average reward, and discusses its validity.
(3.1 Proposed Riemannian Metric Matrix and Proposed Natural Steady Policy Gradient for State Behavior)
Since the function that can be directly adjusted by reinforcement learning is the policy π (a | s; θ), the research of the natural policy gradient so far is only for the policy function, that is, the statistical model Pr (a | s, M (θ)). It was something that was noticed. However, in reality, if the policy changes, the state distribution Pr (s | M (θ)) also changes. Then, from equation (4), the average reward R (θ) is defined by the simultaneous distribution of state behavior (s, a) ∈S × A, so attention is paid to (local) maximization of the average reward, which is the purpose of reinforcement learning. It is considered that Pr (s, a | M (θ)) is a reasonable statistical model to be used.

Because, if the Fisher information matrix of Pr (s, a | M (θ)) is used as the Riemannian metric matrix, the natural policy gradient is the following KL divergence D _KL in the state behavior simultaneous distribution with respect to the policy parameter θ This is because it coincides with the steepest gradient direction of the average reward under the constraint that the minute change is constant.

  The Fisher information matrix Fs, a (θ) corresponding to this statistical model is obtained as shown in the following equation (9) using equation (8):

  And the natural stationary policy gradient (NSG: National Stationarypolicy Gradient), which is the proposed natural policy gradient with the Riemannian metric matrix G (θ) as the Fisher information matrix Fs, a (θ) by the simultaneous distribution of state behaviors, is obtained from Equation (6). Obtained by the following formula:

(3.2 Comparison with the conventional Riemann metric matrix)
The only Riemann metric matrix that has been used for natural policy gradients so far is a matrix proposed by Kakade [Non-Patent Document 2] ad hoc, and the policy Fisher information matrix Fa (s, θ) is weighted with a steady distribution. The following equation (12) was added.

  This is the second term of Formula (9) of Fs, a (θ). If it is assumed that the steady distribution does not change due to the change of the policy parameter, the following relationship is established in the equation (10).

  However, the assumption of the above equation is generally not valid. In other words, Kakade's Riemannian metric matrix ignores the change in the steady distribution expressed by the following equation of the state due to perturbation of the policy parameter θ in the statistical model Pr (s, a | M (θ)) of the simultaneous distribution of state behavior It can be seen that this is the Riemann metric matrix.

  Meanwhile, Bagnell et al. [Non-patent document 16] and Peters et al. [Non-Patent Document 15] and others showed the following contents.

  And Bagnell et al. [Non-Patent Document 16] and Peters et al. [Non-Patent Document 15] et al., Kakade's Riemannian metric matrix becomes a valid Riemannian metric matrix because the reinforcement learning problem, that is, the average reward maximization problem, ultimately results in optimization of the system trajectory from Equation (3). Insist on getting.

  However, the following can be said.

  This is because the statistical model of the system trajectory takes into account the time evolution of the system, but the statistical model of state behavior is not considered, and will be clarified below.

Which Fisher information matrix is suitable for maximizing average reward, which is a reinforcement learning problem? As mentioned earlier, the average reward (equation (4)) is determined by the simultaneous distribution of state actions (system trajectory of 1 hour step) and does not depend on the system progress after 2 hour steps. Therefore, Kakade's Fisher information matrix is A redundant statistical model is assumed, and the Fisher information matrix F _{s, a} (θ) of the simultaneous distribution of state actions is considered to be the most natural Riemannian metric. The numerical comparison experiment of natural gradient using unit matrix I, Fisher information matrix F _{s, a} (θ), Kakade's Riemann metric matrix, etc. is Riemannian metric. Shown in section.

  Furthermore, the following can be said.

(3.3 Similarity between Fisher information matrix and Hessian matrix)
The Fisher information matrix F _{s, a} (θ) and Kakade's Riemann metric matrix are compared with the Hessian matrix, which is a second-order partial derivative with respect to the policy parameter θ of the average reward.

  Comparing Kakade's (Normalized Fisher Information Matrix in Infinite-Time System Trajectory) Riemann Metric Matrix (12) with Hessian H (θ) (16), as Kakade claims [Non-patent [2] Kakade's Riemannian metric matrix holds no information except for the first two terms in curly braces {...} in equation (16) of H (θ).

On the other hand, from equations (9) and (15), the Fisher information matrix F _{s, a} (θ) in the state action clearly holds some information regarding all the terms of H (θ). Therefore, it is suggested that the proposed Fisher information matrix F _{s, a} (θ) can be a valid Riemannian metric in comparison with the Hessian matrix.

It should also be noted that the average reward is not generally quadratic with respect to the policy parameter θ, and especially when θ is far from the optimal parameter θ ^* , the Hessian matrix tends to be an indefinite value matrix. On the other hand, since the Fisher information matrix is not an indefinite value matrix and its semi-definiteness is guaranteed by its definition (equation (8)), the natural gradient method uses the Hessian matrix H (θ) especially in the reinforcement learning problem. Newton-Raphson method, etc. [Non-Patent Document 25] This is considered to be a more general-purpose covariant gradient method. These numerical comparison experiments are described in 5. Shown in section.

If a special reward function that depends on θ is used, F _{s, a} (θ) and the Hessian match (Appendix 2).
(4. Natural stationary policy gradient based on Fisher information matrix F _{s, a} (θ))
Consider an estimation of a natural steady policy gradient, which is a natural gradient of an average reward _{, using a} Fisher information matrix F _{s, a} (θ) of _a state and action simultaneous distribution Pr (s, a | M (θ)) as _a Riemann metric matrix. We show that this estimation results in a regression problem of the reward function r (s, a, s').

  At this time, the following [Theorem 1] holds.

  In view of the above, consider a linear function approximator using a function represented by the following equation as a basis function.

If the immediate reward r _{t + 1} is approximated by least squares with this linear function approximator, it can be seen that the learning parameter ω becomes an unbiased estimation amount of the natural steady policy gradient expressed by the following equation.

  Thus, natural steady-state policy gradient estimation results in a reward function regression problem, so simply by the least-squares method or Morimura et al. By extending the method of [Non-Patent Document 19], the natural steady policy gradient can be estimated even by the gradient method that does not explicitly require the Fisher information matrix.

  In actual implementation, the following points should be noted.

Note that the mounting method described in Non-Patent Document 26 is well-known in this document, so only a summary will be given later. The content of this non-patent document 26 has been filed as Japanese Patent Application No. 2008-077771 by the same patent applicant as the present case.
(5. Numerical experiments)
In this section, the natural steady policy gradient method and the conventional gradient method are applied to the Markov decision process for various arbitrary states.
(5.1 Comparison of weighing)
FIG. 3 is a diagram illustrating the difference on the phase plane of the Riemann metric matrix.

3, (i) shows the case of the proposed Fisher information matrix F _{s, a} (θ), (ii) shows the Kakade Riemannian metric matrix, and (iii) shows the unit matrix I Yes.

  Here, each state s∈ {1, 2} has a mutual transition action A∈ {1, m}, and each state transition is a deterministic two-state Markov decision process (MDP). .

  The policy π is a sigmoid function as in the Markov decision process described later with reference to FIG.

In the figure, the change in the gray scale of the background is the ratio of the steady state distribution of the state 1 and the state 2 expressed in the log scale, and each ellipse has a constant distance Δθ ^T G (θ) Δθ = ε ² . Corresponding to a satisfactory set of Δθ, the natural steady policy gradient indicates the steepest average gradient.

In the proposed method, the direction of the short axis of the ellipse is along the direction of change of the steady distribution, and it can be seen that the change of the steady distribution due to the perturbation of the policy parameter θ can be handled well.
(5. 2 Comparison of learning)
(5.2.1 Experiment settings)
The Markov decision process (MDP) with the number of states | S | ∈ {3, 10, 20, 35, 50, 65, 80, 100} and the number of actions | A | = 2 was set according to the following procedure.

  Regarding the state transition probability p (s' | s, a), the ergodic property of the Markov chain M (θ) assumed in theory is not broken, and it corresponds to a general reinforcement learning task [Non-patent document 22]. Thus, it is designed so that the coupling of each state becomes coarse, in other words, the mixing time of M (θ) increases as the number of states increases.

  A specific configuration method is as follows.

For the reward function r (s, a, s'), the return value of each combination of arguments is temporarily determined according to the standard normal distribution N (μ = 0, σ ² = 1), and the average reward for each MDP setting To equalize the size of the reward function, the reward function was normalized by the following formula so that the maximum value max R (θ) = 1 and the minimum value minR (θ) = 0:

  The policy π (a | s; θ) was set as follows. That is,

(5. 2.2 Setting the gradient method)
A total of four policy gradient methods, the proposed method and three conventional methods, were applied to the above MDP. These differences are related to the Riemann metric matrix G (θ) (formula (6)) used for obtaining the gradient, and these four types of policy gradient methods are as follows.

  This Hessian adjustment method is a general adjustment method that interpolates the partial differential direction and the Newton direction except that the degree of adjustment changes according to λmax [Non-patent Document 27].

  When the degree of adjustment is large, the gradient direction remains the same as the direction of partial differentiation below, so in this experiment, the degree of adjustment can be changed so that the minimum correction is necessary to avoid losing the Hessian matrix as much as possible. It was.

  In implementing each gradient method, the present invention does not discuss the data sampling problem for gradient estimation, and focuses only on the gradient direction, so each gradient is obtained analytically. Of the required values, the derivation of the steady distribution and its partial derivative is not simple. It was shown to. The total number of learning steps T is 300, and the learning rate η of each gradient method in policy parameter update (equation (7)) is determined by the following equation.

If the learning rate η is appropriate, R (θ) always increases with the policy update according to equation (7). Therefore, when the policy update reduces R (θ), the learning rate is adjusted to “η: = η / 2” and the update is attempted again at the same time step. Such adjustment was maintained in a range where ΔR (θ)> = 0. On the other hand, when η ₀ > η holds even in the next step, adjustment is made to “η: = 2η” to prevent learning from stopping.

(5.3 Experimental results and discussion)
FIG. 2 . It is a figure which shows the learning curve of 10 episodes from which each initial setting value differs about MDP of the state number | S | = 100 set according to 1 clause. However, similar results are obtained with other MDP settings.

FIG. 4 shows that the proposed natural stationary policy gradient method succeeded in optimizing the policy parameters uniformly compared with other gradient methods. For example, if each gradient method is compared with respect to a certain episode, it can be confirmed that only the proposed method has been successfully learned and the other gradient methods have fallen into a learning plateau (stagnation). However, there are cases where learning is successful in all gradient methods compared to a specific episode, and learning and the average reward rise are the slowest in the proposed method. The following reasons can be considered. The average reward is determined by the linear combination sum obtained by weighting the reward function with the state behavior simultaneous distribution Pr (s, a | M (θ)) from Equation (4), so Pr (s, a | M (θ)) is generally If it changes greatly, its value will also change greatly. And since the proposed gradient method uses the Fisher information matrix F _{s, a} (θ) of Pr (s, a | M (θ)) as _a Riemannian metric, the policy parameter can be set once with a sufficiently small learning rate η. The KL divergence due to the update Δθ is constant, and Pr (s, a | M (θ)) does not change significantly, so the average reward R (θ) does not change abruptly.

  On the other hand, other gradient methods use other Riemann metrics, so Pr (s, a | M (θ)) can change rapidly, so it is possible to increase the average reward quickly and quickly. is there. However, it seems that other learning methods easily fall into the learning plateau, and the results shown in Fig. 5 are obtained. One of the specific causes is that if Pr (s, a | M (θ)) changes abruptly, the partial differentiation (Equation (5)) is also very easy to change, and it can be very close to zero. It can be considered that the geometric structure of the objective function R (θ) with respect to the parameter space in the gradient method tends to be flat.

  Fig. 4 shows the results of Amari et al. This is consistent with the results of [Non-Patent Document 14].

  FIG. 5 is a diagram showing a learning success rate (Success Rate) by 300 episodes of each gradient method in the MDP of each state number | S | ∈ {3, 10, 20, 35, 50, 65, 80, 100}.

Since the maximum value of the average reward R (θ) was set to 1, learning was successful if R (θ _T )> 0.95.

  In the case of MDP with a small number of states, it can be seen that the proposed natural gradient method and Kakade's natural gradient method did not fall into a plateau compared to other gradient methods, and learned appropriately. This is due to the fact that the Riemann metric matrix used in Kakade's natural gradient method is a metric related to the Fisher information matrix of the statistical model Pr (s, a | M (θ)). .

  On the other hand, when the number of states is large, Kakade's natural gradient method fails to learn compared to the proposed natural gradient method. This is 3. As discussed theoretically in Section 2, the Kakade method is thought to be because the amount of Fisher information Fs (θ) related to the state distribution is ignored.

  Furthermore, it can be confirmed from the figure that the Kakade method is inferior to the pseudo Newton gradient method when the number of states is large. This is because the Kakade method does not capture any information about Fs (θ), while the pseudo-Newton gradient method is not in the form of Fs (θ), but it also has information about the differentiation of the steady distribution due to θ. ) Is equal to-(pseudo-Hesse matrix) "(Equation (16)).

FIG. 6 is a diagram showing how much each gradient method falls into the learning plateau. In FIG. 6, the vertical axis represents the integration of absolute values of the smoothness (discrete curvature) Δ ² R (θ _t ) of the learning curve as a plateau index.

  FIG. 6 confirms that the proposed natural gradient method learned the smoothest in any number of states, and suggests that the proposed method is the most difficult method to fall into a plateau consistent with the results so far. The

From the above numerical experiments, the proposed natural gradient method does not depend on the MDP environment setting (p, r, ψ) and the initial policy parameter value θ ₀ , and in particular, there is a strong correlation between the steady distribution and the policy parameter. It was confirmed that it was possible to learn appropriately without relying on the learning plateau. Therefore, this gradient method is considered to be a natural policy gradient method that is more natural than the natural policy gradient method proposed by Kakade.

  Appendix 1. As a result, the derivation of the equation (14) can be performed as follows.

  In addition, Appendix 2. As such, the coincidence between the Fisher information matrix and the Hessian matrix is shown as follows.

  In addition, Appendix 3. As a result, the derivation of the steady distribution and its partial derivative can be performed as follows.

  From equation (A-2), the steady distribution vector is obtained as follows.

  Similarly, the second order partial differential is obtained by the following equation.

FIG. 7 is a conceptual diagram showing an example of the configuration of the controller of the present invention.
In the example of FIG. 7, the controller performs an action, that is, a process of giving a control signal to the control target, and the state quantity of the control target is observed (for example, a position sensor, an angle sensor, an acceleration sensor, an angular acceleration sensor, etc. ) And estimate the "logarithm partial derivative of the steady distribution" (LSDG) from this observation result, and use this to estimate the "natural steady policy gradient", update the policy parameters, and Update. Then, a control signal (corresponding to the action in the above description) is generated by the updated policy, and the control target is further controlled.
( 6. Overview of implementation of derivation of gradient of steady distribution shown in Non-Patent Document 26)
The implementation of the derivation of the gradient of the policy and the gradient of the steady distribution is detailed in Non-Patent Document 26, but the outline of the implementation method will be summarized below for confirmation.

  Some notations are newly redefined in the following description.

In the following, the Markov decision process (MDP) will be considered, and the controlled object (controller environment) is in a state transition probability (execution (control) u _t when it is in the state x _t at time t). The same applies to the case where the probability is x _{t + 1} ) and the reward function r _{t + 1} = r (x _t , u _t ) (r _{t + 1} = r (x _{t + 1} , x _t , u _t )) Can be discussed). In addition, about this reward function, it shall be predetermined according to the control target of control object. The mapping from the state input xεX to the action output uεU is called a policy and is expressed stochastically as described below. The strategy is defined by the parameter θ.

A discrete time MDP having a finite set of state X∋x and action U∋u is defined by the following state transition probability p and reward function r _{+ 1} .

Here, for simplicity of description, x _{+ 1} is the next state given by action u in state x, and r _{+ 1} is the immediate reward observed in x _{+ 1} . x _{+ k} and u _{+ k} are respectively a state and an action which are k steps from the state x, and vice versa if the subscript is −k. The decision (in MDP) is made according to the following probabilistic policy π parameterized by θεR ^d .

  The following strategy π is assumed to be differentiable with respect to θ for all x∈X and u∈U.

Furthermore, the following assumptions are made.
(Assumption 1)
A Markov chain M (θ) having a state transition probability p and a stochastic policy π as follows is ergodic (irreducible and aperiodic) for all policy parameters.

  Thus, there is only one steady distribution:

  This steady distribution is independent of the initial state and satisfies the following equation.

  Here, the following equation holds.

The purpose of PGR L is to find a policy parameter θ ^* that maximizes the average of immediate rewards, referred to below as “average reward”.

  Under Assumption 1, the average reward is shown to be independent of the initial state x and equal to:

  The policy gradient RL algorithm updates the policy parameter θ in the direction of the average reward R (θ) gradient for θ shown in the following equation.

  Hereinafter, it is often referred to simply as policy gradient (PG). This policy gradient is given as follows.

  Deriving the partial derivative of the logarithm of the steady distribution shown in the following equation is not easy.

  Therefore, the conventional PG algorithm uses another representation of PG.

  Here, the action value function Q and the state value function V are respectively expressed as follows at a discount rate γ∈ [0, 1).

Since the contribution of the second term of Equation ( 23 ) decreases as γ approaches 1, the conventional algorithm approximates PG from only the first term by setting γ˜1. Such bias due to omission becomes smaller when the discount rate γ approaches 1, but the variance of estimation increases.

Here, another approach is proposed. There, the partial differential (LSDG) of the logarithm of the steady distribution of the following formula is estimated, and formula ( 22 ) is used to derive PG.

A striking feature is that no value function needs to be learned, and therefore the algorithm has nothing to do with the trade-off between bias and variance in the selection of discount rate γ.
( 6.1 Estimation of partial derivative of logarithm of stationary distribution)
In the following, we propose an LSDG estimation algorithm, LSDG (λ), based on the least square method. For this purpose, we formulate the inverse process of the Ergoto-like Markov chain M (θ) and show that LSDG can be estimated in the framework of the TD method.
( 6.1.1 Properties of forward and reverse Markov chains)
Using Bayesian theory, the backward probability from the current state to the past state and action pair is expressed by the following equation.

  The following posterior probability q depends on the prior distribution p.

  As will be described below, when the prior distribution p follows the steady distribution d and the policy π, the posterior distribution q is called a steady reverse probability, and a subscript B (θ) is added.

The Markov chain—both M (θ) and B (θ) —is closely related as described in the following two theorems.
(Theorem 1)

(Proof)
The following steady distribution is applied to both sides of the equation ( 24 ).

Then, summing over all possible actions u ₋₁ ∈U, we get:

Then, when the sum is taken for the possible state xεX on both sides of the equation ( 26 ), the following equation is established.

  This establishes the following two points. (I) B (θ) has the same steady distribution as M (θ), and (ii) B (θ) has the same irreducible property as M (θ).

This suggests that (iii) B (θ) has the same aperiodic nature as M (θ). Equation ( 25 ) is directly proved by (i)-(iii).
(Theorem 2)

(Proof)
By substituting the properties of the Markov chain and the formula ( 24 ), the following relationship is obtained.

This proves that the equation ( 27 ) holds in the case of finite K. Since the following equation is derived from Theorem 1, it is immediately proved that the K → ∞ limit of equation ( 27 ) holds.

Theorem 1 and Theorem 2 indicate that samples from the forward Markov chain M (θ) can be used for estimation on the backward Markov chain B (θ) as they are under a state distribution that converges to a steady distribution. So it ’s important. These can be used later.
( 6.1.2 TD (Temporal Difference) learning method for LSDG of Markov chain from reverse direction to forward direction)
LSDG is decomposed as follows using equation ( 24 ).

Equation ( 29 ) implies that the LSDG of state x is an accumulation of infinite intervals of the backward Markov chain B (θ) from the logarithmic partial differential state x of the policy represented by the following equation: .

From equations ( 28 ) and ( 29 ), LSDG can be estimated by TD learning for reverse TD δ as follows for reverse Markov chain B (θ) rather than M (θ).

  Here, the first two terms are the actual observation value one step before the logarithmic partial differentiation of the policy at B (θ) and the LSDG one step before, and the LSDG in the current state is the last term. .

Using the eligibility decay rate λε [0,1] and the backward tracking time step KεN, equation ( 29 ) is generalized as follows:

  Even if the setting is not as described above, if a large λ or K is used, such minimization is more effective for the non-Markov effect as in the case of TD (λ) learning for the conventional value function. Not sensitive.

To fill the gap between theoretical assumptions and real-world applications, one of the following two strategies needs to be taken. (I) If λ˜1, K is not set to a very large integer, (ii ) If K˜t, λ is not set to 1, where t is the current forward Markov chain current Is the time step.
( 6.1.3 LSDG estimation algorithm: restricted inverse TD (λ) least squares method)

This 6.1.3 proposes an LSDG estimation algorithm, LSDG (λ), based on the least square method. This simultaneously attempts to reduce the mean square and meet the constraints.

Therefore, the function to be minimized is the following expression ( 32 ).

Here, the second term on the right side is for the constraint condition of the formula ( 31 ). Therefore, the partial differentiation of the equation ( 32 ) is as follows.

  In a general RL problem, such a correlation exists, so that an instrumental variable method is applied to remove such a bias.

Hereinafter, to indicate the status of the time step t in the real Markov chain M (theta), to change the notation for x _t. The proposed LSDG estimation algorithm, LSDG (λ), makes the time step K going backwards the same as the time step t of the current state x _t under the qualifying decay rate λε [0,1). That is, the following holds.

  FIG. 8 is a diagram showing a procedure for obtaining LSDG (λ) as algorithm 1.

FIG. 9 is a flowchart showing the algorithm 1.
Referring to FIG. 9, first, in step S 100, as a premise of processing, the following settings are made.

  Subsequently, the following processing is performed as initialization processing (step S102).

  The time step t is set to t = 0 (step S104), and the following processing is repeated from t = 0 to t = T−1 (steps S106 to S116).

  First, if t = 0 in step S106, the initial state is observed (step S108), and then the following settings are made (S110).

  On the other hand, if t is not 0 in step S106, the following processing is performed (step S112).

  Following step S110 or S112, the following calculation is performed (step S114).

  In step S116, if t is smaller than T, the process returns to step S106. If t is equal to or greater than T, the process proceeds to step S118, and the following calculation is performed.

  Subsequently, an estimated value of LSDG is obtained by the following calculation.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a conceptual diagram which shows an example of the system 1000 using the controller with which the control method and control program of this invention are applied. It is a figure which shows the structure of the computer 100 in a block diagram format. It is a figure which shows the difference of a Riemann metric matrix. It is a figure which shows a learning curve. It is a figure showing the learning success rate of each gradient method. It is a figure which shows how much each gradient method has fallen into the learning plateau. It is a conceptual diagram which shows an example of a structure of the controller of this invention. FIG. 5 is a diagram showing a procedure for obtaining LSDG (λ) as algorithm 1; 3 is a flowchart showing Algorithm 1.

Explanation of symbols

100 computer, 102 computer, 104 display, 106 FD drive, 108 CD-ROM drive, 110 keyboard, 112 mouse, 118 CD-ROM, 120 CPU , 122 memory, 124 hard disk, 128 communication interface, 200 the controlled device, 1000 systems.

Claims (4)

When the time evolution of the target system is described as a Markov process, the controller learns a policy that is a control law for the state of the system by observing the state quantity of the system,
Control signal generating means for generating a control signal for controlling the system based on the strategy;
State quantity detection means for observing the state quantity of the system;
A natural steady policy gradient estimating means for estimating a natural steady policy gradient that is a natural gradient of an average reward, using a Riemannian metric matrix as a Fisher information matrix of a simultaneous distribution of a state and a control signal value specified by the state quantity,
A controller comprising: policy update means for updating the policy by updating a policy parameter that defines the policy based on an estimation result by the policy gradient estimation means. The natural steady policy gradient estimation means includes:
Reward value acquisition means for acquiring a reward value depending on the state and the control signal in a predetermined relationship;
Based on the state quantity and the control signal at each time step, the logarithmic partial derivative of the steady distribution is estimated, and a linear function that defines the basis function between the state specified by the estimated partial differentiation and the control signal The controller according to claim 1, further comprising: an estimation unit that estimates the natural steady policy gradient by regressing the reward value using an approximater. When the time evolution of the target system is described as a Markov process, the control method learns a policy that is a control law for the state of the system by observing the state quantity of the system,
A control signal generating step for generating a control signal for controlling the system based on the strategy;
A state quantity detection step of observing the state quantity of the system;
And Riemann metric matrix Fisher information matrix of the joint distribution of the state and the control signal value which is specified by the previous SL state quantity and nature steady policy gradient estimating step for estimating a natural constant policy gradient is a natural slope of the average earnings,
A control method comprising: a policy update step of updating the policy by updating a policy parameter that defines the policy based on an estimation result obtained by the policy gradient estimation step. A program for causing a computer to execute a control method for learning a policy that is a control law for the state of the system by observing the state quantity of the system when the time evolution of the target system is described as a Markov process. And
A control signal generating step for generating a control signal for controlling the system based on the strategy;
A state quantity detection step of observing the state quantity of the system;
A natural steady policy gradient estimation step for estimating a natural steady policy gradient that is a natural gradient of an average reward, using a Riemannian metric matrix as a Fisher information matrix of a simultaneous distribution of a state specified by the state quantity and a control signal value;
A control program for causing a computer to execute control processing, including a policy update step of updating the policy by updating a policy parameter that defines the policy based on an estimation result obtained by the policy gradient estimation step.