JP2021527289A - Sum Stochastic Gradient Estimating Methods, Devices, and Computer Programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a gradient estimation method, a gradient estimation device, and a computer program.
A gradient estimation method is a gradient estimation method that includes a calculation graph and estimates the slope of another variable with respect to one variable in the calculation graph, and different gradient estimation is performed at some nodes in the calculation graph. Perform two or more different estimates of the same gradient using the quantity, combine the different estimates so that they are less than the initial estimates, pass the combined estimates to different nodes in the calculation graph, and the gradient estimates. Is used for further calculations.

Description

The present invention relates to a method of estimating the gradient of a variable defined in a calculation graph, a device for making the above estimation, and a computer program.

Most machine learning problems involve optimizing the expected value of the objective function J (x; θ) of some data generation distribution p _Data (x) as a whole, but this distribution is only through the _{sample data points {x i}.} It is accessible.

The most common optimization method is the gradient descent method using the Pathwise derivative calculated by backpropagation.

Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5 (2): 157-166, 1994 A survey on policy search for robotics by Deisenroth, Marc Peter, Neumann, Gerhard, Peters, Jan, et al. Foundations and Trends in Robotics, 2 (1-2): 1-142, 2013 Deisenroth, Marc Peter, Fox, Dieter, and Rasmussen, Carl Edward. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37 (2): 408-423, 2015

In some situations (especially with very long or recurrent computational graphs), this technique can also fall into a random walk due to the explosion of gradient variance. Usually, this phenomenon is regarded as a numerical problem leading to an increase in steps and destabilization of learning (see Non-Patent Document 1).

An object of the present invention is to solve a problem associated with gradient estimation. The present invention is a general purpose gradient estimation method that can be used for any computational graph as an alternative to the backpropagation algorithm.

Gradient estimation methods include computational graphs and estimate the gradient of a variable relative to other variables in the computational graph, with several nodes in the graph using separate gradient estimators for the same gradient. Perform two or more separate estimates, combine the separate estimates so that they are less than the initial number of estimates, pass the combined estimates to different nodes in the graph, and the gradient estimates are used for further calculations. used.

According to the present application, it is possible to provide an alternative and flexible framework for gradient assessment that is more accurate and does not suffer from gradient explosions.

It is a block diagram which shows the hardware configuration of the computing device 1 which concerns on this embodiment. It is a figure explaining the policy gradient evaluation algorithm by PILCO. It is a figure explaining the total propagation algorithm. It is a flowchart explaining the procedure executed by the computing device 1 which concerns on this Embodiment. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a graph of variance. It is a graph of variance. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the example of the route in formula 11. It is a figure which shows the example of the route in formula 11. It is a figure which shows the probability calculation graph of the model reference and LR gradient estimation without a model. It is a figure which shows the probability calculation graph of the model reference and LR gradient estimation without a model. It is a figure which shows the algorithm 3 for demonstrating in detail how it conforms with total propagation. It is a figure which shows the calculation path in the Gauss molding gradient. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the experimental result. It is a figure which shows the general form of an algorithm. It is a figure which shows the back propagation algorithm used in all the neural network applications in machine learning, as well as many other applications. It is a figure which shows the sum propagation algorithm when the gradient estimation is performed by combining the likelihood ratio and the re-parameterized gradient estimator so that it becomes a single gradient estimator.

(Embodiment 1)
FIG. 1 is a block diagram showing a hardware configuration of the computing device 1 according to the present embodiment. The computing device 1 according to the present embodiment is an information processing device such as a personal computer and a server device. The computing device 1 includes a control unit 11, a storage unit 12, an input unit 13, a communication unit 14, an operation unit 15, and a display unit 16. The computing device 1 is described in "PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos", "Total Propagation Algorithm: Supplementary notes", and "Total stochastic gradient algorithms and applications in reinforcement learning" by the present inventors. Implements the disclosed method.

The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The ROM of the control unit 11 stores a control program or the like that controls the operation of each part of the hardware. The CPU of the control unit 11 executes a control program stored in the ROM and various programs stored in the storage unit 12 described later to control the operation of the hardware as in the method disclosed in the above-mentioned paper. The RAM of the control unit 11 stores data that is temporarily used when executing various programs.

The control unit 11 is not limited to the above configuration, and may be one or a plurality of processing circuits or arithmetic circuits including a single-core CPU, a multi-core CPU, a GPU (Graphics Processing Unit), a microcomputer, and a volatile or non-volatile memory. There may be. Further, the control unit 11 may include functions such as a clock for outputting data and time information, a timer for measuring the elapsed time from the application of the measurement start command to the given of the measurement end command, and a counter for counting. good.

The storage unit 12 includes a storage device using a SRAM (Static Random Access Memory), a flash memory, a hard disk, or the like. The storage unit 12 stores various programs executed by the control unit 11, data necessary for executing various programs, and the like. Examples of the program stored in the storage unit 12 include a computer program in which the technique disclosed in the above paper is implemented.

The program stored in the storage unit 12 may be provided by the recording medium M in which the program is readable and recorded. The recording medium M is a portable memory such as an SD (Secure Digital) card, a micro SD card, and a compact flash (registered trademark). In this case, the control unit 11 can read a program from the recording medium M using a reading device (not shown), and the read program can be installed in the storage unit 12. Further, the program stored in the storage unit 12 may be provided by communication via the communication unit 14. In this case, the control unit 11 can acquire a program through the communication unit 14 and install the acquired program in the storage unit 12.

The input unit 13 has an input interface for inputting various data to the device. The control unit 11 acquires data to be processed through the input unit 13.

The communication unit 14 includes a communication interface for connecting to a communication network (not shown) such as the Internet, transmits various types of information notified to the outside, and transmits various types of information transmitted from the outside. Receive. In the present embodiment, the data to be processed is acquired through the input unit 13, but the data to be processed may be acquired through the communication unit 14.

The operation unit 15 includes a user interface such as a keyboard and a touch panel, and receives various operation information and setting information. The control unit 11 executes appropriate control based on the operation information input from the operation unit 15, and stores the setting information in the storage unit 12 as needed.

The display unit 16 includes a display device such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel, and displays information notified to the user based on a control signal output from the control unit 11.

In the present embodiment, the configuration disclosed in the above paper is realized by software processing executed by the control unit 11, but LSI (Large Scale integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Arra). ) Etc. may be mounted separately from the control unit 11. In this case, the control unit 11 realizes the method disclosed in the above paper in the hardware by sending the data to be processed input from the input unit 13 to the hardware.

Further, in the present embodiment, the computing device 1 is described as a single device for simplification, but may be composed of a plurality of computing devices, or may be composed of one or a plurality of virtual machines. It may be configured.

In the present embodiment, the computing device 1 includes an operation unit 15 and a display unit 16, but the operation unit 15 and the display unit 16 are not essential. For example, the computing device 1 may accept an operation through an externally connected computer and output information notified to the external computer.

Hereinafter, the gradient estimation method of the present invention will be described. In the following formula, lowercase letters represent scalars and bold letters represent vectors or matrices. However, in the following description, lowercase letters and bold letters are shown without distinction. Further, in the following description, "C_ ^" represents a character with a hat, and "C_ ~" represents a character with a tilde.

(2.1 Policy search)
Non-Patent Document 2 is referred to as a summary of the policy search method. Note that the policy search is only one application of the algorithm, and is not limited to a specific calculation graph, and can be applied to any calculation graph. The state vector x _{t (e.g.,} the position of the robot and speed) and apply operation / control vector u _{t (e.g.,} motor torque) consider a discrete-time system described by. The episode begins by sampling the states from a fixed initial state distribution x _{0 to} p (x _0). Policy [pi _theta, applied operation _{u t ~p (u t) =} π; determining _{(x t} θ). The application of the operation, the unknown dynamics function _{x t + 1 ~p (x t} + 1) = f (x t, u t) state transitions in accordance with. Both the policy and the dynamics may be stochastic and non-linear. The operation and state transition are repeated up to the maximum T time step, and the locus τ: (x ₀ , u ₀ , x ₁ , u ₁ , ..., X _T ) is generated. Each episode is scored according to the return function G (τ). The return is often decomposed into the sum of the costs for each time step G (τ) = Σ _{t = 0} ^T c (x _t ) (t = 0, ..., T), where c (x). ) Is a cost function. The goal is to optimize the policy parameter θ _{to minimize the expected return J (θ) = Er ~ p (τ; θ)} [G (τ)]. Here, the value V _h (x) = _{Et = h} ^T [Σc (x _t )] is defined.

Learning alternates between executing policies on the system and improving performance on subsequent trials by subsequently updating θ. In the policy gradient method, the gradient d / dθ · J (θ) of the objective function is directly estimated and used for optimization. Some model-based policy search methods use all the data to learn the model of f, indicated by f_ ^, and use it for "mental rehearsal" between trials to optimize the policy. Hundreds of simulated trials can be performed for each real-world trial to significantly improve data efficiency. Here we take advantage of the fact that the derivative of f_ ^ can yield a better gradient estimator than the unmodeled algorithm. The model in this case is stochastic and predicts the state distribution.

(Stochastic gradient descent)
_{Here, a method of calculating the gradient d / dθE x to p (x; θ)} [φ (x)] (for example, the expected return for a policy parameter) of the expected value of an arbitrary function φ (x) with respect to the parameters of the sampling distribution. Will be described.

(Reparameterization Gradient (RP))
Consider sampling from a univariate Gaussian distribution. In one technique, after sampling with zero mean and unit variance ε-N (0,1), this point is mapped to duplicate the sample from the desired distribution (x = μ + σε). Here, it is easy to differentiate the output with reference to the distribution parameters. That is, dx / dμ = 1 and dx / dσ = ε. The sample averaging dφ / dx · dx / dθ gives an unbiased estimate of the expected gradient. This is a normally distributed RP gradient. In the case of a multivariate Gaussian distribution, the Cholesky factor (L, st Σ = LL ^T ) of the covariance matrix can be used instead of σ.

(Likelihood ratio gradient (LR))
The desired gradient can be described as d / dθ · E _{x to p (x; θ)} [φ (x)] = ∫dp (x; θ) / dθφ (x). In general, from the distribution q (x) by executing ∫φ (x) dx = ∫q (x) φ (x) / q (x) dx = _{Ex to} q [φ (x) / q (x)] Any function can be integrated by sampling. The likelihood ratio gradient is directly integrated as follows by extracting q (x) = p (x).

LR gradients are often highly dispersed and need to be combined with dispersion reduction techniques known as control variables (Greensmith et al., 2004). In a general method, a constant reference value b is subtracted from a function value to obtain an estimator _{Ex to} p [d / dθ · (log p (x; θ)) (φ (x) −b)]. If b is irrelevant to the sample, this can significantly reduce the dispersion without introducing a bias. In practice, sample mean is a good choice (b = E [φ (x)]). When estimating the gradient from a batch, the unbiased estimator can be obtained by estimating the reference value without one at each point. That is, ^{_{b i = Σ j ≠ i P}} φ (x j) / (P-1).

(Trajectory gradient estimation)
The probability density p (τ) = p (x ₀ , u ₀ , x ₁ , u ₁ , ..., X _T ) for observing a specific trajectory is p (x ₀ ) π (u ₀ | x ₀ ) p. (X ₁ | x ₀ , u ₀ ) ... It can be described as _{p (x T} | x _T-1 , u _T-1).

To use the RP gradient dynamics _p needs to know or estimate the _{(x t + 1 | u t} | x t). In other words, it is applicable in the case of model criteria. According to such a model, the predicted locus can be differentiated using the chain rule.

To use the LR gradient, log p (τ) can be converted to the sum because p (τ) is a product. It is expressed as _{G h} (τ) = Σ _{t = h} ^T c (x _t). Note that (1) only the motion distribution is determined by the policy parameters, and (2) the motion does not affect the cost obtained in the past time step, and the following gradient estimator can be obtained.

(PILCO)
FIG. 2 is a diagram illustrating a policy gradient evaluation algorithm by PILCO. Here we follow the original PILCO, which uses a Gaussian process dynamics model to predict changes in state from one time step to the next. That is, p (Δx _{t + 1} ^a _{) = gP (x t} , ut) (where x ^{∈ R D} , u _∈ ^{RF F} , Δ x _{t + 1} ^a = x _{t + 1} ^a −x _t ^a ). A separate Gaussian process is learned for each dimension a. Here, the square exponential covariance function ^{_{k a (x_~, x'_~) =}} s a 2 exp (- (x_~-x'_~) T Λ a -1 (x_~-x'_~)) To use. However, s _a and Λ = diag ([ _la1 , _la2 , ..., _{laD + F} ]) are hyperparameters of function variance and length scale, respectively. We also use a Gauss-likelihood function with noise hyperparameters of σ _n. Hyperparameters maximize marginal likelihood by training. Upon sampling from these models, the prediction has the form y = f_ ^ (x) + ε (where ε to N (0, σ _f ² (x) + σ _n ² )). Here, σ _f ² represents the uncertainty of the model and is due to the lack of data in the region. On the other hand, σ _n ² is the learned intrinsic model noise. The trained model noise is not necessarily the same as _{the actual observed noise σ o} ^{2 in the system.} In fact, the latent state is not modeled and the system is approximated by predicting the next observation given the current observation. Further, there is an additional dispersion source in the locus, and the locus is different if the starting position is different.

(Moment matching prediction)
In general, when the Gaussian distribution is mapped by a nonlinear function, the output is awkward and non-Gaussian. However, there are cases where the moment of output distribution can be evaluated analytically. Moment matching (MM) approximates the output distribution as a Gaussian distribution by matching the mean and variance with the true moment. Note that even if the state dimensions are modeled by a separate function fa_ ^, the MM is executed integrally and the state distribution can include covariance.

(Particle prediction)
In general, particle trajectory prediction is simple, predicting at all particle positions, sampling from the output distribution, and repeating. However, the neural network dynamics model is also applied to PILCO by comparison with the method based on Gauss resampling (GR).

(Gauss resampling (GR))
MM is stochastically replicable. In each time step, the average μ_^ = Σ _{i = 1} ^P x _i / P and the variance _{Σ_ ^ = Σ i = 1} ^P (x _i − μ_ ^) (x _{i −} ^{μ_ ^) T} / (P-1) of the particles. ) Is estimated. After that, the particles, fit distribution _{x 'i ~μ _ ^ + Lz} i | z i ~N (0, I) ( where, L is the Cholesky factor of Σ_ ^) is re-sampled from. It is not easy to find the gradient dL = dΣ_ ^. Here, the given symbolic representation is used.

(Hybrid gradient estimation technology)
In the case of the present invention, an RP gradient can be used. However, surprisingly, the RP gradient is hopelessly inaccurate (see Figure 5D). To solve this problem, we obtained a new gradient estimator that combines the model derivative with the LR gradient. In particular, in the method of the present invention, sampling efficiency can be improved by in-batch weighted sampling.

(Model standard LR)
The distribution on the predicted locus is p (τ) = p (x ₀ ) π (u ₀ | x ₀ ) f_ ^ (x ₁ | x ₀ , u ₀ ) ... f_ ^ (x _T | x _T-1) , U _T-1 ). Also, depending on the deterministic policy, it is possible to combine the model and the policy, such as _{p (x t + 1} | x _t ) = f_ ^ (x _{t + 1} | x _t , π (x _{t; θ)).} Is differentiable (dp _{t + 1} / dθ = dp _{t + 1} / du _t · du _t / dθ). The gradient of the model reference is derived as follows.

(Batch weighted LR (BIW-LR))
Here, parallel computing is used to sample multiple particles at the same time. The state distribution is expressed as a mixture distribution q (x _{t + 1} ) = Σ _{i = 1} ^P p (x _{t + 1} | x _{i, t} ; θ) / P. Similar to the derivation of LR, for each time step, the low variance estimator can be derived as follows by intensive sampling in batch.

The following equation is used to estimate the average of returns without one by normalization-weighted sampling.

However, c _{j, t + 1} = p (x _{j, t + 1} | x _{i, t} ) / Σ _{k = 1} ^P p (x _{j, t + 1} | x _{k, t} ). Without normalization, the high variance of the reference value estimation results in an inadequate LR gradient. Incidentally, while calculating the P reference value for each time step, the gradient estimator, there is P ² component. To determine the true unbiased gradient shall be calculated one vent reference value P ² (the one for each particle of each mixture component distribution). The present specification includes an evaluation using only the reference values presented herein (which is known to have eliminated most of the bias).

(RP / LR weighted average)
Most of the calculation _{is spent on the dp (x t + 1} | x _t ; θ) / dθ term. Since these terms are required for both LR and RP gradients, there is no penalty for combining both estimators. Well-known statistical results show that for independent estimators, optimal weighted average estimates are achieved when the weights are proportional to the inverse variance. That is, μ = μ _LR k _LR + μ _RP k _RP (where k _LR = σ _LR _ ^ ^-2 / (σ _LR _ ^ ^-2 + σ _RP _ ^ ^-2 ) and k _RP = 1-k _LR ). ..

In the simple coupling method, the gradients of all trajectories are calculated separately for both estimators and then combined, but in this method, reparameterization gradients are used for the short parts of the trajectories. Opportunities for better gradient estimates are ignored. The novel sum propagation algorithm (TP) of the present invention is superior to this simple method. In TP, a single back pass calculates the union over all possible RP depths, so the low variance estimator is automatically weighted heavily.

FIG. 3 is a diagram illustrating a total propagation algorithm. In Algorithm 2, each backward step evaluates the gradient against policy parameters by using both the LR and RP methods. We also evaluate the ratio based on the variance in the policy parameter space, which is proportional to the variance of the policy gradient estimator. The gradients are combined and the best estimate in the distribution parameter space is passed to the past time step. In this algorithm, the sample variance of the gradient estimator is extracted from particles with different V operators, but other variance estimation methods are also conceivable. For example, it is possible to estimate the variance from the moving average of the gradient magnitude. However, it is possible to use different statistical inferences for the variance, or just a subset of the policy parameters. This algorithm is not limited to the RL problem, but can be applied to general probabilistic calculation graphs, and can also be used for training stochastic models, stochastic neural networks, and the like. In a typical computational graph setting, multiple gradient estimators may be combined at several nodes in the graph by propagating the gradient backwards in the graph. In this case, if the time step parameter t is reduced by 1, this corresponds to the backward movement of the node in the graph while the propagation of the gradient. Statistical values such as variance used to determine the gradient estimator in the combined method may be obtained from any other node in the calculation graph.

FIG. 4 is a flowchart illustrating a procedure executed by the computing device 1 according to the present embodiment. The computing device 1 executes the following process according to the algorithm 2.

The control unit 11 initializes various parameters (step S101). Specifically, the control unit 11 _sets the _{dG T + 1 / dζ T +} 1 = 0, dJ / dθ = 0, G T + 1 = 0. Where ζ is a distribution parameter (eg μ and σ).

The control unit 11 sets the time (time step) t to T (step S102), and executes the following calculation for each particle i (step S103). However, _ct is the cost at time t.

The control unit 11 executes the following calculation using the calculation result of the formula 6 (step S104).

Further, the control unit 11 executes the following calculation for each particle i using the calculation result of the formula 6 (step S105).

Next, the control unit 11 determines whether the time t has reached the predetermined time 1 (step S106). If the time t is not time 1 (S106: NO), the control unit 11 reduces the time t by 1 (step S107) and returns the process to step S103.

(Policy optimization)
It is possible to use any gradient-based optimization procedure, but in this embodiment, a stochastic gradient descent method such as RMSprop is used (an algorithm derived from RMSprop is used). RMSprop uses the moving average of the square of the gradient to normalize its SGD step. In the case of the present invention, since the batch size is large, the expected value of the square is directly estimated from the batch by ^{z = E [g 2} ] = E [g] ^{2 + V [g] (where g is a gradient).} Also, use the mean variance. That is, V [g] is the variance divided by the number of particles P. The gradient step is g / z ^1/2 . Also, the momentum of the parameter γ is used. The fully updated equation looks like this:

Fixed random number seeds can deterministically change stochastic problems and are also known in the RL community as PEGASUS tricks. When the seed is fixed, the RP gradient is the exact gradient of interest and a deterministic pseudo-Newton optimizer such as BFGS can be used.

(experiment)
Experiments were conducted for two purposes: (1) to explain why the RP gradient was not sufficient, and (2) that the newly developed method of the present invention could be comparable to PILCO in terms of learning efficiency. To show.

(Plot the value landscape)
5A-5F illustrate the experimental results. The policy parameter θ is perturbed in a randomly selected fixed direction, and the objective function and the magnitude of the projection gradient are plotted as a function of Δθ. The results of this experiment were perhaps the most novel part of this specification, and came up with the term "the curse of chaos".

Plots were generated in a non-linear cart-pole task. Using 1000 particles, the random seeds were kept fixed to demonstrate that the high variance in Figure 5D was not caused by randomness, but by the chaotic properties of the system. The confidence interval is estimated by Var / P, where Var is the sample variance and P is the number of particles. As will be described later, a more principled method is used to plot the dependence of the variance on P.

FIG. 5D contains a peculiar result, in which the RP gradient behaves well in some regions, but when the policy parameters are perturbed, the variance explodes due to changes such as phase transitions. Dispersion in [Delta] [theta] = 1.5 is to 4 × 10 ⁵ times the [Delta] [theta] = 0, which means that for RP gradient in this region is the accurate are required 4 × 10 ⁸ particles. In practical use, a simple random walk can be derived by optimizing with the RP gradient.

Since the seeds are fixed, the RP gradient in FIG. 5D is the exact gradient of the values in FIG. 5A. Therefore, there is a very small deterministic "noise" on the right side of FIG. 5A. However, the value averaged over 1000 particles is not the true purpose, but it is necessary to average an infinite number of particles. Is there still "noise" when averaging an infinite number of particles? Or will the function be smooth?

The new gradient estimators in FIGS. 5E and 5F suggest that the true purpose is certainly smooth. To provide further evidence, the magnitude of the gradient was estimated from the finite difference of the values in FIG. 5A using a sufficiently large perturbation at θ so that “noise” could be ignored. The fact that two separate methods (one to change the policy parameter θ, the other to keep θ fixed but to estimate the gradient from the trajectory) is convincing that the true purpose is smooth. Give some evidence.

5B and 5C explain the reason for the dispersion explosion when using the RP gradient. FIG. 5B corresponds to the leftmost parameter setting, and FIG. 5C corresponds to the rightmost parameter setting. The plot shows how the value V (x; θ) (residual cumulative cost) changes as a function of position x. Since the random number seed is fixed, the value V is the same as the residual return G. The drawings were created by predicting the locus of each point for 4 particles with different fixed seeds and averaging the cost of the locus. After trying one particle, I tried to predict four particles, for which the values seemed to include stair-like parts, but otherwise it wasn't very interesting compared to the current drawing. Since the average value of the four particles is unstable, at least one of the four particles must have been very unstable within the area shown.

From the center of the initial state distribution, the square is positioned in the center of the average prediction. This is because the axes of the squares are slightly different, but the position p (x1; θ), which is predicted to change θ, changes. The edge lengths correspond to the four standard deviations of the Gaussian distribution p (x1; θ). The speed remained fixed at the average value.

RP estimates d / dθ ∫p (x ₁ ; θ) V (x ₁ ) dx. It samples the points inside the square, calculates the gradient dV / dθ = dV / dx · dx / dθ, and averages it with the sample. In FIG. 5C, finding the gradient of the expected value by differentiating V is quite hopeless. In contrast, the LR gradient (FIG. 5E) uses only the value V, not the derivative of the value V, and does not suffer from this problem. TP (Fig. 5F) effectively combines both estimators.

We do not show plot values and gradients for the Gauss resampling case, but in the end both of these were smooth functions for a fixed random seed. Therefore, resampling is also effective against the "curse of chaos".

6A and 6B are graphs of variance. In FIGS. 6A and 6B, how the variance of the gradient estimator at Δθ = 0 and Δθ = 1.5 depends on the number of particles P is plotted. The variance was calculated by iteratively sampling the estimator many times and calculating the variance from the set of evaluations. RP, TP and LR gradients are compared both with and without batch weighted (BIW) to show that the weighted sampling scheme of the present invention reduces variance. Using a weighted reference value-in practice, a normal LR gradient uses a simpler reference value and has a much higher variance. The RP gradient is omitted in FIG. 6B because the variance was between ^{10 8 and} 10 ^15. The TP gradient combined the BIW-LR and RP gradients.

The results confirm that BIW significantly reduces dispersion. Furthermore, the TP algorithm of the present invention was the best. Importantly, the dispersion of the RP slope for all trajectories in Figure 6B is ¹⁰⁶ greater than other estimates, TP 10-50% for less particles than 250 using the RP gradient short path length reduction Is getting the variance. This is a notable result, because when the gradient estimators are calculated separately, the highest possible accuracy for the combined estimators is the sum of the accuracy of the separate estimators. However, since the total propagation algorithm of the present invention uses the graph structure of calculation, it achieves higher accuracy than the total.

(Learning experiment)
Compare PILCO in episodic learning tasks with the following particle-based methods: RP, RP with fixed seed (RPFS), Gauss resampling (GR), GR with fixed seed (GRFS), model-based batch Weighted likelihood ratio (LR), and total propagation (TP). In addition, two variations of particle prediction are evaluated. _{(1) TP (TP-σ f} ) that adds only noise at each time step while ignoring model uncertainty. (2) TP (TP + σ _n ) in which the predicted noise is increased. 300 particles were used in all cases.

A recent PILCO paper (Non-Patent Document 3): Swing-up and balance of cart poles, and balance of unicycles, performed more learning tasks. The simulation dynamics were set to be the same, and other aspects were the same as the original PILCO. 7A, 7B, 8 and 9 illustrate the experimental results.

The optimizer was run for 600 policy evaluations between each trial. The SGD learning rate and momentum parameters were α = 5 × 10 ^-4 and γ = 0.9. The episode length was 3s for the cart pole and 2s for the unicycle. For unicycle tasks, 2s is not enough to generalize the policy to long trials, but it can still be compared to PILCO. The control frequency was 10 Hz. The cost is of type 1-exp (-(x-t) ^T Q (x-t)), where t is the target. The output from policy_ (x) is constrained by the saturation function sat (u) = 9sin (u) / 8 + sin (3u) / 8, where u = π_ to (x). One experiment consists of (1; 5) random trials, followed by learned trials (15; 30) for each of the cart and unicycle tasks. Each experiment was repeated 100 times and averaged. Each trial was evaluated by running the policy 30 times and averaging, but note that this was done for evaluation purposes only (algorithm access is only one trial). Success was judged by whether the return of the final trial was below the threshold.

(Cart-pole swing up and balance)
This is a standard control theory benchmarking task. The task consists of pushing the cart back and forth and rocking the upright mounted pendulum to maintain its balance. The state space is expressed as x = [s, β, ds / dt, dβ / dt], where s is the cart position and β is the pole angle. The reference noise levels are σ _s = 0.01 m, β = 1 deg, σ _{ds / dt} = 0.1 m / s, and σ _{dβ / dt} = 10 deg / s. The noise is corrected by the multiplier k: σ ₂ = kσ _base ^{2 in different experiments.} The original paper considers direct access to the true state. To determine the setting of the similarity, was set to k = ^{10 -2,} were tested again k∈ {1,4,9,16}. Policy π_ ~ is a radial basis function network (sum of Gaussian) with 50 basis functions. Consider two cost functions. One is the same as the original PILCO, where x contains the sine and cosine and depends on the distance between the tip of the pendulum and the position of the tip when the pendulum is in balance. (Tip Cost). Another cost was Q = diag ([1,1,0,0]) using the raw angle (Angle Cost). This cost is conceptually different from Tip Cost, because there is only one correct direction to swing up the pendulum.

(Balance of unicycle)
The task consists of balancing the unicycle robot, with state dimension D = 12 and control dimension F = 2. The noise was set to a low value. Π_ ~ giving control is linear.

(Learning experiment)
PILCO works well in noisy scenarios, but the addition of noise worsens the results. This exacerbation is most likely to occur due to the accumulation of errors in the MM approximation, previously observed by Vinogradska et al. (2016), who used quadrature for prediction. Particles do not suffer from this problem and using a TP gradient is always better than PILCO in high noise conditions.

On the other hand, at low noise levels, the performance of TP and LR is degraded. If all of the particles are sampled from a small area, it will be difficult to estimate the gradient from the change in return (at the limit of delta variance, the LR gradient cannot even be evaluated). The TP gradient does not suffer much from this problem, because it incorporates information from the RP. Finally, if the prediction uncertainty is very low (eg k = ^10-2 ), model noise can be considered as a parameter affecting learning and increased to obtain a more accurate gradient. See TP + σ _n. However, the model noise variance was multiplied by 100.

In particular, methods using MM, such as PILCO, and GR are superior to others when using Tip Cost. The reason can be the multi-modality of interest-in Tip Cost, the pendulum can be swung up from any direction to solve the task; in Angle Cost, there is only one correct direction. Running MM forces the algorithm to follow a unimodal path, but the particle technique can nevertheless attempt a bimodal swing-up where some particles come from one and stop at the other. There is sex. Therefore, the MM may perform a kind of "distribution reward shaping" that simplifies the optimization problem. Such an explanation was previously made by Gal et al. (2016).

Finally, we point out a surprising TP-σ _{f experiment.} The prediction ignores model uncertainty, but the method achieves a success rate of 93%. It is difficult to explain why learning was successful, but we hypothesize that success can be associated with a zero prior average of GP. In the region with no data, the average of the GP dynamics model goes to 0, which means that the input control signal has no effect on the particles. Therefore, for successful policy optimization, the particles must be controlled to stay in the area where the data resides. Similar results were found by Chatzilygeroudis et al. (2017), who achieved a success rate of 85-90% in the cart-pole task even when using evolutionary algorithms and ignoring model uncertainty. There is.

Most machine learning problems involve optimizing the expected value of the objective function J (x; θ) for some data generation distribution p _Data (x), but this distribution is accessed only through the _{sample data points {x i}.} It is possible. The predictive framework of the present invention is similar to a deep model: p (x ₀ ) is a data generation distribution and p (x _t ; θ) is _{by passing p Data} (x) through the model layer. Desired. The most common optimization method is SGD using the Pathwise derivative calculated by backpropagation. The results of the present invention suggest that in some situations (especially in the case of very deep or recurrent models), this approach can also lead to random walks due to gradient dispersion explosions.

Gradient explosions have been observed for many years in deep learning studies (Doya, 1993; Bengio et al., 1994). This phenomenon is usually regarded as a numerical problem that leads to increased steps and learning instability. Common countermeasures include gradient clipping, ReLU activation function (Nair & Hinton, 2010), and smart initialization. The description of the present invention for this problem is different: not only does the gradient increase, but the gradient variance explodes, which means _{that every sample from x i to p Data} changes the model parameter θ and the entire distribution E. It means that it essentially does not give information about whether to increase the expected value of the purpose for _{pData [J (x)].} While choosing good initialization is one way to deal with this problem, it seems difficult to guarantee that the system will not become chaotic during learning. In econometrics, for example, optimal policies can even result in chaotic dynamics (Deneckere & Pelikan, 1986). Gradient clipping can stop large parameter steps, but random gradients do not fundamentally solve the problem. Considering that chaos does not occur in linear systems (Alligood et al., 1996), the analysis of the present invention suggests why piecewise linear activation, which is less susceptible to chaos such as ReLU, works well in deep learning. ..

While the deep hypothesis of the present invention must still be computationally confirmed, some studies have investigated chaos in neural networks (Kolen & Pollack, 1991; Sompolinsky et al., 1988), but again the book. For the first time, we believe that chaos suggests that gradients can be degenerated when calculated using backpropagation. In particular, Poole et al. (2016) suggested that such a property provides "exponential expressiveness", but believes that this phenomenon can replace the curse.

(Conclusion and future research)
The limitations of optimizing expected values using Pathwise derivatives, such as those calculated by backpropagation, have been explained. In addition, we show how to counteract this curse by adding noise to the calculation and using the likelihood ratio trick. The sum propagation algorithm of the present invention combines a reparameterized gradient for any probabilistic computational graph with any amount of other gradient estimators (even gradients calculated using value functions can be used). Provides an efficient way to do this. There are countless ways to extend the work of the present invention: better optimization, incorporation of natural gradients, etc. The flexible nature of the methods of the invention should facilitate these extensions.

(Embodiment 2)
A definition of a probabilistic computational graph (PCG) is provided. Note that the PCG concept is different from the computational graph concept used to explain the sum propagation algorithm, but instead describes a framework for reasons for gradient estimators. The definition is quite equivalent to the definition of a standard directed graph model, but is more focused on the methods of the invention, emphasizing the invention's interest in calculating gradients rather than performing inference. ing. The main difference is the explicit inclusion of the distribution parameters ζ, mean μ, and covariance Σ for Gaussian, for example.

Definition 1 (Probabilistic Calculation Graph (PCG))
A non-circular graph with node / vertex V and edge E satisfies the following characteristics:
1. 1. Each node i ∈ V corresponds to a set of random variables with a peripheral joint probability density p (x _i ; ζ _{i ), where ζ i} is a perhaps infinite parameter of the distribution. It should be noted that the parameterization is not unique and any parameterization is acceptable.
2. Probability density for each node conditionally dependent on the parent node, _p | a _{(x i} Pa i). Where Pa _i is a random variable in the immediate parent of node i.
3. 3. The joint probability density _{satisfies p (x 1} , ···, x _n ) = Π _{i = 1} ⁿ p (x _i | Pa _i ).
4. Each ζ _i is a function of its parent, ζ _i = f (Pz _i ). Here, Pz _i is the distribution parameter in the parent node i. In particular, p (x _i ; ζ _i = ∫p (x _i | Pa _i ) p (Pa _i ; Pz _i ) dPa _i .

It should be emphasized that there is no stochastic in the mathematical formula of the present invention. Each calculation can be analytically cumbersome, but deterministic. Furthermore, this definition does not exclude deterministic nodes, that is, it emphasizes that the distribution at the nodes can be the Dirac delta variance (mass point). Later, we will use this formula to derive a stochastic estimate of the gradient.

(Derivation of the theorem)
The subject of interest is to calculate the total derivative _{of the distribution parameter at one node ζ i with} respect to the parameter at another node dζ _i / dζ _j. By iterating the rule of total derivative, the sum over the path from node j to node i is derived as follows.

This equation applies to all deterministic computational graphs and is well known in the OJA community, for example. This equation naturally leads to the stochastic gradient theorem of the present invention, explaining that the sum of the paths from A to B can be written as the sum of the paths from A to the intermediate node and from the intermediate node to B. .. 10A and 10B illustrate examples of routes in Equation 11.

Theorem 1 (Sum Stochastic Gradient Theorem)
In a probabilistic calculation graph, i and j are different nodes, IN is an arbitrary set of intermediate nodes, which blocks the path from j to i, that is, IN so that there is no path from j to i. It is for doing so, and does not pass through the node in IN. {A → b} is represented by a set of paths from a to b, {a → b} / c is a set of paths from a to b, and nodes along the paths except b are included in the set c. Can't. In this case, the total derivative dζ _i / dζ _j can be written by the following equation.

Formula 10 and Formula 11 can be combined to give:

A similar theorem can be derived by swapping r ∈ {j → m} / IN and s ∈ {j → m} / IN, respectively. This leads to the following equation.

Equations 12 and 13 are referred to as the sum-gradient equations for the second half and the first half, respectively.

(Gradient estimation on the graph)
The previous section provided a means of decomposing the gradient calculation for the entire graph into a gradient calculation for a narrower graph, and also provided a method for estimating the gradient for subgraphs. Here we clarify a method of how the gradients for a subgraph can be combined into an estimator for the gradient for the entire graph. The task is to estimate the derivative of the expected value at the distal node i with respect to the parameter at node j: d / dζ _j E _{xi to p (xi; ζ i)} [xi]. Since true ζ is difficult to handle, the sampling standard is estimated. Consider sampling the subvariance of p (x; ζ). That is, ζ_^ is sampled so that p (x; ζ) = ∫p (x; ζ_^) p (ζ_^) dζ_^. This can be written as:

ζ_ ^ occurs naturally in the traditional sampling procedure. For the sake of brevity, the sampling is reparameterizable, i.e. further assumed _{that p (ζ m} _ ^; ζ _j ) = f (ζ _m _ ^; ζ _j , z _m ) p (z _m). do. This can be written as:

The term dζ _m _ ^ / dζj is estimated by the Pathwise derivative estimator. The remaining terms d / d ζ _m _ ^ Ex _{i to p (xi; ζ i_ ^)} [xi] are estimated by any other estimator, for example a jump estimator can be used. If the second estimator is also unbiased, then the entire estimator is unbiased.

In summary, for the entire graph, the steps to create a gradient estimator from j to i are:
1. 1. Select a set of intermediate nodes IN that block paths j to i.
2. Construct a Pathwise derivative estimator from j to the intermediate node IN.
3. 3. Construct a total derivative estimator from IN to i and apply the chain rule from i to j.

(Relationship to the policy gradient theorem)
In a typical model without RL problems, the agent operates u~π according probabilistic policy [pi |; running _{(u t x} t _θ), the transition state _{x t,} determining the cost _{c t} (or, On the contrary, ask for a reward). Agent goal is to find the policy parameters theta, which optimizes the expected return ^{_{G = Σ t = 0 H c}} t of each episode. 11A and 11B illustrate the probability calculation graphs for model-based and LR gradient estimation without a model. In the literature, two "gradient theorems", the policy gradient theorem and the deterministic policy gradient theorem, are generally applied.

Qt_ ^ corresponds to the estimator _{of the residual return Σ h = t} ^H-1 _{ch + 1} from the specific state x when the operation u is selected. For Equation 16, any estimator is acceptable, and even a sampling criterion estimate can be used. For Equation 17, Q_ ^ is usually a differentiable surrogate model. Importantly, for the above equation to be valid, Q_ ^ must be an estimator, not a true Q. That is, when estimating the gradient, it must be assumed that the policy parameters only change for the current time step and remain fixed for subsequent time steps. FIG. 11A shows how these two theorems correspond to the same probabilistic computational graph. The intermediate node is the operation selected at each time step. There is a difference in the choice of jump estimates to estimate the total derivative following the intermediate node-the policy gradient theorem uses the LR gradient, while the deterministic policy gradient theorem applies the Pathwise derivative to the surrogate model. use.

(New algorithm)
Typically, when estimating the gradient for a PCG, traditional sampling is performed throughout the graph to obtain one sample, for example for the RL problem, the trajectory is sampled. Such samples are called particles. A batch of such samplings can be used to determine the gradient estimator. The estimated distribution parameters at a node are given by the set of distribution parameters for each sampled particle ζ _ ^ = {ζ _i _ ^} _i ^{P, where P is the number of particles.} For example, if the PCG consists of sequential sampling from a Gaussian distribution, ζ _i _ ^ corresponds to the mean and covariance of Gaussian where the particles were sampled at that node. The following sections take advantage of the option of using a set of particles to estimate a different set of direct distribution parameters Γ for the marginal distribution.

(Density estimation LR (DEL))
With the following description, it is possible to estimate the distribution parameter Γ from the set of sampled particles and try to apply the LR gradient using the estimated distribution ζ_ ^. In particular, the density is approximated as Gaussian by estimating the mean _{μ_^ = Σ i} ^P x _i / P and the variance _{Σ_ ^ = Σ i} ^P (x _{i −} μ_ ^) ^{2 / (P-1).} The gradient ΣiP dlogq (x _i ) / dθ (G _i − b) can then be estimated using standard LR tricks, where q (x) = N (μ_ ^, Σ_ ^). be. To use this method, _{the derivatives of μ_ ^ and Σ_ ^ with respect to the particles x i} must be calculated and the chain rule must be used to convey the gradient to the policy parameters, which is easy. The new method of the present invention is called a DEL estimator. Importantly, note that q (x) is used to estimate the gradient, but not in any way to modify trajectory sampling. This is in contrast to Gauss resampling, where particles are resampled from a Gaussian distribution so fitted and the trajectory distribution is modified.
Advantages of DEL: LR gradients can be used without adding noise to the calculation.
Disadvantages of DEL: The estimator is unbiased and can make density estimation difficult.

(Gauss molding gradient (GS))
So far, all RL methods have used the latter half of the sum gradient equation (Equation 12). Is it possible to make an estimator using the first half of the equation (Formula 13)? FIG. 13 illustrates the calculation path in the Gauss forming gradient. FIG. 13 gives an example of how this can be done. We propose to estimate the density at xm by fitting Gaussian to particles. Then dE [ _cm ] = dΓ _m (gray edge) is estimated by resample the particles from this distribution (or by any other method of integration). This _{leaves the question of how to estimate dΓ m} / dθ (dotted and thick edges). It is easy to use the RP method. To use the LR method, first apply the latter half of the sum gradient equation to the dΓ _m = dθ term Σ _{r ∈ {θ → k} / IN} Π _{(p, t) ∈} r ∂ζ _t / Find ∂ζ _p (dotted line edge) and dΓm / dζ _k (thick line edge). In the scenario under consideration, the first of these terms is a single route and is estimated using RP. The second term is more interesting and is estimated using the LR method. Since the Gaussian approximation is used, the distribution parameter Γ _m is the mean and variance of _{x m , where μ m} = E [x _m ] and Σ m = E [x _m x _m ^T ] -μ _m μ _m ^T. Can be estimated. The LR gradient estimators for these terms can be determined as follows.

In practice, you may make an estimate _{of the sampling criteria ζ k _ ^ and be concerned that the estimator is conditional on the sample ζ k} _ ^, but you are interested in the non-conditional estimates. be. Explain that conditional estimates are equivalent. Note that for variances, μ _m is an estimate of the unconditional mean, so the entire estimate directly corresponds to the estimate of the unconditional variance. For the mean, the iterated expected value rule applies as follows.

This makes it clear that the conditional gradient estimator is an unbiased estimator for the unconditional mean gradient.

(Efficient algorithm for accumulating gradients)
As a specific example, consider a model-based policy gradient method, the PCG of which is given in FIG. In previous studies of the present invention, this algorithm was first conceived and relies decisively on access to a differentiable stochastic model of dynamics. How to apply the GS gradient to this situation will be described. For _{each x k} node, we _{want to perform an LR jump to all x m} nodes after k and calculate the gradient with a Gaussian approximation of the distribution at node m. Accumulate all nodes during the backward path in a back-propagation-like manner. The gradient can be written as dE [ _cm ] / dΓ _m dΓ _m / dζ _k (dζ _k / du _k-1 du _k-1 / dθ) for each k and the path. The term dE [ _cm ] / dΓ _m · dΓ _m / dζ _k is estimated as dE [ _cm ] / dΓ _m z _m d logp (x _k ; ζ _k ) / dζ _k , where zm is the above term x. corresponding to vector summarizing the like _m -b _mu. In addition, dE [ _cm ] / dΓ _m z _{m is} just a scalar amount g _m . Therefore, we use an algorithm that accumulates the sum of all g during the backward path and sums all m nodes at each k node. FIG. 12 illustrates Algorithm 3 for explaining in detail how it fits with sum propagation. The final algorithm essentially simply replaces the normal cost / reward with the modified value, and such techniques are further applied to the unmodeled policy gradient algorithm using probabilistic policies and LR gradients. It is possible. Two interpretations of GS: 1. At a node, perform a Gaussian approximation of the marginal distribution. 2. Perform some type of reward shaping based on the distribution of particles. In particular, it essentially encourages the trajectory distribution to be unimodal so that all of the particles are concentrated on one "island" of the reward rather than split across multiple reward regions-this simplifies the optimization. May become.

(experiment)
Based on the PILCO paper, a model-based RL simulation experiment was performed. Cart-pole swing-up and balance tasks were tested to test coupling with the GS method of the invention as well as total propagation. In addition, to demonstrate the feasibility of this idea, the DEL method was tested on a simpler cart-pole, balance-only task. Particle-based gradients with new estimators of the invention were compared to PILCO. In previous studies of the present invention, the cost function had to be modified to obtain reliable results using particles-one of the main motives of the current experiment was used by the original PILCO. Matching with PILCO results using the same cost as (this will be further detailed later).

(Model-based policy search background)
Consider a model-based analog for a modelless policy search method. The corresponding stochastic calculation graph is given in FIG. 11B. The notation follows previous work of the present invention. After each episode, we train a separate Gaussian process model for each dimension of dynamics using all of the data so that p (Δx _{t + 1} ^a ) = gP (x _{t _ ~).} Here, x_ ~ = [x _t ^T , ^{ut T} ] and ^{x ∈ R D} , _{u ∈} ^{R F.} This model is then used to perform a "mental simulation" between episodes to optimize the policy by gradient descent. Squared exponential covariance function ^{_{k a (x_~, x'_~) =}} s a 2 exp - using ^{((x_~-x'_~) T Λ} a -1 (x_~-x'_~)) .. The noise hyperparameter uses Gaussian likelihood function of sigma _{n, 2} ^2. Hyperparameters {s, Λ, σ _n } are trained by maximizing marginal likelihood. The prediction takes the form of p (x _{t + 1} ^a ) = N (μ (x _t _ ~), σ _f ² (x _t _ ~) + σ _n ² ), where σ _f ² (x _t _ ~) is a model. Uncertainty about, depending on the availability of data within the realm of the state space. In FIG. 11B, the partial derivative from θ to the intermediate node is estimated by the Pathwise derivative, and the total derivative following the intermediate node is estimated by the jump estimator.

(setup)
A cart-pole consists of a cart that can be pushed back and forth and an attached pole. The state space is [s, β, ds / dt, dβ / dt], where s is the cart position and β is the angle. Control is a horizontal force on the cart. The dynamics were similar to the PILCO paper. The setup follows previous work of the present invention.

(Common characteristics in tasks)
The experiment consists of one random episode, followed by 15 episodes with a learned policy, and the policy is optimized between episodes. Each episode length was 3 s and the control frequency was 10 Hz. Each task was evaluated 100 times separately with different random seeds to test reproducibility. Random seeds were shared by different algorithms. Each episode was evaluated 30 times and the cost was averaged, but keep in mind that this was done for evaluation purposes only-the algorithm has access to only one episode. The policy is optimized using a learning rule such as RMSprop from previous studies of the present invention, which normalizes the gradient using a gradient sampling variance from different particles. In model-based policy optimization, 600 gradient steps were performed using 300 particles for each policy gradient evaluation. The learning rate and momentum parameters are α = 5 × 10 ^-4 and γ = 0: 9, respectively, which are the same as in the previous studies of the present invention. The output from the policy is saturated by sat (u) = 9sin (u) / 8 + sin (3u) / 8, where u = π_ to (x). Policy π_ ~ is a radial basis function network (Gaussian sum) with a sum of 50 basis functions and 254 parameters. The cost function is of type 1-exp (-(x-t) ^T Q (x-t)), where t is the target. Consider two types of cost functions: 1) Angle Cost, Q = diag ([1,1,0,0]) where the cost is a diagonal matrix, 2) Tip Cost, from the original PILCO paper. It is a cost and depends on the distance from the tip of the pendulum to the position of the tip when balanced. These cost functions are conceptually different-in Tip Cost, the pendulum can swing up from any direction, and in Angle Cost, there is only one correct direction. The reference observed noise levels are σ _s = 0.01 m, σ _β = 1 deg, σ _{ds / dt} = 0.1 m / s, σ _{dβ / dt} = 10 deg / s, and these are σ ² = kσ _base ^2. It is corrected by the multiplier k ∈ {10 ^{-2, 1}.}

(Cart-pole swing up and balance)
In this task, the pendulum first hangs downwards and then has to swing and balance. Some results have been obtained from previous studies of the present invention: 1) PILCO, 2) reparameterization gradient (RP), 3) Gauss resampling (GR), 4) batch with weighted reference values. Weighted LR (LR), 5) Total propagation (TP) that combines LR and RP. Compared with the new method: 6) Gauss forming gradient (GLR) using only the LR component, 7) Gauss forming gradient (GTP) combining both LR and RP variables using summation propagation. For a description of the sum propagation algorithm, see previous work of the present invention, which is a method of effectively combining multiple gradient estimators on a computational graph. Furthermore, the GTP when the model noise variance was multiplied by 25 was tested (GTP + σn).

(Cart-pole balance in DEL estimator)
This task is much simpler-Paul is initially upright and must be balanced. Experiments have been devised to show that DEL is feasible and may be useful if further developed. Angle Cost and reference noise levels were used.

(result)
14 and 15 illustrate the experimental results. As in previous studies of the present invention, when the noise is low, the method containing the LR component does not work. However, GTP + σn experiments show that adding noise to model prediction can solve the problem. The main important result is that GTP is consistent with PILCO in the Tip Cost scenario. In previous studies of the invention, one concern was that TP was inconsistent with PILCO in this scenario. Just looking at the costs in FIGS. 15B and 15C does not properly show the difference. In contrast, the success rate indicates that TP did not work either. Success rates were measured both by the threshold calibrated in previous studies of the invention (final loss below 15) and by visually classifying all experimental runs. Both methods matched. The peak performer loss in the final episode was 11.14 ± 1.73, GTP: 9.78 ± 0.40, PILCO: 9.10 ± 0.22, which also means that TP was significantly worse. Is shown. Residual experiments converged while peak performers were still improving. PILCO still looks slightly more data efficient, but due to the small amount of data required, the differences have little practical significance. It should also be noted that the variance of TP is smaller in FIG. 15B. Large variances of GTP and PILCO are caused by outliers with large losses. These outliers converge to the local minimum, which utilizes the tail of the Gaussian approximation of the state distribution-this is a suggestion before PILCO searched using the tail of the Gaussian approximation. In contrast.

(Embodiment 3)
The sum-propagation algorithm, like backpropagation, is a general-purpose gradient estimation algorithm for computational graphs, but it overcomes the problem of gradient explosion. An important idea in the algorithm is to combine multiple methods of gradient estimation during the backward path of gradient calculation. Importantly, multiple gradient estimators are aggregated into a smaller set of gradient estimators (eg, all gradient estimators are combined into a single best gradient estimate), and all of the gradient estimators. Is not separate, but a small set of this gradient estimator is passed backwards. Such a method allows a number of gradient estimation techniques to be combined to improve the accuracy of the gradient estimation in the computational graph without causing the growth of the gradient estimator passed backwards, thereby resulting in good computational efficiency. To realize.

(Explanation of framework and algorithm)
The calculation graph is a set of nodes / vertices V and directed edges E, and defines the computational relationship between the variables at the vertices. Each node i receives a variable from its parent node Pa _{i as an input and calculates an output x i} = f (Pa _i ), where the function f can also be stochastic. Since Pa _i and x _i represent a set of one or more variables, they may be vector-valued or tensor-valued. The variable x _i is passed to the child node of node _i and is written as Ch i. FIG. 16 illustrates a general form of the algorithm. A general form of the algorithm is presented in Algorithm 4, where an important novelty is a combination that includes steps 5 and 6. Sum-propagation is similar to the back-propagation algorithm, where the gradient calculated by applying the chain rule is sent to the back of the graph to calculate the gradient for the entire graph. Standard backpropagation is illustrated in FIG. Sum-propagation modifies this procedure by performing multiple gradient estimates at some nodes, combining gradient estimators, and sending the combined estimates backwards with reference to FIG.

FIG. 17 illustrates a backpropagation algorithm used in all neural network applications in machine learning, as well as many other applications. The sum propagation algorithm uses different gradient estimation techniques to _{obtain multiple estimates of dL / dz 2} (eg, reparameterization and likelihood ratio methods), and these estimates are used for smaller gradient estimators. Modify this procedure by joining them into a set and passing them to the back of the computational graph.

FIG. 18 illustrates a total propagation algorithm when gradient estimation is performed by combining likelihood ratios and reparameterized gradient estimators to result in a single gradient estimator. This simply generalizes combining three or more gradient estimators to a number less than the sum of the gradient estimators, and sending the combined gradient estimators backwards.

Claims (25)

A gradient estimation method that includes a calculation graph and estimates the slope of another variable with respect to one variable in the calculation graph.
At some nodes in the calculation graph, perform two or more different estimates of the same gradient with different gradient estimators and combine the different estimates so that they are less than the number of initial estimates. A gradient estimation method in which the combined estimates are passed to different nodes in the calculation graph and the gradient estimates are used for further calculations. The different estimates of the gradient are combined on the basis of the weighted average, and the weight of the weighted average is the variance of the gradient estimates of some other variables in the calculation graph relative to some variables in the calculation graph. The gradient estimation method according to claim 1, which is calculated based on an explicit or implicit estimate of. The gradient estimation method according to claim 2, wherein the weight is set in proportion to the magnitude of the reciprocal of the variance. The gradient estimation method according to any one of claims 1 to 3, wherein the gradient estimator is a likelihood ratio and a reparameterized gradient estimator. The gradient estimation method according to any one of claims 1 to 4, wherein the gradient is used for optimizing parameters in the calculation graph. A gradient estimation method that includes a computational graph and estimates the gradient of a variable relative to a variable in the computational graph, the purpose for both the likelihood ratio and the reparameterization method at some nodes in the computational graph. A gradient estimation method that estimates the gradient of a function and uses both estimates to optimize the parameters in the calculation graph. The gradient estimation method according to claim 6, wherein the likelihood ratio and reparameterization gradient estimators are combined based on a weighted average and the weights are proportional to the reciprocal of the variance of each gradient estimator. The gradient estimation method according to any one of claims 1 to 7, wherein the calculation graph corresponds to a policy search, reinforcement learning, machine learning, or neural network calculation graph. The gradient estimation method according to any one of claims 1 to 8, wherein the combined estimated value is passed to the preceding node in the calculation graph. The gradient estimation method according to any one of claims 6 to 9, wherein the parameter optimization method is a gradient descent or ascending optimization method. The gradient estimation method according to any one of claims 1 to 10, wherein the further calculation is a further gradient estimation of some other variable with respect to some variable. The gradient estimation method according to any one of claims 5 to 11, wherein the combination of the gradient estimates is determined based on the gradient by the previous optimization step. A gradient estimation method that includes a calculation graph and estimates the gradient of another variable with respect to one variable in the calculation graph. The gradient estimation method is a probability at some nodes in the calculation graph. Assuming a parametric form of density, the parameters of the probability density are estimated from the sampled calculations in the calculation graph, the gradient of the expected variable of the node depending on the current variable is estimated, and the expected value is over the entire estimated distribution. A gradient estimation method obtained by multiplying the gradient with some statistical values at the node to obtain a scalar variable, and using the scalar variable to obtain a probability ratio gradient estimator. The gradient estimation method according to claim 13, wherein the parametric form of the probability distribution is a Gaussian distribution. The order of multiplying the gradient with the statistical value and obtaining the likelihood ratio gradient estimate is such that the likelihood ratio gradient estimation is performed prior to the multiplication of the estimated parametric probability distribution with the gradient. The gradient estimation method according to claim 13 or 14, which is replaced. A gradient estimation method according to a combination of the gradient estimation method according to any one of claims 1 to 12 and the gradient estimation method according to claims 13, 14 or 15. An apparatus that executes the gradient estimation method according to any one of claims 1 to 16. A computer program that executes the gradient estimation method according to any one of claims 1 to 16. It is a policy search method in reinforcement learning,
Estimating the gradient of the average total reward for policy parameters, by combining the reparameterization method and the likelihood ratio method at each of the gradient backpropagation steps opposite to the direction in which the state transition occurs according to the policy and dynamics. Estimate the gradient of the average total reward for the parameters
The policy parameters are updated according to the evaluation result.
Policy search method. Further, the weight of the weighted average is set based on the variance of the desired gradient with respect to the policy parameter.
The policy search method according to claim 19. The weights assigned to the gradient estimators according to the re-parameterization method and the likelihood ratio method are set in proportion to the reciprocal of the variance of each gradient estimator.
The policy search method according to claim 20. A policy search device for reinforcement learning
Compute the state in a discrete-time system,
By combining the reparameterization method and the likelihood ratio method at each of the gradient backpropagation steps opposite to the direction in which the state transition occurs according to the policy and dynamics, the gradient of the average total reward for the policy parameters is estimated.
The policy parameters are updated according to the evaluation result.
Policy search device. Further, the weight of the weighted average is set based on the variance of the desired gradient with respect to the policy parameter.
The policy search device according to claim 22. The weights assigned to the gradient estimators according to the re-parameterization method and the likelihood ratio method are set in proportion to the reciprocal of the variance of each gradient estimator.
The policy search device according to claim 23. On the computer
Compute the state in a discrete-time system,
By combining the reparameterization method and the likelihood ratio method at each of the gradient backpropagation steps opposite to the direction in which the state transition occurs according to the policy and dynamics, the gradient of the average total reward for the policy parameters is estimated.
The policy parameters are updated according to the evaluation result.
A computer program that lets a computer perform processing.

Images