JP7378836B2 - Summative stochastic gradient estimation method, apparatus, and computer program - Google Patents

Description

The present invention relates to a method for estimating the slope of a variable defined in a computational graph, an apparatus for performing the estimation, and a computer program.

Most machine learning problems involve optimizing the expectation value of an objective function J(x; θ) over some data-producing distribution p _Data (x), but this _distribution is accessible.

The most common optimization method is gradient descent using pathwise derivatives calculated by backpropagation.

Bengio, Y., Simard, P., and Frasconi, P. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994 Deisenroth, Marc Peter, Neumann, Gerhard, Peters, Jan, et al., A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2):1-142, 2013 Deisenroth, Marc Peter, Fox, Dieter, and Rasmussen, Carl Edward. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408-423, 2015

In some situations (particularly with very long computational graphs or recurrent computational graphs), this approach can also fall into random walks due to exploding gradient variance. Usually, this phenomenon is considered to be a numerical problem that leads to an increase in steps and instability in learning (see Non-Patent Document 1).

The purpose of the invention is to solve the problems associated with gradient estimation. The present invention is a general gradient estimation method that can be used for arbitrary computational graphs as an alternative to backpropagation algorithms.

Gradient estimation methods involve a computational graph and estimate the slope of one variable with respect to other variables in the computational graph, such that at several nodes in the graph, separate slope estimators are used to estimate the slope of the same slope. Perform two or more separate estimates, combine the separate estimates to less than the initial number of estimates, pass the combined estimates to different nodes in the graph, and use the gradient estimate for further calculations. used.

According to the present application, it is possible to provide an alternative flexible framework for gradient evaluation that is more accurate and does not suffer from gradient explosion.

1 is a block diagram showing the hardware configuration of a computing device 1 according to the present embodiment. FIG. FIG. 2 is a diagram illustrating a policy gradient evaluation algorithm by PILCO. FIG. 2 is a diagram illustrating a summation propagation algorithm. 2 is a flowchart illustrating a procedure executed by the computing device 1 according to the present embodiment. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. This is a graph of dispersion. This is a graph of dispersion. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. 11 is a diagram showing an example of a route in Formula 11. FIG. 11 is a diagram showing an example of a route in Formula 11. FIG. FIG. 6 is a diagram showing probability calculation graphs for model-based and model-free LR slope estimation; FIG. 6 is a diagram showing probability calculation graphs for model-based and model-free LR slope estimation; FIG. 7 is a diagram showing Algorithm 3 for explaining in detail how it is compatible with summation propagation. It is a figure which shows the calculation path in a Gaussian shaping gradient. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 3 is a diagram showing experimental results. FIG. 2 is a diagram showing the general form of the algorithm. FIG. 2 illustrates a backpropagation algorithm used in all neural network applications in machine learning, as well as many other applications. FIG. 3 illustrates a summation propagation algorithm where gradient estimation is performed by combining likelihood ratios and reparameterized gradient estimators into a single gradient estimator.

(Embodiment 1)
FIG. 1 is a block diagram showing the hardware configuration of a computing device 1 according to this embodiment. The computing device 1 according to this embodiment is an information processing device such as a personal computer or a server device. The computing device 1 comprises a control unit 11 , a storage unit 12 , an input unit 13 , a communication unit 14 , an operating unit 15 and a display unit 16 . The computing device 1 is described in "PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos", "Total Propagation Algorithm: Supplementary notes", and "Total stochastic gradient algorithms and applications in reinforcement learning" by the present inventors. Implements the disclosed method.

The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The ROM of the control unit 11 stores control programs and the like that control the operations of each part of the hardware. The CPU of the control unit 11 executes a control program stored in the ROM and various programs stored in the storage unit 12, which will be described later, to control the operation of the hardware as in the method disclosed in the above-mentioned paper. The RAM of the control unit 11 stores data temporarily used when executing various programs.

Note that the control unit 11 is not limited to the above configuration, and may be one or more processing circuits or arithmetic circuits including a single-core CPU, multi-core CPU, GPU (Graphics Processing Unit), microcomputer, volatile or non-volatile memory. There may be. The control unit 11 may also include functions such as a clock that outputs data and time information, a timer that measures the elapsed time from the application of the measurement start command until the measurement end command is given, and a counter for counting. good.

The storage unit 12 includes a storage device using an SRAM (Static Random Access Memory), a flash memory, a hard disk, or the like. The storage unit 12 stores various programs executed by the control unit 11, data necessary for executing the various programs, and the like. Examples of programs stored in the storage unit 12 include computer programs implementing the technology disclosed in the above-mentioned paper.

The program stored in the storage unit 12 may be provided by a recording medium M on which the program is readably recorded. The recording medium M is a portable memory such as an SD (Secure Digital) card, a micro SD card, or a compact flash (registered trademark). In this case, the control unit 11 can read the program from the recording medium M using a reading device (not shown) and install the read program into the storage unit 12. Furthermore, the program stored in the storage unit 12 may be provided by communication via the communication unit 14. In this case, the control unit 11 can acquire the program through the communication unit 14 and install the acquired program in the storage unit 12 .

The input unit 13 has an input interface for inputting various data into the device. The control unit 11 obtains data to be processed through the input unit 13 .

The communication unit 14 includes a communication interface for connecting to a communication network (not shown) such as the Internet, and transmits various types of information notified to the outside, and receives various types of information transmitted from the outside. Receive. In this embodiment, the data to be processed is acquired through the input unit 13, but the data to be processed may be acquired through the communication unit 14.

The operation unit 15 includes a user interface such as a keyboard and a touch panel, and receives various operation information and setting information. The control unit 11 executes appropriate control based on the operation information input from the operation unit 15, and stores setting information in the storage unit 12 as necessary.

The display unit 16 includes a display device such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel, and displays information to be notified to the user based on a control signal output from the control unit 11.

In this embodiment, the configuration disclosed in the above-mentioned paper is realized by software processing executed by the control unit 11. ) etc. may be installed separately from the control unit 11. In this case, the control unit 11 implements the method disclosed in the above paper in the hardware by sending the data to be processed inputted from the input unit 13 to the hardware.

Furthermore, in this embodiment, the computing device 1 is described as a single device for simplicity, but it may be configured by multiple computing devices or by one or more virtual machines. may be configured.

In this embodiment, the computing device 1 includes an operation unit 15 and a display unit 16, but the operation unit 15 and display unit 16 are not essential. For example, the computing device 1 may accept operations through an externally connected computer and output information to be notified to the external computer.

The gradient estimation method of the present invention will be explained below. In the formulas below, lowercase letters represent scalars and bold letters represent vectors or matrices. However, in the following description, lowercase letters and bold letters are shown without distinction. Furthermore, in the following description, "C_^" represents a character with a hat, and "C_~" represents a character with a tilde.

(2.1 Policy search)
For a general overview of policy search methods, refer to Non-Patent Document 2. Note that policy search is just one application of the algorithm, and is not limited to a particular computational graph, but can be applied to any computational graph. Consider a discrete time system described by a state vector x _t (eg, robot position and velocity) and an applied motion/control vector u _t (eg, motor torque). An episode begins by sampling states from a fixed initial state distribution x ₀ ~p(x ₀ ). The policy π _θ determines the applied action u _t ~p(u _t )=π(x _t ; θ). Application of the action causes a state transition according to an unknown dynamics function x _t+1 ~p(x _t+1 )=f(x _t , u _t ). Both policies and dynamics may be stochastic and non-linear. The operations and state transitions are repeated up to a maximum of T time steps to generate a trajectory τ: (x ₀ , u ₀ , x ₁ , u ₁ , . . . , x _T ). Each episode is scored according to the return function G(τ). The return is often decomposed into the sum of costs for each time step G(τ)=Σ _t=0 ^T c(x _t )(t=0,...,T), where c(x ) is the cost function. The goal is to optimize the policy parameter θ to minimize the expected return J(θ)=E _r~p(τ;θ) [G(τ)]. Here, the value V _h (x)=E _t=h ^T [Σc(x _t )] is defined.

Learning occurs alternately between executing a policy on the system and subsequently updating θ to improve performance on subsequent trials. In the policy gradient method, the gradient d/dθ·J(θ) of the objective function is directly estimated and used for optimization. Some model-based policy search methods use all the data to learn a model of f, denoted f_^, and use this for "mental rehearsal" between trials to optimize the policy. Hundreds of mock trials can be run for every real trial, greatly increasing data efficiency. Here, we exploit the fact that by differentiating f_^, we can obtain a gradient estimator that is better than the model-less algorithm. The model in this case is probabilistic and predicts the state distribution.

(Stochastic gradient estimation)
Here, how to calculate the gradient d/dθE of the expected value of any function φ(x ₎ with respect to the parameters of the sampling distribution I will explain about it.

(Reparameterization Gradient (RP))
Consider sampling from a univariate Gaussian distribution. One approach, after sampling with zero mean and unit variance ε~N(0,1), maps this point to replicate samples from the desired distribution (x=μ+σε). Here, it is easy to differentiate the output with reference to the distribution parameters. That is, dx/dμ=1 and dx/dσ=ε. The averaging of the samples dφ/dx·dx/dθ gives an unbiased estimate of the slope of the expected value. This is a normally distributed RP slope. In the case of a multivariate Gaussian distribution, the Cholesky factor (L, s.t.Σ=LL ^T ) of the covariance matrix can be used instead of σ.

(Likelihood ratio gradient (LR))
The desired gradient can be written as d/dθ·E _x~p(x;θ) [φ(x)]=∫dp(x;θ)/dθφ(x). In general, from the distribution q(x) by executing ∫φ(x)dx=∫q(x)φ(x)/q(x)dx=E _x~q [φ(x)/q(x)] Any function can be integrated by sampling. The likelihood ratio gradient is obtained by extracting q(x)=p(x) and directly integrating it as follows.

LR gradients are often highly dispersive and need to be combined with variance reduction techniques known as control variables (Greensmith et al., 2004). In a general method, a constant reference value b is subtracted from the function value to obtain the estimated amount E _x~p [d/dθ·(log p(x;θ))(φ(x)−b)]. If b is independent of the sample, this can significantly reduce the variance without introducing bias. In practice, the sample average is a good choice (b=E[φ(x)]). When estimating a gradient from a batch, an unbiased gradient estimator can be obtained by estimating a reference value for each point. That is, b _i =Σ _j≠i ^P φ(x _j )/(P-1).

(trajectory gradient estimation)
The probability density p(τ)=p(x ₀ , u ₀ , x ₁ , u ₁ , ..., x _T ) of observing a specific trajectory is p(x ₀ )π(u ₀ |x ₀ )p (x ₁ |x ₀ , u ₀ )...p(x _T |x _T-1 , u _T-1 ).

Using the RP gradient requires knowing or estimating the dynamics p(x _t+1 |x _t | _ut ). In other words, it is applicable in the case of model criteria. According to such a model, the predicted trajectory can be differentiated using the chain rule.

Note that to use the LR gradient, log p(τ) can be converted to a sum since p(τ) is a product. It is expressed as G _h (τ)=Σ _t=h ^T c (x _t ). Note that (1) only the motion distribution is determined by the policy parameters, and (2) the motion does not affect the cost determined in the past time step, and the following gradient estimator is obtained.

(PILCO)
FIG. 2 is a diagram illustrating a policy gradient evaluation algorithm by PILCO. We follow the original PILCO here, which uses a Gaussian process dynamics model to predict state changes from one time step to the next. That is, p(Δx _t+1 ^a )=gP(x _t , _ut ) (where x∈R ^D , u∈R ^F , Δx _t+1 ^a =x _t+1 ^a −x _t ^a ). A separate Gaussian process is learned for each dimension a. Here, the squared exponential covariance function k _a (x_~, x'_~) = s _a ² exp (-(x_~-x'_~) ^T Λ _a ^-1 (x_~-x'_~)) use. However, s _a and Λ=diag([l _a1 , l _a2 , . . . , l _aD+F ]) are hyperparameters of the function variance and length scale, respectively. Also, a Gaussian likelihood function with a noise hyperparameter σ _n is used. Hyperparameters, through training, maximize the marginal likelihood. Upon sampling from these models, the prediction has the form y=f_^(x)+ε, where ε~N(0,σ _f ² (x)+σ _n ² ). Here, σ _f ² represents the model uncertainty and is due to the lack of data in the region. On the other hand, σ _n ² is learned unique model noise. The trained model noise is not necessarily the same as the actual observed noise σ _o ² in the system. In fact, the latent state is not modeled and the system is approximated by predicting the next observation given the current observation. Furthermore, there are additional sources of variance in the trajectory, and different starting positions result in different trajectories.

(Moment matching prediction)
In general, if a Gaussian distribution is mapped by a nonlinear function, the output is unwieldy and non-Gaussian. However, in some cases it is possible to analytically evaluate the moments of the output distribution. Moment matching (MM) approximates the output distribution as a Gaussian distribution by matching the mean and variance to the true moments. Note that even if the state dimension is modeled with a separate function fa_^, the MM is performed integrally and the state distribution may include covariance.

(Particle prediction)
In general, particle trajectory prediction is simple: predict at every particle position, sample from the output distribution, and iterate. However, we also apply a neural network dynamics model to PILCO by comparison with a method based on Gaussian resampling (GR).

(Gaussian resampling (GR))
MM is probabilistically replicable. At each time step, the particle's mean μ_^=Σ _i=1 ^P x _i /P and variance Σ_^=Σ _i=1 ^P (x i - μ_^)(x _{i -} μ_^) ^T /(P-1 ) is estimated. The particles are then resampled from the fitted distribution x' _i ~μ_^+Lz _i |z _i ~N(0,I), where L is the Cholesky factor of Σ_^. It is not easy to find the gradient dL=dΣ_^. Here we will use the given symbolic representation.

(Hybrid gradient estimation technology)
In the case of the present invention, RP gradients can be used. However, surprisingly, the RP slope is hopelessly inaccurate (see Figure 5D). To solve this problem, we obtained a new slope estimator that combines the model derivative with the LR slope. In particular, the method of the present invention makes it possible to improve sampling efficiency by performing focused sampling within a batch.

(LR based on model)
The distribution on the predicted trajectory is p(τ)=p(x ₀ )π(u ₀ |x ₀ )f_^(x ₁ |x ₀ ,u ₀ )...f_^(x _T |x _T-1 , u _T-1 ). Also, by using a deterministic policy, it is possible to combine a model and a policy as shown in p(x _t+1 | x _t )=f_^(x _t+1 | x _t , π(x _t ; θ)), but this is differentiable (dp _t+1 /dθ=dp _t+1 /du _t ·du _t /dθ). The slope of the model criterion is derived as follows.

(Batch weighted LR (BIW-LR))
Here, we use parallel computation to sample multiple particles at the same time. The state distribution is expressed as a mixture distribution q(x _t+1 )=Σ _i=1 ^P p(x _t+1 |x _i,t ; θ)/P. Similar to the derivation of LR, for each time step, a low variance estimator can be derived by weighted sampling within the batch as follows.

The following equation is used to estimate the one-out average return using normalized weighted sampling.

However, c _j,t+1 =p(x _j,t+1 |x _i,t )/Σ _k=1 ^P p(x _j,t+1 |x _k,t ). Without normalization, the high variance of the reference value estimates will result in poor LR slopes. Note that while the P reference value is calculated for each time step, there is a ^P2 component in the gradient estimator. In order to find the true unbiased gradient, a one-pick reference value of P ² (one for each particle of each mixture component of the distribution) shall be calculated. This specification includes an evaluation using only the reference values presented here (which has been found to remove most of the bias).

(RP/LR weighted average)
Most of the calculation is spent on the dp(x _t+1 |x _t ; θ)/dθ term. Since these terms are needed for both the LR and RP slopes, there is no penalty for combining both estimators. According to well-known statistical results, for independent estimators, an optimal weighted average estimate is achieved if the weights are proportional to the inverse variances. That is, μ=μ _LR k _LR + μ _RP k _RP (where k _LR =σ _LR _^ ^-2 /(σ _LR _^ ^-2 +σ _RP _^ ^-2 ) and k _RP =1-k _LR ). .

A simple combination method would calculate the gradients of the entire trajectory for both estimators separately and then combine them, but this method uses reparameterized gradients for short parts of the trajectory, Opportunities for better gradient estimates are ignored. Our new sum propagation algorithm (TP) outperforms this simple method. In TP, a single backward pass computes the union over all possible RP depths, so low variance estimators are automatically given greater weight.

FIG. 3 is a diagram illustrating the summation propagation algorithm. In Algorithm 2, at each backward step, we evaluate the gradient against the policy parameters by using both LR and RP methods. We also evaluate the ratio based on the variance in the policy parameter space, which is proportional to the variance of the policy gradient estimator. The gradients are combined and the best estimate in the distribution parameter space is passed to past time steps. In this algorithm, the sample variance of the gradient estimate is extracted from particles with different V operators, but other variance estimation methods are also possible; for example, it is also possible to estimate the variance from a moving average of the gradient magnitude. However, it is also possible to use different statistical estimators for the variance, or only a subset of policy parameters. This algorithm is not limited to RL problems, but can also be applied to general probabilistic calculation graphs, and can also be used for training probabilistic models, probabilistic neural networks, etc. In a typical computational graph setting, multiple gradient estimators may be combined at some nodes in the graph by propagating the gradient backward through the graph. In this case, if we reduce the time step parameter t by 1, this will correspond to a backward movement of the node in the graph, while propagating the gradient. Statistical values such as variance used in determining the gradient estimator using the combination method may be obtained from any other node in the calculation graph.

FIG. 4 is a flowchart illustrating a procedure executed by the computing device 1 according to the present embodiment. Computing device 1 executes the following process according to algorithm 2.

The control unit 11 initializes various parameters (step S101). Specifically, the control unit 11 sets dG _T+1 /dζ _T+1 =0, dJ/dθ=0, and G _T+1 =0. where ζ is a distribution parameter (eg, μ and σ).

The control unit 11 sets the time (time step) t to T (step S102), and executes the following calculation for each particle i (step S103). However, c _t is the cost at time t.

The control unit 11 executes the following calculation using the calculation result of Equation 6 (step S104).

Further, the control unit 11 uses the calculation result of Equation 6 to perform the following calculation for each particle i (step S105).

Next, the control unit 11 determines whether the time t has reached a predetermined time 1 (step S106). If the time t has not reached time 1 (S106: NO), the control unit 11 decrements the time t by 1 (step S107) and returns the process to step S103.

(Policy optimization)
Note that any optimization procedure based on gradients can be used, but in this embodiment, a stochastic gradient descent method such as RMSprop is used (an algorithm derived from RMSprop is used). RMSprop uses a moving average of the squared gradient to normalize its SGD steps. In the case of the present invention, since the batch size is large, the expected value of the square is directly estimated from the batch using z=E[g ² ]=E[g] ² +V[g] (where g is the gradient). Also, use the variance of the mean. That is, V[g] is the variance divided by the number of particles P. The gradient step will be g/z ^1/2 . Also, the momentum of the parameter γ is used. The fully updated equation looks like this:

Fixing the random number seed allows us to turn a stochastic problem into deterministic, also known in the RL community as the PEGASUS trick. If the seed is fixed, the RP gradient is the exact gradient of interest and a deterministic pseudo-Newton optimizer such as BFGS can be used.

(experiment)
Experiments were conducted for two purposes: (1) to explain why the RP gradient is not sufficient, and (2) to demonstrate that our newly developed method can be comparable to PILCO in terms of learning efficiency. To show.

(plot the value landscape)
5A-5F illustrate experimental results. We perturb the policy parameter θ in a randomly selected fixed direction and plot the objective function and the magnitude of the projection gradient as a function of Δθ. The results of this experiment, perhaps the most novel part of this paper, led to the term "the curse of chaos."

The plots were generated with a non-linear cart-pole task. 1000 particles were used, while the random number seed was kept fixed to demonstrate that the high dispersion in Figure 5D is not caused by randomness, but rather due to the chaos-like properties of the system. The confidence interval is estimated by Var/P, where Var is the sample variance and P is the number of particles. As described below, a more principled approach is used to plot the dependence of the variance on P.

Figure 5D contains an unusual result, where the RP gradient behaves well in some regions, but when the policy parameters are perturbed, the dispersion explodes due to phase transition-like changes. The variance at Δθ=1.5 is ~4×10 ⁵ times that of Δθ=0, meaning that 4×10 ⁸ particles are required for the RP gradient to be accurate in this region. In practice, a simple random walk is derived by optimizing with the RP gradient.

Since the seed is fixed, the RP slope in Figure 5D is the exact slope of the value in Figure 5A. Therefore, there is extremely small deterministic "noise" on the right side of FIG. 5A. However, a value that is averaged over 1000 particles is not the real goal, as it requires averaging over an infinite number of particles. If you average an infinite number of particles, will there still be "noise"? Or will the function be smooth?

The new slope estimators in Figures 5E and 5F suggest that the true objective is indeed smooth. To provide further evidence, we estimated the magnitude of the slope from the finite difference of the values in Figure 5A using a sufficiently large perturbation in θ such that the "noise" was negligible. The fact that two separate methods (one that varies the policy parameter θ, the other that keeps θ fixed but estimates the gradient from the trajectory) agree is convincing that the real objective is smoothness. Give some evidence.

Figures 5B and 5C explain the reason for the explosion in dispersion when using RP gradients. FIG. 5B corresponds to the leftmost parameter setting, and FIG. 5C corresponds to the rightmost parameter setting. The plot shows how the value V(x; θ) (remaining cumulative cost) varies as a function of position x. Note that since the random number seed is fixed, the value V is the same as the residual return G. The drawings were created by predicting the trajectory of each point for four particles with different fixed seeds and averaging the cost of the trajectories. After trying 1 particle, I ended up predicting 4 particles, for which the values seemed to include some staircase-like parts, but otherwise weren't very interesting compared to the current drawing. Since the average value of 4 particles is unstable, at least one of the 4 particles must have been highly unstable within the region shown.

A square is centered from the center of the initial state distribution to the average prediction. The axes of the squares are slightly different, but this is because when θ changes, the predicted position p(x1; θ) changes. The length of the side corresponds to 4 standard deviations of the Gaussian distribution p(x1; θ). The speed remained fixed at the average value.

RP estimates d/dθ ∫p(x ₁ ; θ)V(x ₁ )dx. This samples the points inside the square, calculates the slope dV/dθ=dV/dx·dx/dθ, and averages it with the samples. In FIG. 5C, finding the slope of the expected value by differentiating V is completely hopeless. In contrast, the LR gradient (FIG. 5E) uses only the value V, rather than its derivative, and does not suffer from this problem. TP (Fig. 5F) effectively combines both estimators.

We will not show the plotted values and slopes for the case of Gaussian resampling, but in the end both of these were smooth functions for a fixed random seed. Therefore, resampling is also effective against the "curse of chaos."

6A and 6B are graphs of dispersion. In FIGS. 6A and 6B, we plot how the variance of the gradient estimator at Δθ=0 and Δθ=1.5 depends on the number of particles P. Variance was calculated by repeatedly sampling the estimator many times and calculating the variance from the set of ratings. RP, TP and LR slopes are compared both with and without batch weighted weighting (BIW) to show that our weighted sampling scheme reduces variance. Used weighted sampling criteria - In practice, normal LR gradients use simpler criteria and have much higher variance. The RP slope is omitted in FIG. 6B because the variance was between 10 ⁸ and 10 ¹⁵ . A TP gradient combined the BIW-LR and RP gradients.

The results confirm that BIW significantly reduces dispersion. Moreover, the TP algorithm of the present invention was the best. Importantly, in Figure 6B, the variance of the RP gradient for all trajectories is 10 ⁶ larger than the other estimators, but TP is reduced by 10-50% for fewer than 250 particles utilizing short path length RP gradients. We have obtained a certain amount of variance. This is a remarkable result because if the gradient estimators are computed separately, the highest possible precision for the combined estimator will be the sum of the precisions of the separate estimators. However, since the summation propagation algorithm of the present invention utilizes the graph structure of computation, it achieves higher accuracy than summation.

(Learning experiment)
We compare PILCO for episodic learning tasks with the following particle-based methods: RP, RP with fixed seed (RPFS), Gaussian resampling (GR), GR with fixed seed (GRFS), and batch model-based. weighted likelihood ratio (LR), and sum propagation (TP). Furthermore, we evaluate two variations of particle prediction. (1) TP (TP-σ _f ) that adds only noise at each time step while ignoring model uncertainties. (2) TP (TP+σ _n ) with increased prediction noise. 300 particles were used in all cases.

Recent PILCO papers (Non-Patent Document 3): Cart pole swing up and balance, and unicycle balance, more learning tasks were performed. The simulation dynamics were set the same and other aspects were similar to the original PILCO. 7A, 7B, 8 and 9 illustrate experimental results.

The optimizer was run for 600 policy evaluations between each trial. The SGD learning rate and momentum parameters were α=5×10 ⁻⁴ and γ=0.9. The episode length was 3s for the cart pole and 2s for the unicycle. Note that for the unicycle task, 2s is not sufficient to generalize the policy to long trials, but it can still be compared with PILCO. The control frequency was 10Hz. The cost is of type 1-exp(-(xt) ^T Q(xt)), where t is the target. The output from policy_(x) is constrained by a saturation function sat(u)=9sin(u)/8+sin(3u)/8, where u=π_(x). One experiment consisted of (1;5) random trials, followed by (15;30) learned trials for the cart and unicycle tasks, respectively. Each experiment was repeated 100 times and averaged. Note that each trial was evaluated by running the policy 30 times and averaging, but this was done for evaluation purposes only (the algorithm only accesses one trial). Success was determined by whether the return on the final trial was below a threshold.

(Cart-pole swing-up and balance)
This is a standard control theory benchmark problem. The task consists of pushing the cart back and forth and swinging an upright pendulum to maintain its balance. The state space is expressed as x = [s, β, ds/dt, dβ/dt], where s is the cart position and β is the pole angle. The reference noise levels are σ _s =0.01 m, β=1 deg, σ _ds/dt =0.1 m/s, and σ _dβ/dt =10 deg/s. The noise is corrected by a multiplier k:σ ₂ =kσ _base ² in different experiments. In the original paper, direct access to the true state is considered. To find a similar setting, we set k=10 ⁻² , but we also tested k∈{1,4,9,16}. The policy π_~ is a radial basis function network (Gaussian summation) with 50 basis functions. Consider two cost functions. One is the same as the original PILCO, where x includes sine and cosine and depends on the distance between the tip of the pendulum (Tip) and the position of the tip when the pendulum is balanced. (Tip Cost). Another cost used raw angles and was Q=diag([1,1,0,0]) (Angle Cost). This cost is conceptually different from the Tip Cost because there is only one correct direction for the pendulum to swing up.

(unicycle balance)
The task consists of a unicycle robot balancing, state dimension D=12, and control dimension F=2. Noise was set to a low value. The control π_~ is linear.

(Learning experiment)
PILCO performs well in no-noise scenarios, but when noise is added, the results deteriorate. This deterioration is most likely caused by the accumulation of errors in the MM approximation and was previously observed by Vinogradska et al. (2016) who used quadrature for prediction. Particles do not suffer from this problem, and using TP gradients always outperforms PILCO in high noise conditions.

On the other hand, at low noise levels, the performance of TP and LR is degraded. If all of the particles are sampled from a small area, it becomes difficult to estimate the slope from the change in return (in the limit of delta variance, the LR slope cannot even be evaluated). TP gradients do not suffer from this problem as much because they incorporate information from the RP. Finally, if the prediction uncertainty is very low (eg k=10 ⁻² ), we can consider the model noise as a parameter that affects learning and increase it to obtain a more accurate gradient. See TP+σ _n . However, the model noise variance was multiplied by 100.

In particular, approaches using MM such as PILCO, and GR outperform others when using Tip Cost. The reason may be the multimodality of the objective - in Tip Cost, the pendulum can be swung up from either direction to solve the task; in Angle Cost, there is only one correct direction. Although running MM forces the algorithm to follow a unimodal path, the particle method can nevertheless attempt a bimodal swing-up where some particles come from one side and stop at the other. There is sex. Therefore, the MM may be performing a type of "distributed reward shaping" that simplifies the optimization problem. Such an explanation was previously made by Gal et al. (2016).

Finally, we point out a surprising TP-σ _f experiment. Although the prediction ignores model uncertainties, the method achieves a success rate of 93%. Although it is difficult to explain why learning was successful, we hypothesize that success may be related to the zero prior mean of GP. In regions where there is no data, the mean of the GP dynamics model tends toward 0, meaning that the input control signal has no effect on the particles. Therefore, in order to succeed in policy optimization, particles must be controlled to stay in the area where data exists. Similar results were found by Chatzilygeroudis et al. (2017), who used an evolutionary algorithm to achieve an 85-90% success rate in a cart-pole task even when ignoring model uncertainty. There is.

Most machine learning problems involve optimizing the expectation of an objective function J(x; θ) for some data-producing distribution p _Data (x), which is only accessible through sample data points {x _i }. It is possible. Our predictive framework is similar to a deep model: p(x ₀ ) is the data-generating distribution and p(x _t ; θ) is by passing p _Data (x) through the model layer. Desired. The most common optimization method is SGD using Pathwise derivatives calculated by backpropagation. Our results suggest that in some situations (particularly for very deep or recurrent models), this approach may also fall into random walks due to the explosion of gradient variance.

Gradient explosion has been observed for many years in deep learning research (Doya, 1993; Bengio et al., 1994). This phenomenon is usually considered a numerical problem leading to increased steps and instability of learning. Common countermeasures include gradient clipping, ReLU activation functions (Nair & Hinton, 2010), and smart initialization. Our explanation for this problem is different: the slope not only grows, but the slope variance explodes, which explains how every sample from x _i ~p _Data changes the model parameter θ and changes the entire distribution E This means that essentially no information is given as to whether to increase the desired expected value for _pData [J(x)]. While choosing a good initialization is one way to address this problem, this seems difficult to ensure that the system does not become chaotic during learning. For example, in econometrics, optimal policies can even lead to chaotic dynamics (Deneckere & Pelikan, 1986). Gradient clipping can stop large parameter steps, but it does not fundamentally solve the problem if the gradients become random. Considering that chaos does not occur in linear systems (Alligood et al., 1996), our analysis suggests why piecewise linear activations that are less sensitive to chaos, such as ReLU, work well in deep learning. .

While the deep hypotheses of the invention still have to be verified computationally, several studies have investigated chaos in neural networks (Kolen & Pollack, 1991; Sompolinsky et al., 1988), but this book is still lacking. We believe that our invention suggests for the first time that chaos can degenerate gradients when computed using backpropagation. Among others, Poole et al. (2016) suggested that such properties result in "exponential expressiveness," and believe that this phenomenon could be an alternative to curses.

(Conclusion and future research)
We have discussed the limitations of optimizing expectations using Pathwise derivatives, such as those computed by backpropagation. Furthermore, we show how to counteract this curse by injecting noise into the calculations and using the likelihood ratio trick. Our summation propagation algorithm combines reparameterized gradients for arbitrary stochastic computational graphs with any quantity of other gradient estimators (even gradients computed using value functions can be used) provide an efficient method for There are countless ways to extend our work: better optimization, incorporating natural gradients, etc. The flexible nature of the methods of the invention should facilitate their expansion.

(Embodiment 2)
Provides a definition of a probabilistic computational graph (PCG). Note that the concept of PCG is different from the concept of computational graphs used to describe the summation propagation algorithm, but instead describes a framework for reasoning about gradient estimators. The definition is quite equivalent to that of a standard directed graph model, but is more focused on our method, emphasizing our interest in computing gradients rather than performing inference. ing. The main difference is the explicit inclusion of the distribution parameters ζ, mean μ, and covariance Σ, for example for the Gaussian.

Definition 1 (Probabilistic Computation Graph (PCG))
An acyclic graph with nodes/vertices V and edges E satisfies the following properties:
1. Each node i∈V corresponds to a set of random variables with marginal joint probability density p(x _i ; ζ _i ), where ζ _i is a possibly infinite parameter of the distribution. Note that parameterization is not unique, and any parameterization is acceptable.
2. The probability density of each node conditionally depends on the parent node and is p(x _i |Pa _i ). Here Pa _i is a random variable in the immediate parent of node i.
3. The joint probability density satisfies p(x ₁ , . . . , x _n )=Π _i=1 ⁿ p(x _i |Pa _i ).
4. Each ζ _i is a function of its parent, ζ _i =f(Pz _i ). Here, Pz _i is a distribution parameter in the parent of node i. In particular, p(x _i ; ζ _i =∫p(x _i |Pa _i )p(Pa _i ; Pz _i )dPa _i .

We would like to emphasize that there is nothing stochastic about the mathematical formulation of the present invention. Each calculation may be analytically unwieldy, but it is deterministic. Furthermore, we emphasize that this definition does not exclude deterministic nodes, i.e. the distribution at the nodes can be a Dirac delta distribution (particle mass). We later use this formulation to derive probabilistic estimates of the slope.

(Derivation of theorem)
We are interested in computing the total derivative of the distribution parameter at one node ζ _i with respect to the parameter at another node dζ _i /dζ _j . By iterating over the total differentiation rule, the summation over the path from node j to node i is derived as follows.

This equation applies to any deterministic computational graph and is also well known, for example, in the OJA community. This equation trivially leads to our stochastic gradient theorem and explains that the summation over the path from A to B can be written as the summation of the paths from A to intermediate nodes and from intermediate nodes to B. . 10A and 10B illustrate examples of paths in Equation 11.

Theorem 1 (summation stochastic gradient theorem)
Let i and j be different nodes in some probabilistic computation graph, and let IN be any set of intermediate nodes, which blocks the path from j to i, i.e. IN is such that there is no path from j to i. It does not pass through the nodes in the IN. {a→b} is represented by a set of paths from a to b, {a→b}/c is a set of paths from a to b, and nodes along the path except for b are included in the set c. I can't. In this case, the total differential dζ _i /dζ _j can be written as the following equation.

Equations 10 and 11 can be combined to give the following:

Note that a similar theorem can be derived by swapping rε{j→m}/IN and sε{j→m}/IN, respectively. This leads to the following equation.

Equations 12 and 13 are referred to as the total gradient equations for the second half and the first half, respectively.

(Gradient estimation on graph)
In the previous section, we provided a means to decompose the gradient computation for an entire graph into a gradient computation for a narrower graph, and also provided a method for estimating the gradient for subgraphs. Here, we demonstrate how the gradients for subgraphs can be combined into an estimator for the gradient for the entire graph. The task is to estimate the derivative of the expectation at distal node i for the parameter at node j: d/dζ _j E _xi~p(xi;ζi) [xi]. Since the true ζ is difficult to handle, we estimate the sampling criterion. Consider sampling a subvariance of p(x;ζ). That is, ζ_^ is sampled so that p(x;ζ)=∫p(x;ζ_^)p(ζ_^)dζ_^. This can be written as:

ζ_^ occurs naturally in the tradition sampling procedure. For simplicity of explanation, we further assume that the sampling is reparameterizable, i.e., p(ζ _m _^; ζ _j ) = f(ζ _m _^; ζ _j , z _m ) p(z _m ) do. This can be written as:

The term dζ _m _^/dζj is estimated by the Pathwise derivative estimator. The remaining term d/dζ _{m_} ^Ex _{i~p(xi;ζi_^)} [xi] can be estimated by any other estimator, for example using a jump estimator. If the second estimator is also unbiased, then the entire estimator is unbiased.

In summary, the steps to create a gradient estimator from j to i for the entire graph are as follows:
1. Select a set of intermediate nodes IN that block the path from j to i.
2. Construct a Pathwise derivative estimator from j to intermediate node IN.
3. Construct the total differential estimator from IN to i and apply the chain rule from i to j.

(Relationship to policy gradient theorem)
_In a typical model-less RL problem _, an agent executes operations u~π(u _{t |} (on the contrary, they seek compensation). The agent's goal is to find the policy parameters θ, which optimize the expected return G=Σ _t=0 ^H c _t for each episode. FIGS. 11A and 11B illustrate probability calculation graphs for model-based and model-free LR slope estimation. In the literature, two "gradient theorems" are commonly applied: the policy gradient theorem and the deterministic policy gradient theorem.

Qt_^ corresponds to an estimate of the residual return Σ _h=t ^H−1 c _h+1 from a particular state x when action u is selected. For Equation 16, any estimator is acceptable, even an estimate of the sampling criterion can be used. For Equation 17, Q_^ is a normally differentiable surrogate model. Importantly, for the above equation to be valid, Q_^ must be an estimator and not the true Q. That is, when estimating the gradient, it must be assumed that the policy parameters are only changed for the current time step and remain fixed for subsequent time steps. FIG. 11A shows how these two theorems correspond to the same probabilistic computation graph. Intermediate nodes are the actions selected at each time step. A difference exists in the choice of jump estimator to estimate the total derivative following an intermediate node - the policy gradient theorem uses the LR gradient, whereas the deterministic policy gradient theorem uses the Pathwise derivative for the surrogate model. use.

(new algorithm)
Typically, when estimating gradients for a PCG, one sample is obtained by performing tradition sampling over the entire graph, and for example, for RL problems, the trajectory is sampled. Such samples are called particles. A batch of such samplings can be used to determine the slope estimator. The estimated distribution parameters at a node are given by the set of distribution parameters for each sampled particle ζ_^={ζ _i _^} _i ^P , where P is the number of particles. For example, if the PCG consists of sequential sampling from a Gaussian distribution, ζ _i corresponds to the mean and covariance of the Gaussian at which the particles were sampled at that node. In the following sections, we exploit the option of using a collection of particles to directly estimate different sets of distribution parameters Γ for the marginal distribution.

(Density estimation LR (DEL))
With the following explanation, one can try to estimate the distribution parameter Γ from the sampled collection of particles and apply the LR gradient using the estimated distribution ζ_^. In particular, the density is approximated as a Gaussian by estimating the mean μ_^=Σ _i ^P x _i /P and the variance Σ_^=Σ _i ^P (x _i −μ_^) ² /(P−1). Standard LR tricks can then be used to estimate the slope ΣiP dlogq(x _i )/dθ(G _i -b), where q(x)=N(μ_^,Σ_^) be. To use this method, we have to compute the derivatives of μ_^ and Σ_^ with respect to particle x _i and use the chain rule to propagate the gradients to the policy parameters, which is easy. The new method of the present invention is called the DEL estimator. Importantly, note that q(x) is used to estimate the slope, but not to modify the trajectory sampling in any way. This is in contrast to the case of Gaussian resampling, where particles are resampled from a Gaussian distribution so fitted, modifying the trajectory distribution.
Advantage of DEL: LR slope can be used without introducing noise into the calculation.
Disadvantages of DEL: The estimator is unbiased, which can make density estimation difficult.

(Gaussian forming gradient (GS))
Until now, all RL methods have used the second half of the sum gradient equation (Equation 12). Can we create an estimator that uses the first half of the equation (Equation 13)? FIG. 13 illustrates the computational path in a Gaussian shaping gradient. Figure 13 gives an example of how this can be done. We propose to estimate the density at xm by fitting a Gaussian to the particle. dE[c _m ]=dΓ _m (gray edge) is then estimated by resampling particles from this distribution (or by any other method of integration). This leaves the question of how to estimate dΓ _m /dθ (dotted and thick edges). It is easy to use the RP method. To use the LR method, first apply the second half of the summation gradient equation to dΓ _m = dθ to obtain the term Σ _{r∈{θ→k}/IN} Π _{(p, t)∈r} ∂ζ _t / Find ∂ζ _p (dotted edge) and dΓm/dζ _k (bold edge). In the scenario considered, the first of these terms is a single path and is estimated using RP. The second term is more interesting and is estimated using the LR method. Since we are using a Gaussian approximation, the distribution parameter Γ _m is the mean and variance of x _m , as μ _m = E[x _m ] and Σm = E[x _m x _m ^T ] − μ _m μ _m ^T It can be estimated. The LR gradient estimator for these terms can be determined as follows.

In practice, we estimate the sampling criterion ζ _k _^ and may be concerned that the estimator is conditional on the sample ζ _k _^, but what we are interested in is the estimate that is not conditional. be. Explain that conditional estimates are equivalent. Note that for the variance, since μ _m is an estimate of the non-conditional mean, the overall estimate corresponds directly to the estimate of the non-conditional variance. For the average, apply the iterated expectation rule as follows.

This makes it clear that the conditional slope estimator is an unbiased estimator of the mean slope that is not conditional.

(Efficient algorithm for accumulating gradients)
As a specific example, we consider a model-based policy gradient method and its PCG is given in FIG. In previous work of the present invention, this algorithm was first conceived and critically relies on access to a differentiable probabilistic model of the dynamics. We explain how GS gradients are applied to this situation. For each x _k node, we want to perform an LR jump to all x _m nodes after k and compute the gradient with a Gaussian approximation of the distribution at node m. Accumulate all nodes during the backward pass in a backpropagation-like manner. Note that for each k and path, the gradient can be written as dE[c _m ]/dΓ _m dΓ _m /dζ _k (dζ _k /du _k-1 du _k-1 /dθ). The term dE[c _m ]/dΓ _m・dΓ _m /dζ _k is estimated as dE[c _m ]/dΓ _m z _m d logp(x _k ;ζ _k )/dζ _k , where zm is the upper term x It corresponds to a vector summarizing _m - b _μ , etc. Note that dE[c _m ]/dΓ _m z _m is just a scalar quantity g _m . Therefore, we use an algorithm that accumulates all g sums during the backward pass and sums all m nodes at each k node. FIG. 12 illustrates Algorithm 3 for explaining in detail how it is compatible with summation propagation. The final algorithm essentially just replaces the normal costs/rewards with modified values, and such techniques can be further applied to model-less policy gradient algorithms using stochastic policies and LR gradients. It is possible. Two interpretations of GS: 1. A Gaussian approximation of the marginal distribution is performed at a certain node. 2. Do some type of reward shaping based on the distribution of particles. In particular, it inherently promotes keeping the trajectory distribution unimodal so that all of the particles are concentrated in one "island" of reward rather than having the distribution split across multiple reward regions - this simplifies optimization. It may become.

(experiment)
Based on the paper by PILCO, we conducted a model-based RL simulation experiment. To test the GS method of the present invention and its coupling with summation propagation, cart-pole swing-up and balance tasks were tested. Furthermore, to demonstrate the feasibility of this idea, we tested the DEL method on a simpler cart-pole balance-only task. The particle-based slope with our new estimator was compared to PILCO. In our previous work, we had to modify the cost function in order to obtain reliable results using particles - one of the main motivations for the current experiments was that the original PILCO used (this will be explained in more detail later).

(Model-based policy search background)
Consider a model-criteria analog to model-free policy search methods. The corresponding probabilistic calculation graph is given in FIG. 11B. The notation follows previous work of the present invention. After each episode, all of the data is used to train a separate Gaussian process model for each dimension of the dynamics, such that p(Δx _t+1 ^a )=gP(x _t _~),. Here, x_~=[x _t ^T , u _t ^T ] and x∈R ^D , u∈R ^F. This model is then used to perform "mental simulations" between episodes to optimize the policy via gradient descent. Using the squared exponential covariance function k _a (x_~, x'_~) = s _a ² exp (-(x_~-x'_~) ^T Λ _a ^-1 (x_~-x'_~)) . Also, a Gaussian likelihood function with a noise hyperparameter σ _n,2 ² is used. The hyperparameters {s, Λ, σ _n } are trained by maximizing the marginal likelihood. The prediction takes the form p(x _t+1 ^a )=N(μ(x _t ＿〜), σ _f ² (x _t ＿〜)+σ _n ² ), where σ _f ² (x _t ＿〜) is the model , and is dependent on the availability of data within the region of the state space. In FIG. 11B, partial derivatives from θ to intermediate nodes are estimated with Pathwise derivatives, and total derivatives following intermediate nodes are estimated with jump estimators.

(set up)
A cart-pole consists of a cart that can be pushed back and forth and an attached pole. The state space is [s, β, ds/dt, dβ/dt], where s is the cart position and β is the angle. The control is a horizontal force on the cart. The dynamics were similar to the PILCO paper. The setup follows our previous work.

(Common characteristics in tasks)
The experiment consists of one random episode followed by 15 episodes with the learned policy, and the policy is optimized between episodes. Each episode length was 3 s and the control frequency was 10 Hz. Each task was evaluated separately 100 times with different random number seeds to test reproducibility. Random number seeds were shared between different algorithms. Note that each episode was evaluated 30 times and costs averaged, but this was done for evaluation purposes only - the algorithm only had access to one episode. The policy is optimized using a learning rule such as RMSprop from our previous work, which normalizes the gradient using the sampling variance of the gradient from different particles. For model-based policy optimization, 600 gradient steps were performed using 300 particles for each policy gradient evaluation. The learning rate and momentum parameters are α=5×10 ⁻⁴ and γ=0:9, respectively, which are the same as in our previous work. The output from the policy is saturated by sat(u)=9sin(u)/8+sin(3u)/8, where u=π_~(x). The policy π_~ is a radial basis function network (Gaussian summation) with 50 basis functions and a summation of 254 parameters. The cost function is of type 1-exp(-(xt) ^T Q(xt)), where t is the target. Consider two types of cost functions: 1) Angle Cost, where the cost is a diagonal matrix with Q=diag([1,1,0,0]), 2) Tip Cost, from the original PILCO paper It is a cost and depends on the distance from tip to tip of the pendulum when balanced. These cost functions are conceptually different - with Tip Cost, the pendulum can swing up from either direction, and with Angle Cost, there is only one correct direction. The standard observation noise level is σ _s = 0.01 m, σ _β = 1 deg, σ _ds/dt = 0.1 m/s, σ _dβ/dt = 10 deg/s, and these are σ ² = kσ _base ² . It is modified by the multiplier k∈{10 ⁻² , 1} as follows.

(Cart-pole swing-up and balance)
In this task, the pendulum initially hangs downward and must then swing to balance. We obtained some results from our previous work: 1) PILCO, 2) Reparameterization gradient (RP), 3) Gaussian resampling (GR), 4) Batch with batch weighted reference values. weighted LR (LR); 5) summation propagation (TP) that combines LR and RP; We compared new methods: 6) Gaussian Shaped Gradient (GLR), which uses only the LR component, and 7) Gaussian Shaped Gradient (GTP), which uses summation propagation to combine both LR and RP variables. For a description of the summation propagation algorithm, see our previous work on how to effectively combine multiple gradient estimators for graphs of computations. Furthermore, GTP was tested when the model noise variance was multiplied by 25 (GTP+σn).

(Kart-Paul balance with DEL estimator)
This task is much simpler - the pole is initially upright and must be balanced. Experiments were devised to demonstrate that DEL is feasible and may be useful if further developed. Angle Cost and reference noise level were used.

(result)
Figures 14 and 15 illustrate experimental results. Similar to our previous work, when the noise is low, methods involving LR components fail. However, the GTP+σn experiment shows that injecting noise into the model predictions can solve the problem. The main important result is that GTP matches PILCO in the Tip Cost scenario. In our previous work, one of the concerns was that TP would not match PILCO in this scenario. Just looking at the costs in FIGS. 15B and 15C does not adequately indicate the difference. In contrast, the success rate indicates that TP was also unsuccessful. Success rate was measured both by a threshold (final loss below 15) calibrated in our previous work as well as by visually classifying all experimental runs. Both methods were in agreement. The loss of peak performer in the final episode was TP: 11.14 ± 1.73, GTP: 9.78 ± 0.40, PILCO: 9.10 ± 0.22, which again indicates that TP was significantly worse. It shows. The remaining experiments converged while the peak performer still improved. PILCO still appears to be slightly more data efficient, but the difference has little practical significance due to the small amount of data required. Note also that the variance of TP is smaller in FIG. 15B. The large variance in GTP and PILCO is caused by outliers with large losses. These outliers converge to a local minimum, which takes advantage of the Gaussian tail of the state distribution - this is contrary to previous suggestions that PILCO uses the Gaussian tail to search. It's a contrast.

(Embodiment 3)
The summation propagation algorithm, like backpropagation, is a general-purpose gradient estimation algorithm for computational graphs, but it overcomes the problem of exploding gradients. The key idea in the algorithm is to combine multiple methods of gradient estimation during the backward pass of gradient computation. Importantly, multiple gradient estimates are aggregated into a smaller set of gradient estimators (e.g., all gradient estimators are combined into a single best gradient estimate), and all of the gradient estimators are This small set of gradient estimators is passed backwards, rather than separately. Such a method allows multiple gradient estimation techniques to be combined to increase the accuracy of gradient estimation in the graph of computation without incurring a proliferation of gradient estimators passed backwards, thereby achieving good computational efficiency. Realize.

(Framework and algorithm description)
A computational graph is a set of nodes/vertices V and directed edges E, and defines computational relationships between variables at the vertices. Each node i receives variables from its parent node Pa _i as input and computes an output x _i =f(Pa _i ), where the function f can also be stochastic. Since Pa _i and x _i represent a set of one or more variables, they may be converted into vector values or tensor values. Variable x _i is passed to the child node of node i and is denoted Ch _i . FIG. 16 illustrates the general form of the algorithm. The general form of the algorithm is presented in Algorithm 4, where the key novelty is the combination comprising steps 5 and 6. Summation propagation is similar to the backpropagation algorithm, which computes the gradient across the graph by applying the chain rule and sending the computed gradient backwards through the graph. Standard backpropagation is illustrated in FIG. Sum propagation modifies this procedure by performing multiple gradient estimates at several nodes, combining the gradient estimators, and sending the combined estimators backwards.

Figure 17 illustrates the backpropagation algorithm used in all neural network applications in machine learning, as well as many other applications. The summation propagation algorithm uses different gradient estimation techniques to obtain multiple estimates of dL/dz ₂ (e.g., reparameterization and likelihood ratio methods) and then subdivides these estimates into smaller gradient estimators. We modify this procedure by joining into sets and passing them backwards in the computational graph.

FIG. 18 illustrates a summation propagation algorithm where gradient estimation is performed by combining likelihood ratios and reparameterized gradient estimators into a single gradient estimator. This easily generalizes combining three or more gradient estimators into less than the total number of gradient estimators and sending the combined gradient estimators backwards.

Claims (22)

A computer-based gradient estimation method comprising a calculation graph and estimating the slope of one variable with respect to another variable in the calculation graph, the method comprising:
performing two or more different estimates of the same gradient using different gradient estimators at some nodes in the computational graph;
Combine the different estimates so that the number is less than the initial estimate,
Pass the combined estimates to different nodes in the calculation graph
executing the process by the computer;
Gradient estimation method, where the gradient estimate is used for further calculations. The different estimates of the slope are combined based on a weighted average, the weight of the weighted average being the variance of the slope estimates of some variables in the computational graph versus some other variables in the computational graph. The gradient estimation method according to claim 1, wherein the gradient estimation method is calculated based on an explicit or implicit estimate of . 3. The gradient estimation method according to claim 2, wherein the weight is set in proportion to the magnitude of the reciprocal of the variance. 4. A gradient estimation method according to any one of claims 1 to 3, wherein the gradient estimator is a likelihood ratio and a reparameterized gradient estimator. The gradient estimation method according to any one of claims 1 to 4, wherein the gradient is used for optimizing parameters in the calculation graph. A gradient estimation method according to any one of claims 1 to 5, wherein the combined estimate is passed to a preceding node in the computational graph. A computer-based gradient estimation method that includes a calculation graph and estimates the slope of a variable with respect to a variable in the calculation graph, the method comprising:
Estimate the gradient of the objective function for both the likelihood ratio and reparameterization methods at some nodes in the computational graph;
Optimize the parameters in the calculation graph using both estimators
A gradient estimation method , wherein processing is executed by the computer . 8. The gradient estimation method of claim 7 , wherein the likelihood ratio and reparameterization gradient estimators are combined based on a weighted average, the weights being proportional to the inverse of the variance of the respective gradient estimators. 9. The gradient estimation method according to claim 5, wherein the parameter optimization method is a gradient descent or ascent optimization method. 7. A gradient estimation method according to any one of claims 1 to 6 , wherein the further calculation is a further gradient estimation of some variables with respect to some other variables. A gradient estimation method according to any one of claims 1 to 6 , wherein the combination of gradient estimates is determined based on gradients from previous optimization steps. A computer-based gradient estimation method comprising a calculation graph and estimating the slope of one variable with respect to another variable in the calculation graph ,
At some nodes in the computational graph, assume a parametric form of the probability density at the nodes;
estimating parameters of the probability density from sampled calculations in the calculation graph;
Estimate the gradient of a node's expected variable depending on the current variable
executing the process by the computer;
Expected values are obtained over the entire estimated distribution,
Further, multiplying the gradient by some statistical value at the node to obtain a scalar variable,
Find the likelihood ratio gradient estimator using the scalar variables
A gradient estimation method , wherein processing is executed by the computer . 13. A gradient estimation method according to claim 12 , wherein the parametric form of a probability distribution is a Gaussian distribution. The order of multiplying the gradient by the statistic and determining the likelihood ratio gradient estimator is such that likelihood ratio gradient estimation is performed prior to multiplying the estimated parametric probability distribution by the gradient. The gradient estimation method according to claim 12 or claim 13 , wherein the gradient estimation method is replaced. A gradient estimation method, which is performed by combining the gradient estimation method according to any one of claims 1 to 11 and the gradient estimation method according to claim 12 , 13 , or 14 . The gradient estimation method according to any one of claims 1 to 15, wherein the calculation graph corresponds to a calculation graph of policy search, reinforcement learning, machine learning, or neural network. An apparatus for carrying out the gradient estimation method according to any one of claims 1 to 16. A computer program for executing the gradient estimation method according to any one of claims 1 to 16. A policy search device in reinforcement learning,
compute states in a discrete-time system,
In a gradient backpropagation step opposite to the direction in which state transitions occur according to the policy and dynamics, estimate the gradient of the average total reward with respect to the policy parameters by combining reparameterization and likelihood ratio methods,
updating the policy parameters according to the evaluation results;
Policy search device. further, setting weights for the weighted average based on the variance of the desired gradient for the policy parameters;
The policy search device according to claim 19 . the weights assigned to gradient estimators according to the reparameterization method and the likelihood ratio method are set proportional to the magnitude of the inverse of the variance of the respective gradient estimator;
The policy search device according to claim 20 . to the computer,
compute states in a discrete-time system,
In a gradient backpropagation step opposite to the direction in which state transitions occur according to the policy and dynamics, estimate the gradient of the average total reward with respect to the policy parameters by combining reparameterization and likelihood ratio methods,
updating the policy parameters according to the evaluation results;
A computer program that causes a computer to perform a process.

Images