JP2018124982A - Control device and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control device that is able to exert optimum control by using a neutral network.SOLUTION: A control device 1 for performing optimal control by path integral includes a neural network section 3 including a machine-learned dynamics model and cost function, an input section 2 that inputs a current state of a control target 50 and an initial control sequence for the control target 50 into the neural network section 3, and an output section 4 that outputs a control sequence for controlling the control target 50, the control sequence being calculated by the neural network section 3 by path integral from the current state and the initial control sequence by using the dynamics model and the cost function. Here, the neural network section 3 includes a second recurrent neural network incorporating a first recurrent neural network including the dynamics model.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a control device and a control method, and more particularly to a control device and a control method using a neural network.

  As one of the optimal controls, path integral control is known (for example, see Non-Patent Document 1). Optimal control can be regarded as a framework for prefetching the future state and reward of the system to be controlled and obtaining an optimum sequence of manipulated variables. Optimal control can be formulated as a constrained optimization problem.

  On the other hand, deep neural networks such as convolutional neural networks have been successfully applied and used for controls such as automatic driving and robot operation.

Model Predictive Path Integral Control: From Theory to Parallel Computationhttps: //arc.aiaa.org/doi/full/10.2514/1.G001921. [Search September 29, 2017], Internet <URL: https: // arc .aiaa.org / doi / full / 10.2514 / 1.G001921> Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel, "Value Iteration Networks", NIPS 2016.

  However, in the conventional optimal control such as Non-Patent Document 1, although it is necessary to specify the dynamics of the system and use a cost function in order to predict the future state and future reward of the system, There is a problem that it is difficult to describe the dynamics and the cost function.

  In addition, there is a problem that optimal control cannot be performed using a deep neural network such as a convolutional neural network. This is because deep neural networks such as convolutional neural networks only grow reflexively, no matter how much they learn.

  The present disclosure has been made in view of the above-described circumstances, and an object thereof is to provide a control device and a control method capable of performing optimal control using a neural network.

  In order to solve the above problem, a control device according to an embodiment of the present disclosure is a control device for performing optimal control by path integration, and the control device has a machine-learned dynamics model and a cost function. An input unit that inputs a neural network, a current state of a control target, and an operation amount series that includes a plurality of operation parameters for the control target as components, and an initial operation amount sequence to the neural network; and the neural network An operation amount sequence for controlling the control target, which is an operation amount sequence calculated by path integration from the current state and the initial operation amount sequence using the dynamics model and the cost function. An output unit for outputting, wherein the neural network has a first recurrence having the dynamics model. And a second recurrent neural network with bets neural network therein.

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM. The system, method, integrated circuit, computer You may implement | achieve with arbitrary combinations of a program and a recording medium.

  According to the control device and the like of the present disclosure, optimal control can be performed using a neural network.

It is a block diagram which shows an example of a structure of the control apparatus in embodiment. It is a block diagram which shows an example of a structure of the neural network part shown in FIG. It is a block diagram which shows an example of a structure of the calculation part shown in FIG. It is a figure which shows an example of a detailed structure of the calculation part shown in FIG. It is a figure which shows an example of the detailed structure of the Monte Carlo simulator part shown to FIG. 3B. It is a figure which shows an example of a detailed structure of the 2nd process part shown to FIG. 3B. It is a flowchart which shows the process of the control apparatus in embodiment. It is a figure which shows an example of the conceptual diagram of the learning process in embodiment. It is a flowchart which shows the outline | summary of the learning process in embodiment. It is a figure which shows the control simulation result in experiment. It is a figure which shows a true cost function. It is a figure which shows the cost function of the learned path | route integral control neural network. It is a figure which shows the cost function of the neural network of the comparative example learned. 10 is a block diagram illustrating an example of a configuration of a neural network unit in Modification 1. FIG.

(Background to obtaining one embodiment of the present disclosure)
There is known optimal control that is control that minimizes an evaluation function that represents good control. Optimal control can be regarded as a framework for prefetching the future state and reward of the system to be controlled and obtaining an optimal sequence of manipulated variables. Optimal control can be formulated as a constrained optimization problem.

  Moreover, path integral control is known as one of the optimal controls (see, for example, Non-Patent Document 1). Non-Patent Document 1 describes that path integration control is performed by mathematically solving path integration as a stochastic optimal control problem using Monte Carlo approximation based on stochastic sampling of trajectories.

  However, in the conventional optimal control such as Non-Patent Document 1, it is necessary to use a model that identifies the dynamics of the system and a cost function in order to predict the future state and future reward of the system. It is difficult to describe dynamics and cost functions. This is because, if the model of the system is completely known, the dynamics composed of complicated equations and a large number of parameters can be described, but there are few such cases. In particular, it is difficult to describe a large number of parameters. Similarly, the cost function used to evaluate rewards is described if it fully knows or can fully simulate the change in the overall state of the environment from the current state of the system to the future state. Although it is possible, this is rare. The cost function is a function that describes what state is desirable in order to perform targeted control using parameters such as weights. For this reason, it is particularly difficult to optimally describe parameters such as weights.

  On the other hand, as described above, in recent years, deep neural networks such as convolutional neural networks have been successfully applied and used in controls such as automatic driving and robot operation. Such a deep neural network is trained to output a desired operation amount by imitation learning or reinforcement learning using teacher data.

  Therefore, it is conceivable to perform optimal control using a deep neural network such as a convolutional neural network. If optimal control can be performed using such a deep neural network, it is considered that the dynamics and cost function necessary for optimal control, or these parameters that are particularly difficult to describe, can be learned. .

  However, optimal control cannot be performed using a deep neural network such as a convolutional neural network. This is because such a deep neural network grows only reflexively, no matter how much it learns. That is, such a deep neural network cannot obtain generalization ability such as prefetching no matter how much it is learned.

  In view of the above circumstances, the inventors have come up with a control device and a control method capable of performing optimal control using a neural network.

  That is, a control device according to one embodiment of the present disclosure is a control device for performing optimal control by path integration, and the control device includes a machine-learned dynamics model and a neural network having a cost function, and a control target An input unit for inputting an initial manipulated variable sequence, which is a manipulated variable sequence including a plurality of manipulated parameters for the control target, to the neural network, and the neural network includes the dynamics model and Using the cost function, an output unit that outputs an operation amount sequence for controlling the control target, which is an operation amount sequence calculated by path integration from the current state and the initial operation amount sequence. The neural network includes a first recurrent neural network having the dynamics model. And a second recurrent neural network having a network therein.

  With this configuration, it is possible to cause a neural network including a double recurrent neural network to perform optimal control by path integration, and thus optimal control can be performed using the neural network.

  Here, for example, the second recurrent neural network includes the first recurrent neural network and the cost function, and the first recurrent neural network includes the current state and the initial manipulated variable sequence. Based on a first processing unit that calculates a state at each time by a Monte Carlo method and calculates a cost of the plurality of states using the cost function, the initial manipulated variable series and the cost of the plurality of states, A second processing unit that calculates an operation amount sequence for the control target, and the second processing unit outputs the calculated operation amount sequence to the output unit, and outputs the initial operation amount sequence to the second recurrent neural network. The second recurrent neural network is fed back as a manipulated variable sequence, and the second recurrent neural network is connected to the first processing unit. And fed back manipulated variable sequence by the second processing unit, the may be from the current state to calculate the cost of a plurality of states in each of the following times of the respective times.

  With this configuration, it is possible to cause a neural network including a double recurrent neural network to perform optimum control by path integration by the Monte Carlo method.

  Further, for example, the second recurrent neural network further includes a third processing unit that generates a random number used in the Monte Carlo method, and the third processing unit outputs the generated random number to the first processing unit and the first processing unit. You may make it output to 2 process parts.

  Further, for example, the cost function may be a cost function model configured by a neural network.

  Further, a control method according to an aspect of the present disclosure is a control method for a control device for performing optimal control by path integration, and the control device includes a neural network having a machine-learned dynamics model and a cost function. An input step of inputting an initial manipulated variable sequence, which is a manipulated variable sequence having a plurality of operational parameters for the controlled object as components, to the neural network; and Using the dynamics model and the cost function, an operation amount sequence calculated by path integration from the current state and the initial operation amount sequence, and an operation amount sequence for controlling the control target. An output step of outputting, wherein the neural network has the dynamics model And a second recurrent neural network having one recurrent neural network therein.

  Here, for example, further including a learning step for machine learning of the dynamics model and the cost function before the input step, the learning step being a state prepared in advance corresponding to the current state of the control target And an initial operation amount sequence prepared in advance corresponding to the initial operation amount sequence for the control target, a state prepared in advance, and an initial operation amount sequence prepared in advance. A step of preparing learning data including a manipulated variable sequence for controlling the control target as teacher data, and by using the teacher data to learn the weight of the neural network by an error back propagation method, Learning the dynamics model and the cost function.

  As a result, a neural network composed of a double recurrent neural network can learn the dynamics and cost function necessary for optimal control, or these parameters.

  Here, for example, the cost function may be a cost function model configured by a neural network.

  Each of the embodiments described below shows a specific example of the present disclosure. Numerical values, shapes, components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements. In all the embodiments, the contents can be combined.

(Embodiment)
Hereinafter, a control device, a control method, and the like in the embodiment will be described with reference to the drawings.

[Configuration of Control Device 1]
FIG. 1 is a block diagram showing an example of the configuration of the control device 1 in the present embodiment. FIG. 2 is a block diagram showing an example of the configuration of the neural network unit 3 shown in FIG.

  The control device 1 is realized by a computer or the like using a neural network, and performs optimal control by path integration on the control target 50. The control device 1 includes an input unit 2, a neural network unit 3 and an output unit 4 as shown in FIG. Here, the control target 50 is a control target system that performs optimal control, such as a vehicle that performs automatic driving or a robot that moves autonomously.

<Input unit 2>
The input unit 2 inputs a current state of the control target and an operation amount series including a plurality of operation parameters for the control target as components and an initial operation amount series to the neural network of the present disclosure.

In the present embodiment, the input unit 2 displays the current state of the control target 50.
And an initial manipulated variable sequence having an initial operation parameter for the control object 50 as a component
Are obtained from the control object 50 and input to the neural network unit 3.
Is a time series of manipulated variables from time t_0 to t_ {N-1}.

<Output unit 4>
The output unit 4 is an operation amount sequence calculated by the path integration from the current state and the initial operation amount sequence using the machine model learned dynamics model and the cost function, and the output unit 4 controls the control target. Output a manipulated variable sequence. The dynamics model may be, for example, a dynamics model configured by a neural network or a function represented by a mathematical expression. Similarly, the cost function may be, for example, a cost function model configured by a neural network or a function represented by a mathematical expression. That is, the dynamics and the cost function may be configured by a neural network or a function including a mathematical expression and a parameter as long as machine learning can be performed in advance.

In the present embodiment, the output unit 4 is the initial manipulated variable sequence acquired from the control object 50 by the input unit 2.
Manipulated variable series updated
Is output to the control object 50. In other words, the control device 1 uses the initial manipulated variable sequence.
Based on the operation amount series, which is an optimum operation amount sequence calculated by prefetching the future state and reward of the control object 50
Is output to the control object 50.

<Neural network part 3>
The neural network unit 3 is constituted by a neural network having a machine-learned dynamics model and a cost function. The neural network unit 3 includes a second recurrent neural network having a first recurrent neural network having a machine-learned dynamics model therein. Hereinafter, the neural network unit 3 of the present disclosure may be referred to as a path integration control neural network.

  Then, the neural network unit 3 uses the machine-learned dynamics model and the cost function to calculate a manipulated variable sequence for controlling the controlled object by path integration from the current state and the initial manipulated variable sequence. .

In the present embodiment, the neural network unit 3 includes a calculation unit 13 as shown in FIG. The calculation unit 13 receives the current state of the control target 50 through the input unit 2.
And the initial manipulated variable series for the control object 50
Are entered. The calculation unit 13 uses a machine-learned dynamics model and a cost function to perform an initial manipulated variable sequence by path integration.
The updated operation amount series is calculated. Then, the calculation unit 13 converts the updated operation amount sequence to the initial operation amount sequence.
The operation amount series further updated from the updated operation amount series is calculated. In this way, the calculation unit 13 recursively updates the operation amount sequence, for example, the operation amount sequence for controlling the control target 50 by recursively updating the operation amount sequence U times.
Is calculated.

  The part of the calculation unit 13 that recursively updates the manipulated variable series corresponds to the recurrent neural network 13a. The recurrent neural network 13a is, for example, a second recurrent neural network.

Further, U times is set to a large number so that the updated manipulated variable series sufficiently converges. Assume that the dynamics model is represented by a function f parameterized by machine learning. The cost function model is a function parameterized by machine learning.
, Φ.

  FIG. 3A is a block diagram illustrating an example of a configuration of the calculation unit 13 illustrated in FIG. FIG. 3B is a diagram illustrating an example of a detailed configuration of the calculation unit 13 illustrated in FIG. 2. FIG. 4 is a diagram illustrating an example of a detailed configuration of the Monte Carlo simulator unit 141 illustrated in FIG. 3B. FIG. 5 is a diagram illustrating an example of a detailed configuration of the second processing unit 15 illustrated in FIG. 3B.

  For example, as illustrated in FIG. 3A, the calculation unit 13 includes a first processing unit 14, a second processing unit 15, and a third processing unit 16. For example, as shown in FIG. 3B, the calculation unit 13 further includes a storage unit 17 that stores the initial manipulated variable series input by the input unit, and outputs the storage unit 17 to the first processing unit 14 and the second processing unit 15. May be.

«First processing unit 14»
The first processing unit 14 includes a first recurrent neural network and a cost function, and causes the first recurrent neural network to calculate a state at each time from a current state and an initial manipulated variable sequence by a Monte Carlo method, The cost of a plurality of states is calculated using a cost function model. Further, the first processing unit 14 calculates the cost of a plurality of states at each time next to each time from the operation amount series fed back to the second recurrent neural network by the second processing unit 15 and the current state. To do.

  In the present embodiment, the first processing unit 14 includes a Monte Carlo simulator unit 141 and a storage unit 142, as shown in FIG. 3B.

  The Monte Carlo simulator unit 141 uses a path integration framework that probabilistically samples a plurality of different time series using Monte Carlo simulation. A time series of states is called a trajectory. For example, as shown in FIG. 4, the Monte Carlo simulator unit 141 uses a machine-learned dynamics model 1411 and a random number input from the third processing unit 16 to calculate a current state and an initial manipulated variable sequence. A time series of states having a state at a time after the present as a component is calculated. Furthermore, the Monte Carlo simulator unit 141 inputs the time series of the calculated state again, and updates the time series of the state. As described above, the Monte Carlo simulator unit 141 calculates the state at each time after the present by recursively updating the state time series N times, for example. In addition, the Monte Carlo simulator unit 141 calculates the cost of the Nth time, that is, the state calculated last in the termination cost calculation unit 1412 and outputs the cost to the storage unit 142 as the termination cost.

More specifically, for example, the dynamics model 1411 is
And the cost function model 1413 is
The terminal cost model 1412 is
It is assumed that α, β, R, and γ are parameters of the dynamics model and the cost function model. In this case, first, the Monte Carlo simulator unit 141
State at time ti
Assign to. k is an index indicating one of a total of K states. These K states are processed in parallel. The Monte Carlo simulator unit 141
And initial manipulated variable series
From the above, the dynamics model 1411
And random numbers
And the state at time ti + 1 after time ti
Is calculated. Further, the Monte Carlo simulator unit 141 calculates the state
The state at time ti
Enter again as K states
Update. The Monte Carlo simulator unit 141 is the state calculated by the terminal cost calculation unit 1412 for the Nth time.
Is input to the termination cost model 1412 and the termination cost obtained is
Is output to the storage unit 142.

  In addition, the Monte Carlo simulator unit 141 uses the cost function model 1413 and the random number input from the third processing unit 16 to evaluate the cost of a plurality of states at each time calculated from the initial manipulated variable sequence. Is calculated.

More specifically, the Monte Carlo simulator unit 141 is a cost function model 1413.
And a random number input from the third processing unit 16
And the initial manipulated variable series
From the cost of multiple states at each time calculated from 1 to N-1th
Is output to the storage unit 142 as an evaluation cost.

  The portion of the Monte Carlo simulator unit 141 that recursively calculates a plurality of states corresponds to the recurrent neural network 141a. The recurrent neural network 141a is, for example, a first recurrent neural network. N times indicates the number of time steps to be read ahead.

The storage unit 142 is, for example, a memory, and an evaluation cost that is a cost of a plurality of states at each time for N times.
Is temporarily stored and output to the second processing unit 15.

<< second processing unit 15 >>
The second processing unit 15 calculates an operation amount sequence for the control target at each time based on the initial operation amount sequence and the costs of a plurality of states. The second processing unit 15 outputs the calculated manipulated variable sequence at each time to the output unit 4 and feeds it back to the second recurrent neural network as an initial manipulated variable sequence.

  In the present embodiment, the second processing unit 15 includes a cost integrating unit 151 and an operation system updating unit 152, for example, as shown in FIG.

The cost integrating unit 151 calculates an integrated cost obtained by integrating the costs of a plurality of states at N times stored in the storage unit 142. More specifically, the cost integration unit 151 uses the following (Equation 1) to integrate the costs of a plurality of states at N times stored in the storage unit 142.
Is calculated.

The operation system update unit 152 calculates the initial operation amount from the initial operation amount series, the costs of a plurality of states at each time for N times accumulated by the cost accumulation unit 151, and the random number input from the third processing unit 16. An operation amount sequence for the control target 50 whose sequence has been updated is calculated. More specifically, the operation system updating unit 152 uses the following (Equation 2) to calculate the initial operation amount series.
And the integrated integrated cost calculated by the cost integrating unit 151
And a random number input from the third processing unit 16
From the above, the operation amount series for the control object 50
Is calculated.

«Third processing unit 16»
The third processing unit 16 generates a random number used in the Monte Carlo method. The third processing unit 16 outputs the generated random number to the first processing unit 14 and the second processing unit 15.

  In the present embodiment, the third processing unit 16 includes a noise generation unit 161 and a storage unit 162, as shown in FIG. 3B.

For example, the noise generator 161 converts Gaussian noise to random numbers.
And stored in the storage unit 162.

The storage unit 162 is a memory, for example, and is a random number.
Is temporarily stored and output to the first processing unit 14 and the second processing unit 15.

[Operation of Control Device 1]
An example of the operation of the control device 1 configured as described above will be described below.

  FIG. 6 is a flowchart showing the processing of the control device 1 in the present embodiment. The control device 1 includes a path integration control neural network that is a neural network of the present disclosure. The path integral control neural network has a machine-learned dynamics model and a cost function. The path integral control neural network is a double recurrent neural network. That is, the path integral control neural network is composed of the second recurrent neural network having the first recurrent neural network having the dynamics model therein as described above.

  First, the control device 1 uses a path integral that is a neural network according to the present disclosure to display the current state of the control target 50 and an operation amount sequence that includes a plurality of operation parameters for the control target as initial components. Input to the control neural network (S11).

  Next, the control device 1 controls the path integration control neural network by path integration from the current state input in S11 and the initial manipulated variable sequence using a machine-learned dynamics model and cost function. An operation amount sequence for controlling the object 50 is calculated (S12).

  And the control apparatus 1 outputs the operation amount series for controlling the control object 50 calculated by the said path | route integral control neural network by S12 (S13).

[Learning process]
In the present disclosure, in order to learn the dynamics and cost function necessary for optimal control using a neural network, or these parameters, attention is paid to a path integration controller which is one of the optimal controllers. Since the function formulated to realize the path integral controller is differentiable, the chain rule which is the differential formula of the composite function can be applied. A deep neural network can be interpreted as a composite function that is a huge set of differentiable functions and can be learned by the chain rule. It was found that an arbitrarily shaped deep neural network can be constructed if the principle of differentiation is followed.

  From the above, the path integral controller is formulated with a differentiable function and can be applied with the chain rule, so it can be realized using a deep neural network that can learn all parameters by backpropagation, that is, error backpropagation. I came up with it. More specifically, a recurrent neural network which is one of deep neural networks can be interpreted as a neural network in which the same function is executed a plurality of times in series, that is, the functions are arranged in series. From this, it was conceived that the path integration controller can be expressed by a recurrent neural network.

  This makes it possible to learn the dynamics and cost function necessary for path integration control, or these parameters, using a neural network. Furthermore, as described above, by using a neural network having learned dynamics, cost function, etc., path integral control, that is, optimum control by path integral can be realized.

  Hereinafter, the learning process of the parameters of the dynamics and cost function necessary for the path integral control will be described.

  FIG. 7 is a diagram showing an example of a conceptual diagram of the learning process in the present embodiment. The neural network unit 3b has a dynamics model and a cost function model before learning. By learning these dynamics model and cost function model, it can be applied to the dynamics model and cost function model that the neural network unit 3 constituting the control device 1 has.

  FIG. 7 shows an example in which the neural network unit 3b performs learning processing for learning the dynamics model and the cost function model by the error back propagation method using the teacher data 5. If there is no teacher data, the learning process may be performed using reinforcement learning.

  FIG. 8 is a flowchart showing an outline of the learning process S10 in the present embodiment.

  In the learning process S10, first, learning data is prepared (S101). More specifically, a state prepared in advance corresponding to the current state of the control object 50, an initial operation amount sequence prepared in advance corresponding to an initial operation amount sequence for the control object 50, and a prepared in advance Learning data including an operation amount sequence for controlling a control target, which is calculated in advance by path integration from the state and an initial operation amount sequence prepared in advance, is prepared. In the present embodiment, an operation history of a skilled person including a set of states and operation sequences is prepared as learning data.

  Next, the computer learns the dynamics model and the cost function model by learning the weight of the neural network unit 3b by the error back propagation method using the prepared learning data as teacher data (S102). More specifically, the computer uses the learning data to cause the neural network unit 3b to perform an operation amount by path integration from a state prepared in advance included in the learning data and an initial operation amount sequence prepared in advance. Let the series be calculated. Then, the computer calculates an error, which is a difference between the manipulated variable sequence calculated by the path integration through the neural network unit 3b and the prepared manipulated variable sequence included in the learning data, using a prepared evaluation function or the like. The parameters of the dynamics model and the cost function model are updated so that the error is reduced. Further, the computer adjusts or updates the parameters of the dynamics model and the cost function model until the error evaluated by an evaluation function prepared in advance in the learning process is minimized or no longer fluctuates.

  As described above, the computer evaluates the evaluation function prepared in advance and repeats the updating of the parameters of the dynamics model so that the error is reduced. Let them learn.

  In the present embodiment, by performing the learning process S10 as described above, the neural network unit 3 used in the control device 1 can learn the dynamics model and the cost function model.

  In addition, when the teacher data includes data that includes a set of (state, operation, and next state), the dynamics model can be independently trained by using the data. It is also possible to incorporate an independently learned dynamics model into the neural network unit 3 and fix only the cost function model using the learning process S10 after fixing the parameters of the dynamics model. Since the method of supervised learning of the dynamics model is known, it is omitted.

  Hereinafter, the neural network unit 3 will be described as a path integration control neural network that is a neural network of the present disclosure.

[Verification by experiment]
The effectiveness of the path integral control neural network having the learned dynamics and cost function model was verified by experiments, and the experimental results will be described.

  As a problem of optimal control, there is a swing control of a single pendulum in which a downward single pendulum is shaken and brought to a handstand state. In this experiment, the dynamics and cost function used for pendulum swing-up control were imitated by using expert teacher data and the effectiveness of the pendulum swing-up control was verified by simulation.

<Teacher data>
In this experiment, the expert is assumed to be an optimal controller having true dynamics and a cost function. The true dynamics is given by the following (formula 3), and the cost function is given by the following (formula 4).

  Here, θ is a pendulum angle, k is a model parameter, and u is a torque, that is, an operation input.

<Experimental result>
FIG. 9 is a diagram showing a control simulation result in this experiment.

  In this experiment, the dynamics and the cost function are expressed by a neural network having one hidden layer. Then, after learning the dynamics independently from the teacher data by the method described above, the cost function was learned so as to produce a desired output by the error back propagation method. The processing path integral control neural network that has performed such learning processing is expressed as “Trained” in the Controllers of FIG. 9. Further, a path integration control neural network in which the dynamics is independently learned with the above-described teacher data and the true cost function shown in (Equation 4) is given without performing the cost function learning is “Freezed” in the Controllers of FIG. It expresses. On the other hand, VIN (Value Iteration Network) shown in Non-Patent Document 2 is expressed as a comparative example in the Controllers of FIG. As shown in Non-Patent Document 2, VIN is a neural network in which a state transition model and a reward model are learned by an error back propagation method. In this experiment, VIN was trained using the above teacher data with the state transition model as the dynamics and the reward model as the cost function.

Further, the item MSE For D _train shown in FIG. 9 indicates an error with respect to the teacher data, and the item MSE For D _test shown in FIG. 9 indicates an error with respect to the evaluation data, that is, a generalization error. The item Success Rate shown in FIG. 9 indicates the success rate of the swing-up, and the case where the control is actually controlled and the successful swing-up is successful is shown as a success rate of 100%. The item traj.Cost S (τ) shown in FIG. 9 indicates the accumulated cost, and indicates the cost of the trajectory until the downward single pendulum is in the upright state in the inverted state. The item trainable params shown in FIG. 9 indicates the number of parameters.

  As shown in FIG. 9, it can be seen that “Trained” has the highest generalization performance. The reason why “Freezed” is lower in generalization performance than “Trained” is because the dynamics learned in the first learning process was not optimized in the second learning process. That is, in “Freezed”, it is considered that the generalization performance is low due to the influence of the dynamics error learned in the first learning process.

  On the other hand, in the comparative example, the success rate of the swing-up control is 0%, and the swing-up is not successful. That is, in the comparative example, it is considered that the number of parameters to be learned becomes too large and a state explosion has occurred. This shows that it is difficult to learn the dynamics and the cost function in the neural network of the comparative example.

  Next, learning results in this experiment will be described with reference to FIGS. 10A to 10C.

  FIG. 10A is a diagram illustrating a true cost function, and the cost function represented by the above (Equation 4) is visualized. FIG. 10B is a diagram illustrating the cost function of the learned path integral control neural network, and the cost function learned in “Trained” in this experiment is visualized. FIG. 10C is a diagram illustrating a cost function of the learned neural network of the comparative example, and the cost function learned in the comparative example in this experiment is visualized.

  As can be seen from a comparison between FIG. 10A and FIG. 10B, the “Trained” cost function, that is, the cost function of the path integral control neural network, is learned to have a shape close to the true cost function.

  On the other hand, as can be seen from FIG. 10C, the cost function of the comparative example has no shape. This indicates that the cost function cannot be learned with the neural network of the comparative example.

  From the above experimental results, it can be seen that the path integration control neural network which is the neural network of the present disclosure can learn the cost function in a shape close to the true cost function. It can also be seen that the path integral control neural network using the learned cost function has high generalization performance.

  From the above, it can be seen that the path integral control neural network, which is the neural network of the present disclosure, can not only learn the dynamics and cost function necessary for optimal control, but also acquire generalization performance and prefetch.

[Effects]
As described above, by using the path integration control neural network including the double recurrent neural network which is the neural network of the present disclosure, the dynamics and cost function necessary for optimal control by path integration, or these parameters are learned. be able to. In addition, since the path integral control neural network can obtain high generalization performance by imitation learning, a control device that can perform prefetching can be realized by using the path integral control neural network. That is, according to the control device and the control method of the present embodiment, it is possible to cause a neural network composed of a double recurrent neural network to perform optimal control by path integration, so that optimal control by path integration using a neural network is possible. It can be performed.

  Furthermore, as described above, for learning of the dynamics and cost function of the path integral control neural network, an existing learning method such as an error back propagation method can be used in learning of the neural network. That is, according to the control device and the control method of the present embodiment, it is possible to easily learn parameters that are difficult to describe, such as dynamics and cost function necessary for optimal control, using an existing learning method.

  Further, according to the control device and the control method of the present embodiment, the path integral control neural network that can be expressed by a differentiable composite function is used, so that continuous control is performed in which the state and operation of the controlled object are performed with continuous values. be able to. Further, according to the control device and the control method of the present embodiment, since the path integral control neural network that can be expressed by a differentiable synthesis function is used, the cost function can be expressed flexibly. That is, the cost function can be expressed not only by a model of a neural network but also by a neural network using a mathematical expression.

(Modification 1)
In the above embodiment, the neural network unit 30 includes only the calculation unit 13 and outputs the operation amount series calculated by the calculation unit 13, but is not limited thereto. The operation amount series calculated by the calculation unit 13 may be averaged and output. Hereinafter, this case will be described as a first modification, focusing on differences from the embodiment.

[Neural network unit 30]
FIG. 11 is a block diagram illustrating an example of the configuration of the neural network unit 30 in the first modification. Elements similar to those in FIG. 2 are denoted by the same reference numerals, and detailed description thereof is omitted.

  The neural network unit 30 shown in FIG. 11 differs from the neural network unit 3 shown in FIG. 2 in that it further includes a multiplier 31, an adder 32, and a delay unit 33.

<Multiplier 31>
Multiplier 31 multiplies the manipulated variable sequence calculated by calculation unit 13 by a weight and outputs the result to adder 32. More specifically, the multiplier 31 multiplies the weight w _i each time the calculating unit 13 updates the operation amount sequence, and outputs to the adder 32. Note that the calculation unit 13 updates the operation amount sequence recursively U times as described above, thereby controlling the operation amount sequence for controlling the control target.
Is calculated. Since the operation amount sequence updated later by the calculation unit 13 is an operation amount sequence with less variation, the weight w _i satisfies the following (Equation 5) and the number of updates of the calculation unit 13 is large. It is determined to be larger.

<Adder 32>
The adder 32 adds and outputs the manipulated variable series multiplied by the weight output from the multiplier 31 and the manipulated variable series multiplied by the weight output from the multiplier 31 before. To do. More specifically, the adder 32 weights and averages all the operation amount sequences output from the calculation unit 13 by adding all the operation amount sequences multiplied by the weight output from the multiplier 31. Average manipulated variable series
Is output as an output of the neural network unit 30.

<Delay unit 33>
The delay unit 33 delays the addition result of the adder 32 for a predetermined time and provides it to the adder 32 at a timing for updating. In this way, the delay unit 33 accumulates all the operation amount sequences multiplied by the weights output from the multiplier 31 in the adder 32, whereby all the operation amounts output from the calculation unit 13 in the adder 32. The series can be weighted and averaged.

  In addition, the other structure and operation | movement of the control apparatus of this modification are as having demonstrated in the other structure and operation | movement of the control apparatus 1 of said embodiment.

[Effects]
According to the control device in the present modification, the manipulated variable series updated by the calculation unit 13 is not output as it is, but the manipulated variable series multiplied by a larger weight is added and output later. As a result, the larger the number of updates, the smaller the variation in the manipulated variable series, which can be utilized. In other words, even if the gradient is weakened by learning the recurrent neural network by the error back propagation method, it can be solved by weighting and averaging the operation amount series updated later.

(Possibility of other embodiments)
The control device and the control method of the present disclosure have been described above in the embodiments. However, the present disclosure is not limited to the above embodiments. For example, another embodiment realized by arbitrarily combining the components described in this specification and excluding some of the components may be used as an embodiment of the present disclosure. Further, the present disclosure also includes modifications obtained by making various modifications conceivable by those skilled in the art without departing from the gist of the present disclosure, that is, the meanings of the words described in the claims. It is.

  The present disclosure further includes the following cases.

  (1) Specifically, the above apparatus is a computer system including a microprocessor, ROM, RAM, a hard disk unit, a display unit, a keyboard, a mouse, and the like. A computer program is stored in the RAM or hard disk unit. Each device achieves its functions by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  (2) A part or all of the constituent elements constituting the above-described apparatus may be configured by one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically, a computer system including a microprocessor, ROM, RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  (3) A part or all of the constituent elements constituting the above-described device may be constituted by an IC card that can be attached to and detached from each device or a single module. The IC card or the module is a computer system including a microprocessor, a ROM, a RAM, and the like. The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  (4) Moreover, this indication may be the method shown above. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.

  (5) In addition, the present disclosure provides a computer-readable recording medium such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD ( It may be recorded on a Blu-ray (registered trademark) Disc), a semiconductor memory, or the like. The digital signal may be recorded on these recording media.

  In addition, the present disclosure may transmit the computer program or the digital signal via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

  The present disclosure may be a computer system including a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating according to the computer program.

  In addition, the program or the digital signal is recorded on the recording medium and transferred, or the program or the digital signal is transferred via the network or the like and executed by another independent computer system. You may do that.

  The present disclosure can be used for a control device and a control method for performing optimal control. The present disclosure provides a control device that allows a deep neural network to learn parameters that are particularly difficult to describe, such as dynamics and cost functions, and that allows the deep neural network to perform optimal control using the learned dynamics and cost functions. It can be used for the control method.

DESCRIPTION OF SYMBOLS 1 Control apparatus 2 Input part 3, 3b, 30 Neural network part 4 Output part 5 Teacher data 13 Calculation part 13a, 141a Recurrent neural network 14 1st process part 15 2nd process part 16 3rd process part 17, 142, 162 Storage Unit 31 multiplier 32 adder 33 delay unit 50 controlled object 141 Monte Carlo simulator unit 151 cost integration unit 152 operation system update unit 161 noise generation unit 1411 dynamics model 1413 cost function model

Claims (7)

A control device for performing optimal control by path integration,
The controller comprises a neural network having a machine-learned dynamics model and a cost function;
An input unit that inputs a current state of the control target and an operation amount sequence having a plurality of operation parameters for the control target as components and an initial operation amount sequence to the neural network;
An operation amount for controlling the control target, which is an operation amount sequence calculated by path integration from the current state and the initial operation amount sequence by the neural network using the dynamics model and the cost function. An output unit for outputting the series,
The neural network comprises a second recurrent neural network having a first recurrent neural network having the dynamics model therein,
Control device. The second recurrent neural network is:
The first recurrent neural network and the cost function; causing the first recurrent neural network to calculate a state at each time from the current state and the initial manipulated variable sequence by a Monte Carlo method; A first processing unit that calculates a cost of the plurality of states using a function;
A second processing unit that calculates an operation amount sequence for the control target based on the initial operation amount sequence and the costs of the plurality of states;
The second processing unit outputs the calculated manipulated variable sequence to the output unit and feeds back to the second recurrent neural network as the initial manipulated variable sequence,
The second recurrent neural network includes, to the first processing unit, the cost of a plurality of states at each time next to each time from the operation amount series fed back by the second processing unit and the current state. To calculate,
The control device according to claim 1. The second recurrent neural network further includes:
A third processing unit for generating random numbers used in the Monte Carlo method,
The third processing unit outputs the generated random number to the first processing unit and the second processing unit;
The control device according to claim 2. The cost function is a cost function model composed of a neural network.
The control device according to claim 1. A control method of a control device for performing optimal control by path integration,
The controller comprises a neural network having a machine-learned dynamics model and a cost function,
An input step for inputting a current state of the control target and an operation amount sequence having a plurality of operation parameters for the control target as components and an initial operation amount sequence to the neural network;
An operation amount sequence calculated by path integration from the current state and the initial operation amount sequence using the dynamics model and the cost function to the neural network, and for controlling the control target An output step for outputting a manipulated variable series,
The neural network comprises a second recurrent neural network having a first recurrent neural network having the dynamics model therein,
Control method. Furthermore, before the input step, a learning step for machine learning the dynamics model and the cost function,
The learning step includes
A state prepared in advance corresponding to the current state of the control target, an initial operation amount sequence prepared in advance corresponding to an initial operation amount sequence for the control target, a state prepared in advance and a state prepared in advance Preparing learning data including an operation amount sequence for controlling the control target, which is calculated in advance by path integration from an initial operation amount sequence, as teacher data;
Learning the dynamics model and the cost function by learning the weight of the neural network using an error back propagation method using the teacher data.
The control method according to claim 5. The cost function is a cost function model composed of a neural network.
The control method according to claim 5 or 6.