JP5152780B2 - Function approximation device, reinforcement learning system, function approximation system, and function approximation program - Google Patents

Description

  The present invention relates to a function approximation device, a function approximation system, and a function approximation program that approximate a function using a neural network. The present invention also relates to a reinforcement learning system to which the function approximation device is applied in order to appropriately control the control device.

  Conventionally, a technique for approximating a function representing a relationship between an input and an output is known and widely used in the field of intelligent information processing such as machine learning, pattern recognition, and control of a robot. In addition, the technique for approximating the function is useful in various engineering fields such as weather forecasting and iron furnace blast furnace control if the function can be approximated with high accuracy. Yes.

As a technique for approximating the function, for example, a linear model (general linear model (GLM)) that approximates the function to a linear polynomial, or a table reference method that approximates by referring to a pre-stored table Techniques such as (Table Look-up) are known.
In addition, a so-called layered neural network technique is known in which the function is approximated by expressing the input / output and the function in a distributed manner by elements (neurons) included in the input layer, the intermediate layer, and the output layer. The layered neural network includes a multi-layer perceptron (MLP), an RBF (Radial Basis Function) network, and a GRBF (Gaussian Radial) that have different characteristics such as learning ability (generalization ability). Basis Function) Network and other algorithms are included.
The linear model is described in, for example, Non-Patent Document 1 and the like, and the layered neural network is described in, for example, Non-Patent Document 2 and the like, both of which are publicly known.

(About selective desensitization)
The present inventors have proposed a new layered neural network to which the selective desensitization method is applied. The selective desensitization method is described in Non-Patent Document 3, etc., and is well known. Here, the layered neural network to which the selective desensitization method is applied will be described below based on the contents described in Non-Patent Document 3.
In the layered neural network to which the selective desensitization method is applied, the problem of associating the target pattern T by integrating the information of the two input patterns S and C, that is, if the input is S and C, the output is T. Consider the task of approximating a function.

FIG. 12 is an explanatory diagram of the minimum configuration of a layered neural network to which the selective desensitization method is applied.
In FIG. 12, a layered neural network 01 to which the selective desensitization method is applied includes an input layer 01a composed of 2n elements, an intermediate layer 01b composed of n elements, and m elements. And an output layer 01c constituted by The 2n elements of the input layer 01a are allocated to the input patterns S and C by n. That is, the input patterns S and C are each composed of n elements (elements). Further, n elements of the input patterns S and C are respectively coupled to n elements of the intermediate layer 01b, and an i-th element (element x _i described later) of the intermediate layer 01b includes The i-th elements (elements s _i , c _i described later) of the input patterns S, C are respectively coupled (i = 1, 2,..., N).

Here, the input pattern S, C _{is, S = (s 1, s} 2, ..., s n), C = (c 1, c 2, ..., c n) and indicated by. In this case, the i-th element of the input patterns S and C is denoted by s _i and c _i . Further, it is assumed that the intermediate pattern that is the output of the intermediate layer 01b is X, and the intermediate pattern X is represented by X = (x ₁ , x ₂ ,..., X _n ). At this time, the i-th element x _i of the intermediate pattern X is represented by the following formula (1).
x _i = g _i (s _i −ave (x _i )) + ave (x _i ) (1)
Here, g _i is a gain (output sensitivity, sensitivity) in the element x _i , and ave (x _i ) is an average output level of the element x _i , that is, for all inputs (elements s _i , c _i ). It is the average value of the output (element x _i ).

The average output level ave (x _i ) can be replaced with the average value (ave (s _i )) of the element s _i when there is no correlation between the gain g _i and the element s _i. It is. Here, there is almost no problem even if both (ave (x _i ), ave (s _i )) do not exactly match. Thus, for example, when the value of the element s _i takes 1 and −1 with equal probability, and the gain g _i and the element s _i are determined independently (no correlation), the average output level It is also possible to set ave (x _i ) to 0 (ave (x _i ) = 0). In this case, the element x _i can be expressed by the following equation (1) ′.
x _i = g _i × s _i (1) ′

Further, it is assumed that a gain vector including the gain g _i as the i-th element is G, and the gain vector G is represented by G = (g ₁ , g ₂ ,..., G _n ). The value of the gain g _i is normally 1, but setting this to 0 is referred to as “desensitization”, and n elements s of the input pattern S are output at the output of the intermediate pattern X. Desensitizing only a part of _i (i = 1, 2,..., n) is referred to as “selective desensitization”.
Then, the combination of the desensitized elements s _i (i = 1, 2,..., N) is changed based on the elements c _i (i = 1, 2,..., N) of the input pattern (context pattern) C. The method is a context modification method based on selective desensitization, which is referred to as “product type context modification”. An input pattern S that has been subjected to product type context modification with the input pattern (context pattern) C is referred to as product type modification S (C).

In this case, the i-th element (gain) g _i of the gain vector G is determined based on the i-th element c _i of the input pattern (context pattern) C. As the simplest determination method, for example, the gain g _i is expressed by the following equation (2) when the value of the element c _i takes 1 and −1 with the same probability as the element s _i. .
g _i = (1 + c _i ) / 2 (2)
Therefore, when c _i = 1, g _i = 1, and when c _i = −1, g _i = 0, and the input pattern (context pattern) C and the gain vector G are the same. I can see it. When there is a correlation between the gain g _i (element c _i ) and the element s _{i, that} is, between the gain vector G (input pattern (context pattern) C) and the input pattern S. If there is, it is desirable to eliminate the correlation by appropriately shuffling the component (element g _i ) of the gain vector G and set ave (x _i ) = 0.

In FIG. 12, a plurality of elements of the intermediate layer 01b are coupled to each of the m elements of the output layer 01c. Here, when the output pattern which is the output of the output layer 01c is Y, and the output pattern Y is expressed by Y = (y ₁ , y ₂ ,..., Y _m ), the j-th output pattern Y The element y _j is expressed by the following equation (3).

Note that w _ji is a coupling load (weighting) from the element x _i to the element y _j (0 ≦ w _ji ≦ 1). Also, sgn (u) is a code for a value u such that sgn (u) = 1 when u> 0, and sgn (u) = − 1 otherwise (u ≦ 0). It is a function.

Further, since there is no correlation between the input patterns S and C and the orthogonality between the input patterns S and C is low, an input pattern with low orthogonality is used as a feedback-type learning rule for the coupling load w _ji. Orthogonal learning capable of associative memory for S and C can be used. That is, when the target pattern T is represented by T = (t ₁ , t ₂ ,..., T _m ) and the j-th element of the target pattern T is represented by t _j , the combined load w _ji is Each time the input patterns S and C are input, the update value Δw _ji expressed by the following equation (4) is added and updated.

Here, ε is a positive constant (eg, 0.3). Further, the formula (4) can be expressed by the following formula (4) ′ by the formula (3).
Δw _ji = ε (t _j −y _j ) x _i (4) ′

(Product model and mutual modification model)
FIG. 13 is an explanatory diagram of a layered neural network to which the selective desensitization method is applied, FIG. 13A is an explanatory diagram of a product type model, and FIG. 13B is an explanatory diagram of a mutual modification model.
Note that the layered neural network to which the selective desensitization method is applied includes a product model 01 ′ shown in FIG. 13A and a mutual modification model 02 shown in FIG. 13B.
In FIG. 13A, the product model 01 ′ includes an input layer 01a ′ composed of 2n elements, an intermediate layer 01b ′ composed of n elements, and an output layer composed of m elements. 01c '. The product model 01 ′ is simplified by omitting the input pattern S of the input layer 01a ′ by the arrow S, and has the same configuration as the layered neural network 01 shown in FIG. Yes.

13B, the mutual modification model 02 includes an input layer 02a composed of 2n elements, an intermediate layer 02b composed of 2n elements, and an output layer 02c composed of m elements. And have. Here, in the mutual modification model 02, each intermediate element 01, 01b ′ of the product model 01, 01 ′ outputs each product (x _i (i = i = , N)), the intermediate layer 02b outputs each element (x _i (i = 1) that outputs the product type modification S (C) of the input pattern S. , 2,..., N)) and an element (x _i ′ (i = 1, 2,..., N)) that outputs the product type modification C (S) of the input pattern C. That is, the intermediate pattern X ′ of the intermediate layer 02b is represented by X ′ = (x ₁ , x ₂ ,..., X _n , x ₁ ′, x ₂ ′,..., X _n ′).
In the mutual modification model 02, the input patterns S and C are mutually subjected to product type context modification. That is, the input pattern S is subjected to product type context modification by the input pattern C to output the product type modification S (C), and the input pattern C is also subjected to product type context modification by the input pattern S to be product type. The modification C (S) is output (see formulas (1) to (4)). Therefore, the mutual modification model 02 can further improve the learning ability (generalization ability) compared to the product model 01 ′ (see Non-Patent Document 3).

Atsushi Kato, “General Linear Models”, “online”, September 2002, Hokkaido University, “August 6, 2007 search”, Internet <URL: http://www.neurosci.aist.go.jp/ ~ kurita / lecture / prnn / prnn.html> Takio Hamada, “Pattern recognition and neural network”, “online”, February 2001, National Institute of Advanced Industrial Science and Technology, “August 7, 2007 search”, Internet <URL: http: //hosho.ees.hokudai. ac.jp/~kato/seminar/020909/slide_bold.pdf> Masahiko Morita and three others, “Information Integration Capability of Layered Neural Networks Applying Selective Desensitization”, “Journal of the Institute of Electronics, Information and Communication Engineers”, December 2004, Volume J87-D-II, No. 12 , P.2242-2252 Soichiro Mori, Hayato Yamana, “Acceleration of learning through parallel reinforcement learning”, “online”, March 2004, Waseda University, “August 8, 2007 search”, Internet <URL: http: // www .yama.info.waseda.ac.jp / publications / 2003 / 0403_mori.pdf> Shinichi Asakawa, “Reinforcement Learning”, “online”, July 2002, Tokyo Women's University, “August 22, 2007 search”, Internet <URL: http://www.twcu.ac.jp/~asakawa/ chiba2002 / lect8-Reinforcement / ReinforcementLearning.pdf>

(Problems of conventional technology)

However, when a function is approximated by a linear model, only a function that is known to be linearized in advance can be approximated with high accuracy. To approximate the function with high accuracy, the order of the linear polynomial is increased, It was necessary to increase the output sample data as much as possible. Further, when the function is approximated by the table reference method, there is a problem that the size of the table to be referred to increases exponentially according to the number of variables. For example, when there are n state variables that can take 100 different values, a table having 100 ⁿ different sizes is required to indicate all states in the table. Therefore, when the function is approximated with high accuracy, as many sample data as possible are required as many as 100 ⁿ types, and there is a problem that enormous time and storage capacity are required for table creation.

Further, when the function is approximated by the multilayer perceptron or the RBF (GRBF) network, the number of sample data necessary for learning increases as the number of elements in the input layer increases as in the case of the table reference method. There is a problem that enormous learning time is required to accurately approximate the function.
In particular, the multi-layer perceptron has a problem that the number of independent input patterns increases, and the more complicated the task, the more the expected learning ability (generalization ability) is lost (Non-Patent Document 3). reference). In addition, it is considered that this problem also exists in the RBF (GRBF) network in which the algorithm differs from the multilayer perceptron.
Therefore, with these conventionally known techniques, it is not possible to accurately approximate a function (nonlinear function) having strong nonlinearity, and an enormous number of sample data is required unless the number of input patterns is small. There was a problem that enormous learning time was required.

  The product models 01 and 01 ′ (see FIGS. 12 and 13A) and the mutual modification model 02 (see FIG. 13B) to which the selective desensitization method is applied have learning ability ( Generalization ability) is high, and it is possible to learn accurately with a small number of learning times (see Non-Patent Document 3). However, in the layered neural networks 01, 01 ', 02 to which the selective desensitization method is applied, the intermediate patterns X, X' (X = S (C), X '= S) for two input patterns S, C. (C), C (S)) is only assumed to be output. That is, when the number of input patterns increases to three or more, the layered neural networks 01, 01 ', 02 to which the selective desensitization method is applied cannot be applied. However, there is a problem that the above problem cannot be solved because there is only a method of approximating the function by applying a table reference method, a layered neural network such as a multilayer perceptron). In addition, the layered neural networks 01, 01 ', 02 to which the conventionally known selective desensitization method is applied have been tested for simple linear functions (for example, f (x, y) = xy). However, there has been a problem that the effectiveness of approximation of a complex nonlinear function has not been verified, and no practical application example has been proposed.

In view of the above circumstances, the present invention has the following description (O01) as a technical problem.
(O01) To accurately approximate a function even when the number of input variables is large, and to reduce the learning cost necessary for the approximation.

In order to solve the technical problem, the function approximating device according to claim 1 comprises:
An input layer configured by an input element to which an input variable value is input, and an intermediate element coupled to the input element, wherein the value of the intermediate variable calculated based on the value input to the input element is An intermediate layer constituted by the intermediate element to be output, and an output element coupled to the intermediate element, wherein the value of the output variable calculated based on the value input to the intermediate element is output In a function approximation device that approximates a function that is a relationship between the input variable and the output variable by a layered neural network having an output layer constituted by output elements,
The input layer constituted by each input element to which each value of three or more input variables is input;
Input variable input means for inputting each value of the three or more input variables;
An input variable set in which any two input variables among the three or more input variables are set, and each value of one input variable of the input variable set and the other input variable of the input variable set are set. Intermediate variable calculation means for calculating each value of the intermediate variable based on each first output sensitivity calculated based on each value of
An output variable calculation means for calculating the value of the output variable based on the intermediate variable and a combined load set according to the importance of the value of the intermediate variable;
A connection weight learning means for learning the connection weight by updating the connection weight based on a difference between a value of the output variable and an actual value of the function stored in advance;
It is provided with.

The invention according to claim 2 is the function approximating device according to claim 1,
Each value of the first intermediate variable based on each value of one input variable of the input variable set and each first output sensitivity calculated based on each value of the other input variable of the input variable set First intermediate variable calculation means for calculating each of the above, each value of the other input variable of the input variable set, each second output sensitivity calculated based on each value of one input variable of the input variable set, , Based on the second intermediate variable calculation means for calculating each value of the second intermediate variable, respectively, the intermediate variable calculation,
Based on the first intermediate variable, the second intermediate variable, and each combined load set in accordance with the importance of each value of the first intermediate variable and the second intermediate variable, the output variable The output variable calculating means for calculating a value;
It is provided with.

The invention according to claim 3 is the function approximating device according to claim 1 or 2,
The intermediate layer includes each intermediate element that outputs each value of a plurality of intermediate variables, and all the input variables of the three or more input variables are at least one of the input variable sets or A plurality of sets of the input variables are configured by being set as the other input variable.

In order to solve the technical problem, the reinforcement learning system of the invention according to claim 4 comprises:
A control device as a target for controlling behavior;
State measuring means for measuring the state of the control device;
Reward acquisition means for acquiring a reward for the behavior;
An action value function calculating means for calculating an action value function that is an evaluation value for evaluating all actions in the measured state based on prediction of a reward that can be acquired in the future;
The function approximation device according to any one of claims 1 to 3, wherein the action value function in the measured state is approximated by regarding a measured value measured in the state as a value of the input variable.
Action selecting means for selecting the action based on the approximated action value function;
Action executing means for executing the selected action;
Control end determination means for determining whether to end control of the control device by determining whether control of the control device has failed based on the reward,
It is provided with.

The invention according to claim 5 is the reinforcement learning system according to claim 4,
The function approximation device according to any one of claims 1 to 3, further comprising: the connection weight learning unit that learns the connection weight when it is determined that the control of the control device has failed.
It is provided with.

In order to solve the technical problem, the function approximation system of the invention according to claim 6 comprises:
An input layer configured by an input element to which an input variable value is input, and an intermediate element coupled to the input element, wherein the value of the intermediate variable calculated based on the value input to the input element is An intermediate layer constituted by the intermediate element to be output, and an output element coupled to the intermediate element, wherein the value of the output variable calculated based on the value input to the intermediate element is output In a function approximation system that approximates a function that is a relationship between the input variable and the output variable by a layered neural network having an output layer constituted by output elements,
The input layer constituted by each input element to which each value of three or more input variables is input;
Input variable input means for inputting each value of the three or more input variables;
An input variable set in which any two input variables among the three or more input variables are set, and each value of one input variable of the input variable set and the other input variable of the input variable set are set. Intermediate variable calculation means for calculating each value of the intermediate variable based on each first output sensitivity calculated based on each value of
An output variable calculation means for calculating the value of the output variable based on the intermediate variable and a combined load set according to the importance of the value of the intermediate variable;
A connection weight learning means for learning the connection weight by updating the connection weight based on a difference between a value of the output variable and an actual value of the function stored in advance;
It is provided with.

In order to solve the technical problem, the function approximation program of the invention according to claim 7 is:
Computer
An input layer configured by input elements to which values of input variables are input and configured by input elements to which values of three or more input variables are input, and an intermediate element coupled to the input elements, An intermediate layer composed of the intermediate element that outputs an intermediate variable value calculated based on a value input to the input element, and an output element coupled to the intermediate element, the intermediate element An input layer configured to output the value of the output variable calculated based on the input value; and an input layer configured to input each value of the three or more input variables. Variable input means,
An input variable set in which any two input variables among the three or more input variables are set, and each value of one input variable of the input variable set and the other input variable of the input variable set are set. Intermediate variable calculation means for calculating each value of the intermediate variable based on each first output sensitivity calculated based on each value of
An output variable calculation means for calculating the value of the output variable based on the intermediate variable and a combined load set according to the importance of the value of the intermediate variable;
A combined load learning means for learning the combined load by updating the combined load based on a difference between a value of the output variable and an actual value of the function stored in advance;
, The function that is the relationship between the input variable and the output variable is approximated.

  According to the invention of claim 1, each first output sensitivity calculated based on each value of one input variable of the input variable set and each value of the other input variable of the input variable set, Based on the above, it is assumed that only the intermediate variable and the output variable are output for two input variables by calculating each value of the intermediate variable. A layered neural network based on the selective desensitization method having high learning ability (generalization ability) can also be applied to a function having three or more input variables that is not a net application target. As a result, even when the number of input variables of the function is large, the learning ability (generalization ability) is improved as compared with conventionally known techniques (linear model, table reference method, layered neural network such as multilayer perceptron). Therefore, the function can be approximated with high accuracy, and the computation cost such as the amount of computation and computation time required for approximating the function can be reduced. According to the invention described in claim 1, when each value of a plurality of intermediate variables is calculated based on a plurality of the input variable sets, each value can be calculated in parallel. It is also possible to perform calculations in parallel by a plurality of hardware. In this case, the intermediate variable can be calculated at high speed, and the calculation time for approximating the function can be reduced.

According to the second aspect of the present invention, one input variable of the input variable set can calculate the first intermediate variable that is selectively desensitized, and the other input variable of the input variable set is The second intermediate variable that has been selectively desensitized can be calculated. That is, two intermediate variables in which the input variables of the input variable set are selectively insensitive to each other can be calculated. As a result, the learning ability (generalization ability) can be improved as compared with the case where the input variables of the input variable set are not selectively desensitized to each other, so that the function can be approximated with high accuracy. In addition, it is possible to reduce computation costs such as computation amount and computation time in learning necessary for approximating the function.
According to the third aspect of the present invention, the plurality of intermediate variables selectively desensitized can be calculated based on all the input variables of the three or more input variables. As a result, the learning ability (generalization ability) can be improved compared to the case where the output variable is computed regardless of the computation of the intermediate variable for some of the input variables. In addition, it is possible to reduce calculation costs such as calculation amount and calculation time in learning necessary for approximating the function.

According to the fourth aspect of the present invention, it is possible to select and execute an appropriate action even for an unknown state by approximating the action value function by the function approximating device. For this reason, it is possible to increase the learning ability (generalization ability) compared to a conventionally known reinforcement learning system that does not approximate the action value function of an unknown state, and to apply to unknown environments including disturbances and observation noises. Can also be high.
According to the fifth aspect of the present invention, in order to learn the combined load when it is determined that the control of the control device has failed and it is determined that the control of the control device is to end, The combined weight can be learned on the basis of an action value function calculated so as to be able to evaluate the state. As a result, it is possible to reduce calculation costs such as calculation amount and calculation time in learning necessary for approximating the function as compared with the case where the connection weight is learned each time each behavior value function in each state is calculated. it can.

  According to invention of Claim 6, each 1st output sensitivity calculated based on each value of one input variable of the said input variable group, and each value of the other input variable of the said input variable group, Based on the above, it is assumed that only the intermediate variable and the output variable are output for two input variables by calculating each value of the intermediate variable. A layered neural network based on the selective desensitization method having high learning ability (generalization ability) can also be applied to a function having three or more input variables that is not a net application target. As a result, even when the number of input variables of the function is large, the learning ability (generalization ability) is improved as compared with conventionally known techniques (linear model, table reference method, layered neural network such as multilayer perceptron). Therefore, the function can be approximated with high accuracy, and the computation cost such as the amount of computation and computation time required for approximating the function can be reduced. According to the invention described in claim 6, when each value of a plurality of intermediate variables is calculated based on a plurality of the input variable sets, it is possible to calculate each value in parallel. It is also possible to perform calculations in parallel by a plurality of hardware. In this case, the intermediate variable can be calculated at high speed, and the calculation time for approximating the function can be reduced.

  According to invention of Claim 7, each 1st output sensitivity calculated based on each value of one input variable of the said input variable set, and each value of the other input variable of the said input variable set, Based on the above, it is assumed that only the intermediate variable and the output variable are output for two input variables by calculating each value of the intermediate variable. A layered neural network based on the selective desensitization method having high learning ability (generalization ability) can also be applied to a function having three or more input variables that is not a net application target. As a result, even when the number of input variables of the function is large, the learning ability (generalization ability) is improved as compared with conventionally known techniques (linear model, table reference method, layered neural network such as multilayer perceptron). Therefore, the function can be approximated with high accuracy, and the computation cost such as the amount of computation and computation time required for approximating the function can be reduced. Further, according to the invention described in claim 7, when each value of a plurality of intermediate variables is calculated based on a plurality of the input variable sets, it can be calculated in parallel, for example, It is also possible to perform calculations in parallel by a plurality of hardware. In this case, the intermediate variable can be calculated at high speed, and the calculation time for approximating the function can be reduced.

Next, specific examples of embodiments of the present invention (hereinafter referred to as examples) will be described with reference to the drawings, but the present invention is not limited to the following examples.
In order to facilitate understanding of the following description, in the drawings, the front-rear direction is the X-axis direction, the left-right direction is the Y-axis direction, the up-down direction is the Z-axis direction, and arrows X, -X, Y, -Y, The direction indicated by Z and -Z or the indicated side is defined as the front side, the rear side, the right side, the left side, the upper side, the lower side, or the front side, the rear side, the right side, the left side, the upper side, and the lower side, respectively.
In the figure, “•” in “○” means an arrow heading from the back of the page to the front, and “×” in “○” is the front of the page. It means an arrow pointing from the back to the back.
In the following description using the drawings, illustrations other than members necessary for the description are omitted as appropriate for easy understanding.

FIG. 1 is an overall explanatory diagram of a function approximation system according to a first embodiment of the present invention.
In FIG. 1, an example of a function approximation system (reinforcement learning system) S according to the first embodiment is a control system configured as a simulator for solving the problem of preventing an inverted pendulum from falling down. The function approximating system S includes a carriage (control device) 1 as an example of a moving body that can move in the left-right direction (Y-axis direction). A rod 2 as an inverted pendulum is supported on the cart 1. The rod 2 is fixedly supported in a state where the base end is rotatable at the central portion of the upper surface of the carriage 1, and the tip end is capable of vibrating in the left-right direction (Y-axis direction).
The carriage 1 is provided with drive wheels 1a, and the carriage 1 can move on a horizontal floor surface 3 by the drive wheels 1a. The drive wheel 1a is driven by a drive unit (drive motor) 1b built in the cart 1, and the drive unit 1b is driven by a control unit (function approximation device) C built in the cart 1. Be controlled. Further, a right end wall 3a and a left end wall 3b are respectively provided at both ends of the floor surface 3 in the left-right direction (Y-axis direction).

(Inverted pendulum equation of motion)
Here, as shown in FIG. 1, the center of gravity of the carriage 1 is a distance away from the center of the floor surface 3 in the left-right direction (Y-axis direction) in the right direction (+ Y direction), that is, the position of the carriage 1. X [m], the angle at which the rod 2 is tilted to the right (+ Y direction) with respect to the vertical direction (Z-axis direction) is θ [deg], and the control unit C drives the drive unit 1b. The force applied to the cart 1 in the right direction (+ Y direction) is F [N].
Further, the mass of the carriage 1 is M [kg], the speed at which the carriage 1 moves in the right direction (+ Y direction) is v [m / s] (v = (d / dt) x), and the rod 2 M [kg], the length from the base end of the rod 2 to the center of gravity, that is, the half length of the rod 2 is L [m], and the rod 2 is in the vertical direction (Z-axis direction) , The angular velocity rotating clockwise is ω [deg / s] (ω = (d / dt) θ), the gravitational acceleration is g [m / s ² ], and the carriage 1 is in the right direction (+ Y direction). Is the acceleration when moving to a (a = (d / dt) v [m / s ² ]), and the angular acceleration when the rod 2 rotates clockwise with respect to the vertical direction (Z-axis direction). and b (b = (d / dt ) ω [deg / s 2]), the friction between the carriage 1 and the pendulum 2, the driving wheel 1a and the floor surface 3 of the carriage 1 When friction and between what a negligible, the equation of motion of the inverted pendulum problem is an issue, the following equation (5-1), can be represented by (5-2).

(M + m) a + (mL cos θ) b−mLω ² sin θ = F (5-1)
(MLcos θ) a + (4/3 × mL ² ) b-mgLsin θ = 0 (5-2)
Further, as a result of solving the variables a and b using the equations (5-1) and (5-2) as simultaneous equations, the variables a and b are expressed by the following equations (6-1) and (6-2). Can show.
a =
{4/3 (F + mLω ² sin θ) −mg sin θ cos θ} /
{4/3 (M + m) -mcos ² θ} (6-1)
b =
{(M + m) gsinθ- (F + mLω ² sinθ) cosθ} /
{4/3 (M + m) -mcos ² θ} L (6-2)

  Further, a certain natural number is k (k = 0, 1, 2,...), A minute time step is t, and the position of the carriage 1 at a certain time k × t, (k + 1) × t is x (k), x (k + 1), velocity is v (k), v (k + 1), angle of the rod 2 is θ (k), θ (k + 1), angular velocity is ω (k), ω (k + 1), and time k × When the acceleration of the carriage 1 at t is a (k) and the angular acceleration of the rod 2 is b (k), each variable x (k + 1), v (k + 1), θ (time (k + 1) × t k + 1) and ω (k + 1) are represented by the following equations (7-1) to (7-4) using a conventionally known Runge-Kutta method (RK method). The Runge-Kutta method is based on a preset initial value to calculate the increment of the target ordinary differential equation and calculate the value of the next point. It is a solution algorithm of.

x (k + 1) = x (k) + t × v (k) (7-1)
v (k + 1) = v (k) + t × a (k) (7-2)
θ (k + 1) = θ (k) + t × ω (k) (7-3)
ω (k + 1) = ω (k) + t × b (k) (7-4)

In Example 1, the mass M of the carriage 1 is 1.0 [kg] (M = 1.0), the mass m of the rod 2 is 0.1 [kg] (m = 0.1), The length L of the rod 2 is 1.0 [m] (L = 1.0), the gravitational acceleration g is 9.8 [m / s ² ] (g = 9.8), and the time step t is 0.02. [S] (t = 0.02) is set in advance. The distance from the right end wall 3a to the left end wall 3b on the floor surface 3 is preset to 4.8 [m]. That is, the position x of the carriage 1 is limited in advance to a range of −2.4 to 2.4 [m] (x = −2.4 to 2.4).
Further, when the first force applied to the carriage 1 is F (0), the force F (0) is set in advance to 20.0 [N] (F (0) = 20.0). Further, when the position of the carriage 1 in the initial state is x (0), the speed is v (0), the angle of the rod 2 is θ (0), and the angular speed is ω (0), each variable x (0) , V (0), θ (0), ω (0) are expressed by the following equations (7-1) ′ to (7-4) ′.

x (0) = 0.0 (7-1) ′
v (0) = 0.0 (7-2) '
θ (0) = − 3.0 to 3.0 (random number set in the range of −3 to 3) (7-3) ′
ω (0) = 0.0 (7-4) ′

  Therefore, in the function approximation system S of the first embodiment, the above formulas (6-1), (6-2), (7-1) to (7-4), (7-1) ′ to (7-4) ′, The constants M, m, L, g, and t and the variables F, a, b, and k, the variables x, v, θ, and ω for each time step t can be calculated. Further, in the function approximation system S of the first embodiment, the control unit C applies the appropriate force F to the carriage 1 based on the calculated variables x, v, θ, ω. The drive of 1b is controlled.

(Description of the control part C of Example 1)
FIG. 2 is a block diagram (function block diagram) illustrating each function provided in the control unit of the cart according to the first embodiment of the present invention.
The control unit C includes a ROM (read only memory) in which programs and data for performing necessary processing are stored, a RAM (random access memory) in which necessary data is temporarily stored, and the ROM is stored in the ROM. It is composed of a CPU (Central Processing Unit) that performs processing according to the programmed program and a microcomputer having a clock oscillator and the like, and realizes various functions by executing the program stored in the ROM Can do.
The control unit C configured as described above can realize various functions by executing programs stored in the ROM. A function approximation program AP1 is stored in the ROM of the control unit C of the first embodiment.

(Function approximation program AP1)
FIG. 3 is a simple explanatory diagram of reinforcement learning.
The function approximating program AP1 has the following functional means (program modules).
C1: Inverted pendulum control start discriminating means The inverted pendulum control start discriminating means C1 has a start state storage means C1A. It is determined whether or not to start an inverted pendulum control process for controlling movement. In the inverted pendulum control process of the first embodiment, the drive of the drive unit 1b is controlled based on the reinforcement learning shown in FIG. 3 so that the rod 2 does not fall.

(About reinforcement learning)
Reinforcement learning refers to a learning method in which an agent acting as an action subject adapts to an environment to be controlled through trial and error using a reward as a clue. Here, as shown in FIG. 3, the agent (control device) is A, the environment is E, a certain time is t, a time next to the time t is t + 1, and actions at the times t and t + 1 are performed. _{Is set} to a _t , a _{t + 1} , the state of the environment E at the time t, t + 1 is set to s _t , _{st t + 1,} and the reward obtained from the environment E at the time t, t + 1 is set to r _t , r _{t + 1} .
In the reinforcement learning, if the current state of the environment E to be controlled is a s _t, the agent A, appeal directly to the environment E taking a certain action a _t. As a result, the state of the environment E changes to s _{t + 1,} the agent A, to obtain a reward r _t from the environment E. Thus, the agent A, which aims to maximize the reward r _t. That is, in the reinforcement learning, instead of teachers, etc. explicitly indicate the correct action a _t with respect to the input as a normal machine learning does not exist, the action a the reward result given from the environment E of _t r _t Learning based on.

Incidentally, the agent A, the in the current state s _t environment E, holds the predicted reward (r _t) which is the expected yield for each action a _t. Thus, the predicted reward _{(r t)} is the priority of the action _{a t,} the evaluation value of the action _{a t.} Then, the agent A is obtained reward r _t is slide into action by trial and error so as to maximize.
However, wherein the reward r _t, is a possibility that there is a delay or noise. For example, not only action a _t the direct state s _t, since also affects the subsequent state (s t _{+ 1),} cases can be considered such that affect all subsequent reward then (r t _{+ 1)} . Therefore, in reinforcement learning, experience-reinforcement-type reinforcement learning that performs evaluation in units of episodes and environment identification-type reinforcement learning algorithm that performs evaluation in units of steps can be selected according to the task. Here, the step, the initial state when the s _0, the agent A, on the basis of the certain state s _t observed in environment E, in unit before actually executing the action a _t An episode is a set of the steps from the initial state s ₀ to a state where the goal has been achieved (s _t ) or a state where it has failed (s _t ).

The experience enhanced reinforcement learning of, preferentially selected the action a _t a high reward r _t in during the learning can be acquired. For this reason, although the start-up of learning is fast, there is a high possibility of falling into a local solution (local minimum). Also, reinforcement learning of the environment identification type, finding the optimal solution by attempting to every action a _t in all states s _t of the environment E. Therefore, in Example 1, reinforcement learning by Q-learning, which is a typical algorithm of the environment identification type, is performed.

(About Q-learning)
Here, the Q-learning, the evaluation value _{Q (s} t, _{a t)} actions _{a t} executed in a state _{s t} at time step t case of the respective action in each state _{(s t)} ( each evaluation value of _{a t) (Q (s t} , a t) with predicted and stored), stored the evaluation values _(Q (s _{t, a} t) on the basis of) certain state (s behavior in _t) the _{(a t)} is an algorithm for selection. Here, a specific algorithm of one episode in the Q-learning is exemplified by the following processes (1) to (5).

(Specific example of Q-learning algorithm)
(1) Agent A observes the state _{s t} environment E.
(2) Agent A performs an action a _t in accordance with any action selection method.
(3) Agent A, as well as to win a reward _{r t} from the environment E, observing the state _{s t + 1} after the state transition.
(4) Agent A updates the action value function Q (s _t , a _t ), which is an evaluation value, based on the following equation (8).
Q (s _t , a _t ) =
(1-α) Q (s _t , a _t )
+ Α [r _t + γmax _{at + 1} (Q (s _{t + 1} , a _{t + 1} ))] (8)
Here, α is a learning rate as a parameter representing the amount of change in the action value function Q (s _t , a _t ) by one update (0 <α ≦ 1), and γ is expected to be obtained in the future. A discount rate as a parameter indicating how much the reward (r _{t + 1} ) to be discounted is reflected in the current evaluation (0 <γ ≦ 1). Further, max _{at + 1} (Q (s _{t + 1} , a _{t + 1} )) is the maximum value of the evaluation values Q (s _{t + 1} , a _{t + 1} ) for all the actions a _{t + 1} in the state s _{t + 1} .
(5) It is determined whether or not the target has been achieved (successful state) or has failed (failure state). If the target is in a successful state or a failed state, the episode is terminated. Otherwise, the time is advanced from t to t + 1 and the processing returns to the processing (2).

In the first embodiment, as the action selection method in the process (2), a greedy selection method that selects an action that can always be expected to have the highest value (Q (s _t , a _t )) is applied. For example, it is possible to apply an ε-greedy selection method for selecting the action of the maximum value. In the first embodiment, the learning rate α and the discount rate γ as reinforcement learning constants are 0.1 (α = 0.1) as the learning rate α and 0.95 (γ = 0.95) as the discount rate γ. Is set in advance.
Note that behavior selection methods such as reinforcement learning, Q-learning, and greedy selection methods are described in Non-Patent Documents 4 and 5 and are well known.

Therefore, the reinforcement learning Q-learning, unless time variation or the like which does not depend on the behavior of the agent A for the environment E, the agent A, tries all act to _{(a t)} in all states _{(s t)} By learning the action value function Q (s _t , a _t ), it becomes possible to select the optimum action (a _t ). However, in order to select the optimum action (a _t), said action value function Q (s _{t, a t)} enormous time is required for converging the learning about. For this reason, for example, when the task is complicated, all trials may not be completed in a finite time.

(Reinforcement learning with a layered neural network with selective desensitization applied)
FIG. 4 is an explanatory diagram of a layered neural network to which the selective desensitization method according to the first embodiment of the present invention is applied.
Therefore, in the first embodiment, in order to improve the learning efficiency of the action value function Q (s _t , a _t ), a multivariate which is an example of a layered neural network to which the selective desensitization method shown in FIG. 4 is applied. A new Q-learning reinforcement learning in which the action value function Q (s _t , a _t ) is approximated by the mutual modification model N is applied.
In FIG. 4, the multivariable mutual modification model N includes an input layer Na composed of 4n elements (input elements), an intermediate layer Nb composed of 12n elements (intermediate elements), and m number of elements. And an output layer Nc composed of elements (output elements).

  In the multivariable mutual modification model N, the input layer 02a of the conventionally known mutual modification model 02 is composed of two input patterns S and C (see FIG. 13B), whereas the input of the first embodiment is used. The layer Na is defined by four input variables (input patterns) x, v, θ, ω, that is, the position x of the carriage 1, the speed v, the angle θ of the rod 2, and the angular speed ω. It is configured. In the first embodiment, the four input variables x, v, θ, and ω are each composed of n elements (4 × n = 4n).

  In the multivariable mutual modification model N, the intermediate layer 02b of the conventionally known mutual modification model 02 includes a product type modification S (C) of the input pattern S and a product type modification C (S of the input pattern C. ) (See FIG. 11B, equations (1), (1) ′, (2)), the intermediate layer Nb of the first embodiment uses the product type modification x (v) of the input variable x. , X (θ), x (ω), product type modification v (θ), v (ω), v (x) of the input variable v, and product type modification θ (ω), θ of the input variable θ. (X), θ (v) and product type modifications ω (x), ω (v), ω (θ) of the input variable ω are output. That is, in the intermediate layer Nb of the first embodiment, the input variables x, v, θ, and ω are mutually subjected to product type context modification, with two each being one set (input variable set), and product type context modification is performed. The twelve product type modifications x (v), v (x), x (θ), θ (x), x (ω), ω (x), v (θ), θ (v), v ( ω), ω (v), θ (ω), ω (θ) are output as intermediate variables (intermediate pattern). In the first embodiment, the twelve product type modifications x (v), v (x), x (θ), θ (x), x (ω), ω (x), v (which are intermediate variables are used. θ), θ (v), v (ω), ω (v), θ (ω), and ω (θ) are each composed of n elements (12 × n = 12n).

In the multivariable mutual modification model N, in the output layer 02c of the conventionally known mutual modification model 02, a plurality of elements in 2n of the intermediate layer 02b are connected to the m elements. Similarly to the case where the weights w _ji are combined in a weighted state and the sign function is calculated and the output pattern Y is output (see FIG. 11B, equation (3)), the first embodiment For the output layer Nc, a plurality of elements of 12n of the intermediate layer Nb are coupled to each of m elements in a state weighted by a coupling load w _ji (w _ji ′). In addition, the sign function is calculated, and the action evaluation function Q (s _t , a _t ) as an output variable (output pattern) is output. In the first embodiment, the output variable Q (s _t , a _t ) is composed of m elements.

(About the correspondence between each element of reinforcement learning and each element of the inverted pendulum problem)
Thus, in Example 1, with respect to the function approximation system S, each element in the reinforcement learning _{Q-learning (A, E,} s t, a t, r t) to the following (1) to (5 ), The inverted pendulum control process can be executed.

(1) The agent A is associated with the control unit C of the carriage 1.
(2) The environment E is associated with the elements 1 to 3 as the configuration of the function approximating system S of the first embodiment. That is, constants M, m, L, t, g and variables a, b, k, etc. for the cart 1 and the rod 2 and relational expressions for the inverted pendulum problem (formulas (5-1), (5-2) ), (6-1), (6-2), (7-1) to (7-4)), the floor surface 3 (the right end wall 3a and the left end wall 3b), and the like.
(3) The states s _t and s _{t + 1} are associated with the input variables x, v, θ, and ω. Here, for example, when the state s _t is represented by s _t = x (k), v (k), θ (k), ω (k), the next state s _{t + 1} is changed to s _{t + 1} = x ( k + 1), v (k + 1), θ (k + 1), and ω (k + 1).
(4) the action _{a t} is associated with the force F. That is, it is associated with the force F corresponding to a value for controlling the drive unit 1b in order to move the carriage 1 so that the rod 2 does not fall down. Here, if the force F in the time k × t was F (k) (k = 0,1,2 , ...), said action _{a t} is, for _example, be represented by a t = F (k) Can do.
(5) The rewards r _t and r _{t + 1} may be values for confirming whether or not the bar 2 is collapsed. For example, when the rod 2 is tilted and the angle θ becomes θ ≦ −180 ° or θ ≧ 180 °, the rewards r _t and r _{t + 1} are set to negative values (r _t <0 , R _{t + 1} <0), otherwise, the rewards r _t , r _{t + 1} are set to 0 or more (r _t ≧ 0, r _{t + 1} ≧ 0).

The inverted pendulum control start determining means C1 according to the first embodiment is configured such that the set values of the constants M, m, L, t, g, α, γ and the variables x, v, It is determined whether or not the inverted pendulum control process is started by determining whether or not initial values of θ, ω, a, b, k, F, w _ji , and w _ji ′ are set.
C1A: Start state storage unit The start state storage unit C1A stores a state in which the inverted pendulum control process is started. The start state storage means C1A according to the first embodiment sets the constants M, m, L, t, and g for the carriage 1 and the rod 2 as the state in which the inverted pendulum control process is started (M = 1. 0, m = 0.1, L = 1.0, t = 0.02, g = 9.8), set values of reinforcement learning constants α, γ (α = 0.1, γ = 0.95), Initial values of each variable x, v, θ, ω, a, b, k, F, w _ji , w _ji ′ (x (0) = v (0) = ω (0) = a (0) = b ( 0) = w _ji (0) = w _ji ′ (0) = 0.0, θ (0) = − 3.0 to 3.0, k = 0, F (0) = 20.0) To do.

In Example 1, the action a _t is, because the is the control of the movement in the lateral direction of the carriage 1 (Y-axis direction), the load coupling for evaluation of the movement in the right direction (+ Y direction) w _{j i} and load coupling for evaluation of movement in the left direction (−Y direction) as w _ji ′ are provided separately so that learning of each of the load couplings w _ji and w _ji ′ does not interfere with each other. Is set in advance. In Example 1, the respective load coupling in the case of untrained _{w ji, w ji} 'initial values _w ji of _{(0), w} ji' in case of the (0), any state _(s t = x (0), v (0), θ (0), ω (0)), the load coupling w _ji (0), w _ji ′ (0) is _expressed as w _ji (0) = w _ji It is preset as indicated by ′ (0) = 0.0.

C2: State measuring unit The state measuring unit C2 measures the states s _t and _{st +1} . The state measuring means C2 according to the first embodiment includes the equations (6-1), (6-2), (7-1) to (7-4), and (7-1) ′ to (7-4) ′. And the constants M, m, L, g, t and the variables a, b, k, F, the values of the input variables x, v, θ, ω for each time step t (input values). The states s _t and s _{t + 1} are measured by calculating x (k), v (k), θ (k), ω (k) (k = 0, 1, 2,...)) (s _t = X (k), v (k), θ (k), ω (k), _{st + 1} = x (k + 1), v (k + 1), θ (k + 1), ω (k + 1)).

C3: action executing means action executing means C3 has an action selection means C3A, and a carriage movement control means C3B, performing the action a _t, that is, the action executing means C3 is the time per step t The cart 1 is moved in the left-right direction (Y-axis direction) according to the force F (k).
C3A: behavior selection means action selection means C3A selects an action _{a t} in accordance with any action selection method. Behavior selection means C3A of Example 1, on the basis of the greedy selection method, the action value function _{Q (s} t, _{a t)} is an evaluation value selects the action _{a t} with the maximum.
C3B: Carriage Movement Control Means The carriage movement control means C3B controls the movement of the carriage 1 in the left-right direction (Y-axis direction) by controlling the drive of the drive unit 1b and rotating the drive wheels 1a. To do.

C4: Reward acquisition means Reward acquisition means C4 acquires the rewards r _t and r _{t + 1} . In the reward acquisition means C4 according to the first embodiment, the rewards r _t and r _{t + 1 for} each time step t include the angle θ of θ ≦ −180 ° or θ ≧ 180 ° when the rod 2 is tilted. If it becomes, it is acquired as a negative value (r _t <0, r _{t + 1} <0). Otherwise, it is acquired as a value of 0 or more (r _t ≧ 0, r _{t + 1} ≧ 0). ). The reward acquisition means C4 is preset so that the values of the rewards r _t and r _{t + 1} increase as the angle θ is closer to 0 [deg].
C5: Action value function calculating means The action value function calculating means C5 calculates the action value functions Q (s _t , a _t ), Q (s _{t + 1} , a _{t + 1} ). The action value function calculating unit C5 according to the first exemplary embodiment calculates the action value function Q (s _{t at} each time step t, that is, at time k × t, (k + 1) × t, based on the equation (8). , A _t ), Q (s _{t + 1} , a _{t + 1} ) (k = 0, 1, 2,...).

C6: Action value function storage means The action value function storage means C6 is calculated by the action value function calculation means C5, and the action value functions Q (s _t , a _t ), Q (s _{t + 1} ) for each time step t. , A _{t + 1} ). The action value function storage unit C6 according to the first exemplary embodiment stores all the states s _t and s _{t + 1} (s _t = x (k), v (k), θ (k) at the time k × t and (k + 1) × t. ), Ω (k), s _{t + 1} = x (k + 1), v (k + 1), θ (k + 1), ω (k + 1)), the behavior value functions Q (s _t , a _t ), Q (s _{t + 1} , a _{t + 1} ) is stored (k = 0, 1, 2,...).
C7: Action value function approximating means The action value function approximating means C7 includes an input variable generating means C7A, an input variable storing means C7B, an intermediate variable calculating means C7C, an intermediate variable storing means C7D, an output variable calculating means C7E, An output variable storage means C7F and an action value function learning means C7G are provided, and an action value function approximating process for approximating the action value function Q (s _t , a _t ) is executed by the multivariable mutual modification model N. . That is, the behavior value function approximation process for calculating an approximate value of the behavior value function Q (s _t , a _t ) in the state s _t is executed.

C7A: Input variable generation means (input variable input means)
The input variable generation means C7A is configured to input values (x (k), v (k), θ (k), ω (k) of the input variables x, v, θ, ω that continuously change at each time step t. ), X (k + 1), v (k + 1), θ (k + 1), ω (k + 1)). The input variable generation means C7A according to the first embodiment uses the input variables x and v calculated by the state measurement means C2 at each time step t in the input layer Na (see FIG. 4) of the multivariable mutual modification model N. , Θ, ω corresponding to the input values, an array of values of the elements of the input variables x, v, θ, ω, so-called state patterns are set.

FIG. 5 is an explanatory diagram for explaining an example of an input variable state pattern indicating the position of the carriage.
In the input variable generation means C7A of the first embodiment, all the 4n elements of the input variables x, v, θ, and ω in the input layer Na are previously set to have a value of −1 or +1, respectively. Is set. Therefore, for example, when the state pattern of the input variable x (the position x of the carriage 1) is S _x and S _x = (s _x1 , s _x2 ,..., S _xn ), n pieces of the input variable x S _xi (i = 1, 2,..., N) can be represented by s _xi = + 1 or s _xi = −1.
In the input variable generation means C7A, the input variables x, v, θ, and ω are preset so that n / 2 elements are +1 and the remaining n / 2 elements are -1. ing. Thus, for example, as shown in FIG. 5, with respect to the state pattern S _x of the input variable x (the position x of the carriage 1), n / 2 pieces in a range corresponding to the position where the center of gravity of the carriage 1 actually exists. It is preset that the element s _xi outputs +1 and the remaining n / 2 elements s _xi output -1.

Further, in the input variable generation means C7A, the state pattern of the input variable θ (the angle θ of the rod 2) seems to have a particularly large influence during learning, so the main value of the input variable θ, for example, When θ is a value in the vicinity of 0.0 [deg] (θ≈0), the state pattern is set in advance so as to change greatly by a slight change in value. That is, the state pattern of the input variable θ is S _θ, and each state pattern corresponding to the input values θ (k), θ (k + 1) of the input variable θ at the time k × t, (k + 1) × t is S. _{When θ (k)} and S _{θ (k + 1)} are set, the input variable θ is (θ (k) ≈0 if the input values θ (k) and θ (k + 1) are in the vicinity of 0.0 [deg]. , Θ (k + 1) ≈0), and a slight change in input value when the time has changed from k × t to (k + 1) × t (θ (k + 1) −θ (k) ≈0). The state pattern S _{θ (k + 1)} changes greatly compared to the state pattern S _{θ (k)} before the change, and if the value is other than that, the state pattern S _{θ (( k + 1)} is set in advance so as not to change significantly compared to the state pattern S _{θ (k)} before the change.

C7B: Input Variable Storage Unit The input variable storage unit C7B stores the input values of the input variables x, v, θ, and ω generated by the input variable generation unit C7A. The input variable storage means C7B according to the first embodiment stores the state patterns S _x , S _v , S _θ , S _ω (see FIG. 4) of the input variables x, v, θ, ω at each time step t. To do.

C7C: Intermediate variable calculation means The intermediate variable calculation means C7C has selective desensitization means C7C1, and calculates the intermediate variables based on the input variables x, v, θ, ω at each time step t. As shown in the intermediate layer Nb, the intermediate variable calculation means C7C according to the first embodiment in FIG. 4 sets the input variables x, v, θ, and ω to 2 each at the time step t. Twelve product type modifications x (v), x (θ), x (ω), v (θ), v (ω), v (x), θ (ω that are mutually product type context modified as a set ), Θ (x), θ (v), ω (x), ω (v), ω (θ) are calculated (output) as intermediate variables (y). That is, the intermediate variable is y (y = (x (v), v (x), x (θ), θ (x), x (ω), ω (x), v (θ), θ (v ), V (ω), ω (v), θ (ω), ω (θ))), the state of the input variables x, v, θ, ω when the intermediate variable state pattern is S _y. From the patterns S _x , S _v , S _θ , S _ω , the state pattern S _y (S _y = (S _x (S _v ), S _v (S _x ), S _x (S _θ ), S)) of the intermediate variable y is obtained. _θ (S _x ), S _x (S _ω ), S _ω (S _x ), S _v (S _θ ), S _θ (S _v ), S _v (S _ω ), S _ω (S _v ), S _θ (S _ω ), S _ω (S _θ ))) is calculated.

C7C1: Selective desensitizing means The selective desensitizing means C7C1 includes first desensitizing means C7C1a, first output sensitivity calculating means C7C1b, second desensitizing means C7C1c, and second output sensitivity calculating means C7C1d. Then, at the output of the intermediate variable y, a selective desensitization process for selectively desensitizing the input variables x, v, θ, and ω is executed. The selective desensitizing means C7C1 according to the first embodiment executes the selective desensitizing process with each of the input variables x, v, θ, ω as two sets. For example, two input variables x, v selected from four input variables x, v, θ, ω (position x and speed v of the carriage 1) are set as one set, and the input variable x is set as a first selection variable, When the input variable v is a second selection variable, the second selection variable v selectively desensitizes the first selection variable x and the first selection variable x selectively selects the second selection variable v. Desensitize.

Specifically, in FIG. 4, first, the state pattern S _x (S _x = (s _x1 , s _x2 ,..., S _xn ) = s _xi (i = 1, 2,...) Of the first selection variable x. n)), the state pattern of the second selection variable v is expressed as S _v (S _v = (s _v1 , s _v2 ,..., s _vn ) = s _vi (i = 1, 2,..., n) ), And G _x (G _x = (g _x1 , g _x2 ,..., G _xn ) = g _xi (i = 1, 2, _v) based on the state patterns S _x and S _v of the selection variables x and v , N)), G _v (G _v = (g _v1 , g _v2 ,..., G _vn ) = g _vi (i = 1, 2,..., N)), and the product type modification x (v), The state pattern of v (x) is _expressed as S _x (S _v ) (S _x (S _v ) = (y _{xv 1} , y _{xv 2} ,..., y _xvn ) = y _xvi (i = 1, 2,... , N)), S _v (S _x ) (S _v (S _x ) = (y _vx1 , y _vx2 ,..., Y _vxn ) = y _vxi (i = 1, 2,..., N)).

Then, the expressions (1) and (2) are applied to the state patterns S _x and S _v of the selection variables x and v. In Example 1, the probabilities that the values (+1 or −1) of the elements s _xi , s _vi (i = 1, 2,..., N) of the state patterns S _x , S _v exist are equal, and It is set in advance so as to be determined independently (no correlation). Therefore, the formula (1) ′ can be applied instead of the formula (1). Here, with respect to the state patterns S _x and S _v of the selection variables x and v, the expressions to which the expressions (1) ′ and (2) are applied are expressed by the following expressions (9-1) and (9-2). , (10-1), (10-2).
y _xvi = g _vi × s _xi (9-1)
g _vi = (1 + s _vi ) / 2 (9-2)
y _vxi = g _xi × s _vi (10-1)
g _xi = (1 + s _xi ) / 2 (10-2)

C7C1a: First desensitizing means The first desensitizing means C7C1a does not reflect (desensitize) the value of each element of the first selection variable to the value of each element of the intermediate variable. In the first desensitizing means C7C1a of the first embodiment, for example, when the input variable x is selected as the first selection variable and the input variable v is selected as the second selection variable, according to the equation (9-1), Each element y _xvi (i = 1, 2,..., N) of the product type modification x (v) is calculated from each element s _xi (i = 1, 2,..., N) of the first selection variable x. At this time, if the gain g _vi becomes 0 (g _vi = 0), the corresponding element y _xvi becomes y _xvi = 0. That is, the value (+1 or −1) of each element s _xi of the first selection variable x becomes the value of each element y _xvi (i = 1, 2,..., N) of the product type modification x (v). It is not reflected and becomes 0. In this specification, this situation is referred to as “desensitized”.
C7C1b: First output sensitivity calculation means The first output sensitivity calculation means C7C1b calculates a gain vector of a gain (first output sensitivity) for desensitizing the first selection variable. For example, when the input variable x is selected as the first selection variable and the input variable v is selected as the second selection variable, the first output sensitivity calculation means C7C1b of the first embodiment is the state pattern of the second selection variable v. Based on S _v , the gain vector G _v = g _vi (i = 1, 2,..., N) for desensitizing the first selection variable x is calculated (see Expression (9-2)).
The first desensitizing means C7C1a and the first output sensitivity calculating means C7C1b constitute first intermediate variable calculating means (C7C1a + C7C1b) of the first embodiment.

C7C1c: Second desensitizing means The second desensitizing means C7C1c does not reflect (desensitize) the value of each element of the second selection variable to the value of each element of the intermediate variable. In the second desensitizing means C7C1c of the first embodiment, for example, when the input variable x is selected as the first selection variable and the input variable v is selected as the second selection variable, according to the equation (10-1), Each element y _vxi (i = 1, 2,..., N) of the product type modification v (x) is calculated from each element s _vi (i = 1, 2,..., N) of the second selection variable v. At this time, when the gain g _xi becomes 0 (g _xi = 0), the corresponding element y _vxi becomes y _vxi = 0. That is, the value (+1 or −1) of each element s _vi of the second selection variable v becomes the value of each element y _vxi (i = 1, 2,..., N) of the product type modification v (x). It is not reflected and becomes 0. That is, it is desensitized.
C7C1d: second output sensitivity calculation means The second output sensitivity calculation means C7C1d calculates a gain vector of gain (second output sensitivity) for desensitizing the second selection variable. For example, when the input variable x is selected as the first selection variable and the input variable v is selected as the second selection variable, the second output sensitivity calculation unit C7C1d according to the first embodiment uses the state pattern of the first selection variable x. Based on S _x , the gain vector G _x = g _xi (i = 1, 2,..., N) for desensitizing the second selection variable v is calculated (see Expression (10-2)).
The second desensitizing means C7C1c and the second output sensitivity calculating means C7C1d constitute second intermediate variable calculating means (C7C1c + C7C1d) of the first embodiment.

C7D: Intermediate variable storage means The intermediate variable storage means C7D stores the value (intermediate value) of the intermediate variable calculated by the intermediate variable calculation means C7C (y = (x (v), v (x) , X (θ), θ (x), x (ω), ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ) )). The intermediate variable storage unit C7D according to the first embodiment performs the state pattern S _y (S _y = (S _x (S _v ), S _v (S _x ), S _x (S _x ), S _x (S _θ ), S _θ (S _x ), S _x (S _ω ), S _ω (S _x ), S _v (S _θ ), S _θ (S _v ), S _v (S _ω ), S _ω (S _v ), S _θ (S _ω ), S _ω (S _θ ))).

C7E: Output variable calculation means The output variable calculation means C7E outputs the intermediate variable y (y = (x (v), v (x), x (θ), θ (x), x (ω) at each time step t. ), Ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ)))) based on the output variable Q (s _t , a _t ) Is calculated. In FIG. 4, the output variable calculation means C7E according to the first embodiment changes the state pattern of the output variable Q (s _t , a _t ) to S _Q (S _Q = (q ₁ , q ₂ ,..., q _m ), (q ₁ ′, q ₂ ′,..., q _m ′) = q _j , q _j ′ (j = 1, 2,..., m)) and the state of the intermediate variable y When the following expression (11) is established for the pattern S _y , the following expressions (11-1) and (11-2) based on the expression (3) using the sign function sgn (u) State pattern S _y (S _y = (S _x (S _v ), S _v (S _x ), S _x (S _θ ), S _θ (S _x ), S _x (S _ω ), S)) of the intermediate variable y from _{ω (S x), S v} (S θ), S θ (S v), S v (S ω), S ω (S v), S θ (S ω), S ω (S θ))) It calculates the state pattern S _Q of the serial output variable.

(State pattern S _y of intermediate variable _y )
S _y
= (Y ₁ , y ₂ ,..., Y _12n )
= (Y _xv1 , y _xv2 ,..., Y _xvn , y _vx1 , y _vx2 ,..., Y _vxn ,
y _xθ1 , y _xθ2 ,..., y _xθn , y _θx1 , y _{θx2 ,.}
y _xω1 , y _xω2 , ..., y _xωn , y _ωx1 , y _ωx2 , ..., y _ωxn ,
y _vθ1 , y _vθ2 ,..., y _vθn , y _θv1 , y _{θv2 ,.}
y _vω1 , y _vω2 , ..., y _vωn , y _ωv1 , y _ωv2 , ..., y _ωvn ,
_yθω1 , _yθω2 , ..., _yθωn , _yωθ1 , _yωθ2 , ..., _yωθn )
... (11)
(When evaluating movement in the right direction (+ Y direction))
q _j = sgn (Σ _i w _ji y _i ) (i = 1, 2,..., 12n) (12-1)
(When evaluating movement in the left direction (-Y direction))
q _j ′ = sgn (Σ _i w _ji ′ y _i ) (i = 1, 2,..., 12n) (12-2)

C7F: Output Variable Storage Unit The output variable storage unit C7F stores the value (output value) of the output variable Q (s _t , a _t ) calculated by the output variable calculation unit C7E. The output variable storage means C7F of Example 1, the said output variable per time step _{t Q (s t, a t} ) state pattern _{S Q (S Q = (q} 1, q 2 of, ..., _{q m} ), (Q ₁ ′, q ₂ ′,..., Q _m ′) = q _j , q _j ′ (j = 1, 2,..., M)).
C7G: Action value function learning means (bond weight learning means)
The behavior value function learning means C7G has a learning determination means C7G1 combined weight calculation means C7G2 and a combined weight storage means C7G3, and performs behavior value function learning processing for learning the behavior value function Q (s _t , a _t ). Run. The action value function learning unit C7G according to the first embodiment is configured to perform the inverted pendulum control process after the end of the inverted pendulum control process stored in the action value function storage unit C6 when the inverted pendulum control process ends in the failure state. The state pattern of the behavior value function Q (s _t , a _t ) is _expressed as T _Q (T _Q = (t ₁ , t ₂ ,..., T _m ), (t ₁ ′, t ₂ ′,..., T _m ′). = T _j , t _j ′ (j = 1, 2,..., M)), the following equations (13-1) and (13-2) based on the equations (4) and (4) ′ coupling weight _{w ji, w ji} 'update value Δw _ji, Δw _ji' by calculating the coupling weight _{w ji,} by repeating the process of updating the _{w ji} ', the action value function Q _{(s t} to approximate , A _t ), the behavior value function learning process is executed. In Example 1, the positive constant ε is preset to 0.3 (ε = 0.3).

(When evaluating movement in the right direction (+ Y direction))
Δw _ji = ε (t _j −q _j ) y _i (13-1)
(When evaluating movement in the left direction (-Y direction))
Δw _ji ′ = ε (t _j ′ −q _j ′) y _i (13-2)
That is, the behavior value function learning means C7G uses the connection weights w _ji and w _ji ′ of the time k × t and (k + 1) × t as w _ji (k), w _ji (k + 1), and w _ji ′ (k ), W _ji ′ (k + 1), the combined loads w _ji , w _ji ′ are changed to w _ji (k), w _ji ′ (k) according to the following equations (14-1), (14-2). ) To w _ji (k + 1), w _ji ′ (k + 1) is repeated (k = 0, 1,...).

w _ji (k + 1) = w _ji (k) + Δw _ji (14-1)
w _ji ′ (k + 1) = w _ji ′ (k) + Δw _ji ′ (14-2)
Note that when the state patterns S _x , S _v , S _θ , S _{ω of the} input variables x, v, θ, ω when the time changes from k × t to (k + 1) × t are similar, the output The state pattern T _Q of the variable Q (s _t , a _t ) does not change greatly, and for example, the same state pattern (T _Q ) may be obtained. Therefore, in the behavior value function learning means C7G of the first embodiment, when the time changes from k × t to (k + 1) × t, the state patterns S _x , S of the input variables x, v, θ, ω. _{v 1} , S _θ , S _ω are similar (for example, 90% or more of the state patterns S _x , S _v , S _θ , S _ω have not changed), and the output variable Q (s _t , a _t ) State pattern _TQ does not change (Q (s _t , a _t ) = Q (s _{t + 1} , a _{t + 1} )), the update processing of the joint loads w _ji , w _ji ′ is omitted. Therefore, the action value function Q (s _{t, a t)} can reduce the computational steps of the process of learning (calculation amount).

C7G1: Learning discriminating means The learning discriminating means C7G1 has the state patterns S _x , S _v , S _{θ of the} input variables x, v, θ, ω when the time changes from k × t to (k + 1) × t. , S _ω are similar (for example, 90% or more of the state patterns S _x , S _v , S _θ , S _ω have not changed), and the output variables Q (s _t , a _t ), Q ( by _{s t + 1, a t +} 1 state pattern _{T Q} in) is equal to or changed, the time k × t, (k + 1 ) × t (k = 0,1,2, the bond in) load _{w ji} , W _ji ′ is discriminated whether or not to be omitted.
C7G2: Combined load calculating means The combined load calculating means C7G2 calculates the combined loads w _ji and w _ji ′ according to the equations (13-1), (13-2), (14-1), and (14-2). Calculate (update).
C7G3: Combined load storage means The combined load storage means C7G3 stores the combined loads w _ji , w _ji ′ calculated by the combined load calculation means C7G2.

C8: Inverted pendulum control end determining means (control end determining means)
The inverted pendulum control end determining means C8 includes an inverted pendulum control time measuring means C8A, a success state determining means C8B, and a failure state determining means C8C, and determines whether or not to end the inverted pendulum control process. The inverted pendulum control termination judgment means C8 of Example 1, the reward r _t of the state where the rod 2 from the start of the inverted pendulum control process has been inverted 180 seconds (successful state) or negative value is obtained It is determined whether or not the inverted pendulum control process is terminated by determining whether or not the state (failure state) is present.
C8A: Inverted pendulum control time measuring means When the inverted pendulum control time measuring means C8A starts the inverted pendulum control process, the elapsed time k × in the inverted pendulum control process is started by starting the timer TM described later. Time t (k = 0, 1,..., k _max (9000)) is measured.

C8B: Success state discriminating means success status determining means C8B is by the inverted pendulum control time measuring means C8A, from the start of the inverted pendulum control process, the reward r _t is a time even without taking a negative value (r _t ≧ 0) By determining whether or not the rod 2 is in an inverted state for 180 seconds (k = k _max = 9000), it is determined whether or not it is the successful state. Here, k _max is the maximum value of the natural number k (k _max = 180 / 0.02 = 9000).
C8C: Failed state discriminating means fails state discrimination means C8C, by the reward r _t as the failure state it is determined whether or not a negative value (r t _<0), whether the a failure state Is determined.
TM: Timer The timer TM measures each time k × t (k = 0, 1,..., K _max (9000)) when the inverted pendulum control process is executed.

(Description of Flowchart of Example 1)
Next, a processing flow of the function approximation program AP1 of the control unit C according to the first embodiment will be described with reference to a flowchart.
(Description of flowchart of main processing of embodiment 1)
FIG. 6 is a flowchart of the main process of the function approximation program according to the first embodiment of the present invention.
The processing of each ST (step) in the flowchart of FIG. 6 is performed according to a program stored in the ROM or the like of the control unit C. In addition, this process is executed by multitasking in parallel with other various processes of the control unit C.

The flowchart shown in FIG. 6 is started when the function approximating program AP1 is started after the cart 1 is powered on.
In ST1 of FIG. 6, the following processes (1) to (6) are executed, and the process proceeds to ST2.
(1) 0.0 is set as the initial value of the position x, velocity v, acceleration a of the carriage 1, the angular velocity ω of the rod 2, and the angular acceleration b (x (0) = v (0) = ω (0) = a (0) = b (0) = 0.0).
(2) A random number in the range of −3.0 ° to 3.0 ° is set as the initial value of the angle θ of the rod 2 (θ (0) = − 3.0 ° to 3.0 °).
(3) 20.0 is set as an initial value of the force F applied to the cart 1 in the right direction (+ Y direction) (F (0) = 20.0).
(4) 0.0 is set as an initial value of the coupling loads ω _ji and ω _ji ′ (ω _ji (0) = ω _ji ′ (0) = 0.0).
(5) The set values of the constants M, m, L, t, g and the set values of the reinforcement learning constants α, γ for the carriage 1 and the rod 2 are set (M = 1.0, m = 0. 1, L = 1.0, g = 9.8, t = 0.02, α = 0.1, γ = 0.95).
(6) Set 1 to the natural number k (k = 1).

In ST2, it is determined whether or not the learned connection weights ω _ji and ω _ji ′ stored in the connection weight storage means C7G3 exist. If yes (Y), the process proceeds to ST3, and, if no (N), the process proceeds to ST4.
In ST3, the learned connection weights ω _ji and ω _ji ′ stored in the connection weight storage means C7G3 are set. Then, the process proceeds to ST4.
In ST4, timing by the timer TM is started. Then, the process proceeds to ST5.

In ST5, formulas (6-1), (6-2), (7-1) to (7-4), (7-1) ′ to (7-4) ′, and constants M, m, L , G, t and each variable a (0), b (0), k, F (0), input variables x, v at time t (k = 1, t = 0.02 [s]) , theta, value of omega (input value x (1), v (1 ), θ (1), ω (1)) by calculating, measuring the state _{s t (s t = x (} 1), v (1), θ (1), ω (1)). Then, the process proceeds to ST6.
In ST6, an action value function approximation process (see a flowchart of FIG. 7 described later) for approximating the action value function Q (s _t , a _t ) is executed by the multivariable mutual modification model N (see FIG. 4). That is, the behavior value function approximation process for calculating the approximate value of the behavior value function Q (s _t , a _t ) at time k × t (k = 1, 2,..., K _max ) in the state s _t is executed. Then, the process proceeds to ST7.
In ST7, on the basis of the greedy selection method, an evaluation value action value function _{Q (s} t, _{a t)} is executed by selecting an action _{a t} with the maximum. That is, the carriage 1 is moved in the left-right direction (Y-axis direction) according to the force F (k) of time k × t (k = 1, 2,..., K _max ). Then, the process proceeds to ST8.

In ST8, the following processes (1) and (2) are executed, and the process proceeds to ST9.
(1) to get the reward _{r t} in accordance with the action _{a t.} Note that reward r _t of the first embodiment, the angle theta and collapse rods 2, theta ≦ -180 °, or, if it becomes theta ≧ 180 °, and is obtained as a negative value (r _t < 0), otherwise, it is obtained as a value greater than or equal to 0 (r _t ≧ 0). Further, the value of the reward r _t is set in advance so that the angle θ of the bar 2 is closer to 0 [deg].
(2) Expressions (6-1), (6-2), (7-1) to (7-4), each constant M, m, L, g, t and each variable x (k), v ( k), θ (k), ω (k), a (k), b (k), k, F (k), and input variables x, v, θ, ω of time (k + 1) × t value (input value x (k + 1), v (k + 1), θ (k + 1), ω (k + 1)) by calculating, measuring the next state _{s t + 1} after action _{a t (s t + 1 =} x ( k + 1), v (k + 1), θ (k + 1), ω (k + 1) (k = 1, 2,..., k _max )).

In ST9, based on the equation (8), the behavior value Q (s _t , a _t ) for the time k × t (k = 0, 1, 2,..., K _max ) is calculated (updated). Execute function update processing. Then, the process proceeds to ST10.
In ST10, and it measures the reward _{r t} from the start of counting of the timer TM negative value without also not acquire once _{(r t} ≧ 0) 180 seconds _(k = _k max = 9000) state, i.e., rod It is determined whether or not 2 is in a successful state by determining whether or not 2 is in an inverted state for 180 seconds. If no (N), the process moves to ST11, and if yes (Y), the process returns to ST1.

In ST11, it is determined whether or not it is a failure state by determining whether or not a negative value of the reward r _t (r _t <0) is obtained. If no (N), the process proceeds to ST12, and if yes (Y), the process proceeds to ST13.
In ST12, +1 is added to the natural number k (k = k + 1). Then, the process returns to ST6.
In ST13, an action value function learning process (see the flowchart of FIG. 8 described later) for learning the action value function Q (s _t , a _t ) is executed. Then, the process returns to ST1.

(Description of Flowchart of Action Value Function Approximation Processing in Embodiment 1)
FIG. 7 is a flowchart of the action value function approximation process of the function approximation program according to the first embodiment of the present invention, and is an explanatory diagram of the subroutine of ST6 in FIG.
In ST101 of FIG. 7, input variables x, v, θ, ω of time k × t (k = 0, 1, 2,..., K _max ) are generated. That is, the state patterns S _x , S _v , S _θ , S _ω of the input variables x, v, θ, ω of the time k × t in the input layer Na (see FIG. 4) of the multivariable mutual modification model N are set ( To generate). Then, the process proceeds to ST102.

In ST102, intermediate variables y (y = (x (v), v (x), x (θ), θ (x), x (ω), ω () are generated from the generated input variables x, v, θ, ω. x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ))). That is, from the state patterns S _x , S _v , S _θ , S _{ω of} the input variables x, v, θ, ω, the state pattern S of the intermediate variable y in the intermediate layer Nb (see FIG. 4) of the multivariable mutual modification model N. _y (S _y = (S _x (S _v ), S _v (S _x ), S _x (S _θ ), S _θ (S _x ), S _x (S _ω ), S _ω (S _x ), S _v ) (S _θ ), S _θ (S _v ), S _v (S _ω ), S _ω (S _v ), S _θ (S _ω ), S _ω (S _θ ))) are calculated (formula (9-1) ), (9-2), (10-1), (10-2)). In ST102, as shown in FIG. 7, twelve product type modifications x (v), v (x), x (θ), θ (x), x (ω), which are intermediate variables y, Twelve STs 102a to 102m for calculating ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ) are executed in parallel.

In ST103, the output variable Q (s _t , a _t ) is calculated from the calculated intermediate variable y. That is, the state pattern _{S y} of the intermediate variable y, and calculates the state pattern _{S Q} of the output variable y at the output layer Nc multivariable cross-modified model N (see FIG. 4) (formula (11), (12-1) , (12-2)). Then, the behavior value function approximation process is terminated, and the process returns to the main process of FIG.

(Explanation of Flowchart of Behavior Value Function Learning Processing in Embodiment 1)
FIG. 8 is a flowchart of the action value function learning process of the function approximation program according to the first embodiment of the present invention, and is an explanatory diagram of the subroutine of ST13 in FIG.
In ST201 of FIG. 8, 1 is set to the variables i and j, and 0 is set to the updated values Δw _ji and Δw _ji ′ of the natural number k and the coupling weights w _ji and w _ji ′ (i = j = 1, k = 0, Δw). _ji = Δw _ji ′ = 0). Then, the process proceeds to ST202.

In ST202, the state patterns S _x , S _v , S of the input variables x, v, θ, ω when the time changes from k × t to (k + 1) × t (k = 0, 1,..., K _max ). _θ and S _ω are similar (for example, 90% or more of the state patterns S _x , S _v , S _θ , and S _ω have not changed), and the state pattern of the output variable Q (s _t , a _t ) It is determined whether or not _TQ has changed (s _t ≈s _{t + 1} and Q (s _t , a _t ) = Q (s _{t + 1} , a _{t + 1} )). If no (N), the process proceeds to ST203, and if yes (Y), the process proceeds to ST211.

In ST 203, stored in the action-value function storage means C6 (after episode termination) action value function _{Q (s t, a} t) state pattern _{T Q} and the time k × stored in the output variable storage means C7F of _{t (k = 0,1, ...,} k max) output variable _{Q (s t, a} t) of by the state pattern _{S Q} of the coupling weights _{w ji,} updated value of _{w ji} '(variation) [Delta] w _ji , Δw _ji ′ (see equations (13-1) and (13-2)). Then, the process proceeds to ST204.
In ST204, the combined loads w _ji and w _ji ′ are updated by the equations (14-1) and (14-2). Then, the process proceeds to ST205.

In ST205, it is determined whether or not the variable i is 12n or more (i ≧ 12n). If no (N), the process moves on to ST206, and if yes (Y), the process moves on to ST207.
In ST206, +1 is added to the variable i (i = i + 1). Then, the process returns to ST203.
In ST207, the variable i is reset to 1 (i = 1). Then, the process proceeds to ST208.

In ST208, it is determined whether or not the variable j is greater than or equal to m (j ≧ m). If no (N), the process moves on to ST209, and if yes (Y), the process moves on to ST210.
In ST209, +1 is added to the variable j (j = j + 1). Then, the process returns to ST203.
In ST210, variable j is reset to 1 (j = 1). Then, the process proceeds to ST211.
In ST211, it is determined whether or not the natural number k is smaller than k _max −1 (k <k _max −1). If yes (Y), the process transfers to ST212, and, if no (N), the action value function learning process ends, and the process returns to the main process in FIG.
In ST212, +1 is added to the natural number k (k = k + 1). Then, the process returns to ST202.

(Operation of Example 1)
In the function approximating system S of the first embodiment having the above-described configuration, the action a _t (added in the right direction (+ Y direction) with respect to the carriage 1) in the inverted pendulum control process (see ST1 to ST12 in FIG. 6). In order to approximate the action value function Q (s _t , a _t ) for selecting the force F), the action value function approximation process (ST6 in FIG. 6, ST101 in FIG. 7) based on the selective desensitization method. ST103) is executed. Therefore, the action-value function update process based on the Q-learning (ST10 in FIG. 6, equation (8) see) the action value function _{Q (s} t, _{a t)} in a state that is not learned _{(s t)} with regard to An approximate value of the action value function Q (s _t , a _t ) can be calculated, and based on the approximated action value function Q (s _t , a _t ), it is possible to select an appropriate action _{a t} in state _{s t} (see ST6, ST7 in FIG. 6).

In the function approximating system S of the first embodiment, when the inverted pendulum control process ends in a failed state, the combined load for approximating the action value function Q (s _t , a _t ). The behavior value function learning process (see ST13 in FIG. 6 and ST201 to ST212 in FIG. 8) for learning w _ji and w _ji ′ is executed. Therefore, if the inverted pendulum control process fails, in the next episode, that is, the next inverted pendulum control process, the action value function Q (s _t , a _{t with the} updated combined weights w _ji , w _ji ′. ) because it can approximate the action value function _{Q (s} t, _{a t)} be accurately approximated, it is possible to select the action _{a t} better (see ST2, ST3 in FIG. 6).

Here, in the multivariable mutual modification model N used in the behavior value function approximating process and the behavior value function learning process, the intermediate layer Nb (see FIG. 4) has four input variables x, v, θ, ω. For twelve ( ₄ P ₂ = 4 × 3 = 12) product-type modifiers x that are mutually product-type context-modified with two input variables (first selection variable, second selection variable) as one set in a permutation combination (V), v (x), x (θ), θ (x), x (ω), ω (x), v (θ), θ (v), v (ω), ω (v), θ (Ω) and ω (θ) are calculated as the intermediate variable y (see ST102 in FIG. 7, equations (9-1), (9-2), (10-1), and (10-2)). Therefore, in the function approximation system S of the first embodiment, there is a conventionally known selective desensitization method that is assumed only when the intermediate patterns X, X ′ and the output pattern Y are output for the two input patterns S, C. A function (Q (s _t ) having three or more input variables (x, v, θ, ω) that is not applicable to the applied layered neural network 01, 01 ′, 02 (see FIGS. 12 and 13). , A _t )), the multivariable mutual modification model N based on the selective desensitization method having high learning ability (generalization ability) can be applied.

As a result, the function approximating system S according to the first embodiment does not execute all trials (all actions (s _t ) in all states (s _t )) like the conventionally known reinforcement learning of Q-learning. Also, the time for learning to converge by executing the action value function approximation process and the action value function learning process based on the multivariable mutual modification model N with high learning ability (generalization ability), so-called learning time is reduced. can do.

(Experimental example)
Here, in order to examine how the learning efficiency has changed by the function approximation system S according to the first embodiment by executing the action value function approximation process and the action value function learning process, the following experiment is performed. Examples 1-3 and Comparative Examples 1-3 were prepared.

(Experimental example 1)
The function approximating system S of Experimental Example 1 is produced with the same configuration as the function approximating system S of Example 1, and in the main processing of Example 1 (see ST1 to ST13 in FIG. 6), the action value A function approximation process and the action value function approximation process are executed. That is, not only the action value function Q (s _t , a _t ) in the state s _t learned by the reinforcement learning of Q-learning but also the action value function Q (s _t , a _t ) not in the state s _t There is approximated, approximated action value function Q (s _{t, a t)} with the based actions a _t is executed, when the inverted pendulum control process has failed, the action value function Q (s _t, a Learning of the connection weights w _ji and w _ji ′ for approximating _{t 2} ) is performed.
In addition, the function approximation system S of Experimental Example 1 was tested under an environment E without disturbance and observation noise (see FIGS. 1 and 3).

(Comparative Example 1)
In the function approximation system S ′ of Comparative Example 1, the table reference method is applied to conventionally known reinforcement learning of Q-learning. Compared with the function approximation system S of Experimental Example 1, The value function approximation process and the behavior value function approximation process are omitted. That is, the function approximating system S ′ of the comparative example 1 stores the action value function Q (s _t , a _t ) in the learned state s _t in a table. Therefore, the state _{s t} learned, the state _{s t} definitive action value function _{Q (s} t, _{a t)} is the based on the appropriate action _{a t} is executed, the state _{s t} that is not learned, action value function Q (s _{t, a t)} is absent, it can not be executed appropriate action a _t to the state s _t is learned being attempted.

In the function approximation system S ′ of Comparative Example 1, the input variables x, v, θ, and ω are equally divided into ten. In other words, ten types of input values that can be taken by the input variables x, v, θ, and ω are preset. In addition, the money function approximation system S ′ of the first comparative example is the same as the function approximation system S of the first embodiment regarding the movement (action a _t ) in the right direction (+ Y direction) and the left direction (−Y direction). According to the connection loads w _ji and w _ji ′ provided separately, the evaluation table for each movement (action a _t ) in the right direction (+ Y direction) and the left direction (−Y direction) is separately provided. Provided. For this reason, the size of the table stored in Comparative Example 1 is indicated by 2 × 10 ⁴ (10 × 10 × 10 × 10 × 2 = 2 × 10 ⁴ ).
Further, the function approximating system S ′ of Comparative Example 1 was tested under the environment E as in Experimental Example 1.

(Experimental example 2)
The function approximating system S of Experimental Example 2 has the same configuration as that of the functional approximating system S of Experimental Example 1, and the experiment is performed under an environment E ′ in which an angular velocity ω _f as a disturbance is added to the environment E of Experimental Example 1. Went. Here, the disturbance (angular velocity) ω _f corresponds to an action in which the user flips the bar 2 with the finger in the left-right direction (Y-axis direction) when actually controlling the inverted pendulum instead of the simulator.

In the environment E ′ of Experimental Example 2, the disturbance ω _f is given each time the angle θ and the angular velocity ω of the rod 2 are kept within a predetermined range for 3 seconds. That is, a state where a value within a predetermined range is maintained for 3 seconds to make a stable state a stable state, and the minimum values of the angle θ and the angular velocity ω for determining whether the state is the stable state are θ _min , ω _Assuming that the maximum value is θ _max and ω _max , the angle θ and the angular velocity ω are maintained when the values in the range of θ _min ≦ θ ≦ θ _max and ω _min ≦ ω ≦ ω _max are maintained for 3 seconds. It is determined that the state is stable and the disturbance ω _f is given. In Experimental Example 2, θ _min = −2.5 [deg], ω _min = −45 [deg] with respect to the minimum values θ _min and ω _min and the maximum values θ _max and ω _max of the angle θ and the angular velocity ω. / S], θ _max = 2.5 [deg], and ω _max = 45 [deg / s] are set in advance (−2.5 ° ≦ θ ≦ 2.5 °, −45 ≦ ω ≦ 45).

Therefore, in the function approximation system S of Experimental Example 2, when the angle θ and the angular velocity ω maintain values in the range of −2.5 ° ≦ θ ≦ 2.5 ° and −45 ≦ ω ≦ 45 for 3 seconds. Is determined to be in the stable state, and the disturbance ω _f is given.
Further, in the environment E ′ of Experimental Example 2, an angular velocity ω _f that increases by 1 [deg / s] every time the disturbance ω _f is applied is defined as the disturbance ω _f in the left-right direction (Y-axis direction). And given randomly. That is, if the value indicating the number of disturbances (number of times of disturbance is given) is N _ω , ω _f = N _ω [deg / s] (when given in the left direction (−Y direction), ω _f = −N _ω [deg / s]) is established and can be added to the value (input value) of the angular velocity ω of the rod 2.

(Comparative Example 2)
The function approximation system S ′ of Comparative Example 2 was subjected to an experiment under the same environment E ′ as in Experiment Example 2 with the same configuration as the function approximation system S ′ of Comparative Example 1. Incidentally, the function approximation system S of Comparative Example 2 ', compared with the function approximation system S of Example 2, since the separated larger the state s _t in order to facilitate the convergence of learning (the input variable x, Since the types of input values of v, θ, and ω are 10), the time required for stabilization becomes long, and the amount of change in the input values of the input variables x, v, θ, and ω even if they are stable Is getting bigger. For this reason, in the environment E ′ of the comparative example 2, in order to loosen the range of the stability condition as compared with the experimental example 2, the minimum values θ _min , ω _min and the maximum values θ _{max of} the angle θ and the angular velocity ω For ω _max , θ _min = −6.0 [deg], ω _min = −150 [deg / s], θ _max = 6.0 [deg], and ω _max = 150 [deg / s]. It is set in advance (−6 ° ≦ θ ≦ 6 °, −150 ≦ ω ≦ 150). That is, in the function approximating system S ′ of Comparative Example 2, the angle θ and the angular velocity ω are stable when the values in the range of −6 ° ≦ θ ≦ 6 ° and −150 ≦ ω ≦ 150 are maintained for 3 seconds. Is determined to be in the state, and the disturbance ω _f is given. Furthermore, since the time required for the function approximation system S ′ of Comparative Example 2 to be in a stable state is long, the function approximation system S ′ is in a successful state with respect to the inversion time of the rod 2 (the state in which the rod 2 is inverted for 180 seconds). The upper limit of the inversion time of 180 seconds is not set.

(Experimental example 3)
The function approximating system S of Experimental Example 3 has the same configuration as that of the functional approximating system S of Experimental Example 1. Experiments were performed under an environment E ″ to which errors Δx and Δθ were added. Here, the observation noises Δx and Δθ are sensors provided in the carriage 1 when actually controlling an inverted pendulum instead of a simulator. This corresponds to so-called sensor noise (position measurement sensor, tilt angle measurement sensor, etc.) In the environment E ″ of Experimental Example 3, the observation noises Δx and Δθ are time k × t (k = 0). , 1, 2,..., K _max ), a random number is generated at each position x, an error Δx in the range of ± 0.05 [m] at the position x, and an error in the range of ± 0.5 [deg] at the angle θ. It is given as Δθ.
(Comparative Example 3)
The function approximation system S ′ of Comparative Example 3 was tested under the same environment E ″ as in Experiment Example 3 with the same configuration as the function approximation system S ′ of Comparative Example 1.

(Experimental result)
Moreover, it shows below about the experimental result of Experimental Examples 1-3 and Comparative Examples 1-3.

(Experimental results of Experimental Example 1 and Comparative Example 1)
FIG. 9 is an explanatory diagram of the experimental results of the experimental example. The horizontal axis indicates the number of trials of the inverted pendulum control process (number of episodes), and the vertical axis indicates the trial time of the inverted pendulum control process (the time during which the bar has been inverted). 9A is a graph for comparing the learning efficiency of Experimental Example 1 with the learning efficiency of Comparative Example 1, FIG. 9A is a graph showing the experimental results of Experimental Example 1, and FIG. 9B is the experiment of Comparative Example 1. It is a graph which shows a result.
As shown in FIGS. 9A and 9B, the function approximation system S of Experimental Example 1 is in a successful state (a state in which the rod 2 is inverted for 180 seconds) after about 10 trials (episodes). It can be seen that the function approximating system S ′ of Comparative Example 1 required about 1300 trials before it became a successful state. That is, it can be seen that the function approximation system S of Experimental Example 1 has a faster learning start and higher learning ability (generalization ability) than the function approximation system S ′ of Comparative Example 1.

In FIG. 9A, the function approximation system S of Experimental Example 1 often shows trials that are in a failed state (a state in which the rod 2 is collapsed) even after the first successful state. The results of this experiment are considered as follows. That is, in the function approximation system S of Experimental Example 1, the input variable (angle) θ is (θ (k)) when the input values θ (k) and θ (k + 1) are in the vicinity of 0.0 [deg]. ≒ 0, θ (k + 1) ≒ 0), a slight change in input value when the time changes from k × t to (k + 1) × t (θ (k + 1)-θ (k) ≒ 0), change If the subsequent state pattern S _{θ (k + 1)} changes significantly compared to the state pattern S _{θ (k)} before the change, and the other input values θ (k) and θ (k + 1), the input value is small. In this change, the state pattern S _{θ (k + 1)} after the change is set in advance so as not to change significantly compared to the state pattern S _{θ (k)} before the change.

When the time changes from k × t to (k + 1) × t (k = 0, 1,..., K _max ) in order to determine whether or not to learn the connection weights w _ji and w _ji ′. State variables S _x , S _v , S _θ , S _ω are similar (for example, 90% or more of the state patterns S _x , S _v , S _θ , S _ω change). and yet not), and the output variable _{Q (s} t, it is determined whether or not the state pattern _{T Q} of _{a t)} does not change (see ST202 of FIG. 8).
Therefore, the function approximating system S of Experimental Example 1 has more opportunities for learning to maintain a stable state (−1 ° ≦ θ ≦ 1 °) for learning of the coupling weights w _ji and w _ji ′. There are fewer opportunities for learning from a non-stable state to a stable state. Therefore, when the initial value θ (0) of the angle θ is started at an angle larger than the stable state (−1 ° ≦ θ ≦ 1 °) (for example, θ (0) = 3 [deg]), It is presumed that the possibility of failure has increased.

Further, in FIG. 9B, after the function approximation system S ′ of the comparative example 1 is in a successful state for the first time, trials that are in a stable state in a stable state are repeated. According to the experimental results, in the function approximation system S ′ of the comparative example 1, learning is converged, that is, an appropriate action value function Q (s _t , a _t ) is set in 2 × 10 ⁴ tables. Therefore, about 1300 trials are necessary, but if an appropriate action value function Q (s _t , a _t ) is set in the table, trials that are stable and succeed after that are performed. You can see that it can be repeated.

(Experimental results of Experimental Example 2 and Comparative Example 2)
FIG. 10 is an explanatory diagram of the experimental results of the experimental example, where the horizontal axis represents the number of trials of the inverted pendulum control process (number of episodes), and the vertical axis represents the trial time of the inverted pendulum control process (the time during which the rod continued to be inverted). FIG. 10A is a graph for comparing the learning efficiency of Experimental Example 2 and the learning efficiency of Comparative Example 2 by taking the number of times disturbance was applied (number of times the stick was played with a finger), and FIG. FIG. 10B is a graph showing the experimental results of Comparative Example 2.

As shown in FIG. 10A, in the function approximating system S of Experimental Example 2, the trial time k × t (k = 0, 1,...) With respect to the number of trials (number of episodes) by about 50 to 100 trials. The waveform of k _max (9000)) (see the solid line in FIG. 10A) corresponds to the waveform of the number of times N _ω (see the dotted line in FIG. 10A) given the disturbance ω _f with respect to the number of trials. That is, it can be seen that the waveforms are similar.
On the other hand, as shown in FIG. 10B, in the function approximation system S ′ of Comparative Example 2, the waveform of the trial time k × t (k = 0, 1,..., K _max (9000)) with respect to the number of trials (FIG. 10B). see the solid line of 10B) is a waveform of times N _omega gave the disturbance omega _f against attempts (so far (waveform until corresponding to dashed see FIG. 10B) come to similar), also attempted to about 6000 times It was found that was necessary.

In FIG. 10B, in the function approximation system S ′ of Comparative Example 2, it seems that sufficient trial time (inversion time) is obtained in about 4000 trials, but up to about 4000 times. It can be seen that since the number of times N _ω given the disturbance ω _f in the trial is very small, it only takes a very long time to reach a stable state. In addition, in the trial from about 4000 times to about 6000 times, the trial time is very short although the disturbance ω _f is hardly given. Therefore, it can be seen that the function approximation system S ′ of Comparative Example 2 required about 6000 trials for the learning to converge.
As a result, the function approximation system S of Example 2, compared with the function approximation system S 'of Comparative Example 2, the disturbance ω adaptability of about 60 to 120 times with respect to _{f (6000/100 = 60,6000 /} 50 = 120).

(Experimental results of Experimental Example 3 and Comparative Example 3)
FIG. 11 is an explanatory diagram of the experimental results of the experimental example, where the horizontal axis represents the number of trials of the inverted pendulum control process (number of episodes), and the vertical axis represents the trial time of the inverted pendulum control process (the time during which the bar continued to be inverted). 11A is a graph for comparing the learning efficiency of Experimental Example 3 with the learning efficiency of Comparative Example 3, FIG. 11A is a graph showing the experimental results of Experimental Example 3, and FIG. 11B is the experiment of Comparative Example 3 It is a graph which shows a result.

  As shown in FIGS. 11A and 11B, the function approximating system S of Experimental Example 3 has succeeded in about 10 trials (episodes), whereas the function approximating system S ′ of Comparative Example 3 is It can be seen that about 1300 trials were required to reach a successful state. In addition, the function approximation system S of Experimental Example 3 can stably obtain a successful state without being affected by the observation noises Δx and Δθ in about 70 trials, whereas Comparative Example 3 It can be seen that the function approximation system S ′ of FIG. 1 does not stably obtain a successful state even after about 1500 trials. That is, it can be seen that the function approximation system S of Experimental Example 3 has higher adaptability to the observation noises Δx and Δθ than the function approximation system S ′ of Comparative Example 3.

Further, in FIG. 11A, the function approximating system S of Experimental Example 3 is a trial that becomes a failed state (a state in which the rod 2 is collapsed) after a successful state is stably obtained in about 70 trials. It can be seen that the number is less than that of Experimental Example 1. The results of this experiment are considered as follows. That is, the input values x (k), v (k), θ (k), ω (k) (k = 0, 1) that the input variables x, v, θ, ω can take due to the observation noises Δx, Δθ. ,..., K _max (9000)) is wider than that of the function approximation system S of Experimental Example 1. For this reason, the function approximation system S of Experimental Example 3 has more opportunities for learning from the non-stable state to the stable state, and is more stable than the function approximation system S of Experimental Example 1. It is inferred that the trial to become a successful state can be repeated.

Further, in FIG. 11B, the function approximation system S ′ of the comparative example 3 repeats trials that stably become a successful state even after the success of the function approximation system S ′ of the comparative example 1 for the first time. I can't understand that The results of this experiment are considered as follows. That is, in the function approximation system S ′ of Comparative Examples 1 to 3, the state _st is largely divided in order to facilitate learning (types of input values of the input variables x, v, θ, and ω). 10), the input values x (k), v (k), θ (k), ω (k) (k = 0, 1,...) Of the input variables x, v, θ, ω even in a stable state. , K _max (9000)) can take a wide range of values, and the movement of the carriage 1 and the rod 2 is larger than that of the function approximation system S of Experimental Example 3. From this state, further the observation noise [Delta] x, when Δθ is added, or to maintain a stable state, it becomes difficult to select appropriate actions a _t Gayori or a stable state from a state not in a stable state. As a result, it is presumed that learning becomes difficult to converge, and it is impossible to repeat trials that stably become successful.

Based on the experimental results, the function approximation system S of the first embodiment has learning ability (generalization ability) compared to the conventionally known function approximation system S ′ in which the action value function approximation process and the action value function learning process are not executed. In addition to being able to greatly increase, the applicability to unknown environments E ′ and E ″ including disturbance ω _f and observation noises Δx, Δθ, etc. can also be greatly increased. The function approximating system S controls the inverted pendulum more appropriately than the conventionally known function approximating system S ′ even when applied to a device (function approximating device) that actually controls the inverted pendulum instead of the simulator. be able to.

In the function approximation system S of the first embodiment, the twelve product type modifications x (v), v (x), x (θ), θ (x), x (ω), For ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), and ω (θ), the input layer Na (see FIG. 4), the state patterns S _x , S _v , S _θ , S _{ω of the} input variables x, v, θ, _ω are given in parallel (see ST102a to ST102m in FIG. 7). ). Therefore, for example, each product type modification x (v), v (x), x (θ), θ (x), x by hardware such as a plurality of LSIs (Large-Scale Integrated circuits). When (ω), ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ) are calculated in parallel, the intermediate Variable y can be calculated at high speed (12 LSIs can be calculated at about 12 times faster, and 6 LSIs can be calculated at about 6 times faster), reducing the calculation time of the action value function approximation process. it can.

(Example of change)
As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to the said Example, A various change is performed within the range of the summary of this invention described in the claim. It is possible. Modification examples (H01) to (H08) of the present invention are exemplified below.
(H01) The function approximating system S according to the first embodiment of the present invention is configured as a simulator for solving the so-called inverted pendulum problem, but is not limited to the simulator, and is also used in a control unit of an actual control device. Applicable. Also, the problem is not limited to the inverted pendulum problem, and can be applied to other problems, for example, a robot that controls bipedal walking, a pattern recognition device for images and sounds, and the like.

(H02) In the embodiment of the present invention, the reinforcement learning of Q-learning is applied to solve the inverted pendulum problem. However, the present invention is not limited to this, and an algorithm other than Q-learning can be applied. It is. That is, it is possible to approximate a function other than the behavior evaluation function Q (s _t , a _t ). Further, depending on the problem, only the function (see ST6 in FIG. 6 and ST101 to ST103 in FIG. 7) that approximates the function by the multivariable mutual modification model N without applying reinforcement learning is extracted, and the problem is solved. It is also possible to configure a function approximating device to be solved. That is, the function approximating device for approximating the function (nonlinear function or the like) as the problem may be configured by a layered neural network (multivariable mutual modification model N or the like) to which the selective desensitization method of the present invention is applied. Is possible.

(H03) In the embodiment of the present invention, in the multivariable mutual modification model N, the input layer Na is configured by four input variables x, v, θ, ω, but the present invention is not limited to this. The case of variables and the case of five or more input variables can also be configured.
In the embodiment of the present invention, the intermediate variable y of the intermediate layer Nb is set to 12 of the four input variables x, v,. Product type modifications x (v), v (x), x (θ), θ (x), x (ω), ω (x), v (θ), θ (v), v (ω), The state pattern S _y of the intermediate variable y is composed of 12n elements (see FIG. 4, equation (11)), but is limited to this. It is also possible not to mutually modify the product type context.
For example, as a minimum configuration, two of the four input variables x, v, θ, ω ((x, v), (θ, ω)) are set as a pair, and the product type context modified two Only the product type modifications x (v) and θ (ω) may be the intermediate variable y, and the state pattern S _y of the intermediate variable y may be configured by 2n elements. In this case, for the state pattern S _y of the intermediate variable y, S _y = (y ₁ , y ₂ ,..., Y _2n ) = (y _xv1 , y _xv2 ,..., Y _xvn , y _θω1 , y _{θω2,. yθωn} ) holds. In the present specification, the layered neural network having the minimum configuration of the present invention is referred to as a “multivariate product model” with respect to the “multivariate mutual modification model N” in the first embodiment.
Furthermore, in the embodiment of the present invention, the output layer Nc is configured by one output variable Q (s _t , a _t ), but is not limited thereto, and may be configured by a plurality of output variables.

(H04) Each product type modification x (v), v (x), as the input variable x, v, θ, ω and intermediate variable y in the multivariable mutual modification model N (see FIG. 4) of the embodiment of the present invention. x (θ), θ (x), x (ω), ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ) The value of each element number n can be arbitrarily set, and for example, n = 1 can be set. In this case, the number of input variables x, v, θ, ω is equal to the number of elements in the input layer Na (4n = 4 × 1 = 4).
Therefore, in the layered neural network to which the selective desensitization method of the present invention is applied, when the number of elements of the input layer (Na) is fixed at n, the multivariable mutual modification model (N ), The maximum value of the number of elements of the intermediate layer (Nb) is n (n−1). That is, when there are n input variables having the number of elements of 1, _n P ₂ = n (n−1) holds for the maximum value. In the multivariate product model having the minimum configuration, the minimum value of the number of elements in the intermediate layer is n / 2. That is, when there are four input variables with the number of elements n / 4, n / 4 × 2 = n / 2 holds for the minimum value.
As a result, in the layered neural network to which the selective desensitization method of the present invention is applied, when the number of elements in the input layer is n, the number of elements in the intermediate layer is n / 2 to n (n -1) It can be composed of a range.

(H05) The set values of the constants M, m, L, t, g, α, γ and the variables x, v, θ, ω, a, b, k, F, w _ji , in the first embodiment of the present invention. The initial value of w _ji ′, the range of possible values, and the like can be arbitrarily changed.
(H06) In the embodiment of the present invention, the behavior value function learning process (see ST13 in FIG. 6 and ST201 to ST212 in FIG. 8) is in the failed state, and the inverted pendulum control process (see ST1 to ST12 in FIG. 6) is in a failed state. Although it is executed only when it ends (see ST11 in FIG. 6), it is not limited to this, and it can also be executed when it ends in a successful state (see ST10 in FIG. 6). In this case, based on the latest action value function Q (s _t , a _t ) updated by the reinforcement learning of Q-learning (see ST9 in FIG. 6, equation (8)), the combined weights w _ji , w _ji Since ′ is learned, the number of calculation steps is increased as compared with the function approximation system S of the first embodiment, but the learning convergence on the action value function Q (s _t , a _t ) can be accelerated.

(H07) In the embodiment of the present invention, the intermediate layer Nb of the multivariable mutual modification model N is one layer. However, the present invention is not limited to this, and the intermediate layer Nb may be a plurality of layers.
(H08) In the embodiment of the present invention, the intermediate variables y (y = (x (v), v (x), x (θ), θ (x), x) are calculated from the input variables x, v, θ, ω. (Ω), ω (x), v (θ), θ (v), v (ω), ω (v), θ (ω), ω (θ))) are calculated. Desensitization is performed (see equations (9-1), (9-2), (10-1), and (10-2)), and the output variable Q (s _t , a _t ) is changed from the intermediate variable y. When it is calculated, it is calculated in the same manner as a conventionally known multilayer perceptron and is not selectively desensitized. However, the present invention is not limited to this, and the output variable Q (s _t , a _t ) is calculated from the intermediate variable y. It is also possible to apply selective desensitization when calculating. For example, when y = (x (v), θ (ω))) is calculated as the intermediate variable y, that is, the input variable x (product type modification x (v )) And the input variable θ (product type modification θ (ω)) selectively desensitized by the input variable ω as an intermediate variable y, the product type modification x (ω) is used as the product type modification x. After selectively desensitizing (v) (after product-type context modification), it is also possible to calculate the output variable Q (s _t , a _t ) based on the coupling weights w _ji , w _ji ′. (See formulas (11), (12-1), (12-2)). Further, when the intermediate variable y is calculated from each of the input variables x, v, θ, and ω, the calculation is performed in the same manner as a conventionally known multilayer perceptron without performing selective desensitization, and the output variable Q is calculated from the intermediate variable y. It is also possible to perform selective desensitization when (s _t , a _t ) is computed.

  A function approximation device and function for approximating a problem as a function (non-linear function, etc.) using a layered neural network (multivariable mutual modification model N, multivariate product model, etc.) to which the selective desensitization method of the present invention is applied. Create an approximation system and a function approximation program, and apply it to the control of learning control devices that control tasks that can be learned in real time, such as autonomous behavior type robots, pattern recognition devices for images and sounds, etc. As a result, the learning time can be greatly reduced, and the adaptability to an unknown environment can be improved.

FIG. 1 is an overall explanatory diagram of a function approximation system according to a first embodiment of the present invention. FIG. 2 is a block diagram (function block diagram) illustrating each function provided in the control unit of the cart according to the first embodiment of the present invention. FIG. 3 is a simple explanatory diagram of reinforcement learning. FIG. 4 is an explanatory diagram of a layered neural network to which the selective desensitization method according to the first embodiment of the present invention is applied. FIG. 5 is an explanatory diagram for explaining an example of an input variable state pattern indicating the position of the carriage. FIG. 6 is a flowchart of the main process of the function approximation program according to the first embodiment of the present invention. FIG. 7 is a flowchart of the action value function approximation process of the function approximation program according to the first embodiment of the present invention, and is an explanatory diagram of the subroutine of ST7 in FIG. FIG. 8 is a flowchart of the action value function learning process of the function approximation program according to the first embodiment of the present invention, and is an explanatory diagram of the subroutine of ST13 in FIG. FIG. 9 is an explanatory diagram of the experimental results of the experimental example. The horizontal axis indicates the number of trials of the inverted pendulum control process (number of episodes), and the vertical axis indicates the trial time of the inverted pendulum control process (the time during which the bar has been inverted). 9A is a graph for comparing the learning efficiency of Experimental Example 1 with the learning efficiency of Comparative Example 1, FIG. 9A is a graph showing the experimental results of Experimental Example 1, and FIG. 9B is the experiment of Comparative Example 1. It is a graph which shows a result. FIG. 10 is an explanatory diagram of the experimental results of the experimental example, where the horizontal axis represents the number of trials of the inverted pendulum control process (number of episodes), and the vertical axis represents the trial time of the inverted pendulum control process (the time during which the rod continued to be inverted) FIG. 10A is a graph for comparing the learning efficiency of Experimental Example 2 and the learning efficiency of Comparative Example 2 by taking the number of times disturbance was applied (number of times the stick was played with a finger), and FIG. FIG. 10B is a graph showing the experimental results of Comparative Example 2. FIG. 11 is an explanatory diagram of the experimental results of the experimental example, where the horizontal axis represents the number of trials of the inverted pendulum control process (number of episodes), and the vertical axis represents the trial time of the inverted pendulum control process (the time during which the bar continued to be inverted). 11A is a graph for comparing the learning efficiency of Experimental Example 3 with the learning efficiency of Comparative Example 3, FIG. 11A is a graph showing the experimental results of Experimental Example 3, and FIG. 11B is the experiment of Comparative Example 3 It is a graph which shows a result. FIG. 12 is an explanatory diagram of the minimum configuration of a layered neural network to which the selective desensitization method is applied. FIG. 13 is an explanatory diagram of a layered neural network to which the selective desensitization method is applied, FIG. 13A is an explanatory diagram of a product type model, and FIG. 13B is an explanatory diagram of a mutual modification model.

Explanation of symbols

A, 1 ... control device,
AP1 ... Function approximation program,
a t _... action,
C: Function approximation device,
C2 ... state measuring means,
C3 ... Action execution means,
C3A ... Action selection means,
C4 ... Reward acquisition means,
C5: Action value function calculation means,
C7A: Input variable input means,
C7C: Intermediate variable calculation means,
C7C1a + C7C1b ... first intermediate variable computing means,
C7C1c + C7C1d: second intermediate variable calculation means,
C7E: Output variable calculation means,
C7G: Means for learning connection weight,
C8 ... Control end determination means,
E ... Environment
N ... layered neural network,
Na ... input layer,
Nb ... intermediate layer,
Nc: output layer,
Q (a _t , s _t ), Q (a _{t + 1} , s _{t + 1} )... Action value function,
Q (a _t , s _t ) ... function, output variable,
r _t , r _{t + 1} ... reward,
S ... Reinforcement learning system, function approximation system,
_{(Q 1, q 2, ...} , q m), (q 1 ', q 2', ..., q m ') ... output element _{s t, s} t _{+ 1 ...} state,
_{(S x1, s x2, ...} , s xn, s v1, s v2, ..., s vn, s θ1, s θ2, ..., s θn, s ω1, s ω2, ..., s ωn) ... input element,
(T ₁ , t ₂ ,..., T _m ), (t ₁ ′, t ₂ ′,..., T _m ′).
w _ji , w _ji ′… bond load,
x, v, θ, ω ... input variables,
y ... intermediate variable,
(Y ₁ , y ₂ ,..., Y _12n ) ... Intermediate element.

Claims (7)

An input layer configured by an input element to which an input variable value is input, and an intermediate element coupled to the input element, wherein the value of the intermediate variable calculated based on the value input to the input element is An intermediate layer constituted by the intermediate element to be output, and an output element coupled to the intermediate element, wherein the value of the output variable calculated based on the value input to the intermediate element is output In a function approximation device that approximates a function that is a relationship between the input variable and the output variable by a layered neural network having an output layer constituted by output elements,
The input layer constituted by each input element to which each value of three or more input variables is input;
Input variable input means for inputting each value of the three or more input variables;
An input variable set in which any two input variables among the three or more input variables are set, and each value of one input variable of the input variable set and the other input variable of the input variable set are set. Intermediate variable calculation means for calculating each value of the intermediate variable based on each first output sensitivity calculated based on each value of
An output variable calculation means for calculating the value of the output variable based on the intermediate variable and a combined load set according to the importance of the value of the intermediate variable;
A connection weight learning means for learning the connection weight by updating the connection weight based on a difference between a value of the output variable and an actual value of the function stored in advance;
A function approximation apparatus comprising: Each value of the first intermediate variable based on each value of one input variable of the input variable set and each first output sensitivity calculated based on each value of the other input variable of the input variable set First intermediate variable calculation means for calculating each of the above, each value of the other input variable of the input variable set, each second output sensitivity calculated based on each value of one input variable of the input variable set, , Based on the second intermediate variable calculation means for calculating each value of the second intermediate variable, respectively, the intermediate variable calculation,
Based on the first intermediate variable, the second intermediate variable, and each combined load set in accordance with the importance of each value of the first intermediate variable and the second intermediate variable, the output variable The output variable calculating means for calculating a value;
The function approximating device according to claim 1, comprising:   The intermediate layer includes each intermediate element that outputs each value of a plurality of intermediate variables, and all the input variables of the three or more input variables are at least one of the input variable sets or 3. The function approximating apparatus according to claim 1, wherein a plurality of sets of the input variables are configured by being set as the other input variable. A control device as a target for controlling behavior;
State measuring means for measuring the state of the control device;
Reward acquisition means for acquiring a reward for the behavior;
An action value function calculating means for calculating an action value function that is an evaluation value for evaluating all actions in the measured state based on prediction of a reward that can be acquired in the future;
The function approximation device according to any one of claims 1 to 3, wherein the action value function in the measured state is approximated by regarding a measured value measured in the state as a value of the input variable.
Action selecting means for selecting the action based on the approximated action value function;
Action executing means for executing the selected action;
Control end determination means for determining whether to end control of the control device by determining whether control of the control device has failed based on the reward,
Reinforcement learning system characterized by having The function approximation device according to any one of claims 1 to 3, further comprising: the connection weight learning unit that learns the connection weight when it is determined that the control of the control device has failed.
The reinforcement learning system according to claim 4, further comprising: An input layer configured by an input element to which an input variable value is input, and an intermediate element coupled to the input element, wherein the value of the intermediate variable calculated based on the value input to the input element is An intermediate layer constituted by the intermediate element to be output, and an output element coupled to the intermediate element, wherein the value of the output variable calculated based on the value input to the intermediate element is output In a function approximation system that approximates a function that is a relationship between the input variable and the output variable by a layered neural network having an output layer constituted by output elements,
The input layer constituted by each input element to which each value of three or more input variables is input;
Input variable input means for inputting each value of the three or more input variables;
An input variable set in which any two input variables among the three or more input variables are set, and each value of one input variable of the input variable set and the other input variable of the input variable set are set. Intermediate variable calculation means for calculating each value of the intermediate variable based on each first output sensitivity calculated based on each value of
An output variable calculation means for calculating the value of the output variable based on the intermediate variable and a combined load set according to the importance of the value of the intermediate variable;
A connection weight learning means for learning the connection weight by updating the connection weight based on a difference between a value of the output variable and an actual value of the function stored in advance;
A function approximation system characterized by comprising: Computer
An input layer configured by input elements to which values of input variables are input and configured by input elements to which values of three or more input variables are input, and an intermediate element coupled to the input elements, An intermediate layer composed of the intermediate element that outputs an intermediate variable value calculated based on a value input to the input element, and an output element coupled to the intermediate element, the intermediate element An input layer configured to output the value of the output variable calculated based on the input value; and an input layer configured to input each value of the three or more input variables. Variable input means,
An input variable set in which any two input variables among the three or more input variables are set, and each value of one input variable of the input variable set and the other input variable of the input variable set are set. Intermediate variable calculation means for calculating each value of the intermediate variable based on each first output sensitivity calculated based on each value of
An output variable calculation means for calculating the value of the output variable based on the intermediate variable and a combined load set according to the importance of the value of the intermediate variable;
A combined load learning means for learning the combined load by updating the combined load based on a difference between a value of the output variable and an actual value of the function stored in advance;
A function approximation program that approximates a function that is a relationship between the input variable and the output variable by causing the function to function as