JP4408252B2 - Control device and program - Google Patents

Description

  The present invention relates to a control device and a program for controlling a multi-degree-of-freedom system such as a robot.

  As a first conventional technique, there is the following control device. This control device is a control device that controls a biped walking robot, and can acquire a control law of a biped walking robot simulator and preferably control a biped walking robot by reinforcement learning that is an autonomous adaptive control method. It is a control device. The control apparatus includes a learning unit that performs reinforcement learning and a control unit that performs control. The neural oscillator network that is the control unit has a high-dimensional state variable, and the biped walking robot simulator also has a high-dimensional state variable. have. At this time, the control unit enables appropriate control by observing not only the state of the biped walking robot simulator but also the internal state of the neural oscillator network (see Non-Patent Document 1). .

  As a second conventional technique, there is the following control device. Similar to Non-Patent Document 1, the present control device includes a learning unit that performs learning and a control unit that performs control. And although the snake type robot is controlled by this control apparatus, the state space of the non-control target is very large. Furthermore, the search for the set of via points in the control device is based on a kind of random search using a genetic algorithm (see Non-Patent Document 2).

Furthermore, as a related prior art, there is a control device having the following learning algorithm (see Non-Patent Document 3). Such a control device uses a kind of algorithm of reinforcement learning called an Actor-Critic algorithm.
Yasushi Nakamura and two others, “Reinforcement learning method for rhythmic movement using neural oscillator network”, IEICE Transactions, March 2004, D-II Vol. J87 No. 3, pp. 893-902 Kazuyuki Ito et al. , 2003 IEEE International Conference on Robotics & Automation Taipei, Taiwan, September 14-19, 2003 Hajime Kimura and 1 other "Actor-Critic algorithm using a history of suitability for Actor--Reinforcement learning under incomplete Value-Function", Journal of Artificial Intelligence, Vol. 15, no. 2, pp. 267-275 (2000)

  However, in the first prior art described above, the neural oscillator network as a control unit has a high-dimensional state variable, and the biped walking robot simulator also has a high-dimensional state variable, and both are observed together. Therefore, the dimension of the learning unit becomes very large, and the learning is difficult. In addition, this control device is difficult to perform high-speed learning, and requires a large amount of CPU power to perform learning.

  Further, the second conventional technique has a problem that learning by the learning unit is difficult due to the size of the state space. That is, in this control apparatus, since the search for the set of via points is a kind of random search using a genetic algorithm, there is a problem that processing efficiency is poor.

  The control device according to the first aspect of the invention observes the state of the controlled device that is the device to be controlled and acquires two or more first-type parameters, and the two or more first acquired by the observation unit. Based on the seed parameter, the feature extraction unit that acquires a smaller number of second type parameters than the first type parameter, and the controlled device based on one or more second type parameters acquired by the feature extraction unit It is a control apparatus provided with the control part to control.

The control device according to the second invention is the control device according to the first invention, wherein the feature extraction unit acquires two or more second-type parameters based on the two or more first-type parameters acquired by the observation unit. And a selecting means for selecting one or more second type parameters from the two or more second type parameters acquired by the converting means.
With such a configuration, the control device compresses a large number of observed parameters, and can control the controlled device at high speed or learn a control rule for that purpose.

The control device according to the third aspect of the invention is the control device according to the second aspect of the invention, wherein the second type parameter has a variable variable, and the conversion means converts the variable of the second type parameter into the control unit. The control result is changed according to the control result or / and the internal state of the control unit.
With this configuration, suitable control can be performed with higher accuracy than the control device of the second invention.

The control device of the fourth invention is the control device of the first to third inventions, wherein the control unit is configured to control the control device based on one or more second-type parameters acquired by the feature extraction unit. An evaluation unit that evaluates the situation of the controlled object and outputs information indicating whether the evaluation is good or bad, and a control unit that controls the controlled device based on the information indicating whether the evaluation is good or bad. It comprises.
Further suitable control is possible by the configuration learned by such control means.
The control device of the fifth invention is the control device of the first to fourth inventions, wherein the observation unit observes the state of the controlled device and the internal state of the control unit, and two or more first types Get parameters.
With such a configuration, more suitable control is possible.

The control device of the sixth invention further comprises a selection information receiving unit that receives selection information that is information for the feature extraction unit to acquire the second type parameter in the control device of the first to fifth inventions. Then, the feature extraction unit also acquires the number of second type parameters that is smaller than the first type parameter using the selection information received by the previous selection information receiving unit.
With such a configuration, it is possible to give information that is a guideline for suitable feature extraction from the outside, and suitable control is possible.

  The control device of the seventh invention is the control device of the second to fifth inventions, obtains the first type parameter obtained by the observation unit or / and the control result of the control unit, and obtains the obtained information. It further includes a long-term observation unit that accumulates two or more, and the conversion means acquires two or more second-type parameters using information accumulated by the long-term observation unit or information generated based on the information. .

The control device of the eighth aspect of the invention is the control device of the second to fifth aspects of the invention, acquires the first type parameter acquired by the observation unit or / and the control result of the control unit, and obtains the acquired information. The information processing apparatus further includes a long-term observation unit that accumulates two or more, and the selection unit selects one or more second-type parameters using information accumulated by the long-term observation unit or information generated based on the information. .
With such a configuration, it is possible to perform suitable control in consideration of observation results over a predetermined time.
Needless to say, the operation of the control device may be realized by software and a computer.

  According to the present invention, it is possible to provide a control device that can control a multi-degree-of-freedom system such as a robot with a small amount of calculation.

Hereinafter, embodiments of the control device and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.
(Embodiment 1)
FIG. 1 is a block diagram of a control system having a control device and a controlled device that is a control target device of the control device. The control system includes a control device 11 and a controlled device 12.
The control device 11 includes an observation unit 1101, a feature extraction unit 1102, and a control unit 1103. Further, the control device 11 may be configured to exist inside the controlled device 12.

  The observation unit 1101 observes the state of the controlled device 12 that is a device to be controlled, and acquires two or more parameters. The parameters acquired here are referred to as first type parameters. There are many first type parameters. The majority is, for example, 58 or more. The observation unit 1101 can be realized using, for example, motion capture. Specifically, for example, when the controlled device 12 is a snake robot, the observation unit 1101 includes hardware that combines a marker mounted on the snake robot and a tracker that detects the marker, Differentiating the output (for example, information on the position at each link and the angle at each joint of the snake robot) can be realized by software or the like that calculates the speed and angular acceleration.

  The feature extraction unit 1102 acquires a smaller number of parameters than the first type parameters based on the two or more first type parameters acquired by the observation unit 1101. The parameter acquired by the feature extraction unit 1102 is referred to as a second type parameter. The number of second type parameters is preferably sufficiently smaller than the number of first type parameters. Specifically, the number of first type parameters is, for example, 58 or more, while the number of second type parameters is three.

  The control unit 1103 controls the controlled device 12 based on one or more second type parameters acquired by the feature extraction unit 1102. The control is, for example, generating a control torque at each joint so as to move the snake robot forward.

  The feature extraction unit 1102 and the control unit 1103 can usually be realized by an MPU, a memory, or the like. The processing procedures of the feature extraction unit 1102 and the control unit 1103 are usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

The controlled device 12 may be anything as long as it is a device to be controlled. The controlled device 12 is, for example, a robot. The robot is, for example, a snake-type robot or a humanoid robot that requires bipedal walking control. The controlled device 12 is a multi-degree-of-freedom system.
The first type parameter and the second type parameter are different types of parameters having different meanings. Normally, each second type parameter is a parameter configured based on two or more first type parameters.
Hereinafter, the operation of this control system will be described. First, operation | movement of the control apparatus 11 is demonstrated using the flowchart of FIG.

  (Step S201) The observation unit 1101 observes the state of the controlled device 12 and determines whether two or more first-type parameters have been acquired. If two or more first type parameters are acquired, the process proceeds to step S202. If two or more first type parameters are not acquired, the process returns to step S201.

(Step S202) The feature extraction unit 1102 acquires one or more second-type parameters based on the two or more first-type parameters acquired in step S201. Here, the number of second type parameters is usually significantly smaller than the number of first type parameters observed in step S201.
(Step S203) The control unit 1103 configures a control signal that is a signal for controlling the controlled device 12 based on the one or more second-type parameters selected in step S202.
(Step S <b> 204) The control unit 1103 gives the control signal configured in step S <b> 203 to the controlled device 12 and controls the controlled device 12. The process returns to step S201.
In the flowchart of FIG. 2, the process is terminated by powering off or a process termination interrupt.
Hereinafter, a specific operation of the control system in the present embodiment will be described.

  Now, the controlled device 12 is, for example, a snake robot. For example, the snake type robot has a form as shown in FIG. Snakes are propelled by utilizing the fact that the friction in the tangential direction of the body axis is small and the friction in the normal direction is large. This snake type robot is a robot in which links with passive wheels are connected so as to satisfy such a condition. More specifically, this snake-type robot is a robot in which ten box-type links having passive wheels on both sides are connected, and the weight and size of each link and the passive wheels are shown in FIG. . 3B, “link mass” is the link weight, “link length” is the link length, “link width” is the link width, “link height” is the link height, and “wheel mass”. Is the weight of the passive wheel and “wheel radius” is the radius of the passive wheel. That is, in FIG. 3B, the weight of the link is “1.0 kg”, and the length, width, and height of the link are “0.15 m”, “0.10 m”, and “0.10 m”, respectively. The weight of the wheel is “0.5 kg” and the radius of the passive wheel is “0.07 m”. In addition, it is assumed that the joints between the links rotate about the z axis and torque for rotation is applied. The snake robot is controlled by the torque that can be applied to each joint.

In addition, the observation unit 1101 observes the state variables (parameters) of the position, angle, and angular velocity of each link (here, 10) of the snake robot. A set of the state variables of the position, velocity, angle, and angular velocity of each link is two or more first type parameters. For example, the number of such state variables is 58. The position and speed of the i-th link are obtained by (Xi, Yi) and (VXi, VYi), respectively (i is an integer from 1 to 10). The joint angle is represented by “Kj” and the angular velocity is represented by “Lj” (j is an integer from 1 to 9). That is, the first type parameters are (Xi, Yi, VXi, VYi), (Kj, Lj) [i is 1 to 10, j is 1 to 9]. Note that the position of each link, which is one of the first type parameters, may be coordinates of a small number of one or more reference points (such as barycentric coordinates) and relative coordinates indicating a relative position with respect to the reference point. .
In such a case, the feature extraction unit 1102 performs the following formulas 1 and 2 based on (Xi, Yi, VXi, VYi) and (Kj, Lj).
Formula 1 is a feature amount (second type parameter) indicating the direction of the snake robot. Expression 2 is a feature amount (second type parameter) indicating the degree of bend of the snake robot.

数式２において、Ｋｊはｊ番目の関節角であり、｜Ｋｊ｜は，Ｋｊの絶対値を示す。 In Equation 1, xi and yi are the x coordinate and the y coordinate of the center of gravity position of the i-th link from the head side of the snake robot. N indicates the number of links of the snake robot. N is 10 here, for example. Arctan is an arc tangent.

In Equation 2, Kj is the j-th joint angle, and | Kj | represents the absolute value of Kj.

  Then, the feature extraction unit 1102 uses (Xi, Yi, VXi, VYi), (Kj, Lj) [i is 1 to 10 and j is 1 to 9]. The second type parameter (s1, s2) is acquired. Then, the feature extraction unit 1102 passes the second type parameters (s1, s2) to the control unit 1103.

  The control unit 1103 controls the controlled device 12 based on the control input signal that is the second type parameter (s1, s2). This control method will be described in the second embodiment. Specifically, the controllable device 12 operates by determining the torque that can be applied to each joint and using the determined torque.

  According to the above specific example, the 58 first type parameters observed by the observation unit 1101 are compressed into two second type parameters by the feature extraction unit 1102. The controlled device 12 (snake robot) can be controlled by the two second type parameters.

As described above, according to the present embodiment, the control device compresses a large number of observed parameters, and can control the controlled device at high speed, or can learn a control law for that purpose. Such parameter compression is realized by paying attention to the property that the motion of a controlled device such as a snake robot is much simpler than that using all of its state variables.
In addition, according to this Embodiment, although the to-be-controlled device was a snake-type robot, it cannot be overemphasized that other multi-degree-of-freedom systems, such as a humanoid robot which requires bipedal walking control, may be sufficient.

  Further, according to the present embodiment, the second type parameters acquired by the feature extraction unit 1102 are a feature amount indicating the direction of the snake robot and a feature amount indicating the bend state of the snake robot. Other parameters may be used as long as they represent the characteristics of the control device. The second type parameter is a control parameter and is not necessarily a parameter expressing a specific motion.

  In the present embodiment, the feature extraction unit 1102 further includes a selection information reception unit that receives selection information that is information for acquiring the second type parameter. The feature extraction unit 1102 is received by the selection information reception unit. Alternatively, the selection information may be used to obtain a smaller number of second type parameters than the first type parameters. Here, the selection information is, for example, the above-described Formula 1, Formula 2 itself, or information sufficient to specify them.

Furthermore, the processing of the control device in the present embodiment may be realized by software. Then, this software may be distributed by software download or the like. Further, this software may be recorded and distributed on a recording medium such as a CD-ROM. This also applies to other embodiments in this specification. Note that the software that implements the control device in the present embodiment is the following program. In other words, this program causes the computer to observe the state of the controlled device that is the device to be controlled, obtains two or more first type parameters, and two or more first type obtained in the observation step. Based on the parameter, a feature extraction step for obtaining a smaller number of second type parameters than the first type parameter, and controlling the controlled device based on one or more second type parameters obtained in the feature extraction step A program for executing a control step.
(Embodiment 2)
FIG. 4 is a block diagram of a control system including a control device and a controlled device that is a device to be controlled by the control device. The control system includes a control device 41 and a controlled device 12.
The control device 41 includes an observation unit 4101, a feature extraction unit 4102, a control unit 4103, and a selection information reception unit 4104.
The feature extraction unit 4102 includes conversion means 41021 and selection means 41022. The control unit 4103 includes an evaluation unit 41031 and a control unit 41032. Further, the control device 41 may be configured to exist inside the controlled device 12.

The observation unit 4101 observes the state of the controlled device 12 that is the device to be controlled and the internal state of the control unit 4103, and acquires two or more parameters . The parameters acquired here are referred to as first type parameters. There are many first type parameters. For example, the number is 2400 or more. The internal state of the control unit 4103 is, for example, a state variable of the neural oscillator network. The neural oscillator network will be described later. The observation unit 4101 can be realized using, for example, motion capture. Specifically, for example, when the controlled device 12 is a snake robot, the observation unit 4101 includes hardware that combines a marker attached to the snake robot and a tracker that detects the marker, and an output of the hardware ( For example, software for calculating speed and angular acceleration by differentiating the information on the position of each link and the angle at each joint of the snake robot, and hardware such as an oscilloscope for acquiring the internal state of the control unit 4103 Can be realized from.
The feature extraction unit 4102 acquires a second type parameter that is smaller in number than the first type parameter, based on the two or more first type parameters acquired by the observation unit 1101.

  The conversion unit 41021 acquires two or more second type parameters based on the two or more first type parameters acquired by the observation unit 1101. A specific method for acquiring two or more second type parameters based on two or more first type parameters will be described later.

The selection unit 41022 selects one or more second type parameters from the two or more second type parameters acquired by the conversion unit 11021. The selection unit 41022 may select all the one or more second type parameters acquired by the conversion unit 41021. In such a case, the selection unit 41022 does not perform any processing.
The control unit 4103 controls the controlled device 12 based on one or more second type parameters acquired by the feature extraction unit 4102.

ここで、Ｔｊはｊ番目の関節［ｊ＝１，...，９］にかけられるトルクの大きさを表す。Ｔｊを算出する式は、数式４である。

The evaluation unit 41031 evaluates the status of the controlled device 12 based on the one or more second type parameters acquired by the feature extraction unit 4102, and indicates information indicating whether the evaluation is good or bad (such value). If necessary, it is called “output value”. The status of the controlled device 12 regarding the controlled state may be a result after the controlled device 12 is controlled, or a control signal or the like given to the controlled device 12. The information indicating whether the evaluation is good or bad is, for example, a flag that can take one of information “1” indicating that the evaluation is good, information “0” indicating that the evaluation is bad, or the maximum when the evaluation is the best. The value can be an intermediate value depending on the degree, the minimum value at the worst and the meantime. More specifically, when propelling the snake robot in the x-axis direction, the following output value (r) can be set. Formula 3 for calculating the output value (r) is shown in Formula 3.

Here, Tj represents the magnitude of torque applied to the j-th joint [j = 1,..., 9]. The equation for calculating Tj is Equation 4.

In Equation 4, K _m , K _s , S _t , and K _d are gains for muscle force, gains for stiffness, snake hardness, gains for differences in angular velocity, and are predetermined constants. P _j and Q _j are the j-th link angle and the link angular velocity, respectively. The angular velocity (Q _j ) of the link is obtained by differentiating the link angle (P _j ).

  The control unit 41032 controls the controlled device 12 based on the information indicating whether the evaluation is good or not output from the evaluation unit 41031. Further, the control unit 41032 changes the control rule for constructing the control signal based on the second type parameter selected by the selection unit 41022 and the output value which is the evaluation output by the evaluation unit 41031 for such control. (learn. Then, the control means 41032 constructs a control signal based on the learned control rule. In the case where the controlled device 12 is a snake robot, the control means 41032, for example, uses a snake type when the sum of output values given by the above r becomes large within a predetermined number of times (for example, within 100 times). A feature amount (second type parameter) indicating the shape of the robot is accumulated, and the snake robot is controlled so as to have the shape.

  The selection information reception unit 4104 receives selection information that is information for the feature extraction unit 4102 to acquire the second type parameter. The selection information includes a formula (the above formula 1) for calculating a feature quantity (second type parameter) indicating the direction of the snake robot and a feature quantity (second type parameter) indicating the degree of bend of the snake robot. This is a mathematical formula for calculating (the above mathematical formula 2) or the like, or sufficient information for specifying them. The selection information input means may be anything such as a keyboard, mouse, or menu screen. The selection information receiving unit 4104 can be realized by a device driver for input means such as a keyboard, control software for a menu screen, or the like.

The feature extraction unit 4102, the control unit 4103, the conversion unit 41021, the selection unit 41022, the evaluation unit 41031, and the control unit 41032 can be usually realized by an MPU, a memory, or the like. The processing procedure of the feature extraction unit 4102 or the like is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).
Hereinafter, the operation of the control device 41 configuring the present control system will be described with reference to the flowchart of FIG.

  (Step S501) The observation unit 1101 observes the state of the controlled device 12 and determines whether two or more first-type parameters have been acquired. If two or more first type parameters are acquired, the process proceeds to step S502. If two or more first type parameters are not acquired, the process returns to step S501.

  (Step S502) The conversion unit 41021 of the feature extraction unit 4102 acquires two or more second type parameters based on the two or more first type parameters acquired in step S501. Here, the number of the second type parameters may be larger than or the same as the number of the first type parameters.

(Step S503) The selection unit 41022 of the feature extraction unit 4102 selects one or more second type parameters from the two or more second type parameters acquired in step S502. The number of second type parameters selected here is preferably significantly smaller than the number of first type parameters. For example, it is preferable that the number of first type parameters is 2400 or more and the number of selected second type parameters is three.
(Step S504) The control means 41032 constitutes a control signal based on the second type parameter selected in step S503. A specific method for configuring the control signal will be described later.
(Step S <b> 505) The control unit 41032 gives the control signal configured in step S <b> 504 to the controlled device 12 and controls the controlled device 12.

(Step S506) The evaluation unit 41031 outputs an evaluation of the control rule by the control unit 41032 based on the second type parameter selected in step S503 (or in addition to the control signal output from the control unit 41032).
(Step S507) The control unit 41032 changes (learns) a control rule for forming a control signal based on the second type parameter selected in step S503 and the evaluation output by the evaluation unit 41031 in step S506. The process returns to step S501. In step S507, the control unit 41032 does not directly use the second type parameter, but changes (learns) the control rule for configuring the control signal based on the evaluation output by the evaluation unit 41031. Also good.

In the flowchart of FIG. 5, the process ends when the power is turned off or the process ends. Further, the evaluation unit 41031 and the control unit 41032 can be learned by an algorithm described in Non-Patent Document 3.
Hereinafter, a specific operation of the control system in the present embodiment will be described. Now, the controlled device 12 is, for example, a snake robot shown in FIG.

  The observation unit 4101 observes the position variable, the velocity, the angle, the angular velocity state variable, and the state variable of the neural oscillator network of each link (here, 10) of the snake robot. The state variable of the neural oscillator network is an internal state of the control unit 4103 and is a first type parameter observed by the observation unit 4101, and the number thereof is 2400 or more.

Here, the control unit 4103 is a model of the spinal cord, and is implemented by a neural oscillator network in which neuron elements are connected in a network form. For example, as shown in FIG. 6, the neural oscillator network is formed by connecting 100 ganglia composed of four types of elements, that is, left and right MN, EIN, LIN, and CCIN, as shown in FIG.
In addition, each element performs time development by, for example, Equation 5 below.

また、Ｕは各素子に対する外部入力（＝トニック入力）で、数式７によって計算される。

Here, C1, C2, C3, C4, C5, w0, and wi are predetermined constants, and Ψ + and Ψ− indicate a set of indices of the coupled elements. max (x1, x2) is a function that outputs the larger of x1 and x2. The operation of the element can be simulated by numerical integration of the above differential equation at time t on the computer. Alternatively, analog hardware according to the above differential equation may be used. DT is a step size of numerical integration. v is the output of the element. (I, I +, I−) are called internal state variables, and when implemented on a computer, they are updated (numerical integration) as shown in Equation 6.

U is an external input (= tonic input) for each element, and is calculated by Equation 7.

  The policy is given as the probability P that the control signal (u) is output when the three second-type parameters (s1, s2, s3) are given. Expected values may be used. Specifically, as an example of P, a normal distribution having an average value determined from (s1, s2, s3) and a variance given by the parameter θ0 is used.

Further, the control signal output by the control unit 4103 is the output (see FIG. 7) of the motor neurons (NM _L , MN _R ) existing on the left and right of the ganglion. This control signal is given as the torque of each joint of the snake robot (controlled device 12), and the body of the snake robot is bent. In this example, the controlled device 12 is modeled on the assumption that 10 links are connected, and the torque applied to the i-th joint is calculated by Equation 8 from the output of the 10i segment motor neuron.

In Equation 8, K _m , K _s , S _t , and K _d are gain for muscle strength, gain for stiffness, snake hardness, gain for angular velocity difference, and are predetermined constants. P _i and Q _i are the angle of the i-th link and the angular velocity of the link, respectively. The angular velocity (Q _i ) of the link is obtained by differentiating the link angle (P _i ).

The control unit 4103 is controlled by the above-described tonic input that models a signal from the brainstem. When an input of the same size is given to the left and right neurons, the output (control of the motor neuron is controlled by the size of the input. Signal) changes. By giving the changed control signal to the snake robot (controlled device 12) by, for example, Expression 8, the body shape of the snake robot changes. On the other hand, if inputs of different sizes are given to the left and right neurons, a difference in output is generated between the left and right motor neurons. By giving such a difference in output to the snake-type robot (controlled device 12), the traveling direction changes through the shape change of the snake-type robot according to the difference.
Hereinafter, the processing until the control signal to be given to the snake robot after the observation unit 4101 acquires the first type parameter will be described in detail.

  First, the conversion means 41021 converts 2400 or more first-type parameters observed by the observation unit 4101 as follows, and acquires second-type parameters. Here, it is assumed that the number of second type parameters is the same as the number of first type parameters, for example. Here, the second-type parameters include, for example, at least a feature amount indicating the direction of the snake robot, a feature amount indicating the snake bend condition, and a feature amount indicating the state of the neural oscillator network. Specifically, it is assumed that the feature amounts from Equation 4 to Equation 6 are included.

Next, selection means 41022 is the second type parameter conversion unit 41021 obtains, selects three of the two parameters:. The three second-type parameters are a feature value indicating the direction of the snake robot, a feature value indicating the snake bend condition, and a feature value indicating the state of the neural oscillator network. These second type parameters are selected by, for example, a designer's a priori knowledge (the knowledge is usually held by the selection means 41022), but may be obtained by other various means.

Here, the feature quantity s1 (second type parameter) indicating the direction of the snake-like robot is calculated by Equation 9. A feature quantity s2 (second type parameter) indicating the bend state of the snake robot is calculated by Expression 10. A feature quantity s3 (second type parameter) indicating the state of the neural oscillator network is calculated by Expression 11.

数式１０において、Ｋｊはｊ番目の関節角であり、｜Ｋｊ｜は，Ｋｊの絶対値を示す。

すなわち，数式１１は神経振動子ネットワーク内の全運動ニューロン（ＭＮＬ_ｋ，ＭＮＲ_ｋ）の出力の和である
選択手段４１０２２は、３つの第二種パラメータ（ｓ１，ｓ２，ｓ３）を選択し、当該３つの第二種パラメータを制御手段４１０３２および評価手段４１０３１に渡す。 In Equation 9, xi and yi are the x-coordinate and y-coordinate of the centroid position of the i-th link from the head side of the snake robot. N indicates the number of links of the snake robot. N is 10 here, for example. Arctan is an arc tangent.

In Equation 10, Kj is the j-th joint angle, and | Kj | represents the absolute value of Kj.

That is, Formula 11 is the sum of the outputs of all motor neurons (MNL _k , MNR _k ) in the neural oscillator network. Selection means 41022 selects three second type parameters (s1, s2, s3) Three second type parameters are passed to the control means 41032 and the evaluation means 41031.

Next, the control means 41032 receives three second type parameters (s1, s2, s3) and outputs a control signal according to the policy. The strategy is defined by the following equation 12 when three second type parameters (s1, s2, s3) are given.

ここで、ＵＢｒとＵＢｌはそれぞれＵｒとＵｌの平均値である。 The policy is given as the probability P that the control signal (u) is output when the three second-type parameters (s1, s2, s3) are given. Expected values may be used. Specifically, as an example of P, a normal distribution having an average value determined from (s1, s2, s3) and a variance given by the parameter θ0 is used. Further, the following Equation 13 is an example of calculating an average value when the control signal u is given by a two-dimensional vector of (Ur, Ul).

Here, UBr and UBL are the average values of Ur and Ul, respectively.

ここで，β，ＳＢｉ，ｊ（ｉは１からＭまで，ｊは１から３までの整数）は定数で、ｄｉは強化学習によって獲得されるパラメータである。Ｍはあらかじめ決められた定数である。 The evaluation means 41031 receives three second type parameters (s1, s2, s3), and calculates the good state of the system based on the second type parameters (s1, s2, s3). A function for the evaluation means 41031 to perform this calculation is called a state value function. The state value function is a function obtained using, for example, a TD (λ) algorithm often used in a reinforcement learning method that is a known technique. The evaluation unit 41031 substitutes the second type parameters (s1, s2, s3) for the state value function to obtain an output value.
For example, the output value V is calculated as in Expression 14.

Here, β, SBi, j (i is an integer from 1 to M, j is an integer from 1 to 3) are constants, and di is a parameter obtained by reinforcement learning. M is a predetermined constant.

Then, the control unit 41032 learns the policy parameter based on the output value of the evaluation unit 41031. That is, the control unit 41032 determines a policy parameter based on the output value of the evaluation unit 41031 so that the snake robot proceeds smoothly, and configures a control signal. The policy parameter learning process will be described with reference to the flowcharts shown in FIGS.
The policy parameter learning described with reference to FIGS. 8 and 9 is a control and learning procedure. As an example of learning, learning based on an algorithm described in Non-Patent Document 3 is performed.

(Step S801) Initialization is performed. Initialization is the following process. That is, the initial values of the parameters θi (policy parameters) and di of the control means 41032 and the evaluation means 41031 are determined. The auxiliary parameters Θi and Di for learning these parameters are set to zero. Also, t is set to 0.
(Step S802) The observation unit 4101 observes the controlled device 12 and the control unit 4103, and outputs the first type parameter X (t).

(Step S803) The feature extraction unit 4102 acquires the first type parameter X (t), converts it to the second type parameter by the conversion means 41021, and selects s1 (t), s2 (t), s3 (t by the selection means 41022. ) Is selected and output to the control unit 4103. At the same time, the control unit 4103 acquires r (t) calculated by Expression 3. Note that r (t) is r at time t.
(Step S804) The control means 41032 acquires s1 (t), s2 (t), and s3 (t), and generates a control signal.
(Step S805) The learning subroutine (described with reference to FIG. 9) of the control unit 4103 is executed.
(Step S806) The controlled device 12 is controlled by the control signal generated in step S804.
(Step S807) As t = t + 1, the process returns to Step S802.
In the flowchart of FIG. 8, the process ends when the power is turned off or the process is terminated.
Next, a learning subroutine of the control unit 4103 will be described using the flowchart of FIG.
(Step S901) It is determined whether or not time t is zero. If it is not 0, the process goes to step S902, and if it is 0, the process returns to the upper function.

数式１５において、γはあらかじめ決められた定数（０<γ<１）である。
（ステップＳ９０３）評価手段４１０３１は、以下の数式１６を計算し、パラメータをｄｉ＝ｄｉ＋η_ｄ×δ（ｔ−１）×Ｄｉのように更新する。

ここで，λはあらかじめ決められた定数（０<λ<１）である。また，η_ｄは学習係数と呼ばれる小さな正数である。
（ステップＳ９０４）制御手段４１０３２は、以下の数式１７を計算し、パラメータをθｉ＝θｉ＋η_θ×δ（ｔ−１）×Θｉのように更新する。

また、ここでのη_θは学習係数と呼ばれる小さな正数である。さらに、ここで、数式１８は、パラメータθｉについての変微分を表す記号である。

なお、ヘビ型ロボットの制御課題において制御信号は左右のニューロン群に対するトニック入力ＵｒとＵｌで，それぞれ、数式１９により定義した。

(Step S902) The evaluation means 41031 uses the second type parameter at time t-1 and the second type parameter at time t according to the mathematical formula (V (s1, s2, s3)), and V (s1 (t-1) , S2 (t-1), s3 (t-1)) and V (s1 (t), s2 (t), s3 (t)), and further calculated by Equation 3 at time t-1. The TD error is calculated by Equation 15 using (t−1).

In Equation 15, γ is a predetermined constant (0 <γ <1).
(Step S903) The evaluation unit 41031 calculates the following Expression 16, and updates the parameter as di = di + η _d × δ (t−1) × Di.

Here, λ is a predetermined constant (0 <λ <1). Η _d is a small positive number called a learning coefficient.
(Step S904) the control unit 41032 calculates the following formula 17, to update the parameters as _{θi = θi + η θ × δ} (t-1) × Θi.

Further, η _θ here is a small positive number called a learning coefficient. Furthermore, here, Equation 18 is a symbol representing a variable differential with respect to the parameter θi.

In the control task of the snake robot, the control signals are tonic inputs Ur and Ul for the left and right neuron groups, which are defined by Equation 19, respectively.

Here, C6 is a predetermined constant. Also, ε1 (t−1) and ε2 (t−1) are random numbers generated from a normal distribution with an average of 0 and a variance b (the algorithm for generating such random numbers is a publicly known one). , Equation 20 can be calculated.

The results of controlling the snake robot with the above control device are shown below. Specifically, experimental results when the friction coefficient with the ground changes at time 5000 seconds are shown in FIGS. The horizontal axis of FIGS. 10 and 11 indicates time, and the vertical axis indicates the direction of the snake-like robot and the values of policy parameters (here, two policy parameters θ1 and θ2). The direction of the snake-like robot is such that the target direction is zero. After the coefficient of friction changes, the snake-like robot temporarily changes its direction, but after that, it turns out that it is moving in the direction of travel as the policy parameters converge. From the above, it can be seen that the present control apparatus has obtained a good control law for the propulsion motion in the target direction (in this case, the direction parallel to the x-axis) of the snake robot. In addition, it can be seen that control can be performed in a form suitable for the new environment even though the environment (in this case, for example, the friction coefficient) has changed at time 5000 seconds.
Note that the selection information receiving unit 4104 receives, for example, information on the above-described mathematical expression.

  As described above, according to the present embodiment, the control device compresses a large number of observed parameters, and can control the controlled device at high speed, or can learn a control law for that purpose. Further, both the state of the controlled device and the internal state of the control unit are observed, and the evaluation means evaluates the status of the controlled device based on the one or more second type parameters. By simply outputting a simple value based on information indicating whether the evaluation is good or bad (for example, r in Formula 3), the control means 41032 can learn and perform suitable control. Therefore, the controlled device can be controlled without inputting complicated information to the control device. That is, in the control device according to the prior art, when trying to learn the control law for the controlled device at high speed, the control device must hold various complicated information. As mentioned earlier, a lot of information is needed when the environment changes. On the other hand, in the control device according to the prior art, if the initial setting is simple, a large number of states are observed if the controlled device is to be controlled appropriately. As a result, control takes time. According to the present embodiment, the control speed is increased (CPU load can be reduced) while simplifying the initial setting. This is because parameter conversion is performed from many observed type 1 parameters to type 2 parameters, and the reinforcement learning method in the control unit (evaluation means for outputting information indicating whether the evaluation is good or bad, and the evaluation means output by the evaluation means). This is because control is performed by a method realized by control means for controlling the controlled device based on information indicating good or bad. In other words, according to the present embodiment, control in which a designer can learn a control rule for a multi-degree-of-freedom system at high speed even in a situation where a controller adaptively discovers a control rule without designing an optimal control rule. Equipment can be provided.

  In addition, according to the present embodiment, as described above, the control unit includes a neural oscillator network, that is, a network structure in which a plurality of elements are connected in a network shape and has self-feedback, The internal state of the control unit is the state of the elements constituting the network structure. By using the neural oscillator network as a control unit, it is possible to suitably control a motion based on a periodic motion, such as a biped walking of a humanoid robot or a cheek motion of a snake robot. The same applies to other embodiments. However, feature extraction based on the conversion from the first type parameter to the second type parameter, which is the main technique of the present invention, effectively accelerates control learning regardless of whether or not a neural oscillator network is used. To do.

Furthermore, the software that implements the control device according to the present embodiment is the following program. In other words, this program causes the computer to observe the state of the controlled device and the internal state of the control unit, obtain two or more first type parameters, and two or more first type obtained in the observation step. Based on the parameter, a feature extraction step for obtaining a smaller number of second type parameters than the first type parameter, and controlling the controlled device based on one or more second type parameters obtained in the feature extraction step A control sub-step for obtaining two or more second-type parameters based on the two or more first-type parameters obtained in the observation step; Selection sub-step for selecting one or more second-type parameters from two or more second-type parameters acquired in the conversion sub-step And the control step evaluates the status of the controlled device based on the one or more second type parameters acquired in the feature extraction step, and outputs information indicating whether the evaluation is good or bad And a control substep for controlling the controlled device based on information indicating whether the evaluation is good or bad and output in the evaluation substep.
(Embodiment 3)
FIG. 12 is a block diagram of a control system including a control device and a controlled device that is a control target device of the control device. The control system includes a control device 121 and a controlled device 12.
The control device 121 includes an observation unit 4101, a feature extraction unit 12102, a control unit 4103, and a selection information reception unit 4104. The feature extraction unit 12102 includes conversion means 121021 and selection means 41022.

  The conversion unit 121021 changes the variable of the second type parameter according to the control result of the control unit 4103 or / and the internal state of the control unit 4103. In the present embodiment, the second type parameter has a variable variable. The conversion unit 121021 can be usually realized by an MPU, a memory, or the like. The processing procedure of the conversion means 121021 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

数式２１において、αｉ（ｉは１から９の整数）が変数である。 Hereinafter, the operation of the control device 121 in this control system will be described. The operation of the control device 121 is generally the same as the operation of the control device 41 in the second embodiment (described with the flowchart of FIG. 5). The control device 121 is different from the operation of the control device 41 only in that a process for changing the variable of the second type parameter is added according to the control result of the control unit 4103 and / or the internal state of the control unit 4103. Moreover, the process which changes the variable which this 2nd type parameter has is added after step S508, for example. Note that the process of changing the variable of the second type parameter may be performed at any timing.
Hereinafter, a specific operation of the control system in the present embodiment will be described. Here, only the process of changing the variable of the second type parameter performed by the conversion unit 121021 will be described.
First, the second type parameter (s2 ′) having a variable is expressed by, for example, Equation 21.

In Equation 21, αi (i is an integer from 1 to 9) is a variable.

ここでΔｉは乱数を用いて決めることが考えられる。
具体的には、変換手段１２１０２１は、例えば、以下の図１３に示すフローチャートにより、第二種パラメータの有する変数（αｉ）を変更する。 In addition, when the control result of the control unit 4103 is good, for example, when the sum of r in Formula 3 within a certain time is larger than a predetermined threshold value, the conversion unit 121021 The variable of the parameter is fixed, and on the other hand, if it is smaller than the threshold value, the variable of the second type parameter is randomly changed. For example, the variable (αi) of the second type parameter is changed as shown in Equation 22.

Here, Δi may be determined using a random number.
Specifically, the conversion unit 121021 changes the variable (αi) of the second type parameter, for example, according to the flowchart shown in FIG.

  (Step S1301) The selection information receiving unit 4104 receives the following initial values. Specifically, the selection information receiving unit 4104 receives initial values of the parameters αi [i = 1,..., 9]. The selection information receiving unit 4104 receives the value “1” of n. In addition, the selection information receiving unit 4104 receives a threshold value F for performance evaluation and a learning end time T.

(Step S1302) The control device 121 performs control in the same procedure as the learning algorithm in FIGS. 8 and 9 of the second embodiment, using the second type parameter s2 ′ having the parameter αi instead of s2. First, the initial values of the parameters θi (policy parameters) and di of the control means 41032 and the evaluation means 41031 are determined. The auxiliary parameters Θi and Di for learning these parameters are set to zero. Also, t is set to 0. Further, R is set to 0. Note that R is a variable (buffer) for storing the accumulated reward.
(Step S1303) The observation unit 4101 observes the controlled device 12 and the control unit 4103, and outputs the first type parameter X (t).

(Step S1304) The feature extraction unit 12102 acquires the first type parameter X (t), converts it to the second type parameter by the conversion unit 121021, and selects s1 (t), s2 (t), s3 (t by the selection unit 41022. ) Is selected and output to the control unit 4103. At the same time, the control unit 4103 obtains (r (t)) calculated by Expression 3. Then, R = R + r (t).
(Step S1305) The control unit 41032 acquires s1 (t), s2 (t), and s3 (t), and generates a control signal.
(Step S1306) The learning subroutine (FIG. 9) of the control unit 4103 is executed.
(Step S1307) The controlled device 12 is controlled by the control signal generated in step S1305.
(Step S1308) The conversion unit 121021 determines whether or not t = T. If t = T, the process goes to step S1309, and if t = T, the process goes to step S1303.
(Step S1309) The conversion means 121021 determines whether or not n = 1. If n = 1, go to step S1310, and if n = 1, go to step S1313.
(Step S1310) The conversion means 121021 records αmaxi = αi [i = 1,..., 9].
(Step S1311) The conversion means 121021 generates αi according to “αi = αmaxi + Δi”. Here, Δi is determined using a random number.
(Step S1312) The conversion means 121021 increments n by 1. The process returns to step S1302.
(Step S1313) The conversion unit 121021 determines whether or not R> F. If R> F, go to step S1310, and if not R> F, go to step S1311.
The search for αi may be realized by using a search technique such as a genetic algorithm which is a known technique. In addition, the long-term observation unit described in the fourth embodiment may be used in changing the variables of the conversion means.

  As described above, according to the present embodiment, the control device compresses a large number of observed parameters, and can control the controlled device at high speed, or can learn a control law for that purpose. Further, both the state of the controlled device and the internal state of the control unit are observed, and the evaluation means evaluates the status of the controlled device based on the one or more second type parameters. By simply outputting a simple value based on information indicating whether the evaluation is good or bad (for example, r in Formula 3), the control means 41032 can learn and perform suitable control. Therefore, the controlled device can be controlled without inputting complicated information to the control device. That is, normally, when trying to learn a control law for a controlled device at high speed, the control device must hold various complex information. Further, the second type parameter handled by this control apparatus has a variable variable, and the conversion means changes the variable of the second type parameter according to the control result of the control unit or / and the internal state of the control unit. This makes it possible to perform suitable control with higher accuracy. Furthermore, according to the present embodiment, even in a situation where a controller adaptively finds a control rule without designing an optimal control rule by a designer, control that can learn a control rule for a multi-degree-of-freedom system at high speed. Equipment can be provided.

Note that the software that implements the control device in the present embodiment is the following program. In other words, this program causes the computer to observe the state of the controlled device that is the device to be controlled, obtains two or more first type parameters, and two or more first type obtained in the observation step. Based on the parameter, a feature extraction step for obtaining a smaller number of second type parameters than the first type parameter, and controlling the controlled device based on one or more second type parameters obtained in the feature extraction step The second type parameter has a variable variable, and the feature extraction step includes two or more first type parameters acquired in the observation step. The second type parameter is acquired, and the variable included in the second type parameter is used as a control result or / and a control value in the control step. Parts and converting sub step of changing in response to the internal state of a program having a selection sub-step of selecting one or more second-type parameters from two or more second type parameters acquired by the conversion sub-step.
(Embodiment 4)
FIG. 14 is a block diagram of a control system including a control device and a controlled device that is a device to be controlled by the control device. The control system includes a control device 141 and a controlled device 12.

  The control device 141 includes an observation unit 4101, a feature extraction unit 14102, a control unit 4103, a selection information reception unit 4104, and a long-term observation unit 14105. The feature extraction unit 14102 includes conversion means 141021 and selection means 141022. Further, the control device 141 may exist inside the controlled device 12.

  The long-term observation unit 14105 acquires the first type parameter acquired by the observation unit 4101 and / or the control result of the control unit 4103, and accumulates two or more of the acquired information. Further, the long-term observation unit 14105 generates information to be given to the conversion unit 141021 and / or the selection unit 141022 based on the accumulated information, and gives the generated information to the conversion unit 141021 and / or the selection unit 141022. Note that the information generated by the long-term observation unit 14105 may be the same information as the accumulated information.

  The converting unit 141021 uses the two or more first type parameters acquired by the observation unit 4101 and the information accumulated by the long-term observation unit 14105 or the information generated based on the information to convert two or more second type parameters. get.

  The selection unit 141022 uses the information accumulated by the long-term observation unit 14105 or the information generated based on the information from the two or more second type parameters acquired by the conversion unit 141021, to convert one or more second type parameters. select.

The feature extraction unit 14102, the long-term observation unit 14105, the conversion unit 141021, and the selection unit 141022 can usually be realized by an MPU, a memory, or the like. The processing procedures of the feature extraction unit 14102, the long-term observation unit 14105, the conversion unit 141021, and the selection unit 141022 are usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).
Hereinafter, the operation of the control device 141 in this control system will be described with reference to the flowchart of FIG.

(Step S1501) The observation unit 1101 observes the state of the controlled device 12 and determines whether two or more first-type parameters have been acquired. If two or more first type parameters are acquired, the process proceeds to step S1502. If two or more first type parameters are not acquired, the process returns to step S1501.
(Step S1502) The conversion unit 141021 of the feature extraction unit 14102 acquires information used for parameter conversion from the long-term observation unit 14105.

(Step S1503) The conversion unit 141021 acquires one or more second type parameters based on the two or more first type parameters acquired in Step S1501 and the information acquired in Step S1502. Here, the number of the second type parameters may be larger than or the same as the number of the first type parameters.
(Step S1504) The selection unit 141022 of the feature extraction unit 14102 acquires information used for parameter selection from the long-term observation unit 14105.

(Step S1505) The selection unit 141022 selects one or more second type parameters from the one or more second type parameters acquired in step S1503 based on the information acquired in step S1504. The number of second type parameters selected here is preferably significantly smaller than the number of first type parameters. For example, it is preferable that the number of first type parameters is 2400 or more and the number of selected second type parameters is three.
(Step S1506) The control unit 41032 configures a control signal based on the second type parameter selected in step S1505. A specific method for configuring the control signal will be described later.
(Step S 1507) The control means 41032 gives the control signal configured in step S 1507 to the controlled device 12 and controls the controlled device 12.
(Step S1508) The evaluation unit 41031 outputs an evaluation of the control rule by the control unit based on the second type parameter selected in step S503 (or in addition to the control signal output by the control unit).
(Step S1509) The control unit 41032 changes (learns) a control rule for forming a control signal based on the second type parameter selected in step S503 and the evaluation output by the evaluation unit 41031 in step S507.
(Step S1510) The long-term observation unit 14105 acquires the first type parameter acquired in step S1501 and r in Expression 3.
(Step S1511) The long-term observation unit 14105 accumulates the information acquired in Step S1510.

そして、以下の図１６に示すフローチャートに示す動作により、制御装置１２１は動作する。 (Step S1512) The long-term observation unit 14105 generates information used for parameter conversion or / and parameter selection based on the accumulated information (one or more information). A specific information generation method will be described later. The process returns to step S1501.
Note that the processing is ended by powering off or interruption for aborting the processing in the flowchart in FIG.
Hereinafter, a specific operation of the control system in the present embodiment will be described. Now, the controlled device 12 is, for example, a snake robot shown in FIG.
Further, for example, the following s4 (Equation 23) and s5 (Equation 24) are prepared as second type parameters other than the above s1, s2, and s3.

And the control apparatus 121 operate | moves by the operation | movement shown in the flowchart shown in the following FIG.

  (Step S1601) The selection information receiving unit 4104 receives input of various initial values. Specifically, the selection information receiving unit 4104 receives information for selecting three from s1 to s5. Note that a set of three second-type parameters is S. Then, the selection information receiving unit 4104 receives the value “1” of n. The selection information receiving unit 4104 receives an appropriate T.

(Step S1602) Hereinafter, control is performed in the same procedure as in the second embodiment, using the set S instead of the set (s1, s2, s3). That is, first, the initial values of the parameters θi (policy parameter) and di of the control means 41032 and the evaluation means 41031 are determined. The auxiliary parameters Θi and Di for learning these parameters are set to zero. Also, t is set to 0.
(Step S1603) The observation unit 4101 observes the controlled device 12 and the control unit 4103, and outputs the first type parameter X (t).

なお、この報酬の和の値をＦ（ｎ）とする。
（ステップＳ１６１０）長期観測部１４１０５は、ｎ＝１であるか否かを判断する。ｎ＝１ならばステップＳ１６１１に行き、ｎ＝１でなければステップＳ１６１２に行く。
（ステップＳ１６１１）長期観測部１４１０５は、Ｆｍａｘ＝Ｆ（１）を保存する。長期観測部１４１０５はＳｍａｘ＝Ｓを保存してステップＳ１６１２に行く。
（ステップＳ１６１２）長期観測部１４１０５は、Ｓｍａｘの要素をひとつ入れ替えた集合Ｓを生成する。
（ステップＳ１６１３）ｎを１、インクリメントする。ステップＳ１６０２に戻る。
（ステップＳ１６１４）長期観測部１４１０５は、Ｆ（ｎ）>Ｆであるか否かを判断する。Ｆ（ｎ）>ＦならばステップＳ１６１５に行き、Ｆ（ｎ）>ＦでなければステップＳ１６１２に行く。
（ステップＳ１６１５）長期観測部１４１０５はＦｍａｘ＝Ｆ（ｎ），選択情報受付部はＳｍａｘ＝Ｓを保存する。ステップＳ１６１２に行く。
なお、公知技術である遺伝的アルゴリズムなどの探索手法を用いて，Ｓの探索を実現しても良い。 (Step S1604) The feature extraction unit 4102 acquires the first type parameter X (t), converts it to the second type parameter by the conversion unit 41021, selects the set S by the selection unit 41022, and outputs it to the control unit 4103. At the same time, the control unit 4103 acquires r (t) calculated by Expression 3.
(Step S1605) The control means 41032 acquires S and generates a control signal.
(Step S1606) The learning subroutine (FIG. 9) of the control unit 4103 is executed.
(Step S1607) The controlled apparatus 12 is controlled by the control signal generated in step S1305.
(Step S1608) The conversion unit 141021 determines whether or not t = T. If t = T, the conversion unit 141021 proceeds to step S1609. If t = T is not satisfied, the process goes to step S1603.
(Step S1609) The long-term observation unit 14105 calculates and records the sum of rewards shown in Formula 25 at each time from 0 to T seconds.

Note that the value of the sum of the rewards is F (n).
(Step S1610) The long-term observation unit 14105 determines whether n = 1. If n = 1, go to step S1611. If n = 1, go to step S1612.
(Step S1611) The long-term observation unit 14105 stores Fmax = F (1). The long-term observation unit 14105 stores Smax = S and goes to step S1612.
(Step S1612) The long-term observation unit 14105 generates a set S in which one element of Smax is replaced.
(Step S1613) n is incremented by one. The process returns to step S1602.
(Step S1614) The long-term observation unit 14105 determines whether or not F (n)> F. If F (n)> F, go to step S1615, and if F (n)> F, go to step S1612.
(Step S1615) The long-term observation unit 14105 stores Fmax = F (n), and the selection information receiving unit stores Smax = S. Go to step S1612.
Note that the search for S may be realized by using a search technique such as a genetic algorithm which is a known technique.

  As described above, according to the present embodiment, the control device compresses a large number of observed parameters, and can control the controlled device at high speed, or can learn a control law for that purpose. Moreover, suitable control can be performed in consideration of observation results over a predetermined time. Furthermore, according to the present embodiment, even in a situation where a controller adaptively finds a control rule without designing an optimal control rule by a designer, control that can learn a control rule for a multi-degree-of-freedom system at high speed. Equipment can be provided.

Furthermore, the software that implements the control device according to the present embodiment is the following program. That is, this program causes the computer to observe the state of the controlled device and the internal state of the control unit, and obtain two or more first-type parameters;
Based on two or more first-type parameters acquired in the observation step, a feature extraction step for acquiring a smaller number of second-type parameters than the first-type parameters, and one or more second parameters acquired in the feature extraction step Based on the seed parameter, the control step for controlling the controlled device, the first type parameter acquired in the observation step or / and the control result of the control step are acquired, and the acquired information is accumulated for a long period of time. A program for executing an observation step, wherein the feature extraction step includes two or more first type parameters acquired in the observation step, information accumulated in the long-term observation step, or information generated based on the information A transformation sub-step for obtaining two or more second-type parameters based on
A selection sub that selects one or more second-type parameters using information accumulated in the long-term observation step or information generated based on the information from two or more second-type parameters acquired in the conversion substep A step, wherein the control step evaluates the status of the controlled device based on the one or more second type parameters acquired in the feature extraction step, and indicates whether the evaluation is good or bad And a control substep for controlling the controlled device based on the information indicating the quality of the evaluation output in the evaluation substep.
In addition, the computer which performs all the programs mentioned above may be single, and plural may be sufficient as it. That is, centralized processing may be performed, or distributed processing may be performed.

In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.
The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the control device according to the present invention has an effect that a multi-degree-of-freedom system such as a robot can perform efficient control based on control parameters that are smaller than the degree of freedom. It is useful as a control device for controlling the degree of freedom system.

Block diagram of a control system in the first embodiment Flow chart for explaining the operation of the control device Diagram showing the snake robot Block diagram of a control system in the second embodiment Flow chart for explaining the operation of the control device Diagram showing the same ganglion Diagram showing the neural oscillator network Flow chart for explaining the operation of the control device Flow chart for explaining the operation of the control device Figure showing the results of the experiment Figure showing the results of the experiment Block diagram of a control system in the third embodiment Flow chart for explaining the operation of the conversion means Block diagram of a control system in the fourth embodiment Flow chart for explaining the operation of the control device Flow chart for explaining the operation of the control device

Explanation of symbols

11, 41, 121, 141 Control device 12 Controlled device 1101, 4101 Observation unit 1102, 4102, 12102, 14102 Feature extraction unit 1103, 4103 Control unit 4104 Selection information reception unit 11021, 41021, 121021, 141021 Conversion unit 14105 Long-term observation Part 41022, 141022 selection means 41031 evaluation means 41032 control means

Claims (16)

A state of a controlled device that is a device to be controlled and is a multi-degree-of-freedom system having two or more links and a joint connecting the two or more links, and for each state variable of the two or more links An observation unit that obtains two or more of the first type parameters as first type parameters;
By compressing the parameters based on the two or more first type parameters acquired by the observation unit, the number of parameters is less than the two or more first type parameters, and has a different meaning from the first type parameters. A feature extraction unit that obtains a second type parameter that is a parameter that is a parameter that represents a characteristic of the controlled device;
Based on one or more second-type parameters acquired by the feature extraction unit, a control unit for controlling the controlled device ,
The feature extraction unit includes:
Conversion means for acquiring two or more second type parameters based on two or more first type parameters acquired by the observation unit;
The control apparatus which comprises the selection means which selects the said 1 or more 2nd type parameter from the 2 or more 2nd type parameter which the said conversion means acquired. The second type parameter has a variable variable;
The converting means includes
Wherein the second type parameter has variable of the controlled device according to claim 1 wherein the change in accordance with the internal state of the controller is a state variable of the control results and / or control model is the situation for the controlled Control device. The controller is
Evaluation means for evaluating the status of the controlled device based on one or more second-type parameters acquired by the feature extraction unit;
Based on the evaluation in the evaluation means, learning of a control law for the controlled device is performed, and the controlled device is controlled based on the control law and one or more second type parameters acquired by the feature extraction unit. The control apparatus according to claim 1, further comprising a control unit. The observation unit is
The observing the internal state of the controller is a state variable of the state and control model of the controlled device, the control device according to any one of claims 3 2 or more first type parameters claims 1 to retrieve. The controller is
The control apparatus according to claim 4 , wherein a network structure having a plurality of elements connected in a network and having self-feedback is included, and an internal state of the control unit is a state of elements constituting the network structure. The feature extraction unit further includes a selection information receiving unit that receives selection information that is information for acquiring the second type parameter,
The feature extraction unit includes:
The control device according to any one of claims 1 to 5 , wherein a second type parameter having a smaller number than the first type parameter is acquired also using the selection information received by the previous selection information receiving unit. A first-type parameter acquired by the observation unit or / and a control result that is the status of the controlled device being controlled are further acquired, and further includes a long-term observation unit that accumulates two or more of the acquired information.
The converting means includes
The control device according to any one of claims 1 to 6 , wherein two or more second-type parameters are acquired also using information accumulated by the long-term observation unit or information generated based on the information. A first-type parameter acquired by the observation unit or / and a control result that is the status of the controlled device being controlled are further acquired, and further includes a long-term observation unit that accumulates two or more of the acquired information.
The selection means includes
The control device according to any one of claims 1 to 7 , wherein one or more second-type parameters are selected also using information accumulated by the long-term observation unit or information generated based on the information. On the computer,
A state of a controlled device that is a device to be controlled and is a multi-degree-of-freedom system having two or more links and a joint connecting the two or more links, and for each state variable of the two or more links An observation step of obtaining two or more of the first type parameters as first type parameters;
By compressing the parameters based on the two or more first-type parameters acquired in the observation step, the number of parameters is smaller than the two or more first-type parameters, and has a different meaning from the first-type parameters. A feature extraction step of obtaining a second type parameter that is a parameter having a characteristic of the controlled device;
A program for executing a control step for controlling the controlled device based on one or more second-type parameters acquired in the feature extraction step ,
The feature extraction step includes
A conversion sub-step of acquiring two or more second-type parameters based on the two or more first-type parameters acquired in the observation step;
A program comprising a selection sub-step for selecting the one or more second-type parameters from the two or more second-type parameters acquired in the conversion sub-step. The second type parameter has a variable variable;
The conversion substep includes:
The program according to claim 9 , wherein the variable of the second type parameter is changed according to a control result that is a state of the controlled device and / or an internal state of the control unit that is a state variable of the control model. . The control step includes
An evaluation sub-step for evaluating the status of the controlled device based on the one or more second-type parameters acquired in the feature extraction step;
Based on the evaluation in the evaluation sub-step, learning of a control law for the controlled device is performed, and the controlled device is controlled based on the control law and one or more second type parameters acquired in the feature extraction step. The program according to claim 9 , further comprising a control substep. In the observation step,
The program according to any one of claims 9 to 11 , wherein two or more first type parameters are acquired by observing a state of the controlled device and an internal state of a control unit which is a state variable of a control model. The controller is
The program according to claim 12 , wherein a network structure having a plurality of elements connected in a network and having self-feedback is included, and an internal state of the control unit is a state of elements constituting the network structure. On the computer,
Further executing a selection information receiving step of receiving selection information that is information for acquiring the second type parameter in the feature extraction step;
In the feature extraction step,
The program according to any one of claims 9 to 13 , wherein the number of second type parameters smaller than the first type parameters is acquired also using the selection information received in the previous selection information receiving step. On the computer,
Obtaining a control result which is the first type parameter acquired in the observation step and / or a controlled result of the controlled device, and further executing a long-term observation step of storing two or more of the acquired information;
The conversion substep includes:
The program according to any one of claims 9 to 14 , wherein two or more second-type parameters are acquired also using information accumulated in the long-term observation step or information generated based on the information. On the computer,
Obtaining a control result which is the first type parameter acquired in the observation step and / or a controlled result of the controlled device, and further executing a long-term observation step of storing two or more of the acquired information;
The selection sub-step includes
The program according to any one of claims 9 to 15 , wherein one or more second-type parameters are selected also using information accumulated in the long-term observation step or information generated based on the information.

Images