JP4644833B2 - Biped walking movement device - Google Patents

Description

  The present invention relates to a bipedal walking movement apparatus using a dynamic control device that is effective in realizing a periodic movement of bipedal walking that is a nonlinear control target.

  When trying to control a robot or the like, there may be a situation in which state variables necessary for control cannot be directly measured due to sensor noise or the inability to place a sensor in the first place.

  In such a case, it is common to use a state observer (observer) (for example, refer nonpatent literature 1) and a Kalman filter (for example, refer nonpatent literature 2 and nonpatent literature 3). However, when the target dynamics are nonlinear, it may be difficult to estimate the hidden state with these methods. In recent years, observers applicable to nonlinear systems have been proposed, but there is a problem that they cannot be applied unless specific conditions are satisfied (see, for example, Non-Patent Document 4 and Non-Patent Document 5).

  Also, an extended Kalman filter (updates the state distribution using a local linear model) (for example, see Non-Patent Document 3) or a Monte Carlo filter (approximates the state distribution by a large number of particles generated by the Monte Carlo method) (for example, A method such as Non-Patent Document 6) is known as a state estimation method in a nonlinear system, but in any case, the state distribution must be obtained explicitly.

That is, the conventional controller basically needs to observe the current state and give a control output by direct mapping from the current state.
DG Luenberger, “An introduction to observers”, IEEE Trans., AC, Vol. 16, pp. 596--602, 1971. RE Kalman and RS Bucy, “New results in linear filtering and prediction theory”, Trans., ASME, Series D, J. of Basic Engineering, Vol.83, No.1, pp. 95--108, 1961. FL Lewis, “Optimal Estimation: with an Introduction to Stochastic Control Theory”, John Wiley \ & Sons, 1977. Kiyotaka Shimizu, Shunsuke Suzuki, Tetsufumi Tanaka, “Nonlinear Observer by Gradient Descent Method (State Observer for Nonlinear Systems)”, IEICE Transactions A, Vol. J83-A, No.8, pp. 956-- 964, 2000. H. Nijimeijer and TL Fossen, “Directions in Nonlinear Observer Design”, Springer-Verlag, London, 1999. G. Kitagawa, “Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Models”, Journal of Computational and Graphical Statistics, Vol.5, pp. 1--25, 1996.

  An object of the present invention is to provide a biped walking movement device using a dynamic control device that can easily learn a state observer with respect to a periodic motion by giving the controller an internal state and dynamics. It is to be.

  In the present invention, the purpose of learning is not to reduce the estimation error at a certain moment, but to reduce the estimation error throughout the task period. Therefore, we construct a nonlinear state observer using the policy gradient method (reinforcement learning) framework.

  In one aspect of the present invention, the input to the state observer dynamics is considered as the behavior of the learner, and the state is appropriately determined from the current observable state (observation output), state observer state, and controller output. The law of behavior that leads the state of the observer to the state of the controlled object is obtained by the policy gradient method.

Therefore, according to one aspect of the present invention, a biped walking movement device, a right leg comprising a right upper thigh and a right lower thigh connected to the upper right thigh through a right knee joint, and a left upper thigh a left leg and a left lower leg connected through the parts and left upper thigh to the knee joints, detects the ground of the right leg or left leg, right knee right lower leg makes with the upper right thigh A first sensor group for detecting the angle and the left knee angle formed by the left lower leg part with respect to the left upper leg part , connected to the right leg and the left leg, and driving the right leg and the left leg to walk biped For detecting the waist part, the pitch angle sensor for detecting the pitch angle of the waist part, the right joint angle formed by the waist part and the upper right thigh part, and the left joint angle formed by the waist part and the left upper thigh part the second further a sensor group, lumbar, based on the state machine against the right lower leg and the left lower leg, the right lower leg and left Receiving a lower leg control unit for generating a lower leg control signal for driving the thigh, the detection result of the second sensor unit and the pitch angle sensor, generates a thigh control signals for driving the upper right thigh and upper left thigh And a dynamic control device for strengthening the central pattern generator having an internal state that performs periodic time development respectively corresponding to the upper right thigh and the left upper thigh It performs state estimation upper right thigh and left upper thigh on the basis of the learning, as the output of the PD servo system with respect to the target value corresponding to the internal state, the control means for generating on the thigh control signal for the upper right thigh and upper left thigh wherein, based on the upper leg control signal and the crus control signal, further comprising a drive means for driving the right leg and the left leg, the state machine of the lower leg control unit, each of the right leg and left leg A state machine that sequentially transitions between a first knee flexed state, a first knee stretched state, a second knee flexed state, and a second knee stretched state, The transition to the first knee extension state and the transition from the second knee flexion state to the second knee extension state are based on the angle formed by the left upper thigh and the upper right thigh based on the detection result of the second sensor group. Transition occurs on the condition that the value is below a predetermined value, the transition from the first knee extension state to the second knee flexion state and the transition from the second knee extension state to the first knee flexion state are as follows: Based on the detection result of one sensor group, it is generated according to the grounding of the left leg or the right leg, and the driving means is a target angle for the right knee angle and the left knee angle that is predetermined according to the state of the state machine. And the output of the PD servo system based on the right knee angle and the left knee angle detected by the first sensor group And knee drive means for outputting torque applied to the right knee and the left knee .

Preferably, the control means includes a feedback controller that updates a feedback parameter based on a value function and a reward sequence obtained during reinforcement learning based on state information of the upper right thigh and the left upper thigh, and feedback control based on the internal state of the central pattern generator that changes in accordance with the output from the vessel, to produce a thigh control signal.

  Preferably, reinforcement learning is performed by a policy gradient method.

  In the present invention, since the behavior rule itself has dynamics represented by internal variables and its differential equations, the behavior law and the property of pulling in the physical system can be used, and when state estimation is performed for periodic motion It is possible to facilitate learning of the state observer. Furthermore, it is possible to realize a controller having a robust property against sensor input noise and time delay.

  Furthermore, according to the present invention, the degree of freedom of the biped walking movement device is increased, and even when the state space becomes high-dimensional, it is possible to easily learn the state observer for the periodic motion. A biped walking movement apparatus using a control apparatus dynamic control apparatus can be provided.

  Embodiments of the present invention will be described below with reference to the drawings.

  In the following description, a case where the present invention is applied to biped walking control will be described as an example. However, the present invention is not necessarily limited to such a case. The present invention provides a control system effective for a system that performs the above. In particular, the present invention provides a control system suitable for application to an underactuated (mechanical) system that performs periodic motion.

[Embodiment 1]
(System configuration of the present invention)
FIG. 1 is a conceptual diagram showing an example of a biped walking movement system 1000 using the dynamic control device of the present invention.

  Referring to FIG. 1, system 1000 includes dynamic control device 100, body 40 provided on the upper portion of dynamic control device 100, and legs that are driven and controlled by dynamic control device 100. The leg portion includes a right leg 10r and a left leg 101, and each leg includes sensors 20r and 20l provided in the vicinity of the ground contact surface. On the other hand, the body part 40 is provided with a sensor 30.

The sensor 20 _r detects information such as the right leg angle θ _r and the angular velocity dθ _r / dt with respect to the center line 4 of the torso 40, and the sensor 20 _l detects the left leg angle θ _l and the angular velocity dθ _{l with} respect to the center line 4. Information / dt is detected and notified to the dynamic control device 100, respectively. Further, the sensor 30 detects information such as the pitch angle θ _p and the angular velocity dθ _p / dt of the body 40 with respect to the vertical direction 2 and notifies the dynamic control device 100 of the information.

  The dynamic control device 100 controls the operations of the right leg 10r and the left leg 10l based on information from the sensors 20r, 20l, and 30.

  FIG. 2 is a block diagram showing a configuration of the dynamic control apparatus 100 shown in FIG.

  Referring to FIG. 2, dynamic control device 100 has a communication interface 106 that receives signals from sensors 20r, 20l, and 30 and a memory for storing control parameters and information from the sensors that will be described later. Based on the dynamic behavior rule acquired by learning using the information from the device 104, the sensor 20r, the sensor 20l, and the sensor 30, the arithmetic processing unit 102 that generates a control signal, and the control signal from the arithmetic processing unit 102 And a drive unit 108 for controlling the driving of the right leg 10r and the left leg 10l.

  Hereinafter, a preparation process and a control operation for the control operation of the dynamic control device 100 will be described.

(1-1. Dynamic behavior rules)
First, “dynamic behavior rules” will be described as a premise for explaining the control operation of the present invention.

  FIG. 3 is a conceptual diagram for explaining such a dynamic behavior rule.

  The “dynamic behavior rule” is a framework in which the behavior rule itself as shown in FIG. 3A has dynamics represented by internal variables and their differential equations.

  To express this more concretely, as shown in FIG. 3 (2), the observer has an internal variable and its differential equation, or the controller as shown in FIG. The case of having a differential equation is considered.

  A control device that controls a control object based on such a “dynamic behavior rule” will be referred to as a “dynamic control device”.

  Below, the case where bipedal walking movement is implement | achieved by the dynamic control apparatus 100 by using the framework of FIG. 3 (3) first is considered.

  Since the behavioral rule has an internal state, the behavioral law and the physical property of the physical system can be used, which is considered effective when performing periodic motion and state estimation. In addition, it can be expected to have some robust property against noise and time delay of the sensor input.

  When such a behavior rule is acquired by learning, it becomes a problem that the internal state of the behavior rule becomes a hidden variable. However, when the internal state of the behavior rule is drawn into the state of the physical system, each state uniquely corresponds. However, it is necessary to handle the hidden state in the transient state.

  Therefore, in the present invention, as will be described later, a policy gradient method that can be applied even in an environment where a hidden variable exists is used as a dynamic behavior rule acquisition method.

  Below, the learning system of the present invention, a central pattern generator (CPG) constituting a dynamic behavior rule, and a feedback controller to the CPG will be described.

(1-2. Learning system)
In the following description, a dynamic behavior rule is applied to biped walking motion using a three-link biped walking robot model as an example of periodic motion.

  FIG. 4 is a functional block diagram illustrating processing performed by the arithmetic processing unit 102 described in FIG. As will be described below, the arithmetic processing unit 102 functions as a learning system.

  As shown in FIG. 4, this learning system basically configures a dynamic behavior rule by the CPG processing unit 1026 and the feedback controller 1022. The state x used for learning is represented by the following equation.

However, as described above, θ _r and θ _l are angles of the left and right legs 10 r and 10 l from the vertical direction of the robot, respectively, and θ _p is a pitch angle of the body 40.

  That is, the learning system has a feature that the state space is configured only by signals directly obtained from the robot, and the internal state of the CPG is not used.

  Here, it is assumed that all states are observed and y = x.

(1-3. Central pattern generator (CPG))
In the learning system realized by the arithmetic processing unit 102, a neural oscillator model represented by the following formula is used as the configuration of the CPG processing unit 1026 that constitutes the dynamic behavior rule. Such a neural oscillator model is disclosed in, for example, Kiyoshi Matsuoka, “Sustained oscillations generated by mutually inhibiting neurons with adaptation.”, Biologial Cybernetics, Vol. 52, pp. 367-376, 1985. is there.

Here, variables: z and p are states in a neuron, q is a neuron output, z ₀ is a continuous input, constant β is a neuron fatigue coefficient, τ and τ ′ are time constants of z and p, and ω is an antagonist This is the coupling coefficient between neurons. Further, a is an output term from a feedback controller described later.

  FIG. 5 is a conceptual diagram showing a CPG based on a neural oscillator model expressed by equations (1) to (12). In FIG. 5, among the states z and p in the neuron, those that perform a positive connection with each other are indicated by white circles, and those that perform a negative connection are indicated by black circles.

FIG. 6 is a diagram showing the waveforms of variables z ₁ and z ₂ that constitute the output q of such a CPG. In this calculation, τ = 0.05, τ ′ = 0.6, β = 2.5, ω = 2.0, z ₀ = 0.1, and a = 0 are used as examples.

It can be seen that the internal state variables z ₁ and z _{2 of} the CPG change periodically according to the models represented by the equations (1) to (12).

Further, in the PD servo processing unit 1028 of the learning system of FIG. 4, the difference in the output of each neuron is set as the target joint angle θ ^d of the servo system of both legs as shown below.

However, θ _l ^d is the target joint angle of the left leg, and θ _r ^d is the target joint angle of the right leg.

  The torque input u to the robot output as a result of the PD servo processing unit 1028 uses the output of the following PD servo system.

However, u _l is torque input to the left leg, u _r is the torque input to the right leg. K _p is a position gain, and K _d is a speed gain.

(1-4. Feedback controller 1022)
The feedback controller 1022 to the CPG described above is represented by the following probability distribution (17).

Here, x is a state vector to be controlled, and w is a parameter vector.
Therefore, the actual value v _j of the j-th output is given by the following equation (18).

  However, nj (t) to N (0,1), where N (0,1) represents a normal distribution with an average of 0 and a variance of 1.

Here, in order to saturate the output, the output saturation processing unit 1024 determines the final controller output a _j (t) using the function d () as shown in the following equation (19).

  However, in the following description, as an example, the following formula is used as d ().

Here, a _j (j = 1 to 4) corresponds to the extensor and flexor muscles of the neural oscillators of the left and right legs of Expression (1), respectively.

(2. Policy gradient method)
The “policy gradient method” is one type of reinforcement learning method in which action selection is performed according to a parameterized probabilistic policy, and policy parameters are updated little by little in a direction of improving the policy. The following describes how to learn behavioral rules using the policy gradient method.

(2-1. Temporal Difference error in continuous time / state system)
The dynamics of the continuous time / state system is expressed by the following formula (20).

However, x∈X⊂R ⁿ represents a state, and u∈U⊂R ^m represents a control input.

  It is assumed that the reward is given by the following equation (21) as a function of the state and the control input.

  Under a certain control law π (u (t) | x (t)), the value function of the state x (t) is defined by the following equation (22).

  Where τ is the time constant of the value function. Further, a constraint condition of the following expression (23) is given from the time differentiation of both sides of the expression (22).

  Let V (x (t)) = V (x (t); w) be the predicted value of the value function. However, w is a parameter of a predicted value of an evaluation value.

  If the prediction is correct, Expression (23) is satisfied. If the prediction is not correct, learning is performed so as to reduce the prediction error shown in the following equation (24).

  The above equation is a TD error in a continuous time system.

(2-2. General theory of policy gradient method)
When learning is performed based on evaluation of value functions such as dynamic programming or greedy policy, the environment needs to be a Markov decision process. It is difficult to guarantee the Markov decision process by the ability of. However, the policy gradient method can be applied even when the environment is a non-Markov decision process (POMDP) by considering the cumulative reward sequence obtained during the trial, together with the value function.

  Here, when the policy πw having the parameter w is used, the following equation (25) is established.

  However, the following equation (26) holds.

  Here, κ is the time constant of eligibility trace. The temporal difference error δ and the policy eligibility trace e (t) provide that an unbiased estimate of the gradient for the policy parameter w of the value function can be determined. For such policy gradient method, for example, literature: Motoki Kimura, Shigenobu Kobayashi, “actor-critic algorithm using appropriateness history for actor-reinforcement learning under imperfect value-function”, artificial It is described in Journal of Intelligence Society, Vol.15, No.2, pp. 267-275, 2000.

  Therefore, the parameter update rule is as shown in the following equation (27).

  Where η is a learning rate.

(2-3. Learning dynamic behavior rules)
In the following, dynamic behavior rules are acquired using the above-described policy gradient method.

  Here, it is assumed that a desired dynamic behavior rule is acquired by learning the feedback controller shown in Expression (17).

(2-3-1. Update of value function)
First, a normalized Gaussian network (NGnet) according to the following equation (28) is used as a method for expressing a value function in a continuous state calculated by the value function processing unit 1032. The normalized Gaussian function network will be described later.

Here, b _i ^c () is a basis function applied to x in the normalization processing unit 1030, and w _i ^c is a parameter of the value function.

And Erijibiriti trace e _i ^c for the parameters w _i ^c, update equation parameters w _i ^c with TD error is as each of the following formula (29) and (30).

Where α is the learning rate of the value function, and κ ^c is the time constant of the eligibility trace.

(2-3-2. Update of feedback controller)
When the probabilistic feedback controller 1022 shown in Expression (17) is used, the eligibility regarding the average μ _j and standard deviation σ _j of the j-th output is as follows as in the second term on the right side of Expression (26). Given to.

  Here, as in the following equations (33) and (34), the mean μ is represented by a normalized Gaussian function network, and the standard deviation σ is represented by a sigmoid function.

  However, the notation is as follows.

  Eliability corresponding to these parameters is obtained by the following equations (35) and (36).

  When the above equations (35), (36) and equations (26), (27) are taken into account, feedback parameter update rules such as the following equations (37) and (38) are obtained.

  However, the notation is as follows.

  In addition, in the equations (35) and (36), since the parameter σ is a denominator, easiness diverges when σ approaches zero. Therefore, the following equation is used instead of equation (26) for updating the eligibility trace.

  However, the notation is as follows.

(2-4. Specific example)
In the learning system shown in FIG. 4, the results of numerical simulation will be described below.

  In this simulation, the biped walking movement system 1000 (biped walking robot) shown in FIG. 1 has a leg length of 0.2 m, a mass of both legs of 0.5 kg, and a body of 0.1 kg. Furthermore, considering that there is no knee joint, when swinging the swing leg, it was set so that the foot tip could pass through the ground.

  Each learning parameter is as follows.

  In addition, the basis functions of NGnet are assumed to be arranged in a grid pattern in a state space that is expected to be necessary when the robot actually performs a walking motion.

  As a result, a total of 5184 (= 12 × 6 × 12 × 6) was equally arranged in the following ranges.

  The reward function is expressed by the following formula.

However, the term r _H (t) relating to the waist height of the robot and the term r _S (t) relating to the walking speed are expressed by the following equations.

Here, h ₁ is the height of the waist of the robot, h'the waist height of the offset, f _l, is f _r is the height of the left and right legs. Therefore, the first term on the right side of Equation (41) is an amount related to the robot's potential energy, and the second term on the right side is an amount related to the kinetic energy of the robot.

In the simulation described below, the parameters are set to k _s = 0.06, k _H = 0.5, and h ′ = 0.15. The CPG parameters used are those described in (1-2. Learning system).

  The time increment of the robot and CPG dynamics on the computer was 1 msec, and the time increment of the learning system was 10 msec.

  In the simulation, the end condition of one learning trial was as follows.

i) 17700 msec elapsed (after about 100 steps of walking)
ii) At the time of fall
(2-5. Acquisition of flat ground walking and robustness against environmental changes)
FIG. 7 is a diagram showing a learning curve in which the total amount of rewards acquired in one trial is taken for each number of trials. FIG. 7 shows a learning curve when the ground inclination is 0 °.

  FIG. 7 shows that learning has converged at about 350 times, and a steady walking motion has been acquired.

  FIG. 8 is a diagram showing a walking locus corresponding to the learning curve of FIG.

  In FIG. 8, (1) shows a walking trajectory before learning, and (2) shows a walking trajectory after 600 learnings. It can be seen that, after learning 600 times, the stride is increased, the walking speed is improved, and a good walking locus is obtained.

  Moreover, even when the environment is changed by using each learning parameter learned by performing 600 learning trials and adding an inclination of several degrees, it is possible to maintain a walking motion to some extent. Furthermore, it is possible to adapt to a new environment by performing several learning trials. This is presumably because the internal state of the behavior rule (herein, the internal state of the CPG) and the state of the robot constitute a robust limit cycle by pulling in.

FIG. 9 is a diagram showing a limit cycle between the internal state of the CPG and the state of the robot in the walking acquired in FIG. 7 as a time change of the internal state z _{1 of the} CPG and the leg angle.

FIG. 10 is a diagram showing the limit cycle between the internal state of the CPG and the state of the robot in the walking acquired in FIG. 7 as a relationship of the leg angle, the angular velocity of the leg, and the internal state z ₁ of the CPG.

  It can be seen that the control system of the present invention can continue the periodic motion even against external disturbances.

(2-6. Relationship between reward and acquired exercise)
Table 1 shows the relationship of the walking speed of the robot when the speed term coefficient k _s in the reward function of Expression (41) is changed.

Here, the coefficient of the term relating to the waist height of the robot was set to k _H = 0.5 as in the previous section.

  From Table 1, it can be seen that increasing the speed term also increases the walking speed of the robot, and thus controls the robot by changing the learning reward without explicitly using a controller composed of the dynamics of the robot. I can confirm that I can do it.

(2-7. Robustness against sensor noise and time delay)
Using the parameters obtained by learning the walking acquired in FIG. 7 as a teacher signal as initial values and further using the learning system shown in FIG. Acquired bipedal movement with no behavior rules. Using this and the walking motion acquired by learning in FIG. 7, the sensor noise of the controller and the robustness against time delay were compared.

The sensor noise is N (0, 0.01) for x ₁ and x ₃ , N (0, 0.09) is used for x ₂ and x ₄ , and the time delay is 20 msec. went.

FIG. 11 is a diagram showing simulation results for sensor noise and time delay.
In FIG. 11, (1) shows walking with a CPG and (2) shows a walk constituted by a controller without CPG. In FIG. 11, (a) is walking under normal conditions, (b) is walking with sensor noise, and (c) is walking when there is a time delay, and in FIG. , “→” represents the traveling direction of the robot.

  In the case of walking composed of behavior rules without CPG, the walking motion could not be maintained in both cases of noise and time delay. However, in the learning system shown in FIG. 4, there is noise and time delay. But you can walk.

  Therefore, it can be seen that a controller that is robust against sensor noise and time delay can be configured by configuring a behavior rule having an internal state.

(3. Function approximation by normalized Gaussian function network)
The normalized Gaussian function network used to represent the value function and feedback controller described in 2-3-1 will be described below.

NGnet is composed of a three-layer network, and the intermediate element is a normalized Gaussian function.
For the input vector x = (x ₁ ,..., X _n ) ^T , the activation function of the kth unit is as follows:

Here, c _k is the center of the activation function, and M _k is a matrix that determines the shape of the activation function. Here, the activation function φ _k (x) normalized as shown in the following equation (43) so that the sum is 1 at each point is defined as a basis function b _k (x).

  Here, K is the number of basis functions.

By performing such normalization, in the portion where the center point c _k are densely arranged, b _k (x) becomes a local basis functions, the end portions of the distribution of c _k b _k (x ) Becomes a global basis function like a sigmoid function.

  The output of the network is expressed by the following equation (44) by the inner product of the basis function and the weight.

  This output becomes the output of the normalization processing unit 1030 in FIG.

[Modification of Embodiment 1]
In the above description, the control by the structure of FIG. 3 (3) was demonstrated. Hereinafter, control according to the configuration of FIG. 3B will be described as a modification of the first embodiment.

  FIG. 12 is a system corresponding to the configuration of FIG. 3 (2), and is a diagram showing a configuration of the entire system including a controller and a controlled object.

  In FIG. 12, the state observer 2002 includes a state observer dynamics 2004 and a reinforcement learner 2006 based on the policy gradient method (reinforcement learning).

  The reinforcement learner 2006 in the state observer 2002 learns based on the observation output y to be controlled, the output based on the dynamics 2004 of the state observer, and the control output u of the controller 2010 as described below. Unit output U is output. The output function processing unit 2030 gives the output of the state observer 2002 to the reward calculator 2020 based on the estimated state from the state observer 2002. The reward calculation unit 2020 calculates a reward based on the output of the state observer 2002, the observation output y from the observation target, and the learner output U, and gives the reward to the reinforcement learner 2006.

(Learning state observers using the policy gradient method)
Below, the structure of the state observer 2002 of the modification of Embodiment 1 is demonstrated.

  When an estimated value of a state is expressed by adding “^” to the head of x (= xi) (hereinafter referred to as “x hat” in the text), the following state observer is considered.

  Here, first, as in a normal observer or Kalman filter, it is assumed that the target dynamics f (x, u) is known or can be obtained by learning, and based on the observation output y of the target system, the current estimated state x hat and control The policy gradient method is used to learn how to approximate the estimated state to the true state from the output u.

  Here, the purpose of the learning device is to minimize the error between the output y hat of the state observer (with “^” added to the head of y) and the output y of the target system.

  Therefore, the reward function calculated by the reward calculation unit 2020 is as follows.

  However, Q and R are parameters that determine the shape of the reward function. As a result, the learning device obtains a feedback input U of the following notation to the dynamics 2004 of the state observer.

  Here, the feedback input U is expressed by the following probability distribution.

Therefore, the realization value U _j of the j-th output is given by the following equation.

However, n _j (t) to N (0, 1), and N (0, 1) represents a normal distribution with an average of 0 and a variance of 1 as described above. The update of the probability distribution π for generating the feedback input U is performed in the same manner as in the case of learning bipedal locomotion.

  Even with such a configuration, it is possible to provide a dynamic control device capable of facilitating the learning of the state observer with respect to the periodic motion, and a biped walking movement device using such a dynamic control device. .

[Embodiment 2 (application to multi-free time system)]
(5.1) Five-link biped walking robot model In the second embodiment, a configuration in which the learning system configured in FIG. 4 is applied to a multi-degree-of-freedom system will be described.

  From the learning results for the 3-link biped walking robot model described in the first embodiment, by using the policy gradient method as the learning method, it is possible to obtain a desirable behavior rule by considering only the state of the physical system. I understand that.

  FIG. 13 is a diagram for explaining a five-link biped walking robot handled in the second embodiment and a simulation model thereof.

  Based on the results of the first embodiment, in the following, the simulation model (FIG. 13B) of the five-link real robot having the knee joint shown in FIG. Apply the proposed method to acquire walking motion.

  As shown in FIG. 13B, in the simulation model of the second embodiment, the right lower leg 10rd (link 1), the upper right thigh 10ru (link 2), the waist 40 (link 2), the left upper leg 10lu (link 4), and the lower left This is a five-link system of thigh 10ld (link 5).

  The biped walking robot shown in FIG. 13B also includes a dynamic control device 100, a torso 40 provided on the top of the dynamic control device 100, and legs that are driven and controlled by the dynamic control device 100. . The leg has an upper right thigh 10ru, a right lower thigh 10rd, a left upper thigh 10lu, and a left lower thigh 10ld.

  The biped robot shown in FIG. 13 (b) is provided on each leg with sensors 20r and 20l provided near the ground contact surface of the right and left legs, a sensor 20rk provided on the right knee part, and a left knee part. It is assumed that the sensor 20lk is provided. On the other hand, the body 40 is provided with a sensor (not shown). As in the first embodiment, the dynamic control device 100 drives the upper right thigh 10ru and the left upper thigh 10lu by the drive unit 108. The left and right knees are also provided with a drive unit (not shown) and drives the right lower leg 10rd and the left lower leg 10ld based on a torque signal from the dynamic control device 100.

The sensor 20rk detects information such as the angle θ ^r _hip and the angular velocity dθ ^r _hip / dt of the upper right thigh 10ru with respect to the center line 4 of the torso 40, and the sensor 20lk detects the angle θ ^l of the left upper thigh 10lu with respect to the center line 4. Information on _hip and angular velocity dθ ^l _hip / dt is detected and notified to the dynamic control device 100, respectively. Further, the sensor 20r detects information on the right leg angle θ ^r _knee and angular velocity dθ ^r _knee / dt with respect to the extension line 6r of the upper right thigh 10ru and the ground contact state of the right foot, and the sensor 20l detects the left upper leg 10lu. Information on the left leg angle θ ^l _knee and angular velocity dθ ^l _knee / dt and the ground contact state of the left foot with respect to the extension line 61 is detected and notified to the dynamic control device 100, respectively.

The sensor provided in the body 40 detects information such as the pitch angle θ _p and the angular velocity dθ _p / dt of the body 40 with respect to the vertical direction 2, and notifies the dynamic control device 100 of the detected information.

  Table 1 shows the physical parameters of the 5-link biped robot model.

(5.2) Learning system for multi-degree-of-freedom system The learning system for the 5-link biped walking robot model shown in FIG.

In the dynamic control device 100, the state of the knee joint is not used for learning, and the state of the hip joint (θ ^r _hip, θ ^l _hip , dθ ^r _hip / dt, dθ ^l _hip is used as in FIG. 4 of the first embodiment. / dt) and the state variables related to the pitch angle states (θ _p , dθ _p / dt) only.

  On the other hand, for the knee joint, a state machine 1040 that switches the target joint angle of the knee joint based on the ground contact information and the state of the hip joint is used as a controller. The state machine 1040 will be described later.

  FIG. 14 is a diagram illustrating a configuration of a learning system according to the second embodiment in the dynamic control device 100. The same parts as those of the dynamic control device 100 according to the first embodiment shown in FIG. However, in FIG. 14, the feedback controller 1022 ′ has both the function of the feedback controller 1022 and the function of the output saturation processing unit 1024, and the value function processing unit 1032 ′ The function and the function of the normalization processing unit 1030 are also included.

  Torque to the left upper thigh 10lu and upper right thigh 10ru is given from the PD servo processing unit 1028.

  Therefore, the operations of the value function processing unit 1032 ′, the feedback controller 1022 ′, the CPG processing unit 1026, and the PD servo processing unit 1028 differ only in the target to be driven, and the basic operation is the same as in the first embodiment. It is.

  On the other hand, torque input to the knee joint is given by the PD servo processing unit 1042 using the target joint angle determined by the state machine 1040.

(Knee joint controller by state machine 1040)
15 is a concept for explaining the operation of the state machine 1040. FIG. 15 (a) shows the state of the right knee, and FIG. 15 (b) shows the state of the left knee.

  The state repeats the state transition of knee flexion → knee extension → knee flexion → knee extension.

For example, in the right knee, the knee flexion → knee extension in the first half is the “swing state” of the leg, and the target angles are θ ₁ and θ ₂ , respectively. The knee flexion → knee extension in the second half of the right knee is the “standing state” of the leg, and the target angles are θ ₃ and θ ₄ , respectively.

On the other hand, in the left knee, the knee flexion → knee extension in the first half is the “standing state” of the leg, and the target angles are θ ₃ and θ ₄ , respectively. The knee flexion → knee extension in the second half of the left knee is the “swing state” of the leg, and the target angles are θ ₁ and θ ₂ , respectively.

The hip joint angle θ ^r _hip , θ ^r _hip (hereinafter, collectively referred to as “θ _hip ”) and ground contact information are used to determine the knee joint target angle (θ ^r _knee hat, [theta] ^l _knee hat: "[theta] _knee hat") is generally determined, and torque u _knee to the knee joint is output by the PD servo shown below.

Here, K _p is a position gain, and K _d is a speed gain. However, in the simulation, K _p = 12.0 and K _d = 0.15.

As shown in FIG. 15, the state machine switches the four target angles (θ ₁ , θ ₂ , θ ₃ , θ ₄ ) according to the set conditions. Here, θ ₁ = 1.11, θ ₂ = 0.56, θ ₃ = 0.52, and θ ₄ = 0.26. The transition from the knee bent state to the extended state is performed based on the following conditions using the hip joint.

  The transition from the knee extended state to the bent state is performed based on the ground contact information of the sole. In the simulation, for example, b = 0.15.

(Floor model)
Hereinafter, a floor reaction force model used in the simulation will be described.

  x and y represent the positions of the leg ends, and xg and yg are ground contact points, the floor reaction force at the time of ground contact is modeled by the following equation.

Here, F _x and F _y are floor reaction forces in the horizontal and vertical directions.

In the simulation described below, the coefficients of the above equations are k _x = 3000, b _x = 10, k _y = 30000, and b _y = 100. Further, when the floor reaction force satisfies F _x > μF _y , it is defined that the sole can slide on the floor surface, where μ is a coefficient of static friction, and μ = 1.0.

(5.3) Simulation Hereinafter, the simulation result in the second embodiment will be described.

  The CPG parameters used in this simulation require a lower frequency than the characteristics used in the three-link biped walking robot model in FIG. 4 in consideration of changes in the leg length of the robot model. Do as follows.

τ = 0.4, τ ′ = 0.8, β = 1.3, ω = 2.0, z ₀ = 0.1
For the basis functions of the normalized Gaussian function network NGnet, x = (x ₁ , x ₂ , x ₃ , x ₄ ) ^T is arranged as follows in a lattice pattern for each state variable.

  25600 basis functions were evenly arranged in the following ranges.

  As for the reward function, equation (41) is used, and ks = 0.06, kH = 0.5, and h ′ = 0.3 in accordance with the change of physical parameters.

  Other learning parameters and the like are the same as those in the first embodiment except for βσ in equation (38) (σ is a superscript) = 0.02.

  As for the end condition of one learning trial, the following conditions are used as in the first embodiment.

・ After 17700 msec ・ When falling (however, reward of r = −1 is given at the same time)
(5.4) Learning Results Also in the simulation of the second embodiment, as with the learning with the 3-link biped walking robot, the bipedal walking motion is acquired when the robot continues to walk without falling down for 10 consecutive trials. Define that. As a result of 10 simulations, we succeeded in acquiring bipedal locomotion in all trials. In addition, the average number of trials required for exercise acquisition was 899 times.

  FIG. 16 is a diagram showing the relationship between the number of trials and the total sum of earned rewards as an example of the learning process, and FIG. 17 is a diagram showing walking trajectories before and after learning.

  In FIG. 17, (1) is a walking trajectory before learning, and (2) is a walking trajectory after 1500 learning.

  This result shows that even when the degree of freedom of the biped robot increases and the state space becomes high-dimensional, the desired periodic motion can be acquired with a small number of trials by observing only a small number of state variables.

  According to the results of FIG. 16, a conventional example using reinforcement learning for a biped robot with five links (for example, literature: Y. Nakayama, M. Sato, and S. Ishii. Reinforcement Learning for biped robot. In Proceedings of the 2nd International Symposium on Adaptive Motion of Animals and Machines, pp. Thp-II-5, 2003.)

  With the configuration as described above, a dynamic control device capable of facilitating the learning of the state observer for periodic motion even when the degree of freedom of the biped robot increases and the state space becomes high-dimensional. And the biped walking movement apparatus using such a dynamic control apparatus can be provided.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a conceptual diagram which shows an example of the biped walking movement system 1000 using the dynamic control apparatus of this invention. It is a block diagram which shows the structure of the dynamic control apparatus 100 shown in FIG. It is a conceptual diagram for demonstrating a dynamic action rule. It is a functional block diagram which shows the process which the arithmetic process part 102 performs. It is a conceptual diagram which shows CPG by a neural oscillator model. It is a diagram showing a waveform of a variable z _1, z ₂ constituting the output q of the CPG. It is a figure which shows the learning curve which took the sum total of the reward acquired by 1 trial for every number of trials. It is a figure which shows the locus | trajectory of the walk corresponding to the learning curve of FIG. It is a figure which shows the limit cycle between the internal state of CPG and the state of a robot as a time change of the internal state z1 of CPG and a leg angle. The limit cycle between the internal state and the robot status CPG, illustrates leg angle, angular velocity of the leg, as a relation of the internal state z ₁ of CPG. It is a figure which shows the simulation result with respect to sensor noise and time delay. It is a figure which shows the structure of the whole system including a controller and a control object. FIG. 6 is a diagram for explaining a 5-link biped walking robot handled in Embodiment 2 and a simulation model thereof. It is a figure which shows the structure of the learning system of Embodiment 2 in the dynamic control apparatus 100. This is a concept for explaining the operation of the state machine 1040. It is a figure which shows the relationship between the number of trials of an example of a learning process, and the sum total of acquisition reward. It is a figure which shows the walk locus | trajectory before learning after learning.

Explanation of symbols

  10r, 10l leg, 20r, 20l, 30 sensor, 40 trunk, 100 dynamic control device, 102 arithmetic processing unit, 104 storage device, 106 communication interface, 108 drive unit, 1000 biped walking movement system.

Claims (3)

A right leg comprising an upper right thigh and a right lower thigh connected to the upper right thigh via a right knee joint ;
A left leg and a left lower leg connected through the joints of the left knee to the upper left thigh and the left upper thigh,
The ground contact of the right leg or the left leg is detected, and the right knee angle formed by the right lower leg part with respect to the upper right thigh part and the left knee angle formed by the left lower leg part with respect to the left upper leg part are detected. A first sensor group for
A waist part connected to the right leg and the left leg, for driving the right leg and the left leg to perform biped walking;
A pitch angle sensor for detecting the pitch angle of the waist;
A second sensor group for detecting a right joint angle formed by the waist and the upper right thigh and a left joint angle formed by the waist and the left upper thigh ;
The waist is
On the basis of the right lower leg and the state machine against the left crus, a crus controller that generates a lower leg control signal for driving the right lower leg and the left lower leg,
A dynamic control device for receiving a detection result of the second sensor group and the pitch angle sensor and generating an upper leg control signal for driving the upper right thigh and the left upper thigh,
The dynamic control device includes:
Estimating the state of the upper right thigh and the left upper thigh based on reinforcement learning with respect to a central pattern generator having an internal state corresponding to the upper right thigh and the left upper thigh, respectively, that periodically develops. , as the output of the PD servo system with respect to the target value corresponding to the internal state includes a control means for generating said on thigh control signal for the upper right thigh and the left upper thigh portion,
Drive means for driving the right leg and the left leg based on the upper leg control signal and the lower leg control signal ,
The state machine of the lower leg control unit is:
A state machine that sequentially transitions between a first knee flexion state, a first knee stretch state, a second knee flexion state, and a second knee stretch state for each of the right leg and the left leg. And
The transition from the first knee flexion state to the first knee extension state and the transition from the second knee flexion state to the second knee extension state are based on the detection results of the second sensor group. A transition occurs on the condition that the angle formed by the left upper thigh and the upper right thigh is below a predetermined value,
The transition from the first knee extension state to the second knee flexion state and the transition from the second knee extension state to the first knee flexion state are based on the detection results of the first sensor group. Occurs in response to the grounding of the left leg or the right leg,
The driving means includes a predetermined target angle for the right knee angle and the left knee angle determined according to a state of the state machine, and the right knee angle and the left knee angle detected by the first sensor group. A bipedal walking movement device including knee driving means for outputting torque applied to the right knee and the left knee as an output of the PD servo system based on the above . The control means includes
Based on the right upper thigh and status information of the upper left thigh, on the basis of the compensation sequence obtained with the value function in the reinforcement learning comprises a feedback controller to update the feedback parameters,
Based on the internal state of the central pattern generator that changes in accordance with the output from the previous SL feedback controller generates the thigh control signal, bipedal walking mobile apparatus according to claim 1.   The biped walking movement apparatus according to claim 2, wherein the reinforcement learning is performed by a policy gradient method.