WO2023184933A1 - 基于神经振荡器的机器人节律运动控制方法及系统 - Google Patents

基于神经振荡器的机器人节律运动控制方法及系统 Download PDF

Info

Publication number
WO2023184933A1
WO2023184933A1 PCT/CN2022/125984 CN2022125984W WO2023184933A1 WO 2023184933 A1 WO2023184933 A1 WO 2023184933A1 CN 2022125984 W CN2022125984 W CN 2022125984W WO 2023184933 A1 WO2023184933 A1 WO 2023184933A1
Authority
WO
WIPO (PCT)
Prior art keywords
robot
phase
joint position
neural
frequency
Prior art date
Application number
PCT/CN2022/125984
Other languages
English (en)
French (fr)
Inventor
张伟
陈燕云
盛嘉鹏
方兴
谭文浩
宋然
李晓磊
程吉禹
Original Assignee
山东大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 山东大学 filed Critical 山东大学
Publication of WO2023184933A1 publication Critical patent/WO2023184933A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/08—Control of attitude, i.e. control of roll, pitch, or yaw
    • G05D1/0891—Control of attitude, i.e. control of roll, pitch, or yaw specially adapted for land vehicles

Definitions

  • the invention belongs to the field of robot control technology, and in particular relates to a robot rhythmic motion control method and system based on a neural oscillator.
  • MPC Model Predictive Control
  • WBC Whole Body Cryotherapy
  • model-free reinforcement learning which has emerged in recent years, has successfully achieved autonomous learning of the motion strategy of leg-footed robots.
  • the reward function usually cannot directly express the desired rhythmic movement behavior; furthermore, even reasonable rewards must be carefully designed and adjusted to meet the needs, because minimal adjustments to the reward function may also lead to reinforcement learning behaviors. Huge differences; therefore, the design of reward functions that enable unbiased learning by robots is often time-consuming and difficult.
  • the present invention proposes a robot rhythmic motion control method and system based on a neural oscillator.
  • the control structure designed by the present invention consisting of a neural oscillator and a pattern forming network can ensure the formation of the desired rhythmic motion behavior. ;
  • the designed action space with incremental joint positions can effectively accelerate the rhythmic motion reinforcement learning training process.
  • the present invention provides a method for controlling rhythmic motion of a robot based on a neural oscillator, including:
  • control instructions are obtained to control the robot;
  • the preset reinforcement learning network includes an action space, a pattern forming network and a neural oscillator; the action space is used to obtain joint position increments based on the acquired current state; the pattern forming network is used to obtain joint position increments based on the acquired current state; The position increment is used to obtain the control instruction of the target joint position; the neural oscillator is used to adjust the phase change time of the robot's sole trajectory between the swing phase and the standing phase according to the acquired phase and frequency; according to the target joint position The control instructions and the timing of the phase transition of the robot's sole trajectory between the swing phase and the stance phase control the robot.
  • phase is represented by sine and cosine functions.
  • the joint position increment is added to the target joint position at the previous moment to obtain the target joint position at the current moment; according to the target joint position at the current moment , calculate the joint torque.
  • the maximum joint position increment is determined by the maximum motor speed and time step.
  • the neural oscillator outputs frequency to modulate the phase ⁇ of each leg.
  • the front leg is in the support phase
  • the phase ⁇ [ ⁇ ,2 ⁇ ) the front leg is in the swing phase.
  • phase at the current moment is:
  • ⁇ t ( ⁇ t-1 +2 ⁇ *f*T)%2 ⁇
  • ⁇ t represents the phase at the current moment
  • ⁇ t-1 represents the frequency at the previous moment
  • f represents the frequency
  • T represents the time step.
  • the robot's motion problem is regarded as a Markov decision process, and frequency terms and phase terms are added to the reward term.
  • the present invention also provides a robot rhythmic motion control system based on a neural oscillator, including:
  • a data acquisition module configured to: acquire the current state of the robot, as well as the phase and frequency generated by the neural oscillator;
  • the control module is configured to: obtain control instructions based on the obtained current status, phase and frequency, and the preset reinforcement learning network to control the robot;
  • the preset reinforcement learning network includes an action space, a pattern forming network and a neural oscillator; the action space is used to obtain joint position increments based on the acquired current state; the pattern forming network is used to obtain joint position increments based on the acquired current state; The position increment is used to obtain the control instruction of the target joint position; the neural oscillator is used to adjust the phase change time of the robot's sole trajectory between the swing phase and the standing phase according to the acquired phase and frequency; according to the target joint position The control instructions and the timing of the phase transition of the robot's sole trajectory between the swing phase and the stance phase control the robot.
  • the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the program, the implementation of the first aspect is achieved.
  • the steps of the neural oscillator-based robot rhythmic motion control method are described in detail below.
  • the present invention also provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the neural oscillator-based robot rhythmic motion control method described in the first aspect is implemented. A step of.
  • the control structure composed of neural oscillators and pattern forming networks designed by the present invention can ensure the formation of desired rhythmic movement behavior; at the same time, the designed action space with joint position increments can effectively accelerate the rhythmic movement reinforcement learning training process and solve the problem of When learning existing model-free reinforcement learning, reward function design is time-consuming and difficult.
  • Figure 1 is the RL learning framework of Embodiment 1 of the present invention.
  • Figure 2 is a diagram of the rhythmic movement mechanism of vertebrate animals in Embodiment 1 of the present invention.
  • Rhythmic motion is widely present in human and animal locomotor behaviors, such as walking, running, and turning.
  • the flexibility to change movement patterns is crucial for animals to successfully navigate harsh environments. Therefore, studying the mechanisms by which organisms drive different rhythmic movements is an important topic in biology and robotics.
  • Physiological research has found that the central pattern generator, the biological neural circuit in the spinal cord, plays a key role in the generation of rhythmic movement, and it can generate appropriate rhythmic information to modulate the output of motor neurons.
  • Command information from the motor areas of the midbrain and sensory afferent information from proprioceptors and exteroceptors can change the rhythm pattern to adapt to different movement scenarios.
  • some researchers designed simple phase oscillators to provide rhythm information to obtain rhythmic motor behavior instructions.
  • Model-based methods originated earlier, have rich theoretical basis, and have achieved good control effects in specific scenarios.
  • Fukuoka et al. designed a basic phase oscillator to generate plantar trajectories.
  • Blosch et al. accurately modeled the robot's drive system, allowing the algorithm to achieve good motion performance on the hardware platform.
  • Carlo et al. proposed a new reimplementation (Model Predicted Control, MPC) method using a simplified dynamic model.
  • Beled and others used a state machine to generate a reference trajectory for the robot's soles, and used MPC to plan the ground reaction force, further improving the robot's motion performance.
  • these methods require a large amount of accurate prior knowledge about the robot structure and dynamics.
  • Hanoha et al. applied an end-to-end RL framework to train the robot to learn to walk. proposed a hierarchical controller architecture in which the high-level controller is trained by RL while the low-level controller provides predefined fixed motion gaits. This architecture uses traditional control methods to accelerate the RL learning process, but it also limits the robot's movement capabilities. Huang Bo and others achieved stable walking and fall recovery of the robot by carefully designing the reward function. Hickman et al. defined a series of reward functions to specify and implement bipedal gait, but this also requires rich prior knowledge as basic support.
  • this embodiment adds a biological neural oscillator, namely a Rhythm Generator (RG), to the existing reinforcement learning framework to achieve In order to naturally stimulate the rhythmic movement pattern of the leg-footed robot.
  • a biological neural oscillator namely a Rhythm Generator (RG)
  • RG Rhythm Generator
  • this embodiment uses an RG network to adjust the phase transition time of the robot's foot trajectory between the swing phase and the standing phase, and a Pattern Formation (PF) network to output the robot's 12 motor control commands.
  • PF Pattern Formation
  • the RG network determines the duration of the flexor and extensor phases, while the PF network is responsible for generating information that periodically activates flexor and extensor motor neurons.
  • this embodiment instantiates the proposed bionic rhythmic motion control structure by encouraging the robot to lift its feet during the swing phase and maintain contact with the ground during the support phase. The existence of periodic rhythm signals in the legs ensures the formation of animal-like rhythmic movement behavior of the leg-footed robot.
  • the training process can focus on training the legged robot to complete main motion tasks, such as forward motion, left and right motion, and steering motion.
  • phase estimation of the legs provided by the RG network can also improve the robot platform's accurate estimation of the body velocity state when the strategy is deployed on a real robot.
  • state estimation technology of quadruped robots requires contact phase information of the robot legs in contact with the ground to fuse inertial sensor (Inertial Measurement Unit, IMU) measurement information and joint state information to complete the estimation of the whole body state, or use force sensors to achieve regression Detection of contact information.
  • IMU Inertial Measurement Unit
  • the addition of sensors will increase the overall cost and power consumption of the robot and reduce system robustness.
  • the RL strategy in this embodiment outputs the joint position increment, which is added to the target joint position command at the previous moment to obtain the motor control command at the current moment; this new action space
  • the design can speed up rhythmic motion training because the action range that can be explored by the RL strategy is limited to the vicinity of the joint position at the current moment; under the limit of the maximum motor speed, all target joint position commands that can cause the robot joints to exceed the maximum motor speed are important for training.
  • the process is not conducive to the training process; the design of the action space avoids the exploration and selection of some invalid motor commands, thus greatly speeding up the training process.
  • This embodiment provides a method for controlling rhythmic motion of a robot based on a neural oscillator, aiming to naturally stimulate the rhythmic motion behavior of a quadruped robot, which is inspired by the biological motion mechanism and accelerates the RL learning process.
  • the proposed learning framework can be divided into two parts: the bionic control structure composed of RG and PF networks, and the design of a new action space—joint position increment, as shown in Figure 2 Illustration of the rhythmic movement mechanism of vertebrate animals.
  • the movement problem of the quadruped robot is regarded as a partially observable Markov decision process (POMDP) ⁇ S, A, R, P, ⁇ >; where S and A represent the state and action space respectively. ; represents the reward function; P(s t+1 ⁇ s t ,a t ) represents the transition probability; ⁇ (0,1) represents the reward discount coefficient.
  • the quadruped robot takes an action a in the current state s, obtains a scalar reward r, and then moves to the next state s t+1 , determined by the state transition probability distribution P(s t+1 ⁇ s t ,a t ).
  • the overall goal of quadruped robot training is to find an optimal strategy To maximize future discount rewards, ⁇ * is:
  • the input status Contains 3D control commands (includes forward speed Lateral speed and steering angle rate ), three-dimensional linear velocity v of the base, three-dimensional angular rate ⁇ of the base, three-dimensional rotation direction ⁇ g (expressed as the direction of the gravity vector in the IMU coordinate system), 12-dimensional joint position q, 12-dimensional joint velocity 12-dimensional joint position error q e (joint position q minus target joint position ), the 8-dimensional RG phase ⁇ and the 4-dimensional RG frequency f generated by the RG network (where the RG phase ⁇ is represented by sine and cosine functions);
  • Output action Contains 4-dimensional RG frequency f and 12-dimensional joint position increment ⁇ q. Then calculate the target joint position command according to formula (2) Finally, a set of PD controllers are used to calculate the joint torque, i.e. Among them, K p and K d are set to fixed values in simulation and deployment, and the target joint speed then set to 0.
  • the RL strategy in this embodiment outputs joint position increment ⁇ q t , and the target joint position q t at the current moment is defined as:
  • the maximum joint position increment ⁇ q max is given by the maximum motor speed and time step T, defined as
  • the RG network outputs frequency f to modulate the phase ⁇ of each leg, which is defined as:
  • ⁇ [0,2 ⁇ ) means that the front leg is in the support phase when ⁇ [0, ⁇ ), ⁇ [ ⁇ ,2 ⁇ ) means that the current leg is in the swing phase; ⁇ t -1 represents the previous moment.
  • RG phase is used for plantar contact estimation, which is crucial for the state estimator to obtain accurate base linear velocity.
  • the design of the PF network is similar to the function in previous work, i.e., taking the robot state as input state to output motor commands.
  • the action space is defined as joint position increments, and the movement behavior it produces is largely regulated by the RG network.
  • the reward function encourages the robot to follow upper-level control commands and maintain balance and rhythmic motion.
  • the target linear velocity of the base as The target angular velocity of the base is
  • the rotation direction of the base (representing the roll-pitch-yaw angle of the base) is ⁇
  • the joint position of the standing posture is q ref
  • the speed of the sole of the foot is v f
  • the distance between the sole of the foot and the ground is h f
  • the binary foot contact index is I f
  • the real binary foot contact index provided by the physical simulator is
  • the original output of the RL strategy is o
  • the swing phase leg is defined as ⁇ swing
  • the support phase leg is defined as ⁇ stance
  • the l 1 norm is defined as
  • the l 2 norm is defined as
  • this article expresses the command coefficient shared between reward items as and
  • the reward r t at each time step is defined as the sum of the following reward terms:
  • the plantar contact phase can be directly modulated by the RL strategy to promote the robot to form rhythmic movement behavior.
  • Items 9 to 13 encourage the robot to achieve smooth and efficient motion behavior.
  • Item 14 estimates that the robot is more in the support phase to reduce energy consumption.
  • Item 15 is used to reduce the difference between the plantar contact estimate provided by the RG network and the real plantar contact provided by the physical simulator, which plays an important role in the accurate estimation of the robot state by the state estimator during the deployment phase.
  • course learning is introduced to allow the robot to prioritize learning the main tasks (follow commands and maintain balance) and prevent the robot from falling into local optimal strategies such as standing still due to excessive constraint reward coefficients.
  • domain randomization is used to overcome the gap between simulation and real deployment, and the robot is prompted to obtain a more robust control strategy by changing the physical parameters of the robot and adding sensor noise.
  • the upper and lower limits of the randomized physical parameters and the range of sensor noise are shown in Table 1. All parameters and noise are average samples.
  • the reinforcement learning PPO hyperparameter settings are shown in Table 2.
  • This embodiment provides a robot rhythmic motion control system based on a neural oscillator, including:
  • a data acquisition module configured to: acquire the current state of the robot, as well as the phase and frequency generated by the neural oscillator;
  • the control module is configured to: obtain control instructions based on the obtained current status, phase and frequency, and the preset reinforcement learning network to control the robot;
  • the preset reinforcement learning network includes an action space, a pattern forming network and a neural oscillator; the action space is used to obtain joint position increments based on the acquired current state; the pattern forming network is used to obtain joint position increments based on the acquired current state; The position increment is used to obtain the control instruction of the target joint position; the neural oscillator is used to adjust the phase change time of the robot's sole trajectory between the swing phase and the standing phase according to the acquired phase and frequency; according to the target joint position The control instructions and the timing of the phase transition of the robot's sole trajectory between the swing phase and the stance phase control the robot.
  • the working method of the system is the same as the neural oscillator-based robot rhythmic motion control method in Embodiment 1, and will not be described again here.
  • This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the program, it implements the neural-based method described in Embodiment 1. Steps in a method for controlling robotic rhythmic motion of oscillators.
  • This embodiment provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the steps in the neural oscillator-based robot rhythmic motion control method described in Embodiment 1 are implemented.

Landscapes

  • Engineering & Computer Science (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Manipulator (AREA)

Abstract

一种基于神经振荡器的机器人节律运动控制方法及系统，包括：获取机器人的当前状态，以及由神经振荡器产生的相位和频率；依据获取的当前状态、相位和频率，以及预设的强化学习网络，得到控制指令，对机器人进行控制；其中，预设的强化学习网络中包括动作空间、模式形成网络和神经振荡器；由神经振荡器和模式形成网络组成的控制结构，能确保期望的节律运动行为的形成；同时，设计的关节位置增量的动作空间能有效加速节律运动强化学习训练进程，解决了现有无模型强化学习学习时，奖励函数设计耗时、困难的问题。

Description

基于神经振荡器的机器人节律运动控制方法及系统
本发明要求于2022年3月31日提交中国专利局、申请号为202210334049.5、发明名称为“基于神经振荡器的机器人节律运动控制方法及系统”的中国专利申请的优先权，其全部内容通过引用结合在本发明中。
技术领域
本发明属于机器人控制技术领域，尤其涉及一种基于神经振荡器的机器人节律运动控制方法及系统。
背景技术
为实现四足机器人的运动控制，一些传统控制方法，如模型预测控制(Model Predictive Control，MPC)和全身控制(Whole BodyCryotherapy，WBC)通过引入感官反馈和复杂的控制理论来获得更好的运动性能。虽然这些方法在步态控制上取得了一定的成效，但它们的实现通常需要丰富的专业知识和漫长的设计过程。
发明人发现，近年来兴起的无模型强化学习(Reinforcement Learning，RL)成功实现了腿足式机器人运动策略的自主学习。然而，奖励函数通常无法直接表述期望的节律运动行为；再者，即使是合理的奖励也必须经过精心的设计和调节才能满足需要，因为对奖励函数的极小调节，也可能导致强化学习行为的巨大差异；因此，能实现机器人实现无偏差的学习的奖励函数的设计通常很耗时且困难。
发明内容
本发明为了解决上述问题，提出了一种基于神经振荡器的机器人节律运动 控制方法及系统，本发明设计的由神经振荡器和模式形成网络组成的控制结构，能确保期望的节律运动行为的形成；同时，设计的关节位置增量的动作空间能有效加速节律运动强化学习训练进程。
第一方面，本发明提供了一种基于神经振荡器的机器人节律运动控制方法，包括：
获取机器人的当前状态，以及由神经振荡器产生的相位和频率；
依据获取的当前状态、相位和频率，以及预设的强化学习网络，得到控制指令，对机器人进行控制；
其中，预设的强化学习网络中包括动作空间、模式形成网络和神经振荡器；所述动作空间，用于依据获取的当前状态，得到关节位置增量；所述模式形成网络，用于根据关节位置增量，得到目标关节位置的控制指令；所述神经振荡器，用于根据获取的相位和频率，调整机器人足底轨迹在摆动阶段和站立阶段之间相变的时间；依据目标关节位置的控制指令和机器人足底轨迹在摆动阶段和站立阶段之间相变的时间对机器人进行控制。
进一步的，相位由正弦和余弦函数表示。
进一步的，根据关节位置增量，得到目标关节位置的控制指令时：所述关节位置增量与前一时刻的目标关节位置相加，获得当前时刻的目标关节位置；依据当前时刻的目标关节位置，计算关节扭矩。
进一步的，最大关节位置增量由最大电机速度和时间步长决定。
进一步的，神经振荡器输出频率来调制每条腿的相位φ，相位φ∈[0,π)时当前腿处于支撑相阶段，相位φ∈[π,2π)时当前腿处于摇摆相阶段。
进一步的，当前时刻的相位为：
φ t＝(φ t-1+2π*f*T)％2π
其中，φ t表示当前时刻的相位；φ t-1表示前一时刻的频率；f表示频率；T表示时间步长。
进一步的，将机器人的运动问题视为马尔可夫决策过程，在奖励项中添加频率项和相位项。
第二方面，本发明还提供了一种基于神经振荡器的机器人节律运动控制系统，包括：
数据采集模块，被配置为：获取机器人的当前状态，以及由神经振荡器产生的相位和频率；
控制模块，被配置为：依据获取的当前状态、相位和频率，以及预设的强化学习网络，得到控制指令，对机器人进行控制；
其中，预设的强化学习网络中包括动作空间、模式形成网络和神经振荡器；所述动作空间，用于依据获取的当前状态，得到关节位置增量；所述模式形成网络，用于根据关节位置增量，得到目标关节位置的控制指令；所述神经振荡器，用于根据获取的相位和频率，调整机器人足底轨迹在摆动阶段和站立阶段之间相变的时间；依据目标关节位置的控制指令和机器人足底轨迹在摆动阶段和站立阶段之间相变的时间对机器人进行控制。
第三方面，本发明还提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现了第一方面中所述的基于神经振荡器的机器人节律运动控制方法的步骤。
第四方面，本发明还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现了第一方面中所述的基于神经振荡器的机器 人节律运动控制方法的步骤。
与现有技术相比，本发明的有益效果为：
本发明设计的由神经振荡器和模式形成网络组成的控制结构，能确保期望的节律运动行为的形成；同时，设计的关节位置增量的动作空间能有效加速节律运动强化学习训练进程，解决了现有无模型强化学习学习时，奖励函数设计耗时、困难的问题。
附图说明
构成本实施例的一部分的说明书附图用来提供对本实施例的进一步理解，本实施例的示意性实施例及其说明用于解释本实施例，并不构成对本实施例的不当限定。
图1为本发明实施例1的RL学习框架；
图2为本发明实施例1的脊柱动物节律运动机制图解。
具体实施方式：
下面结合附图与实施例对本发明作进一步说明。
应该指出，以下详细说明都是例示性的，旨在对本申请提供进一步的说明。除非另有指明，本文使用的所有技术和科学术语具有与本申请所属技术领域的普通技术人员通常理解的相同含义。
节律运动广泛存在于人类和动物的运动行为中，例如行走，奔跑和转向等。灵活地改变运动模式对动物在恶劣环境中的顺利通行至关重要。因此，研究生物驱动不同节律运动的机制是生物学和机器人学的重要课题。生理学研究发现，中央模式发生器，即生物在脊髓中的神经回路，在节律运动的产生中起关键作用，它能产生合适的节律信息来调制运动神经元的输出。来自中脑运动区的命 令信息和从本体感受器和外感受器的感觉传入信息可以改变节律模式以适应不同的运动场景。受此启发，一些研究人员通过设计简单的相位振荡器来提供节律信息以获得有节律的运动行为指令。
目前，四足机器人运动控制方法主要有基于模型和基于学习的控制方法两类。基于模型的方法起源较早，理论依据丰富，在特定场景下获得了良好的控制效果。福冈等人设计了基本的相位振荡器来生成足底轨迹。布洛施等人对机器人的驱动系统进行了精确建模，使算法在硬件平台获得了良好的运动性能。为获得更加鲁棒的控制性能，卡罗等人提出了一种新的用简化的动力学模型重新实现了(Model Predicted Control，MPC)方法。在卡罗方法的基础上，贝烈德等人使用状态机来生成机器人足底参考轨迹，并通过MPC来规划地面反作用力，进一步提升了机器人运动性能。然而，这些方法需要大量关于机器人结构和动力学的准确的先验知识。
近年来，数据驱动的学习方法已成为一种用于机器人自主学习运动行为的有效替代方案。哈诺哈等人应用端到端的RL框架来训练机器人学会了行走。达等人提出了一种分层控制器架构，其中，高级控制器由RL进行训练，而低级控制器则提供预先定义的固定运动步态。这种架构利用传统的控制方法来加速了RL学习进程，但同时也限制了机器人的运动能力。黄博等人通过精心设计奖励函数，实现了机器人的稳定行走和摔倒恢复。西克曼等人定义了一系列奖励函数来指定并实现了双足步态，但这也需要丰富的先验知识作为基础支撑。
实施例1：
为了解决四足机器人节律运动控制的问题，受脊椎动物节律运动调节机制启发，本实施例在现有的强化学习框架下，添加生物神经振荡器，即节律发生 器(Rhythm Generator，RG)，实现了自然地激发出腿足机器人的节律运动模式。具体的，本实施例使用一个RG网络来调整机器人足底轨迹在摆动阶段和站立阶段之间相变的时间，和一个模式形成(Pattern Formation，PF)网络输出机器人的12个电机控制命令。在哺乳动物神经系统内也存在类似的控制结构。其中，RG网络确定屈肌和伸肌阶段的持续时间，而PF网络则负责产生周期性激活屈肌和伸肌运动神经元的信息。从工程实现的角度，本实施例通过鼓励机器人在摇摆相时脚抬在支撑相时足底与地面保持接触实例化了所提出的仿生节律运动控制结构。腿部周期性节律信号的存在确保了腿足式机器人类似动物般的节律运动行为的形成。在本实施例提出的控制结构下，训练过程可以专注于训练腿足式机器人完成主要运动任务，如向前运动、左右运动和转向运动等。
此外，值得注意的是，RG网络提供的腿的相位估计还可以提高策略部署在真实机器人上时机器人平台对本体速度状态的准确估计。目前，四足机器人的状态估计技术需要机器人腿与地面接触的接触相位信息来融合惯性传感器(Inertial Measurement Unit，IMU)测量信息和关节状态信息以完成全身状态的估计，或使用力传感器来实现退步接触信息的检测。然而，传感器的加持会增加机器人的总体成本和功耗，并降低系统鲁棒性。
与先前的研究工作直接输出机器人关节位置不同，本实施例中的RL策略输出关节位置增量，与前一时刻的目标关节位置命令相加以获得当前时刻的电机控制命令；这个全新的动作空间的设计可以加快节律运动训练速度，因为RL策略可探索的动作范围被限定在了当前时刻关节位置附近；在最大电机速度的限制下，一切能导致机器人关节超过最大电机速度的目标关节位置命令对于训练过程不利于训练过程；该动作空间的设计避免了一些无效的电机命令的探索和 选择，因而极大了加速了训练过程。
本实施例提供了一种基于神经振荡器的机器人节律运动控制方法，旨在自然地激发四足机器人的节律运动行为，其灵感来自生物的运动机制，并加速RL学习进程。如图1所示，本实施例中，所提出的学习框架可分为两部分：由RG和PF网络组成的仿生控制结构，以及一个新的动作空间的设计—关节位置增量，如图2为脊柱动物节律运动机制图解。
本实施例中，将四足机器人的运动问题视为一个部分可观察的马尔可夫决策过程(POMDP)<S,A,R,P,γ>；其中，S和A分别表示状态和行动空间；
Figure PCTCN2022125984-appb-000001
表示奖励函数；P(s t+1∣s t,a t)表示过渡概率；γ∈(0,1)表示奖励折扣系数。四足机器人在当前状态s下采取一个行动a，获得一个标量奖励r，然后转移到下一个状态s t+1，由状态转移概率分布决定P(s t+1∣s t,a t)。四足机器人训练的总体目标是找到一个最优策略
Figure PCTCN2022125984-appb-000002
使得未来的折扣奖励最大，Φ *为：
Figure PCTCN2022125984-appb-000003
如图1所示，本实施例中，输入状态
Figure PCTCN2022125984-appb-000004
包含3维的控制命令
Figure PCTCN2022125984-appb-000005
(包括前向速度
Figure PCTCN2022125984-appb-000006
横向速度
Figure PCTCN2022125984-appb-000007
和转向角速率
Figure PCTCN2022125984-appb-000008
)、基座的三维线性速度v、基座的三维角速率ω、三维旋转方向θ g(表示为IMU坐标系下重力矢量的方向)、12维关节位置q、12维关节速度
Figure PCTCN2022125984-appb-000009
12维关节位置误差q e(关节位置q减去关目标关节位置
Figure PCTCN2022125984-appb-000010
)、由RG网络产生的8维RG相位φ和4维RG频率f(其中，RG相位φ由正弦和余弦函数表示)；
输出动作
Figure PCTCN2022125984-appb-000011
包含4维RG频率f和12维关节位置增量Δq。然后根据公式(2)计算目标关节位置命令
Figure PCTCN2022125984-appb-000012
最后，利用一组PD控制器计算关节扭矩， 即
Figure PCTCN2022125984-appb-000013
其中，K p和K d在仿真和部署中设定为固定值，目标关节速度
Figure PCTCN2022125984-appb-000014
则设置为0。
本实施例中的RL策略输出关节位置增量Δq t，当前时刻目标关节位置q t定义为：
Figure PCTCN2022125984-appb-000015
其中，
Figure PCTCN2022125984-appb-000016
为前一时刻目标关节位置。
由于机器人运行性能限制，给定的目标关节位置指令必然无法超越电机的运行性能。因此，在实践中，最大关节位置增量Δq max由最大电机速度
Figure PCTCN2022125984-appb-000017
和时间步长T决定，定义为
Figure PCTCN2022125984-appb-000018
本实施例中，RG网络输出频率f来调制每条腿的相位φ，定义为：
φ t＝(φ t-1+2π*f*T)％2π        (4)
其中，φ∈[0,2π)表示φ∈[0,π)时当前腿处于支撑相阶段，φ∈[π,2π)则表示当前腿处于摇摆相阶段；φ t-1表示前一时刻的频率；f表示频率；T表示时间步长；％表示求余运算。
处于摇摆相阶段时，鼓励机器人将相应脚抬高，而处于支撑相阶段时则奖励机器人将相应脚保持与地面接触。由于RG频率是f是非负的，因此四足机器人的步进周期必须在摇摆相和支撑相之间循环交替，这为节律运动行为的出现提高了信息保障。在实际部署中，使用RG相位来进行足底接触估计，这对于状态估计器获得准确的基座线速度至关重要。
PF网络的设计类似于先前工作中的功能，即将机器人状态作为输入状态以输出电机命令。然而，在实施例中，动作空间定义为关节位置增量，且它产生 的运动行为在很大程度上受到RG网络的调节作用。
本实施例中，奖励函数鼓励机器人遵循上层控制命令并保持平衡和维持节律性运动。将基座的目标线速度表示为
Figure PCTCN2022125984-appb-000019
基座目标角速度为
Figure PCTCN2022125984-appb-000020
基座旋转方向(表示基座的横滚-俯仰-偏航角)为θ，站立姿态的关节位置为q ref，足底速度为v f，足底与地面的距离为h f，RG网络提供的二进制足底接触指标为I f，物理模拟器提供的真实的二进制足底接触指标为
Figure PCTCN2022125984-appb-000021
RL策略的原始输出为o，摇摆相的腿定义为· swing，支撑相腿定义为· stance，l 1范数定义为|·|，l 2范数定义为||·||。为简单起见，本文将奖励项之间共享的命令系数表示为
Figure PCTCN2022125984-appb-000022
和
Figure PCTCN2022125984-appb-000023
每个时间步的奖励r t定义为以下奖励项的和：
1、前向速度：
Figure PCTCN2022125984-appb-000024
2、横向速度：
Figure PCTCN2022125984-appb-000025
3、角速率：
Figure PCTCN2022125984-appb-000026
4、平衡：
Figure PCTCN2022125984-appb-000027
5、身体扭转：
Figure PCTCN2022125984-appb-000028
6、足底侧滑：
Figure PCTCN2022125984-appb-000029
7、足底支撑：
Figure PCTCN2022125984-appb-000030
8、足底清空：
Figure PCTCN2022125984-appb-000031
9、足底z轴速度：
Figure PCTCN2022125984-appb-000032
10、关节限制：
Figure PCTCN2022125984-appb-000033
11、关节力矩：
-0.0012*c x*|τ t|           (15)
12、关节速度：
Figure PCTCN2022125984-appb-000034
13、策略输出平滑：
-0.016*c x*||o t-o t-1||        (17)
14、RG频率：
Figure PCTCN2022125984-appb-000035
15、RG相位：
Figure PCTCN2022125984-appb-000036
除第14项和15项为本实施例新提出的在RG和PF网络结构下有效的奖励函数外，其余所有奖励函数均已在先前的工作中得到印证。第1项到第5项使机器人能够遵循命令指令并保持基座平衡。第6项到8项是刺激机器人形成节律性运动模式的关键，根据腿的不同腿阶段相应地奖励机器人周期性抬脚或与地面保持接触，实现节律性运动的形成。值得注意的是，在前人的研究工作中，足底接触相位由物理仿真器的足底接触检测函数提供，而在实施例中，足底接触相位由RG网络的输出计算得到。换言之，在本实施例提出的RG和PF网络结构下，足底接触相位可以直接受到RL策略的调制以促使机器人形成节律性运动行为。第9项到第13项鼓励机器人获得平滑且高效率的运动行为。第14项估计机器人更多地处于支撑相阶段以缩减能耗。第15项用于缩小RG网络提供的足底接触估计与物理仿真器提供的真实足底接触之间的差异，这在部署阶段中状态估计器对机器人状态的准确估计发挥重要作用。
本实施例中，引入课程学习让机器人优先学习主要任务(遵循命令并保持平衡)并防止机器人陷入因约束项奖励系数过大而静止不动等局部最优策略。训练过程从附加到奖励函数第项4和第15项(即公式(8)和公式(19))中的乘法课程k c＝0.3(k c∈[0.3,1])开始，然后逐渐增大k c以使运动行为逐渐满足其他约束条件；其中，
Figure PCTCN2022125984-appb-000037
定义为
Figure PCTCN2022125984-appb-000038
k d∈[0,1]表示
Figure PCTCN2022125984-appb-000039
达到最大值1的速度。
本实施例中用域随机化来克服仿真到现实部署的差距，通过改变机器人物理参数和添加传感器噪声来促使机器人获得更加鲁棒的控制策略。随机化物理 参数的上下限和传感器噪声的范围如表1所示。所有参数和噪声均为平均采样。强化学习PPO超参数设置如表2所示。
表1 随机化物理参数上下限和传感器噪声
Figure PCTCN2022125984-appb-000040
表2 PPO超参数设置
Figure PCTCN2022125984-appb-000041
Figure PCTCN2022125984-appb-000042
实施例2：
本实施例提供了一种基于神经振荡器的机器人节律运动控制系统，包括：
数据采集模块，被配置为：获取机器人的当前状态，以及由神经振荡器产生的相位和频率；
控制模块，被配置为：依据获取的当前状态、相位和频率，以及预设的强化学习网络，得到控制指令，对机器人进行控制；
其中，预设的强化学习网络中包括动作空间、模式形成网络和神经振荡器；所述动作空间，用于依据获取的当前状态，得到关节位置增量；所述模式形成网络，用于根据关节位置增量，得到目标关节位置的控制指令；所述神经振荡器，用于根据获取的相位和频率，调整机器人足底轨迹在摆动阶段和站立阶段之间相变的时间；依据目标关节位置的控制指令和机器人足底轨迹在摆动阶段和站立阶段之间相变的时间对机器人进行控制。
所述系统的工作方法与实施例1的基于神经振荡器的机器人节律运动控制方法相同，这里不再赘述。
实施例3：
本实施例提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现了实施例1所述的基于神经振荡器的机器人节律运动控制方法中的步骤。
实施例4：
本实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程 序被处理器执行时实现了实施例1所述的基于神经振荡器的机器人节律运动控制方法中的步骤。
以上所述仅为本实施例的优选实施例而已，并不用于限制本实施例，对于本领域的技术人员来说，本实施例可以有各种更改和变化。凡在本实施例的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本实施例的保护范围之内。

Claims (10)

  1. 基于神经振荡器的机器人节律运动控制方法，其特征在于，包括：
    获取机器人的当前状态，以及由神经振荡器产生的相位和频率；
    依据获取的当前状态、相位和频率，以及预设的强化学习网络，得到控制指令，对机器人进行控制；
    其中，预设的强化学习网络中包括动作空间、模式形成网络和神经振荡器；所述动作空间，用于依据获取的当前状态，得到关节位置增量；所述模式形成网络，用于根据关节位置增量，得到目标关节位置的控制指令；所述神经振荡器，用于根据获取的相位和频率，调整机器人足底轨迹在摆动阶段和站立阶段之间相变的时间；依据目标关节位置的控制指令和机器人足底轨迹在摆动阶段和站立阶段之间相变的时间对机器人进行控制。
  2. 如权利要求1所述的基于神经振荡器的机器人节律运动控制方法，其特征在于，相位由正弦和余弦函数表示。
  3. 如权利要求1所述的基于神经振荡器的机器人节律运动控制方法，其特征在于，根据关节位置增量，得到目标关节位置的控制指令时：所述关节位置增量与前一时刻的目标关节位置相加，获得当前时刻的目标关节位置；依据当前时刻的目标关节位置，计算关节扭矩。
  4. 如权利要求3所述的基于神经振荡器的机器人节律运动控制方法，其特征在于，最大关节位置增量由最大电机速度和时间步长决定。
  5. 如权利要求1所述的基于神经振荡器的机器人节律运动控制方法，其特征在于，神经振荡器输出频率来调制每条腿的相位φ，相位φ∈[0,π)时当前腿处于支撑相阶段，相位φ∈[π,2π)时当前腿处于摇摆相阶段。
  6. 如权利要求5所述的基于神经振荡器的机器人节律运动控制方法，其特 征在于，当前时刻的相位为：
    φ t＝(φ t-1+2π*f*T)％2π
    其中，φ t表示当前时刻的相位；φ t-1表示前一时刻的频率；f表示频率；T表示时间步长。
  7. 如权利要求1所述的基于神经振荡器的机器人节律运动控制方法，其特征在于，将机器人的运动问题视为马尔可夫决策过程，在奖励项中添加频率项和相位项。
  8. 基于神经振荡器的机器人节律运动控制系统，其特征在于，包括：
    数据采集模块，被配置为：获取机器人的当前状态，以及由神经振荡器产生的相位和频率；
    控制模块，被配置为：依据获取的当前状态、相位和频率，以及预设的强化学习网络，得到控制指令，对机器人进行控制；
    其中，预设的强化学习网络中包括动作空间、模式形成网络和神经振荡器；所述动作空间，用于依据获取的当前状态，得到关节位置增量；所述模式形成网络，用于根据关节位置增量，得到目标关节位置的控制指令；所述神经振荡器，用于根据获取的相位和频率，调整机器人足底轨迹在摆动阶段和站立阶段之间相变的时间；依据目标关节位置的控制指令和机器人足底轨迹在摆动阶段和站立阶段之间相变的时间对机器人进行控制。
  9. 一种电子设备，其特征在于，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现了如权利要求1-7任一项所述的基于神经振荡器的机器人节律运动控制方法中的步骤。
  10. 一种计算机可读存储介质，其特征在于，其上存储有计算机程序，该 程序被处理器执行时实现了如权利要求1-7任一项所述的基于神经振荡器的机器人节律运动控制方法中的步骤。
PCT/CN2022/125984 2022-03-31 2022-10-18 基于神经振荡器的机器人节律运动控制方法及系统 WO2023184933A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210334049.5A CN114740875A (zh) 2022-03-31 2022-03-31 基于神经振荡器的机器人节律运动控制方法及系统
CN202210334049.5 2022-03-31

Publications (1)

Publication Number Publication Date
WO2023184933A1 true WO2023184933A1 (zh) 2023-10-05

Family

ID=82279146

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/125984 WO2023184933A1 (zh) 2022-03-31 2022-10-18 基于神经振荡器的机器人节律运动控制方法及系统

Country Status (2)

Country Link
CN (1) CN114740875A (zh)
WO (1) WO2023184933A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114740875A (zh) * 2022-03-31 2022-07-12 山东大学 基于神经振荡器的机器人节律运动控制方法及系统
CN114859737B (zh) * 2022-07-08 2022-09-27 中国科学院自动化研究所 四足机器人步态过渡方法、装置、设备和介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006289602A (ja) * 2005-03-17 2006-10-26 Sony Corp ロボット装置及びその制御方法
US20130138020A1 (en) * 2011-11-30 2013-05-30 Honda Motor Co., Ltd Walking assist device
CN108772836A (zh) * 2018-05-08 2018-11-09 东南大学 基于cpg的脊柱型多足机器人及其仿生运动控制方法
CN109866904A (zh) * 2019-04-09 2019-06-11 哈尔滨工程大学 一种仿生水母类水下机器人的运动及速度控制方法
CN110286592A (zh) * 2019-06-28 2019-09-27 山东建筑大学 一种基于bp神经网络的机器鱼多模态运动方法及系统
CN113616193A (zh) * 2021-07-07 2021-11-09 燕山大学 一种基于自适应cpg参数辨识的节律协同特征刻画方法
CN114740875A (zh) * 2022-03-31 2022-07-12 山东大学 基于神经振荡器的机器人节律运动控制方法及系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4587738B2 (ja) * 2003-08-25 2010-11-24 ソニー株式会社 ロボット装置及びロボットの姿勢制御方法
EP3768472A1 (en) * 2018-04-22 2021-01-27 Google LLC Systems and methods for learning agile locomotion for multiped robots
CN111208822A (zh) * 2020-02-17 2020-05-29 清华大学深圳国际研究生院 一种基于强化学习和cpg控制器的四足机器人步态控制方法
CN113093779B (zh) * 2021-03-25 2022-06-07 山东大学 基于深度强化学习的机器人运动控制方法及系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006289602A (ja) * 2005-03-17 2006-10-26 Sony Corp ロボット装置及びその制御方法
US20130138020A1 (en) * 2011-11-30 2013-05-30 Honda Motor Co., Ltd Walking assist device
CN108772836A (zh) * 2018-05-08 2018-11-09 东南大学 基于cpg的脊柱型多足机器人及其仿生运动控制方法
CN109866904A (zh) * 2019-04-09 2019-06-11 哈尔滨工程大学 一种仿生水母类水下机器人的运动及速度控制方法
CN110286592A (zh) * 2019-06-28 2019-09-27 山东建筑大学 一种基于bp神经网络的机器鱼多模态运动方法及系统
CN113616193A (zh) * 2021-07-07 2021-11-09 燕山大学 一种基于自适应cpg参数辨识的节律协同特征刻画方法
CN114740875A (zh) * 2022-03-31 2022-07-12 山东大学 基于神经振荡器的机器人节律运动控制方法及系统

Also Published As

Publication number Publication date
CN114740875A (zh) 2022-07-12

Similar Documents

Publication Publication Date Title
WO2023184933A1 (zh) 基于神经振荡器的机器人节律运动控制方法及系统
JP4587738B2 (ja) ロボット装置及びロボットの姿勢制御方法
Huang et al. Learning physical human–robot interaction with coupled cooperative primitives for a lower exoskeleton
Nakanishi et al. Learning from demonstration and adaptation of biped locomotion
Bhounsule et al. Low-bandwidth reflex-based control for lower power walking: 65 km on a single battery charge
Santos et al. Gait transition and modulation in a quadruped robot: A brainstem-like modulation approach
Shahbazi et al. Unified modeling and control of walking and running on the spring-loaded inverted pendulum
Abreu et al. Learning low level skills from scratch for humanoid robot soccer using deep reinforcement learning
Gams et al. Adaptation and coaching of periodic motion primitives through physical and visual interaction
JP5052013B2 (ja) ロボット装置及びその制御方法
CN108572553A (zh) 一种四足机器人的运动闭环控制方法
Matos et al. Towards goal-directed biped locomotion: Combining CPGs and motion primitives
Park et al. Evolutionary-optimized central pattern generator for stable modifiable bipedal walking
Sheng et al. Bio-inspired rhythmic locomotion for quadruped robots
Liu et al. Workspace trajectory generation method for humanoid adaptive walking with dynamic motion primitives
Ren et al. Computational models to synthesize human walking
Schumacher et al. Natural and robust walking using reinforcement learning without demonstrations in high-dimensional musculoskeletal models
Geng et al. Planar biped walking with an equilibrium point controller and state machines
Mombaur et al. Control of motion and compliance
Li et al. Central pattern generator based gait control for planar quadruped robots
Bao et al. Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey
Liu et al. A trajectory generation method for biped walking based on neural oscillators
Teixeira et al. Biped locomotion-improvement and adaptation
CN117572877B (zh) 一种双足机器人步态控制方法、装置、存储介质及设备
Kouppas et al. Balance control of a bipedal robot utilizing intuitive pattern generators with extended normalized advantage functions

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22934751

Country of ref document: EP

Kind code of ref document: A1