CN118114746B - Variance minimization reinforcement learning mechanical arm training acceleration method based on Belman error - Google Patents

Abstract

The present invention provides a Bellman error-based variance minimization reinforcement learning robot arm training acceleration method for robot arm control, comprising the following steps: establishing an engineering problem into a reinforcement learning environment model, and obtaining and measuring the position and posture data of the robot arm during movement, such as the joint angle, angular velocity, end effector position, end effector speed, and obstacle position, by using a position sensor and a rotary encoder. The data is transformed by a neural network to form the characteristics of the robot arm state. The variance minimization algorithm based on the projected Bellman error is used for training to improve the control strategy of the robot arm. Through repeated iterative training, the optimal control strategy of the robot arm is finally obtained, and the performance of the robot arm in specific tasks and application scenarios is improved. By reducing the variance of the gradient estimate, the method can accelerate the speed of convergence to the optimal strategy, improve the accuracy and efficiency of the robot arm training, and improve the performance of the automatic control system.

Description

A method for accelerating the training of robotic arms using reinforcement learning based on variance minimization of Bellman error

Technical Field

The present invention relates to the field of computer text sentiment analysis, and in particular to a Bellman error-based variance minimization reinforcement learning robot arm training acceleration method.

Background technique

With the continuous evolution of science and technology, robotic arms have gradually become an indispensable technology in many fields. In complex industrial environments, the control and training of robotic arms face some challenges, such as complex work tasks, uncertain environmental conditions, and the need to quickly adapt to new tasks. In response to these challenges, researchers focus on exploring optimization methods for robotic arm control, especially training acceleration algorithms based on reinforcement learning. Many reinforcement learning methods may face the problem of large variance in gradient estimation during training, resulting in reduced training efficiency and longer training time.

In view of this, it is necessary to design a variance minimization reinforcement learning robotic arm training acceleration method based on Bellman error to solve the above problems.

Summary of the invention

The purpose of the present invention is to provide an efficient Bellman error-based variance minimization reinforcement learning manipulator training acceleration method.

To achieve the above-mentioned invention object, the present invention provides a Bellman error-based variance minimization reinforcement learning robot arm training acceleration method, comprising the following steps:

Step S1: Establish a reinforcement learning environment model according to the operation requirements of the robot arm and instantiate the trained neural network model;

Step S2: Use the position sensor and rotary encoder to obtain and measure the state information of the robot arm , the status information At least the joint angles of the robot arm , joint angular velocity , end effector position , end effector speed and obstacle location ;

Step S3: The status information of the robot arm and optional actions Input into the neural network model to get the corresponding feature vector , using a linear method and with the help of the ε-greedy strategy, select actions , and save the action The corresponding eigenvector ;

Step S4: Agent performs actions ,Earn rewards , enter the next state , Used to indicate the next state, using step S3, to obtain the state Next action and the eigenvector ;

Step S5, the robot arm updates the parameters of the robot arm control strategy by using the method of minimizing the variance of the projected Bellman error;

Step S6, repeating steps S2 to S5 until the robot arm reaches the target position or the iteration reaches the maximum number of times.

As a further improvement of the present invention, the step S1 is specifically as follows: according to the task requirements of the robot arm, a reinforcement learning environment model is established; a data set with marked state features is used to fully train the neural network model; all state information of the robot arm is recorded. and a collection of optional actions Input into the fully trained neural network model in sequence to obtain the corresponding feature vector .

As a further improvement of the present invention, step S2 is specifically: the rotary encoder obtains the state information s of the robot arm, and the state information s of the robot arm at least includes the angle of the robot arm relative to the vertical direction. , Angle of the robot arm in the direction of rotation , angular velocity of the upper end of the robotic arm , angular velocity of the joint of the robot arm , end effector position , end effector speed and obstacle location .

As a further improvement of the present invention, step S3 is specifically: obtaining all the current optional actions of the robot arm , the status information and all optional actions Input into the trained neural network model in sequence to obtain all optional feature vectors ,in, is a set of optional actions. is the feature vector in state s; all optional action-state functions are calculated using linear methods ,Right now ,in, is the feature weight parameter vector, Transpose the sign of a vector; use Method selection action , and save the corresponding feature vector .

As a further improvement of the present invention, step S4 is specifically as follows: after the robot arm performs action a, it obtains an immediate reward and enters the next state , and get all the current optional actions of the robot arm , all status information and optional actions Input the trained neural network model and obtain the state All optional eigenvectors ,in, Used to indicate the status Under all the optional action sets, a linear method is used to calculate all the optional actions in the state Next ,Right now ,in, is the feature weight parameter vector, Transpose the sign of a vector; use Method selection action , and obtain its corresponding eigenvector .

As a further improvement of the present invention, in step S4, the specific reward form of the robot arm obtaining the instant reward is as follows:

Goal Achievement Rewards : The closer the end effector is to the target position, the higher the reward, as follows:

Obstacle Avoidance Reward : The farther the end of the robotic arm is from the obstacle, the positive reward is given, as follows:

Smoothness Bonus : Give positive rewards for smooth changes in joint angles and end-effector positions as follows:

Energy consumption bonus : Give positive rewards for actions with low energy consumption, as follows:

Collision Penalty : If the robot collides with an obstacle, a large negative reward is given, as follows:

Action Penalty : Negative rewards are given for actions where the joint angle and end effector position change too much or the speed is too fast, as follows:

The reward r is the above reward to Sum.

As a further improvement of the present invention, the minimization target of the optimization process of the variance minimization method based on the projected Bellman error is:

Formula 1

in, Indicates error, error , represents the expected error, For reward, , They are used to represent the expected symbol and eigenvector respectively, and are defined To estimate , the formula 1 is transformed into:

Formula 2

Using the stochastic gradient descent method, Update, the update formula is as follows:

Formula 3

Formula 4

Formula 5

in, , represents the feature weight parameter vector, is the optimal action set at time t+1, , represents the state and adjustable parameters at time t+1, For action, , , , They are used to represent the error, state, adjustable parameters and Bellman error expected estimate at time t, respectively. , represents the Bellman error expectation The valuation of , and They are , and The learning rate.

The beneficial effects of the present invention are as follows: the present invention establishes the engineering problem into a reinforcement learning environment model, and obtains and measures the position and posture data of the joint angle, angular velocity, end effector position, end effector speed and obstacle position of the robot arm during the movement. The data is transformed by the neural network to form the characteristics of the robot arm state, and then the variance minimization algorithm based on the projected Bellman error is used for training to improve the control strategy of the robot arm. Through repeated iterative training, the optimal control strategy of the robot arm is obtained, and the performance of the robot arm in specific tasks and application scenarios is improved. At the same time, by reducing the variance of the gradient estimation, the speed of convergence to the optimal strategy can be accelerated, and the speed of the robot arm obtaining the optimal strategy can be improved, and it has good scalability and adaptability. The method enables the robot arm to learn the optimized control strategy more quickly and effectively, thereby improving the performance of the entire system. By optimizing the control strategy, the accuracy and efficiency of the robot arm training are improved, and a more flexible and efficient solution is provided for the robot arm in terms of autonomous decision-making and rapid response, thereby improving the performance of the automatic control system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a Bellman error-based variance minimization reinforcement learning robotic arm training acceleration method according to the present invention.

FIG2 is a schematic diagram of a simplified environment of the Bellman error-based variance minimization reinforcement learning robotic arm training acceleration method of the present invention.

Figure 3 is a comparison between the Bellman error-based variance minimization reinforcement learning robotic arm training acceleration method and the traditional classical training method.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and/or processing steps closely related to the scheme of the present invention are shown in the drawings, while other details that are not closely related to the present invention are omitted.

Referring to FIG. 1 to FIG. 3 , the present invention provides a Bellman error-based variance minimization reinforcement learning robot arm training acceleration method for robot arm control training, which specifically includes the following steps:

Step S1. Build a reinforcement learning model environment according to the actual operation requirements, considering the current operation requirements as follows: drive the free end of the robotic arm to reach the target height. The robotic arm has two elbows and a drive head. It is an under-actuated system, and the motion control of the robotic arm is relatively unstable. The reinforcement learning environment is shown in Figure 2. The robotic arm is an intelligent agent that needs to be trained. At the drive head, you can choose to apply clockwise torque, no torque, and counterclockwise torque. The state information includes the angle of the first robotic arm 1 relative to the vertical direction. and the angle of the first robot arm 1 relative to the second robot arm 2 , the angular velocity of the first robot arm 1 and the angular velocity of the second robot arm 2 , end effector position , end effector speed and obstacle location ; Instantiate the trained neural network, input all the choices into the neural network, and you can get the corresponding features. You can also use tile coding encoder to encode and get the features. Among them, the neural network can try a variety of models, such as fully connected neural network and convolutional neural network, etc. The basic model reference is given below:

Input layer: contains 13 neurons, each neuron corresponds to a state information, including the angle of robot arm 1 relative to the vertical direction , the angle of robot arm 1 relative to robot arm 2 , the rotation direction and angular velocity of the two robotic arms and , end effector position , end effector speed and obstacle location .

Hidden layer: There are three layers in total, with a dimension of (128, 256, 128), and each layer uses the ReLU activation function.

Output layer: The output layer contains 64 neurons, giving the features .

Step S2: The robot arm uses a position sensor and a rotary encoder to obtain and measure the angle of the first robot arm 1 (i.e., the upper end of the robot arm) relative to the vertical direction. and the angle of the first robot arm 1 relative to the second robot arm 2 (i.e. the angle in the direction of rotation of the robot arm), the angular velocity of the first robot arm 1 (equivalent to the angular velocity of the upper end of the robot arm), the angular velocity of the second robot arm 2 (equivalent to the angular velocity of the robot arm joint connection) , end effector position , end effector speed and obstacle location Status information The robot arm includes a first robot arm 1, a second robot arm 2 and a robot arm joint 3 connecting the two, as shown in FIG2 .

Step S3: Obtain all currently available actions of the robot arm, and input the state information and all available actions of the robot arm into the trained neural network model in sequence to obtain all available feature vectors. ,in is the current status information of the robot arm, is a set of optional actions for the robot arm. To take action in state s The characteristic vector of . Use linear methods to calculate all optional action state functions ,Right now ,in, is the feature weight parameter vector, Transpose the sign of the vector. Then use Method to choose the appropriate action , that is, the robot arm has The probability of randomly selecting optional actions is The probability of choosing The action with the largest value and save the corresponding feature vector In our experiments, we set The size is 0.01.

Step S4: The robot performs an action Get instant rewards , and enter the next state At the same time, the position sensor and rotary encoder obtain all the current optional actions of the robot arm, input all the state information and optional actions into the trained neural network model, and obtain the state All optional eigenvectors , Represents in state The set of optional actions under the state, all actions should satisfy the possibility sum to 1. Next, a linear method is used to calculate all optional actions in the state Next ,Right now , in this formula, represents the feature weight parameter vector, is the vector transpose operation. Finally, use Method selection action , and obtain its corresponding eigenvector The specific form of the instant reward for the robotic arm is as follows:

Goal Achievement Rewards : The closer the end effector is to the target position, the higher the reward, as follows:

Obstacle Avoidance Reward : The farther the end of the robotic arm is from the obstacle, the positive reward is given, as follows:

Smoothness Bonus : Give positive rewards for smooth changes in joint angles and end-effector positions as follows:

Energy consumption bonus : Give positive rewards for actions with low energy consumption, as follows:

Collision Penalty : If the robot collides with an obstacle, a large negative reward is given, as follows:

Action Penalty : Negative rewards are given for actions where the joint angle and end effector position change too much or the speed is too fast, as follows:

The final reward r is the above reward to Sum.

S5. The robot arm updates the parameters using the variance minimization algorithm based on the projected Bellman error. The minimization objective of the optimization process of this method is:

Formula 1

in Indicates error, error , Represents the feature weight parameter vector, initialized as dimension vector, It also corresponds to the dimension of the feature vector. represents the expectation of the error, i.e., the Bellman error, For reward, , are used to represent the expected symbol and eigenvector respectively. It is not easy to calculate, so we define To estimate the approximation , initialized to 0. Then formula 1 can be expressed as:

Formula 2

Using the stochastic gradient descent method, Update, the update formula is as follows:

Formula 3

Formula 4

Formula 5

in, , represents the feature weight parameter vector, error , represents the Bellman error expectation The valuation of , and They are , and The learning rate with the best effect can be selected through experiments.

S6. The robot arm will determine whether the action reaches the target or whether the training reaches the maximum number of iterations. If so, the training will end; if not, steps S2-S5 will continue to be repeated.

The invention is compared with the traditional training method, and the statistical results are shown in Figure 3. In Figure 3, the variance minimization reinforcement learning training acceleration method for projected Bellman error proposed by the invention (expressed as improved GQ(0) in Figure 3) has a significantly faster convergence speed than the traditional temporal difference control algorithm Q-learning and the classic GQ(0) algorithm, which effectively improves the training efficiency and shortens the training time.

In summary, the present invention establishes the engineering problem into a reinforcement learning environment model, and uses position sensors and rotary encoders to obtain and measure the position and posture data of the robot arm during the movement, such as the joint angle, angular velocity, end effector position, end effector speed and obstacle position. The data is transformed by a neural network to form the characteristics of the robot arm state, and then the variance minimization algorithm based on the projected Bellman error is used for training to improve the control strategy of the robot arm. Through repeated iterative training, the optimal control strategy of the robot arm is obtained, and the performance of the robot arm in specific tasks and application scenarios is improved. At the same time, by reducing the variance of the gradient estimation, the speed of convergence to the optimal strategy can be accelerated, and the speed of the robot arm obtaining the optimal strategy can be improved, and it has good scalability and adaptability. This method enables the robot arm to learn the optimized control strategy more quickly and effectively, thereby improving the performance of the entire system. Optimizing the control strategy improves the accuracy and efficiency of the robot arm training, provides a more flexible and efficient solution for the robot arm in terms of autonomous decision-making and rapid response, and improves the performance of the automation control system.

The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims (6)

1. A Bellman error-based variance minimization reinforcement learning robot arm training acceleration method, characterized in that it includes the following steps: Step S1: Establish a reinforcement learning environment model according to the operation requirements of the robot arm and instantiate the trained neural network model; Step S2: Use the position sensor and rotary encoder to obtain and measure the state information of the robot arm , the status information At least the joint angles of the robot arm , joint angular velocity , end effector position , end effector speed and obstacle location ; Step S3: The status information of the robot arm and optional actions Input into the neural network model to get the corresponding feature vector , using a linear method and with the help of the ε-greedy strategy, select actions , and save the action The corresponding eigenvector ; Step S4: Agent performs actions ,Earn rewards , enter the next state , Used to indicate the next state, using step S3, to obtain the state Next action and the eigenvector ; Step S5, the robot arm updates the parameters of the robot arm control strategy using a method based on the variance minimization method of the projected Bellman error; Step S6, repeating steps S2 to S5 until the robot reaches the target position or the iteration reaches the maximum number of times; The step S5 is specifically as follows: the minimization target of the optimization process based on the variance minimization method of the projected Bellman error is: Formula 1 in, Indicates error, error , represents the expected error, For reward, , They are used to represent the expected symbol and eigenvector respectively, and are defined To estimate the Bellman error expectation , the formula 1 is transformed into: Formula 2 Using the stochastic gradient descent method, Update, the update formula is as follows: Formula 3 Formula 4 Formula 5 in, , represents the feature weight parameter vector, is the optimal action set at time t+1, , represents the state and adjustable parameters at time t+1, For action, , , , They are used to represent the error, state, adjustable parameters and Bellman error expected estimate at time t, respectively. , represents the Bellman error expectation The valuation of , and They are , and The learning rate. 2. The Bellman error-based variance minimization reinforcement learning robot training acceleration method according to claim 1 is characterized in that: the step S1 specifically comprises: establishing a reinforcement learning environment model according to the task requirements of the robot; fully training the neural network model with a data set with marked state features; and storing all the state information of the robot arm. and a collection of optional actions Input into a fully trained neural network model to obtain the corresponding feature vector . 3. The Bellman error-based variance minimization reinforcement learning robot arm training acceleration method according to claim 1 is characterized in that: the step S2 is specifically: the rotary encoder obtains the state information s of the robot arm, and the state information s of the robot arm at least includes the angle of the robot arm relative to the vertical direction , Angle of the robot arm in the direction of rotation , angular velocity of the upper end of the robotic arm , angular velocity of the joint of the robot arm , end effector position , end effector speed and obstacle location . 4. The Bellman error-based variance minimization reinforcement learning robot arm training acceleration method according to claim 1 is characterized in that: the step S3 specifically comprises: obtaining all current optional actions of the robot arm , the status information and all optional actions Input into the trained neural network model in sequence to obtain all optional feature vectors ,in, is a set of optional actions. To take action in state s The feature vector of ; use linear methods to calculate all optional action state functions ,Right now ,in, is the feature weight parameter vector, Transpose the sign of a vector; use Method selection action , and save the corresponding feature vector . 5. The Bellman error-based variance minimization reinforcement learning robot arm training acceleration method according to claim 1 is characterized in that: the step S4 is specifically: after the robot arm performs action a, it obtains an immediate reward and enters the next state , and get all the current optional actions of the robot arm , all status information and optional actions Input the trained neural network model and obtain the state All optional eigenvectors ,in, Used to indicate the status Under all the optional action sets, a linear method is used to calculate all the optional actions in the state Next ,Right now ,in, is the feature weight parameter vector, Transpose the sign of a vector; use Method selection action , and obtain its corresponding eigenvector . 6. The Bellman error-based variance minimization reinforcement learning robot arm training acceleration method according to claim 1, characterized in that: in step S4, the specific reward form of the robot arm obtaining the instant reward is as follows: Goal Achievement Rewards : The closer the end effector is to the target position, the higher the reward, as follows: Obstacle Avoidance Reward : The farther the end of the robotic arm is from the obstacle, the positive reward is given, as follows: Smoothness Bonus : Give positive rewards for smooth changes in joint angles and end-effector positions as follows: Energy consumption bonus : Give positive rewards for actions with low energy consumption, as follows: Collision Penalty : If the robot collides with an obstacle, a large negative reward is given, as follows: Action Penalty : Negative rewards are given for actions where the joint angle and end effector position change too much or the speed is too fast, as follows: The reward r is the above reward to Sum.