CN113062601B - A trajectory planning method for concrete placing robot based on Q-learning - Google Patents

Abstract

The invention relates to a novel track planning scheme of an intelligent concrete distributing robot, which is suitable for the autonomous pouring control of the concrete distributing robot, avoids the complex inverse kinematics interpolation calculation and belongs to the field of intelligent manufacturing. The invention designs a universal track planning frame, which comprises a rapid movement process that a material distribution robot is reset from an initial state to a path starting point and from a path end point to the initial state; and the material distributing robot performs a continuous concrete pouring process from the pouring path starting point to the pouring path end point. In the process of rapid movement, a simple interior point method is adopted to perform inverse solution optimization with time optimization as a target, and a cubic polynomial is adopted to fit a track. In the continuous concrete pouring process, an error band with a certain area is formed on a path to be poured at the tail end of the distributing robot, the formed path error band is divided into regions by using a Q learning algorithm, reward values are given to the divided regions according to pouring targets and constraints, Q value training is carried out on a given grid, finally, action sequences of all joints of the robot are formed, the robot action is directly obtained, and the complex track planning process based on kinematic inverse solution optimization is avoided.

Description

A Q-learning-based trajectory planning method for concrete placing robots

technical field

The invention relates to a novel trajectory planning method of an intelligent concrete placing robot, which is suitable for the autonomous pouring control of a concrete placing robot, avoids complex interpolation calculation, and belongs to the field of intelligent construction.

Background technique

The concrete placing robot is a construction industrial robot that transports concrete to the construction site, and plays a very important role in the development process of urban modernization construction. With the continuous improvement of the efficiency requirements of the cloth robot in engineering construction, the research on the intelligent control of the cloth robot has gradually developed. Intelligent control is inseparable from the path and trajectory planning of the manipulator. For industrial robots, the path planning mainly refers to the trajectory of the end of the manipulator, and the trajectory planning is expressed as the displacement, velocity and acceleration of the joint motion of each boom that is operated. curve, etc. The specific work of the distribution robot is to realize the displacement of the pumping port at the end of the boom, which can be analyzed according to the mechanism of the industrial robot. The cloth pouring surface is generally divided into a non-revolving horizontal plane, a vertical cylindrical surface, and a non-revolving space plane or curved surface. According to the different pouring surfaces, the target pouring route of the cloth is also different. In order to simplify the pouring route planning, the general route is composed of straight lines.

In recent years, intelligent real-time dynamic autonomous planning has played an extremely important role in industrial robots and other operations. Aiming at the problem of concrete distribution planning, combining the autonomous concrete distribution with the intelligent real-time dynamic path planning of the distribution robot will help to further improve the pouring efficiency and quality of the distribution task. At present, for the large-scale distributing boom system, its construction conditions are relatively bad, the traditional trajectory planning method has a large amount of calculation, and it is difficult to determine the optimal performance index. , The pouring path of the working section is often planned and judged according to the limited perspective of the on-site staff, and the quality of construction pouring is overly dependent on the operating experience and technical level of the workers, and the degree of automation is low, which cannot meet the trajectory motion requirements of the pouring end of the large-scale placing boom system. . Reinforcement learning is a branch of machine learning methods. Its basic principle is to imitate the learning process of living organisms. The agent will gain learning experience in the continuous interaction with the external environment, and gradually train the autonomous planning ability of the agent. Technically speaking, the reinforcement learning method is similar to the learning and operation process of the staff, and it is an effective way to solve the autonomous path planning needs of the working end of the cloth robot.

SUMMARY OF THE INVENTION

The invention designs a trajectory planner based on Q-learning for the autonomous planning and control precision requirements of the intelligent redundant cloth robot. The path required for trajectory planning of the placing robot is a known condition. First, the joint trajectory of the placing machine is planned from the initial position to the starting point of the path using the time-optimized interior point method; when planning the trajectory of the known straight path, according to the pouring accuracy It is required to establish an error zone with a bandwidth of 2 times the precision centered on the known pouring path, based on the Q-learning algorithm in reinforcement learning, divide the error zone into a grid, and divide the grid according to the working conditions of the placing robot and general indicators. The corresponding scalar reward value is given, and the distribution training is carried out with the goal of the maximum reward value, and the optimal observation-action-reward sequence in the error zone environment is obtained, and the joint planning trajectory of the cloth robot is formed according to the action sequence.

The intelligent concrete placing robot has redundancy, and the robot is continuously pouring straight or curved paths during the placing operation. Therefore, the trajectory planning scheme of the placing robot should be designed in Cartesian space, and continuous path casting and redundant degrees of freedom should be considered. planning issues below. According to the existing planning method, the DH coordinate system transformation method should first be used to establish a positive kinematics model for the robot. On the premise that the pouring path in the Cartesian space is known, the path needs to be discretized according to a certain number of values, that is, divided into multiple paths. , perform the kinematic inverse solution operation on the starting point and the end point of each discrete path segment, that is, perform the inverse kinematics derivation of the robot from the pouring position of the end, and obtain the change angle of each joint of the manipulator. Due to the existence of redundant degrees of freedom, the number of transformable joints of the robot is greater than the number of degrees of freedom in the motion space, and the inverse kinematics solution operation will generate multiple solutions. Generally, an objective optimization algorithm is used to select the best inverse solution, and the optimization objective is mostly set to the most time-sensitive solution. optimal or energy optimal. After completing the kinematic inverse solution operation, a series of joint angle values will be obtained, and data fitting will be performed on them. Complete the trajectory planning task.

Traditional methods have many problems for manipulators with redundancy and continuous linear motion. First, after the trajectory planning is completed, the forward kinematics operation is performed on the planned trajectory again, that is, the path in the Cartesian space is deduced from the trajectory of each joint space. The path deviation is large, especially in the super-long and large-scale mechanical arm of the placing robot, the trajectory error of the traditional method can reach about 1m. Therefore, the traditional concrete placing robot mostly uses manual operation of the pouring end to control the actual path range, which leads to the poor autonomy of the placing robot and it is difficult to achieve the goal of intelligent manufacturing. In view of this situation, the existing research focuses on the fitting of higher-order polynomials or the research on segmental fitting of path points, such as the three-five-cubic polynomial interpolation method, which requires high mathematical skills and is difficult to calculate. It is difficult to realize real-time planning of industrial robots. Due to the existence of redundancy, the inverse kinematics solutions are mostly solved by optimization algorithms, and the optimization algorithms are mostly aimed at the optimization of time or energy. In the design of the objective function, weights and constraints can also be added to couple, so as to achieve multi-objective Optimization requirements, this algorithm is complex, the amount of calculation is large, and it greatly depends on the correct given performance indicators, while the working conditions of the cloth robot are complex, and the performance indicators and constraints change with the construction environment. Therefore, the algorithm design is difficult and difficult to change. The internal structure of complex algorithms cannot find a generalized fixed algorithm framework.

SUMMARY OF THE INVENTION

The purpose of the invention is to overcome the deficiencies of the prior art and disclose a trajectory planning method for a concrete placing robot based on Q-learning.

The technical scheme adopted by the present invention to solve its technical problems is:

According to the characteristics of redundant concrete placing robots, a general trajectory planning framework is designed. The trajectory planning of the placing robot is divided into two parts: one is the rapid movement of the placing robot from the initial state to the starting point of the path, and the reset from the end point of the path to the initial state. The other part is the process of continuous concrete pouring by the placing robot from the beginning of the pouring path to the end of the pouring path. In the process of fast motion, since there is no need to consider the intermediate path, a simple interior point method is adopted to carry out the inverse solution optimization aiming at the time optimization, and a cubic polynomial is used to fit the trajectory. In the process of continuous concrete pouring, the path required to be poured at the end of the placing robot forms an error band of a certain area, and the error band is set according to the given pouring accuracy conditions. Using the Q-learning algorithm, the formed path error zone is divided into regions, and the divided regions are given reward values according to the pouring goals and constraints, and the given grids are trained to finally form the action sequence of each joint of the robot, which is directly obtained. Robot actions, avoiding the complex trajectory planning process.

The beneficial effects of the present invention are: according to the working characteristics of the cloth robot, a trajectory planner is designed, which is more versatile and generalized, and the internal parameters are mainly set scalar reward values, which are easy to change and test. The trajectory planning is carried out by designing the error zone of the pouring path, and the error size can be set independently according to the working accuracy requirements, so as to ensure that the actual path deviation is within the working accuracy requirements, and avoid the excessively large distance between the path points in the traditional interpolation planning method. error problem. Using the Q-learning algorithm for training can directly obtain the action values of each joint of the robot, avoiding the complex multi-objective optimization inverse solution operation and data fitting process. The online learning method is adopted for planning, which improves the autonomy of the concrete placing machine and easily achieves the goal of unmanned construction machinery in intelligent construction.

Description of drawings

Fig. 1 is the overall structure diagram of the intelligent redundant concrete distribution robot (previously applied for patent 2020111625562 "A three-joint rotary concrete distribution robot");

Figure 2 is a structural diagram of a trajectory planner for a concrete placing robot based on Q-learning;

Fig. 3 is the flow chart of the trajectory planning of the continuous straight line pouring process;

FIG. 4 is an example diagram of linear trajectory planning using a trajectory planner.

Detailed ways

Figure 1 shows the overall structure of the designed intelligent redundant concrete placing robot, which is mainly composed of five modules, including column assembly (1), pipe assembly (2), pipe clamp (3), cantilever support (4), Turntable assembly (5). The design adopts three rotary joints, namely three turntable assemblies (5). For the plane casting with two degrees of freedom, the design has one redundant degree of freedom.

Figure 2 is a summary of the overall structure of the trajectory planner, including the planning method for the fast-moving part and the continuous concrete pouring part.

Figure 3 describes in detail the idea layer and technical layer of the trajectory planning part of the continuous concrete pouring process. This part adopts the Q-learning method. Figure 3 summarizes and summarizes its application method and process.

In Figure 2, an overall design framework for trajectory planning of concrete placing robots based on Q-learning is shown. The design framework is divided into layers, including structural layer, idea layer, technical layer and output layer. In the structural layer, a Q-learning-based trajectory planner for a concrete placing robot is divided into two parts: one part is the rapid movement process of the placing robot from the initial state to the starting point of the path and the reset from the end point of the path to the initial state; the other part is the distributing robot The process of continuous concrete pouring by the robot from the start of the pour path to the end of the pour path.

Based on the division of the structure layer, the following ideas are adopted: in the process of fast motion, since there is no need to consider the intermediate path in general, the traditional inverse kinematics trajectory planning idea is adopted to carry out the inverse solution optimization aiming at the time optimization. The interior point method can be used to fit the trajectory using a cubic polynomial, and finally the trajectory curve of each joint angle can be produced. In the process of continuous concrete pouring, the idea of forming a certain area of error band for the path required by the end of the placing robot is used to carry out the process. Planning, the error bandwidth is set according to the given pouring accuracy conditions, most of which are 2 times the accuracy. The technology adopts the Q-learning algorithm to divide the formed path error band into regions, and the divided regions are given according to the pouring goals and constraints. The reward value is trained on a given grid, and the action sequence of each joint of the robot is finally produced, and the robot action is directly obtained, avoiding the complex trajectory planning process.

Compared with the rapid movement process, the continuous concrete pouring process includes two processes, one is the rapid movement process, and the other is the continuous concrete pouring process. The rapid movement process means that when the placing robot first starts to work, the initial position is not necessarily at the starting point of the set pouring track, so the rapid movement process is set to make the initial position of the placing robot return to the starting point of the pouring track.

In Figure 3, the trajectory planning process of the placing robot during the continuous linear concrete pouring process of the placing robot is shown.

The first step is to plan the path according to the construction environment. Generally, the pouring trajectory is set as a straight line, and then the construction accuracy requirements are determined, and a trajectory planning error zone with the path as the area centerline and 2 times the accuracy as the area width is established. Considering that in the process of path optimization, it is not necessary to find the shortest path, but to find a sub-optimal solution under the condition of weighing the efficiency and the quality of the route. The establishment of the error band here sacrifices part of the accuracy, while the error band width is certain. Under the conditions, the efficiency and route quality are balanced, and the error is controllable, so it can meet the requirements of trajectory planning.

The second step is to divide the error zone into a grid, and establish a dynamic reward value model R according to the grid area. R is in the form of a matrix, in which the reward value of each grid is stored. In this application, the placement robot is set to pour the end Towards the target end point of the path is a positive reward, otherwise it is a negative reward, and the reward for the area outside the error band is set to negative infinity to ensure that the cloth robot always moves in the error band area during the planning process.

The third step is to quantify the reward value in the R matrix so that the subsequent planner can learn.

3.1 Set the reward value to be subtracted by one unit for each action of the robot to ensure the optimal energy requirement of the robot, that is, to achieve the pouring trajectory requirement with the least action.

3.2 The scalar reward deviating from the path center value and the scalar reward of the completed action distance are added to the R matrix, that is, the trajectory error of the cloth robot in each state is stored in the R matrix, and the error value is directly regarded as the equivalent scalar negative reward, and The distance between the pouring position of the robot in each state and the starting point of the path is regarded as an equivalent scalar positive reward to ensure that the cloth robot moves to the target point within the error range.

3.3 Set the grid where the path end point is located as the maximum reward value to ensure that the planner always moves towards the target end point during the tracing planning.

The fourth step is to establish the initial R matrix according to the above requirements, and establish the cloth robot action matrix a. The actions include three types of forward motion, static and backward motion. Therefore, the action matrix size is 3×3×3, which represents the cloth robot. There are 27 states under all feasible actions. In order to prevent the dead zone of long-term static state, the (0, 0, 0) state is removed, that is, the static state of all joints, that is, the action matrix a includes 26 feasible action states of the cloth robot.

The fifth step is to establish a dynamic Q matrix according to the specified reward matrix R and action matrix a. The Q matrix is equivalent to the action-value function. Input the current state of the robot and the next action to be performed, and get all the rewards for reaching the target point. Here, the Q value of all grids is initialized to 0, and the Q value of the grid where the end of the marker is located is 100 units.

According to the above steps, adopt the Q learning update strategy formula Q*(s,a)=E[R+γmaxQ*(s',a')|s,a] The dynamic Q matrix is continuously trained and iterated, here:

The scalar reward deviating from the path center value and the scalar reward of the completed action distance are added to the R matrix, that is, the trajectory error of the cloth robot in each state is stored in the R matrix, and the error value is directly regarded as the equivalent scalar negative reward. The distance between the pouring position of the robot and the starting point of the path in each state is used as an equivalent scalar positive reward to ensure that the placing robot moves to the target point within the error range;

Action state s, including three rotation joint angles, namely three turntable assembly (5) angles;

Action matrix a, the action matrix includes 3 types of forward motion, static and backward motion. The size of the action matrix is 3×3×3, which represents 27 states under all feasible actions of the cloth robot. In order to prevent the occurrence of long-term static state death area, remove the (0,0,0) state, that is, all joints are still in the static state, that is, the action matrix a includes 26 feasible action states s for the cloth robot;

Expected E in mathematical probability;

Q*(s',a') is the maximum Q value obtained in one training, γ is the learning rate, where the value is 0.9, until the set number of iterations reaches the upper limit or when the Q matrix converges, the update is stopped, at this time the default The obtained Q matrix is the optimal matrix, according to which a series of optimal (action value, reward value) mapping relationships can be obtained, so as to obtain the three joint action value sequences of the cloth robot obtained by trajectory planning within the set error band, In this way, the trajectory planning of straight line pouring is completed, the complex kinematic inverse solution optimization operation is avoided, and the optimal path point of the target pouring trajectory and the state-action sequence of the placing robot are obtained.

As shown in FIG. 4 , through the Q-learning trajectory planner of the present application, the optimal continuous path planning points are obtained, and the part between the path points can be obtained by using a conventional curve fitting method to obtain the optimal continuous trajectory. The concrete continuous pouring trajectory obtained by the method proposed in this application is composed of a series of discrete points, and each discrete point represents a motion state of the placing robot.

In Figure 4, the straight-line path obtained by the trajectory planner of the concrete placing robot based on Q-learning, all path points are all the states reached by the placing robot, all within the set error band, which achieves the autonomous planning of intelligent construction machinery , Error controllable requirements.

Claims (1)

1. The utility model provides a concrete cloth robot path planning method based on Q study, to the characteristics of redundancy concrete cloth robot, its characterized in that designs a general path planning frame, divides cloth robot path planning into two parts: one part is a rapid movement process of the cloth robot from an initial state to a path starting point and from a path end point to an initial state; the other part is the process that the material distribution robot carries out continuous concrete pouring from the starting point of the pouring path to the end point of the pouring path;

the design adopts three rotary joints, namely three turntable assemblies (5);

the track planning process of the material distribution robot in the concrete continuous straight pouring process comprises the following steps:

firstly, planning a path according to a construction environment, and establishing a track planning error band which takes the path as a regional center line and takes 2 times of precision as the regional width;

secondly, dividing grids after the error band is determined, establishing a dynamic reward value model R according to grid areas, wherein the reward value of each grid is stored in a matrix form, setting the pouring tail end of the cloth robot to face a path target terminal point as a positive reward, otherwise setting the pouring tail end of the cloth robot as a negative reward, and setting the reward of areas outside the error band as a negative infinity;

thirdly, quantizing the reward values in the R matrix;

3.1, setting that the reward value is subtracted by one unit every time the robot acts once, and ensuring the optimal energy requirement of the robot, namely, the pouring track requirement is met by the least actions;

3.2 adding scalar rewards deviating from the central value of the path and scalar rewards of finished action distances into the R matrix, namely storing the track error of the distributing robot in each state in the R matrix, directly taking an error value as an equivalent scalar negative reward, and taking the distance between a pouring position of the robot in each state and the starting point of the path as an equivalent scalar positive reward to ensure that the distributing robot moves to a target point in an error range;

3.3 setting the grid where the path end point is located as the maximum reward value;

fourthly, establishing an initial R matrix according to the requirements, and establishing a cloth robot action matrix a, wherein the action comprises 3 types of forward movement, static movement and backward movement, so that the specification of the action matrix is 3 multiplied by 3, the action matrix represents 27 states of the cloth robot under all feasible actions, and in order to prevent a long-term static state dead zone, the (0,0,0) state is removed, namely all joint static states are removed, namely the action matrix a comprises 26 feasible action states s of the cloth robot;

fifthly, establishing a dynamic Q matrix according to a specified reward matrix R and an action matrix a, wherein the Q matrix is equivalent to an action-value function, and inputting the current state of the robot and the next action to be performed to obtain all reward values reaching a target point;

according to the steps, adopting a Q reinforcement learning updating strategy formula Q (s, a) ═ E [ R + gamma maxQ (s ', a') | s, a ] dynamic Q matrix to train and iterate continuously, wherein:

scalar rewards deviating from the central value of the path and scalar rewards of finished action distances are added into the R matrix, namely, the track error of the distributing robot in each state is stored in the R matrix, the error value is directly used as the equivalent scalar negative reward, the distance between the pouring position of the robot in each state and the starting point of the path is used as the equivalent scalar positive reward, and the distributing robot is ensured to move to the target point in the error range;

the action state s comprises three rotation joint angles, namely three angles of the turntable assembly (5);

the motion matrix a comprises 3 types of forward motion, static motion and backward motion, the specification of the motion matrix is 3 multiplied by 3, the motion matrix represents 27 states of the cloth robot under all feasible motions, in order to prevent a long-term static state dead zone, a (0,0,0) state is removed, namely all joint static states are avoided, namely the motion matrix a comprises 26 feasible motion states s of the cloth robot;

expectation in mathematical probability E;

q (s ', a') is the maximum Q value obtained in one training, gamma is a learning rate, the value is 0.9, the updating is stopped until the set iteration times reach the upper limit or the Q matrix is converged, the obtained Q matrix is an optimal matrix by default, and a series of optimal (action value and reward value) mapping relations can be obtained according to the optimal matrix, so that 3 joint action value sequences of the distributing robot obtained by path planning in a set error zone are obtained, the path planning of straight line casting is completed, the complex inverse solution optimization operation of kinematics is avoided, and the optimal path point and the state-action sequence of the distributing robot of the target casting path are obtained.

Images