CN108927806A - A kind of industrial robot learning method applied to high-volume repeatability processing - Google Patents

Abstract

The present invention provides a kind of industrial robot learning methods applied to high-volume repeatability processing, it is characterised in that: the learning method is learnt based on learning model comprising following steps: S001, sensor acquisition state information；S002, learnt according to the information of acquisition；S003, judge whether processing quality and process-cycle reach requirement, terminate to learn if reaching requirement, otherwise resurvey status information and relearn.Method of the invention goes to learn and improves control strategy according to sensing data; reach good control at high speeds; robot debugging efforts can be simplified; and may be implemented in high-volume, scale repeatability processing in apply; and solve robot and lack concussion under high speed operation caused by precise kinetic model in traditional mode of learning, improve industrial machine task efficiency.

Description

A Learning Method for Industrial Robots Applied to Mass Repetitive Processing

technical field

The invention relates to the technical field of industrial robots, in particular to an industrial robot learning method applied to large-scale repetitive processing.

Background technique

An industrial robot is a highly nonlinear system, and accurate modeling of its dynamic characteristics is difficult to achieve. Previous robots usually only consider kinematics without considering dynamic models. When only using the dynamic model, on the one hand, the maximum speed and acceleration of each point are usually set lower than the actual speed and acceleration, which is the maximum torque of the actuator when the dynamic characteristics are considered, but this It also leads to the underutilization of the performance of the actuator. On the other hand, failure to consider the dynamic characteristics not only affects the working efficiency of industrial robots, but also tends to produce strong The vibration, which not only affects the processing quality of the robot, but also affects the life of the robot. In addition, the accurate modeling of industrial robot dynamics still has the problem of difficult identification of robot parameters. If the consistency of the robot is not good, the friction coefficient of each component will be different, resulting in errors in dynamic parameters, and incorrect dynamics The parameters will make the debugging of the robot more cumbersome, and it will be difficult to achieve large-scale and large-scale applications.

Contents of the invention

Aiming at the defects or deficiencies in the prior art, the present invention provides an industrial robot learning method applied to large-scale repetitive processing, which learns and improves the control strategy according to sensor data, achieves good control at high speed, and can simplify the robot Debugging work, and can be applied in large-scale, large-scale repetitive processing, and solve the vibration of robots under high-speed work caused by the lack of accurate dynamic models in traditional learning methods, and improve the work efficiency of industrial robots.

In order to achieve the above object, the technical solution adopted by the present invention is to provide a learning method for industrial robots applied to large batches of repetitive processing. The learning method is to learn based on a learning model, which includes the following steps:

S001. The sensor collects status information;

S002. Learning according to the collected information;

S003 , judging whether the processing quality and the processing cycle meet the requirements, and if the requirements are met, the learning is ended, otherwise, the state information is collected again and the learning is performed again.

As a further improvement of the present invention, the learning model is composed of an environment unit, a robot learning unit and a processing execution unit;

Wherein, the environment unit is composed of a processing workpiece state measurement sensor and a robot state end measurement observer, and the processing workpiece state measurement sensor collects visual information of the processed workpiece, and the visual information includes at least the geometric shape and smooth surface of the workpiece degree information; the position, velocity, acceleration and joint torque information of the robot state terminal measurement observer gathering robot;

The state observation unit, the state observation unit obtains the information collected by the environmental unit through a communication line, and converts the obtained information into a data format;

The data processing unit receives and processes the information converted into a data format by the state observation unit; the data processing unit includes a reward calculation unit and a function update unit, wherein the reward calculation unit sets an instant reward through a reward function setting unit r, the reward calculation unit calculates the information of the state observation unit, and after the calculation is completed, the result parameters are sent to the function update unit, and the function update unit uses neural network training to update the acquired parameters until the final Learning parameters, storing the final learning parameters, making behavioral decisions through the neural network, and then performing reinforcement learning to a deterministic strategy to drive the robot to work.

As a further improvement of the present invention, the reinforcement learning assumes that the robot is defined as a strategy π from state information to behavior, and the cumulative reward obtained from time t is defined as: According to the cumulative return passed

Find the expected return; among them, Q ^π (st _t , a _t ) represents the expected return when taking action a _t in state st _t according to strategy π;

Combining the cumulative return and the formula for obtaining the expected return, the recursive formula for the expected return is obtained:

Continuously use the last updated strategy to make decisions according to the recursive form formula.

In the present invention, the method of reinforcement learning is adopted, and the strategies of reinforcement learning are divided into deterministic strategies and uncertain strategies. output probability, the expected return Q can be calculated by formula (4):

Among them, μ represents the definite behavior.

As a further improvement of the present invention, the reinforcement learning adopts a deterministic strategy reinforcement learning method, and its specific process includes the following steps:

S201, initialize the behavior network μ(s|θ ^μ ), the parameters are expressed as θ ^Q and the evaluation network Q(s, a|θ ^Q ), the parameters are expressed as θ ^μ , and initialize the target network Q′(s, a|θ ^{Q ′} ) and μ′(s|θ ^μ′ ), the parameters are θ ^Q′ ←θ ^Q , θ ^μ′ ←θ ^μ .

S202, initialize the buffer container R;

S203, receiving the state information s _t of the state observation unit;

S204, select the execution behavior at according to the current strategy and apply certain _noise ;

S205, observe the obtained reward r _t , and observe the next state information s _t+1 ;

S206, store the quadruple <s _t , a _t , rt, s _t+1 > in the buffer container R;

S207, randomly select a batch of quaternion samples from the buffer container for training;

S208, updating evaluation network parameters;

S209, updating behavioral network parameters;

S210, judging whether the number of learning times exceeds a preset value or whether the processing quality is good enough;

S211, transmit the parameters of the evaluation network and the behavior network to the host for storage, and end the learning.

As a further improvement of the present invention, when updating the evaluation network parameters in the step S208, the objective function y _t is first set as: y _t = _r ( _st ,at )+γQ( _st+1 ,μ(st _{t +1} )|θ ^Q ), and then use the formula min _a L(θ ^Q )＝ E[(Qst, atθQ-yt)2] to calculate the parameters to update the evaluation network, where at represents the behavior at time t, and Q represents the cumulative Reward, θ ^Q represents the parameters of the behavioral network, E represents the expected value of the sum of the squares of the errors between the actual reward and the target of multiple sets of data, L(θ ^Q ) represents the error under the parameter θ ^Q , μ(s _t+1 ) Denotes a deterministic policy in state _t+1 .

6. The industrial robot learning method applied to large-scale repetitive processing according to claim 3, characterized in that: when updating the behavioral network parameters in the step S209, the gradient method is used To update the behavioral network, and update the target network using the following formula group to update;

θ′←τθ+(1-τ)θ′

θ ^Q′ ←τθ ^Q +(1-τ)θ ^Q′

θ ^μ′ ←τθ ^μ +(1-τ)θ ^μ′

withτ<<0.05

Indicates the derivative of θ ^μ , Indicates the derivative of α, Indicates that taking θ ^μ as a variable, find the derivative of J with respect to θ ^μ .

The beneficial effects of the present invention are:

1. The method of the present invention uses reinforcement learning to learn by collecting processing information, reduces robot debugging work, and optimizes the control strategy of industrial robots, including the trajectory planning function under a given path and the motor control strategy under a given trajectory, Solve the vibration of robots under high-speed work caused by the lack of accurate dynamic models in the traditional learning method, and improve the work efficiency of industrial robots.

2. This learning method is to learn the control strategy under high-speed work, learn and improve the control strategy according to the sensor data, and achieve good control under high speed.

Description of drawings

Fig. 1 is the structural representation of learning model of the present invention;

Fig. 2 is a flow chart of the learning method of the present invention;

Fig. 3 is a flowchart of the reinforcement learning of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

The learning method of the present invention is obtained based on the learning model structure, which is also an industrial robot system; as shown in Figure 1, it is a schematic diagram of the learning model structure of the present invention; the model consists of an environment unit, a robot learning unit and a processing execution unit The environment unit includes at least a processing quality measurement unit, the robot learning unit includes a state observation unit, a data processing unit, and a decision-making unit, and the processing execution unit includes at least a robot and a positioner.

The working process of each unit of the learning model of the present invention is:

The environment unit, in this implementation, is the processing quality measurement unit, which is composed of the processing workpiece state measurement sensor and the robot state end measurement observer. The processing workpiece state measurement sensor mainly collects the visual information of the processed workpiece, including the geometric shape and Surface smoothness. The robot state end measurement observer can also be a robot state end measurement sensor, which is used to collect information such as the position, speed, acceleration, and joint torque of the robot.

A state observation unit, the state observation unit obtains the information collected by the processing quality measurement unit through the communication line, and converts the obtained information into a data format.

The data processing unit receives and processes the information converted into a data format by the state observation unit; the data processing unit includes a reward calculation unit and a function update unit, wherein the reward calculation unit sets the instant reward r through the reward function setting unit, and the reward calculation unit The information of the state observation unit is calculated. After the calculation is completed, the result parameters are sent to the function update unit. The function update unit uses the neural network training method to update the acquired parameters until the final learning parameters are obtained, and the final learning parameters are stored. Make behavioral decisions through the neural network, and then perform reinforcement learning to a deterministic strategy to drive the robot to work. Specifically, the reward calculation unit sets the instant reward r through the reward function setting unit, specifically based on the function r=α*velocity+β*position _error +γ*acceleration+u*R*u ^T +…, by adjusting α, β, The parameter of γ is to change the proportion of the measured index, where α represents the weight of speed in the measurement of reward function, β represents the weight of position error, γ represents the weight of acceleration, R represents positive definite matrix, u represents voltage, current, etc., reward After the calculation of the calculation unit is completed, it is sent to the function update unit for update. In this implementation, the function update unit preferably uses the neural network training method to update the parameters. After obtaining the final learning parameters, it stores the learning results and uses the neural network to do the update. Make behavior decisions, drive robots to work, and then continue to collect new information. In this way, according to specific processing tasks, reinforcement learning will adjust the voltage, position, speed, acceleration, etc. in different states according to the required performance indicators. This process includes trajectory planning functions and motor control strategies.

In the present embodiment, based on the model as shown in Figure 1, a method applicable to the learning of large batches of repetitive processing robots is designed. The flow chart of the method is shown in Figure 2, and the specific steps are as follows:

S001. The sensor collects status information;

S002. Learning according to the collected information;

S003 , judging whether the processing quality and the processing cycle meet the requirements, and if the requirements are met, the learning is ended, otherwise, the state information is collected again and the learning is performed again.

In this embodiment, step S001, the sensor collects state information; the robot state end measurement observer in the processing information measurement unit collects the position, speed, acceleration, current, voltage, vibration rate and torque information of the robot joint and the end of the mechanical arm The processing workpiece state measurement sensor mainly collects the visual information including the geometric shape and surface smoothness of the processing workpiece; in this process, the visual information is processed in gray scale to avoid the influence of light. Normalize the position, speed, acceleration and other information, and unify the length of the information to facilitate input into the neural network for processing.

In this embodiment, step S002, learning according to the collected information, the machine learning unit performs deep reinforcement learning according to information such as current, voltage, moment, torque, vibration and stereo vision obtained by the state observation unit. In this embodiment, according to The collected information is preferably learned in a model-free reinforcement learning manner.

In the process of reinforcement learning, it is based on the discrete equivalent time series, so that the robot interacts with the environment; specifically, it is assumed that within each equivalent time t, the state observation unit inputs the observation state to the machine learning unit, and the machine learning unit is based on the current state. The state is used as the input information, which is input into the neural network, and the output result is the definite behavior of the motor. The robot is defined as a strategy π from state information to behavior, and the cumulative reward obtained from time t is:

The strategies obtained by reinforcement learning can be divided into deterministic strategies and uncertain strategies. The uncertain strategy outputs the probability of each behavior, and the deterministic strategy directly outputs a certain behavior.

In the present invention, the purpose of using reinforcement learning is to learn a deterministic strategy π, that is, directly learn the strategy from state input to output action, that is, for the behavioral network, the input is state information, and the output is action. Maximize the expected return Q obtained from the initial state, which can be expressed by formula (2):

Q ^π (s _t , a _t )＝E _π [R _t |s _t , a _t ] (2)

Among them, Q ^π ( _st , a _t ) represents the expected reward when taking action a _t in state _st according to policy π. Combining formulas (1) and (2), the recursive form of expected return can be obtained, such as formula (3):

This means that we can continuously use the last updated policy to make decisions during the learning process.

In the present invention, the method of reinforcement learning is adopted, and the strategies of reinforcement learning are divided into deterministic strategies and uncertain strategies. output probability, the expected return Q can be calculated by formula (4):

Among them, μ represents the definite behavior. In this process, formula (4) is a recursive form of formula (2). Formula (2) is the overall thought representation of expected return in the present invention, and formula (4) is a practical implementation, it allows the calculation of the cumulative reward at this time to use the value of the last cumulative reward and the value of the instant reward at this moment. value so that it can be easily programmed on a computer.

In this embodiment, the specific method of reinforcement learning using a deterministic strategy includes the steps shown in Figure 3:

S201, initialize the behavior network μ(s|θ ^μ ), the parameters are expressed as θ ^Q and the evaluation network Q(s, a|θ ^Q ), the parameters are expressed as θ ^μ , and initialize the target network Q′(s, a|θ ^{Q ′} ) and μ′(s|θ ^μ′ ), the parameters are θ ^Q′ ←θ ^Q , θμ ^′ ←θ ^μ .

Specifically, initialize the neural network evaluation network Q(s, a|θ ^Q ) and behavior network μ(s|θ ^μ ), the structure of the neural network, and its neural parameters are expressed as: θ ^Q , θ ^μ , representing the neural network The proportion of each neuron in . Among them, θ ^Q represents the parameters of the evaluation neural network, θ ^μ represents the behavior network, the input of μ( _s |θ ^μ ) is the state information from the state observation unit, and the output is a certain behavior at . The input of the evaluation neural network Q(s, a|θ ^Q ) is the state information from the state observation unit and the output of the behavior network μ(s|θ ^μ ), and the output is the value function of taking a certain behavior in this state A value that reflects how good or bad the current strategy is. Initialize the target network. The structure of the target network is the same as that of the behavioral network and the evaluation network. The parameters of the neural network come from the slower copies of the behavioral network and the evaluation network. The update of the target network is slower than that of the behavioral network and the evaluation network. The stability of the online learning process.

S202, initialize the buffer container;

S203, receiving the state information of the state observation unit;

S204. Select an execution behavior according to the current strategy and apply certain noise;

Specifically, for each discrete time point of a processing cycle, the behavior at _t is selected according to the current strategy and a certain amount of random noise, namely: at ＝μ(s _t |θ ^μ )+N( _t ). Among them, μ (s) = argmax _a Q(s, a).

S205, observe the obtained reward, and observe the next state information;

Specifically, the current state information s _t is used as input, which is input into the behavior network, and the behavior network outputs a specific behavior value a _t , and the state information and the behavior value of the behavior network are used as input, input into the evaluation network, and the evaluation network outputs a reward r _t , the state observation unit collects the state information _st+1 at the next moment, and obtains the quadruple information ( _st , at , _rt , _{st +1} ).

S206, store (s _t , at , r _t , s _{t +1} ) in the buffer container, ( _{st t} , at , r _t , s _{t +1} ) means that after taking action at state s _t The size of the reward r _t obtained and the state at the next moment;

S207, randomly select a batch of samples from the buffer container for training;

S208, updating evaluation network parameters;

Specifically, the objective function is set as: y _t = r(s _t , a _t )+γQ(s _t+1 , μ(s _t+1 )|θ ^Q ), and through the formula min _θ L(θ ^Q )= E[(Q(s _t , a _t |θ ^Q )-y _t ) ² ] updates the evaluation network; Q represents the expected cumulative reward, θ ^Q represents the parameters of the behavior network, and E represents the relationship between the actual reward and the target of multiple sets of data The expected value of the sum of squares of the error, L(θ ^Q ) represents the parameter, μ(s _t+1 ) represents the deterministic strategy in the state s _t+1 ;

S209, updating behavioral network parameters;

Specifically, using the gradient method

To update the behavioral network, and update the target network using the following formula group to update;

θ′←τθ+(1-τ)θ′

θ ^Q′ ←τθ ^Q +(1-τ)θ ^Q′

θ ^μ′ ←τθ ^μ +(1-τ)θ ^μ′

withτ<<0.05

in, Indicates the derivative of θ ^μ , Indicates the derivative of α, Indicates that taking θ ^μ as a variable, find the derivative of J with respect to θ ^μ .

S210, judging whether the number of learning times exceeds a preset value or whether the processing quality is good enough; specifically, when the number of learning times reaches a predetermined number of times (such as 1 million times) or the learned strategy has met the application requirements (the processing quality is very good), quit learning .

S211, transmit the parameters of the evaluation network and the behavior network to the host for storage, and end.

After the learning is over, the parameters of the evaluation network and the behavior network are transmitted to the host storage, and the (st _t , a _t , _rt , _st+1 ) information obtained during processing is also transmitted to the host storage. The host computer transmits the obtained information to other trained robots. After the robot learning unit obtains the trained neural network parameters, it adopts the same neural network structure, fixes these parameters so that they cannot be changed, and only the parameters of the last two layers can be changed. Adjust the parameters of the last two layers according to the actual processing situation of the robot.

In this implementation, r represents immediate reward and R represents cumulative return.

The learning method and the reinforcement learning process of the present invention all collect the real-time information/status information of the industrial robot, and are in the working engineering of the robot (including the high-speed motion process and the heavy load process), taking into account the inertial force, centrifugal force, frictional force, The impact of gravity and joint torque on the robot's work, and the vibration generated during the working process are also measured by the sensor at the end of the robot's state to collect information such as the robot's position, speed, acceleration, and joint torque, and update and generate the optimal strategy in real time. Effective It ensures the consistency of the robot and avoids the occurrence of wrong decisions.

To sum up, the method of the present invention collects processing information and uses reinforcement learning to learn, reduce the robot debugging work and optimize the control strategy of industrial robots, including the trajectory planning function under a given path and the motor control under a given trajectory. The strategy solves the vibration of the robot under high-speed work caused by the lack of an accurate dynamic model in the traditional learning method, and improves the work efficiency of the industrial robot. This learning method is to learn the control strategy under high-speed work, learn and improve the control strategy according to the sensor data, and achieve good control under high speed.

The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be assumed that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field of the present invention, without departing from the concept of the present invention, some simple deduction or replacement can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims (6)

1. An industrial robot learning method applied to large batches of repetitive processing, characterized in that: the learning method learns based on a learning model, and it comprises the steps of: S001. The sensor collects status information; S002, learning according to the collected information; S003 , judging whether the processing quality and the processing cycle meet the requirements, and if the requirements are met, the learning is ended, otherwise, the state information is collected again and the learning is performed again. 2. The industrial robot learning method applied to large batches of repetitive processing according to claim 1, wherein the learning model is composed of an environment unit, a robot learning unit and a processing execution unit; wherein the environment unit includes at least a processing unit The quality measurement unit, the robot learning unit includes a state observation unit, data processing unit and decision-making unit, and the processing execution unit includes at least a robot and a positioner; The environment unit is composed of a processing workpiece state measurement sensor and a robot state end measurement observer. The processing workpiece state measurement sensor collects visual information of the processed workpiece, and the visual information includes at least the geometric shape and surface smoothness information of the workpiece ; The position, speed, acceleration and joint torque of the robot are collected by the robot state terminal measurement observer; The state observation unit, the state observation unit obtains the information collected by the environmental unit through a communication line, and converts the obtained information into a data format; The data processing unit receives and processes the information converted into a data format by the state observation unit; the data processing unit includes a reward calculation unit and a function update unit, wherein the reward calculation unit sets an instant reward through a reward function setting unit r, the reward calculation unit calculates the information of the state observation unit, and after the calculation is completed, the result parameters are sent to the function update unit, and the function update unit uses neural network training to update the acquired parameters until the final Learning parameters, storing the final learning parameters, making behavioral decisions through the neural network, and then performing reinforcement learning to a deterministic strategy to drive the robot to work. 3. The industrial robot learning method applied to large batches of repetitive processing according to claim 2, characterized in that: the reinforcement learning is defined as a strategy π by assuming that the robot is defined from state information to behavior, and the cumulative value obtained from time t Returns are defined as: _Calculate the expected return by Q ^π _{( st t} , a _{t )} ^＝ E _π [R _{t |} The expected return when the behavior a _t is taken below; combining the cumulative return and the formula for the expected return, the recursive formula of the expected return is obtained: Continuously use the last updated strategy to make decisions according to the recursive form formula. 4. the industrial robot learning method that is applied to large batches of repetitive processing according to claim 2, is characterized in that: described reinforcement learning adopts the reinforcement learning mode of deterministic strategy, and its concrete process comprises the following steps: S201, initialize the behavior network μ(s|θ ^μ ), the parameters are expressed as θ ^Q and the evaluation network Q(s, a|θ ^Q ), the parameters are expressed as θ ^μ , and initialize the target network Q′(s, a|θ ^{Q ′} ) and μ′(s|θ ^μ′ ), the parameters are θ ^Q′ ←θ ^Q , θ ^μ′ ←θ ^μ . S202, initialize the buffer container R; S203, receiving the state information s _t of the state observation unit; S204, select the execution behavior at according to the current strategy and apply certain _noise ; S205, observe the obtained reward r _t , and observe the next state information s _t+1 ; S206, store the quadruple <s _t , a _t , r _t , s _t+1 > in the buffer container; S207, randomly select a batch of quaternion samples from the buffer container for training; S208, updating evaluation network parameters; S209, updating behavioral network parameters; S210, judging whether the number of learning times exceeds a preset value or whether the processing quality is good enough; S211, transmit the parameters of the evaluation network and the behavior network to the host for storage, and end the learning. 5. The industrial robot learning method applied to large batches of repetitive processing according to claim 3, characterized in that: when updating the evaluation network parameters in the step S208, the objective function y _t is first set to: y _t =r (s _t , a _t )+γQ(s _t+1 , μ(s _t+1 )|θ ^Q ), and then through the formula min _θ L(θ ^Q )＝E[(Q(s _t , a _t |θ ^Q )-y _t ) ² ] Calculate the parameters to update the evaluation network, where a _t represents the behavior at time t, Q represents the expected cumulative reward, θ ^Q represents the parameters of the behavior network, and E represents the relationship between the actual reward and the target of multiple sets of data. The expected value of the sum of the squares of the errors between , L(θ ^Q ) represents the error under the parameter θ ^Q , μ(s _t+1 ) represents the deterministic policy under the state s _t+1 . 6. The industrial robot learning method applied to large-scale repetitive processing according to claim 3, characterized in that: when updating the behavioral network parameters in the step S209, the gradient method is used To update the behavioral network, and update the target network using the following formula group to update; θ′←τθ+(1-τ)θ′ θ ^Q′ ←τθ ^Q +(1-τ)θ ^Q′ θ ^μ′ ←τθ ^μ +(1-τ)θ ^μ′ withτ<<0.05 Indicates the derivative of θ ^μ , Indicates the derivative of α, Indicates that taking θ ^μ as a variable, find the derivative of J with respect to θ ^μ .