CN108415254B - Control method of waste recycling robot based on deep Q network - Google Patents

Abstract

The present invention discloses a control method and device for a waste recycling robot based on a deep Q network, characterized in that: the sensor system is used to sense the position information of objects in front of the robot, represented by image information; the control system is used to control the robot's grasping arm to grasp objects and place objects in a storage mechanism; the operation system receives information from the control system and performs various actions; the drive system is used to provide power for the operation system to perform various actions of the control system; the sensor system collects environmental information and drive system information, and transmits the environmental information and drive system information to the control system, which calculates and processes the received information, and sends information to the operation and drive system to drive the robot to perform corresponding actions. The present invention uses the reinforcement learning algorithm in the field of artificial intelligence, and can autonomously learn and update the parameters of the neural network so that the robot can achieve the control effect of recycling items.

Description

Control method of waste recycling robot based on deep Q network

technical field

The invention belongs to the field of artificial intelligence and control technology, and in particular relates to a method for controlling a waste recycling robot based on a deep Q network, which can carry out self-learning and complete the grasping control of objects by the robot.

Background technique

In recent years, the application of artificial intelligence in family life has become more and more extensive, forming the concept of smart home. Among them, the sweeping robot is a small automatic control robot with artificial intelligence, which is used to clean the home. At present, sweeping robots have been well used in the market. The application of sweeping robots has partially liberated people from household chores and has been well received by people.

However, the current cleaning objects of sweeping robots are mainly aimed at the dust on the ground, and can only clean the ground by vacuuming, so it is only suitable for household cleaning of a single ground environment. For some large wastes, such as discarded bottles and cans, Most sweeping robots will be helpless and can only mark them as obstacles and bypass them directly.

Obviously, a sweeping robot that can only clean the dust on the ground cannot fully meet the needs of larger occasions and more complex environments (such as road surfaces), which limits the scope of use of the sweeping robot.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a control method and device for a waste recycling robot based on a deep Q network. Through the improvement of the control method and self-learning, it can adapt to the new environment faster, ensure the effectiveness of the strategy update, and achieve faster Adapt to the needs of different environments and different cleaning objects, and greatly expand the scope of application.

The technical scheme of the present invention is: a waste recycling robot device based on a deep Q network, comprising a sensing system, a control system, an operation system and a driving system, and is characterized in that:

The sensing system: including a camera and an image acquisition device, used for sensing the position information of objects in front of the robot, represented by image information;

The control system: used to control the robot grasping arm to grasp objects and place objects in the storage mechanism, and control the rotation angle of the rotating mechanism;

The operating system: including a robot grabbing arm, a rotating mechanism, and a storage mechanism, which are used to receive information from the control system and perform various actions;

The drive system: including a motor and a battery, used to provide power for the operating system to perform various actions of the control system;

The sensing system collects environmental information and driving system information, and transmits the environmental information and driving system information to the control system. action.

Another technical solution of the present invention is: a control method of a waste recycling robot device based on a deep Q network: the method steps are:

(1) Obtain environmental information through the sensing system, including visual environmental information and non-visual information;

(2) According to the environmental information obtained in the step (1), initialize the neural network parameters, including environmental state information and reward information, and initialize various parameters of the reinforcement learning algorithm;

(3) Process the image information fed back by the surrounding environment, process the image information into grayscale images through digital processing, use deep convolutional networks for feature extraction and training, and convert high-dimensional environmental visual information into low-dimensional feature information. The dimension feature information and the non-visual information are used as the input state _st of the current value network and the target value network;

(4) The action of the robot is controlled by the output of the current value network; in the state s _t , the action a _t is obtained by calculating the action value function Q(s, a) in the reinforcement learning algorithm according to the current value network, and the robot executes the action a _t After , obtain a new environmental state _{st +1} and an immediate reward rt;

⑸Update the current value network parameters and the target value network parameters, and update the parameters by stochastic mini-batch gradient descent update method;

其中Q(s′,a′；θ_i ^-)表示下一个状态下的状态动作值，Q(s,a；θ_i)为当前状态下的状态动作值，γ为回报函数的折扣因子，γ(0≤γ≤1)，E()为梯度下降算法中的损失函数，r为立即奖赏值，θ表示网络参数；The calculation method of the current value network loss function:

where Q(s',a'; θ _i ^- ) represents the state action value in the next state, Q(s, a; θ _i ) is the state action value in the current state, γ is the discount factor of the reward function, γ (0≤γ≤1), E() is the loss function in the gradient descent algorithm, r is the immediate reward value, and θ is the network parameter;

The target value network is copied and obtained by the current value network after each execution of N million steps;

⑹ Check whether the learning termination condition is met, if not, return to step ⑷ to continue the cycle, otherwise end; the learning termination condition is that the item falls off, or the set number of steps is completed.

In the above technical solution, in the step (4), an experience pool E is set, and the experience pool E is the state information and reward information that the robot obtains feedback from the environment after interacting with the environment, specifically: according to the action value function Q(s, a ) select an action and execute it, save the current state s, action a, the immediate reward r obtained by executing the action, and the next state s' reached as a tuple in the experience pool E, and repeat the above steps for three to fifty thousand steps, are stored in the experience pool E, and the parameters of updating the current value network and the target value network in the step (5) need to be sampled from the experience pool E.

In the above technical solution, the samples sampled from the experience pool E in step (5) need to be selected according to their priority level, and the priority level is set as: whenever the content is stored in the experience pool E, the priority level of the sample is updated once, and the formula is updated. for:

Among them, t is the number of times the sample is selected, β is the influence degree of the use priority, and p _i is the probability that the ith sample is selected. After the sample priority is calculated, it is normalized. The formula is:

In the above technical scheme, the current value network is composed of three layers of convolutional neural networks and one layer of fully connected layers, and the activation function is the relu function; it is used to process the image information processed by the sensor system, wherein the convolutional neural network extracts After the image features, the action value function Q(s,a) is output through the activation function relu, and the action is selected by the ε-Greedy greedy strategy according to the action value function Q(s,a).

A further technical solution is to set "the output of the current value network to control the action of the robot" as: randomly select a number of samples from the experience pool E, and use its state s as the input of the first hidden layer of the current value network. The value network outputs the action value function Q(s,a), and selects the action a _t to take according to the action value function. After the robot performs the action a _t , it obtains a new environment state s _t+1 and an immediate reward r _t , and passes The current value network loss function adjusts the parameters of the current value network.

In above-mentioned technical scheme, in described step (3):

The state S is expressed as: the environmental state perceived by the sensing system is the position information of the objects in the current field of view of the robot, which is presented in the form of images;

Action a is expressed as: the set of operations that can be performed in the current state, including the angle and direction operations of the robot grabbing the item;

The immediate reward r is: the evaluation of the actions taken by the robot in the current state. If the object does not fall off after the robot grabs the object, it will give a reward of +1; if the object is successfully placed in the storage mechanism, the reward will be +1000, If the item is dropped, the reward will be -1000, otherwise the reward will be 0.

The advantages of the present invention are:

1. According to the interaction between the robot and the size of the object using the sensing system, the present invention obtains the grasping strategy of the robot facing different objects through the calculation of the reinforcement learning method, so that the robot can face objects of different shapes, and all can be smoothly Put it in the storage mechanism to realize the recovery of various wastes;

2. By adopting a prioritized deep reinforcement learning control method in the control system of the robot, processing the environmental information obtained from the sensing system, then selecting the appropriate action, and using the sensing system to transmit the control signal of the control system to the job and The drive system enables the robot to grasp objects of different shapes more accurately;

3. The robot can train objects of different shapes, which greatly improves its applicability. After sufficient training, it can be widely used in various scenarios, such as road surfaces and commercial complexes;

4. This method can effectively deal with the control problem with continuous action space;

5. It can effectively avoid the loss of items in the training and application process and speed up the training process;

6. The use of convolutional neural network can effectively extract image features, so that the system can better find the appropriate action.

Description of drawings

Below in conjunction with accompanying drawing and embodiment, the present invention is further described:

1 is a block diagram of the information transmission structure of the robot device in the first embodiment of the present invention;

2 is a structural block diagram of a priority deep reinforcement learning controller in Embodiment 1 of the present invention;

FIG. 3 is a schematic structural diagram of a deep Q network in Embodiment 1 of the present invention.

Detailed ways

Below in conjunction with accompanying drawing and embodiment, the present invention is further described:

Embodiment: Referring to Figures 1 to 3, a waste recycling robot device based on a deep Q network includes a sensing system, a control system, an operation system and a drive system, and is characterized in that:

The sensing system: including a camera and an image acquisition device, used for sensing the position information of objects in front of the robot, represented by image information;

The control system: used to control the robot grasping arm to grasp objects and place objects in the storage mechanism, and control the rotation angle of the rotating mechanism;

The operating system: including a robot grabbing arm, a rotating mechanism, and a storage mechanism, receiving information from the control system, and executing various actions;

The drive system: including a motor and a battery, used to provide power for the operating system to perform various actions of the control system;

The sensing system collects environmental information and driving system information, and transmits the environmental information and driving system information to the control system. Actions to complete the grab and drop of various sizes of items.

The specific implementation process is as follows:

最大(0≤γ≤1为折扣因子)，该策略即为最优策略。但是在现实环境中，环境的状态转移概率函数p和回报函数r未知。智能体要学习到最优策略，只有立即奖赏r_t可用，这样可以直接采用策略梯度方法优化损失函数。当前值网络选择行动，采用TD(TemporalDifference)误差来计算损失，并通过随机梯度下降方法更新当前值网络参数，寻找最优策略。控制结构如图2所示。In this embodiment, the overall control framework of the control system is a Deep Q-Network (DQN) in deep reinforcement learning, and the Q-Learning algorithm in the field of reinforcement learning is used for control. Assuming that at each time step t=1,2,..., the robot sensor system observes the state of the Markov decision process as s _t , the control system chooses the action a _t , obtains the immediate reward r _t of the environmental feedback, and makes the system transfer to The next state, s _t+1 , has a transition probability p(s _t , at , s _{t +1} ). The goal of an agent in a reinforcement learning system is to learn a policy π such that the cumulative discounted reward obtained in future time steps

Maximum (0≤γ≤1 is the discount factor), the strategy is the optimal strategy. But in the real environment, the state transition probability function p and reward function r of the environment are unknown. In order for the agent to learn the optimal policy, only the immediate reward _rt is available, so that the policy gradient method can be directly used to optimize the loss function. The current value network selects the action, uses the TD (TemporalDifference) error to calculate the loss, and updates the current value network parameters through the stochastic gradient descent method to find the optimal strategy. The control structure is shown in Figure 2.

In different environments, the network structure of the control system is the same, and the algorithm parameters also use the same set of parameters. The discount factor of the reward function is γ=0.99. A 3-layer convolutional neural network is used to extract the image information collected by the sensor system. The network parameters of the convolutional neural network are fixed, and the value network and policy network are composed of 3 hidden layers and an output layer. composition. In each experiment, the initial state of the environment where the robot is located is a random initial state, and the robot starts learning from the random initial state. If the control fails, the robot re-learns until the robot can successfully grasp the item.

Step 1: Obtain the environment information where the robot is located.

The sensor system of the robot collects information through cameras and various image acquisition devices. The robot obtains the image information of the surrounding environment of the robot through the sensor, and controls the action of the robot through the sensor.

Step 2: Obtain the initial environment state information and reward information of the robot, and initialize the parameters of the algorithm.

Initialize the neural network parameters and reinforcement learning algorithm parameters in the control system, where the neural network parameters include the weights and biases of the feedforward network.

Step 3: Process the visual information of the environmental feedback.

The state of the robot is sensed through the sensor system. The image information is processed into grayscale images through digital processing, and the high-dimensional environmental visual information is converted into low-dimensional feature information. The low-dimensional feature information and sensor-perceived non-visual information serve as the input state s _t of the policy network and value network.

Step 4: Populate the Experience Pool

After the robot interacts with the environment, it obtains the status information and reward information of the environment feedback. The high-dimensional visual information of the environmental feedback is processed in step 3, and a processed output is generated. After repeating the operation four times, the output is obtained as the current value network input, and the action is selected and executed according to the action value function. The current state s, action a , The immediate reward r obtained by performing the action and the next state s' reached are stored as a tuple in the experience pool E, and step 4 is repeated for fifty thousand steps. After each step, the priority is updated for the experience sample, and the priority update formula is:

Among them, t is the number of times the sample is selected, β is the influence degree of the use priority, and p _i is the probability that the ith sample is selected. After calculating the sample priority, normalize it, the formula is:

Step 5: The actions of the robot are controlled by the current value network.

Four samples are randomly selected from the experience pool E, and their state s is used as the input of the first hidden layer of the current value network, and the action value function Q(s, a) is output by the current value network. Taking action a _t , after the robot performs action a _t , it obtains a new environmental state s _t+1 and an immediate reward r _t . And adjust the parameters of the current value network through the error function (the current value network loss function _{Li (θ i )} ).

The current value network consists of a three-layer convolutional neural network and a fully connected layer, and the activation function is the relu function. It is used to process the image information processed by the sensor system. After the convolutional neural network extracts the image features, the action value function is output through the activation function, and the action is selected by the ε-Greedy strategy according to the action value function.

Step 6: Save the current state s, the action a, the immediate reward r obtained by executing the action, and the next state s' reached into the experience pool E as a tuple.

Step 7: Update the current value network parameters and target value network parameters of the control system.

The robot continuously interacts with the environment through step 4, and samples a batch of samples to update the current value network and the target value network. The specific update method is as follows:

其中Q(s′,a′；θ_i ^-)表示下一个状态下的状态动作值，Q(s,a；θ_i)为当前状态下的状态动作值，该方法使用了强化学习中的Q-Learning算法，并采用RMSProp梯度下降方法(设置动量参数γ为0.95)来更新当前值网络参数。The current value network loss function _{Li (θ i )} is calculated as:

Where Q(s',a'; θ _i ^- ) represents the state action value in the next state, Q(s, a; θ _i ) is the state action value in the current state, this method uses the Q in reinforcement learning -Learning algorithm, and adopt RMSProp gradient descent method (set momentum parameter γ to 0.95) to update current value network parameters.

The target value network is replicated from the current value network after every ten thousand steps.

Step 8: View Control Results

Check whether the learning termination condition is met, if not, return to step 5 to continue the loop. Otherwise end the algorithm.

In the real environment, the initial state of the robot is initialized to the environmental state of the position of the item in front of the robot, and the position of the item is a random position. The control system of the robot makes decisions on the actions the robot needs to take in one step by processing the state and feedback information of the collected environment, and uses these data to update the current value network and the target value network until the robot encounters a termination state, the robot starts again. study. The robot executes 100 episodes in the environment (the episodes are set to have a limited length), and if they can successfully grab the item, the learning is determined to be successful.

The states, actions, and immediate rewards proposed in this embodiment are respectively expressed as:

Status: The environmental status perceived by the sensing system is the position information of the items in the robot's current field of view, which is presented in the form of images.

Action: Action is a set of operations that can be performed in the current state. In this example, the action is divided into the angle and direction operations of the robot grabbing the item.

Immediate reward: Immediate reward is the environment's evaluation of the action taken by the robot in the current state. The reward function in the present invention is defined as: if the object does not fall off after grabbing the object, a reward of +1 is given; if the object is successfully placed in the designated position, the reward is +1000, and if the object is dropped, the reward is -1000 , otherwise the reward is 0.

In the simulation process, after 50 million steps of simulation operation, if encountering the end of the plot, a random new initial environment state will continue to execute, and the number of simulated steps will be accumulated. The average value of the rewards for the most recent 100 episodes is used as the criterion to measure the success of vehicle control.

Claims (5)

1. A kind of waste recycling robot control method based on deep Q network, its method steps are: (1) Obtain environmental information through the sensing system, including visual environmental information and non-visual information; (2) According to the environmental information obtained in the step (1), initialize the neural network parameters, including environmental state information and reward information, and initialize various parameters of the reinforcement learning algorithm; (3) Process the image information fed back by the surrounding environment, process the image information into grayscale images through digital processing, use deep convolutional networks for feature extraction and training, and convert high-dimensional environmental visual information into low-dimensional feature information. The dimension feature information and the non-visual information are used as the input state _st of the current value network and the target value network; (4) The action of the robot is controlled by the output of the current value network; in the state s _t , the action a _t is obtained by calculating the action value function Q(s, a) in the reinforcement learning algorithm according to the current value network, and the robot executes the action a _t After , obtain a new environmental state _{st +1} and an immediate reward rt; ⑸Update the current value network parameters and the target value network parameters, and update the parameters by stochastic mini-batch gradient descent update method; The calculation method of the current value network loss function:

in

Represents the state action value in the next state, Q(s, a; θ _i ) is the state action value in the current state, γ is the discount factor of the reward function, 0≤γ≤1, E() is the gradient descent algorithm in The loss function of , r is the immediate reward value, and θ is the network parameter; The target value network is copied and obtained by the current value network after each execution of N million steps; ⑹ Check whether the learning termination condition is met, if not, return to step ⑷ to continue the cycle, otherwise end; the learning termination condition is that the item falls off, or the set number of steps is completed; In the step (4), an experience pool E is set, and the experience pool E is the state information and reward information that the robot can obtain feedback from the environment after interacting with the environment, specifically: selecting an action according to the action value function Q(s, a) and executing it , save the current state s, action a, the immediate reward r obtained by executing the action, and the next state s' reached as a tuple in the experience pool E, and repeat the above steps for 30,000 to 50,000 steps, all of which are stored in the experience pool In E, the parameters of updating the current value network and the target value network in the step (5) need to be sampled from the experience pool E; In the step (5), the samples sampled from the experience pool E need to be selected according to their priority level, and the priority level is set as: whenever the content is stored in the experience pool E, the priority level of the sample is updated once, and the update formula is:

Among them, t is the number of times the sample is selected, β is the influence degree of the use priority, and p _i is the probability that the ith sample is selected. After the sample priority is calculated, it is normalized. The formula is:

2. The control method according to claim 1, characterized in that it comprises a sensing system, a control system, an operation system and a drive system, The sensing system: including a camera and an image acquisition device, used for sensing the position information of objects in front of the robot, represented by image information; The control system: used to control the robot grasping arm to grasp objects and place objects in the storage mechanism, and control the rotation angle of the rotating mechanism; The operating system: including a robot grabbing arm, a rotating mechanism, and a storage mechanism, which are used to receive information from the control system and perform various actions; The drive system: including a motor and a battery, used to provide power for the operating system to perform various actions of the control system; The sensing system collects environmental information and driving system information, and transmits the environmental information and driving system information to the control system. action. 3. control method according to claim 1 is characterized in that: described current value network is made up of three-layer convolutional neural network and one-layer fully connected layer, and activation function is relu function; The obtained image information, in which the convolutional neural network extracts the image features and outputs the action value function Q(s,a) through the activation function relu, and selects the action with the ε-Greedy greedy strategy according to the action value function Q(s,a). 4. The control method according to claim 3 is characterized in that: "the output of the current value network controls the action of the robot" as: randomly extract several samples from the experience pool E, and use its state s as the current value network. The input of the first hidden layer, the action value function Q(s, a) is output by the current value network, and the action a _t to be taken is selected according to the action value function. After the robot performs the action a _t , it obtains a new environment state s _{t +1} and an immediate reward _rt , and the parameters of the current value network are adjusted by the current value network loss function. 5. control method according to claim 1, is characterized in that: in described step (3): The state S is expressed as: the environmental state perceived by the sensing system is the position information of the objects in the current field of view of the robot, which is presented in the form of images; Action a is expressed as: the set of operations that can be performed in the current state, including the angle and direction operations of the robot grabbing the item; The immediate reward r is: the evaluation of the actions taken by the robot in the current state. If the object does not fall off after the robot grabs the object, it will give a reward of +1; if the object is successfully placed in the storage mechanism, the reward will be +1000, If the item is dropped, the reward will be -1000, otherwise the reward will be 0.

Images