KR20220065232A - Apparatus and method for controlling robot based on reinforcement learning - Google Patents

Abstract

A robot control apparatus that controls a robot comprises: a receiving unit for receiving a plurality of product images in which products moving according to a predetermined sequence in a workspace are photographed; a deep learning model learning unit that trains a deep learning model to estimate 6D pose information of a product based on a plurality of product images; a reinforcement learning model learning unit that trains a reinforcement learning model so that the reinforcement learning model derives robot behavior information based on 6D pose information of the estimated product; and a control unit for controlling an arm of the robot based on the behavior information of the robot. Objectives of the present invention are to predict a next position of the product being moved in the workspace through the deep learning model, derive action information to be taken by the robot for the product to be located in the next position through the reinforcement learning model, and control the arm of the robot based on the derived action information of the robot.

Description

Apparatus and method for controlling a robot based on reinforcement learning

The present invention relates to an apparatus and method for controlling a robot based on reinforcement learning.

Recently, with the increase in the introduction of smart factories, interest in industrial robots and collaborative robots is increasing. In order to build such a smart factory, it is essential to develop an intelligent robot that recognizes the environment, judges the situation, and operates autonomously.

On the other hand, in order to perform a specific task using a robot, 3D information matching the image is generated based on the image captured by the robot and the work object, which normally performs the work in the work space, and utilizes the 3D information This entails controlling the position, motion, and trajectory of the robot. In particular, the automatic pick-up system using a robot must recognize the shape of the work object and determine the object.

The conventional automatic pickup system has a problem in that it is impossible to recognize an object having a shape that has not been input in advance unless image pattern information on the object to be picked up is previously input. In addition, since a gripper that picks up an object is manufactured in advance to match the shape of the object to which image pattern information is input, when the shape of the object is changed, the gripper must also be changed accordingly.

In addition, the conventional automatic pickup system is mainly used limitedly only in product classification because pick and place is possible through a robot only in a state where an object to be picked up is stopped.

Korean Patent Publication No. 2012-0027253 (published on March 21, 2012)

The present invention is to solve the problems of the prior art described above, predicting the next position of a product to be moved in a work space through a deep learning model, and predicting the behavior information to be taken by the robot with respect to the product to be located in the next position using a reinforcement learning model is derived through , and based on the derived robot's behavior information, we want to control the robot's arm.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the robot control device for controlling the robot according to the first aspect of the present invention is a receiving unit for receiving a plurality of product images in which products moving according to a preset sequence in a work space are photographed ; a deep learning model learning unit for learning the deep learning model so that the deep learning model estimates 6D pose information of the product based on the plurality of product images; a reinforcement learning model learning unit for learning the reinforcement learning model so that the reinforcement learning model derives behavior information of the robot based on the 6D pose information of the estimated product; and a controller configured to control the arm of the robot based on the behavior information of the robot.

A method of controlling a robot performed through a robot control device according to a second aspect of the present invention comprises: receiving a plurality of product images in which products moving according to a preset sequence in a work space are photographed; training the deep learning model based on the plurality of product images so that the deep learning model estimates 6D pose information of the product; training the reinforcement learning model so that the reinforcement learning model derives behavior information of the robot based on the 6D pose information of the estimated product; and controlling the arm of the robot based on the behavior information of the robot.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the present invention predicts the next position information where the product will be located in the work space using a deep learning model and a reinforcement learning model, and the robot in the predicted next position information is It is possible to derive the behavioral information of the robot to be taken, and to control the arm of the robot based on the behavioral information of the robot.

1 is a block diagram of a robot control apparatus according to an embodiment of the present invention.
2 is a diagram for explaining a deep learning model according to an embodiment of the present invention.
3 is a diagram for explaining a reinforcement learning model according to an embodiment of the present invention.
4A to 4B are diagrams for explaining a method of controlling a robot according to an embodiment of the present invention.
5 is a flowchart illustrating a method for controlling a robot according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, detailed contents for carrying out the present invention will be described with reference to the accompanying configuration diagram or process flow diagram.

1 is a block diagram of a robot control apparatus 10 according to an embodiment of the present invention.

Referring to FIG. 1 , the robot control apparatus 10 may include a receiver 100 , a deep learning model learner 110 , a reinforcement learning model learner 120 , and a controller 130 . However, the robot control device 10 shown in FIG. 1 is only one embodiment of the present invention, and various modifications are possible based on the components shown in FIG. 1 .

The robot control device 10 may include not only a personal computer such as a desktop or a laptop computer, but also a mobile terminal capable of wired/wireless communication. A mobile terminal is a wireless communication device that guarantees portability and mobility, and includes not only smartphones, tablet PCs, and wearable devices, but also Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, Ultrasonic, infrared, and Wi-Fi ( WiFi) and Li-Fi (LiFi) may include various devices equipped with a communication module. However, the robot control device 10 is not limited to the form shown in FIG. 1 or those exemplified above.

Hereinafter, FIG. 1 will be described with reference to FIGS. 2 to 4B .

The receiver 100 may receive from a camera (not shown) a plurality of product images in which products moving according to a preset sequence in the work space are photographed. Here, the working space may mean, for example, a conveyor on which a product is moved, but is not necessarily limited thereto. In addition, the preset sequence includes a plurality of processes according to time series, such as processing, packaging, and inspection of a product, and may be stored in a memory (not shown).

Meanwhile, the plurality of product images may include an RGB image and a depth image.

The deep learning model learning unit 110 may train the deep learning model based on the plurality of product images so that the deep learning model estimates 6D pose information of the product. Here, the 6D pose information of the product may include x-axis information, y-axis information, z-axis information, pitch information, roll information, and yaw information for the robot to recognize the product.

The deep learning model learning unit 110 may train the deep learning model to estimate the 6D pose information of the product after N time points (eg, N seconds) when the product is moved in the work space.

Referring to FIG. 2 , the deep learning model 20 may be a network structure in which a plurality of convolutional neural networks (CNNs) and at least one long short term memory (LSTM)-based neural network are coupled to each other may be connected.

The LSTM-based neural network used in the deep learning model 20 may operate as a recurrent neural network (RNN), but is not limited thereto. That is, the LSTM-based neural network can be replaced with other types of neural networks. Here, the recurrent neural network is an artificial neural network constructed by connecting the network at the reference time point (t) and the next time point (t+1). used

A plurality of CNNs may extract feature data from a plurality of product images, and an LSTM-based neural network may estimate 6D pose information of a product based on the extracted feature data.

For example, each of the plurality of CNNs included in the first network structure receives an RGB image and a depth image for a product photographed at time T, and each of the plurality of CNNs included in the second network structure is at time T+1. The RGB image and depth image of the photographed product are received, and each of the plurality of CNNs included in the N-th network structure receives the RGB image and the depth image of the product photographed at the T+N time point. At this time, the RGB image and the depth image taken in sequence are simultaneously input to a plurality of CNNs included in each network structure.

Each of the plurality of CNNs included in the first network structure extracts first feature data from the RGB image and the depth image for the product photographed at time T, and the LSTM-based neural network included in the first network structure extracts the extracted first feature. Based on the data, 6D pose information of the product at time T may be estimated.

Each of the plurality of CNNs included in the second network structure extracts second feature data from the RGB image and the depth image of the product photographed at the time T+1, and the LSTM-based neural network included in the second network structure extracts the extracted first 2 Based on the feature data, 6D pose information of the product at the time T+1 may be estimated.

Each of the plurality of CNNs included in the N-th network structure extracts the N-th feature data from the RGB image and the depth image for the product photographed at T+N time, and the LSTM-based neural network included in the N-th network structure extracts the extracted first Based on the N feature data, 6D pose information of the product in T+N may be estimated.

The deep learning model learning unit 110 may compare the 6D pose information of the product estimated using a mean square error through a loss function and the actual position information of the product.

For example, the deep learning model learning unit 110 may train the deep learning model 20 by comparing the 6D pose information of the product estimated by the deep learning model 20 and the actual position information of the product. For example, the deep learning model learning unit 110 is a deep learning model ( 20) can be learned. 1, the reinforcement learning model learning unit 120 can train the reinforcement learning model to derive the behavior information of the robot based on the 6D pose information of the product estimated by the deep learning model. . Here, the behavior information may include position values of joints of the end-effector of the robot according to time. The reinforcement learning model learning unit 120 may train the reinforcement learning model based on the 6D pose information of the product estimated from the deep learning model and the actual 6D pose information on the end effector of the robot. Here, the end effector refers to a part (eg, a gripper) having a function of directly controlling a work object when the robot works.

Referring to FIG. 3 , the reinforcement learning model 30 includes an Actor 301 that is a neural network that determines behavior information of a robot, and Critic 303 that is a neural network that evaluates a behavioral value for behavior information. can do.

The 6D pose information of the product estimated from the deep learning model and the actual 6D pose information on the end effector of the robot may be input as input values of the actor 301 and the crit 303 .

The actor 301 may determine the robot's behavior information based on the 6D pose information of the product estimated from the deep learning model and the actual 6D pose information on the end effector of the robot.

The crit 303 may evaluate the behavioral value of the behavioral information of the robot determined by the actor 301 based on the 6D pose information of the product estimated from the deep learning model and the actual 6D pose information on the end effector of the robot.

When the robot is controlled according to the behavior information of the robot determined by the actor 301 in the reinforcement learning environment in the reinforcement learning environment, the reinforcement learning model 30 may determine a reward for the behavior information of the robot based on the control result of the robot.

For example, the reinforcement learning model 30 may determine a reward for behavior information of the robot as a positive reward value when the robot controls a product according to a preset rule in the reinforcement learning environment. For example, when the robot arm accurately picks up a product, the reinforcement learning model 30 may determine a reward for the robot's behavior information as a positive reward value.

For example, when the robot fails to control a product according to a preset rule in the reinforcement learning environment in the reinforcement learning environment, the reinforcement learning model 30 may determine a reward for the robot's behavior information as a negative reward value. For example, the reinforcement learning model 30 may determine a reward for the robot's behavior information as a negative reward value when the robot's arm fails to pick up a product or collides with an obstacle.

The crit 303 may evaluate the behavioral value of the behavioral information of the robot determined by the actor 301 based on the reward for the behavioral information of the robot determined by the reinforcement learning model 30 .

The actor 301 may modify the behavior information of the robot based on the behavior value of the behavior information of the robot evaluated by the crit 303 .

4A to 4B together, the receiving unit 100 receives a real-time product image obtained by photographing a product 46 moving past a specific position in the working space from a camera 40 that captures a specific position in the working space 42 . can receive

The behavior information derivation unit (not shown) may input a real-time product image to the pre-learned deep learning model 20 , and estimate 6D pose information of the product through the pre-learned deep learning model 20 .

The behavior information derivation unit (not shown) inputs the 6D pose information of the product estimated through the pre-learned deep learning model 20 into the reinforcement learning model 30, and the behavior information of the robot through the reinforcement learning model 30 can be derived. For example, the behavior information derivation unit (not shown) predicts 6D pose information at N viewpoints of the product 46 moving in the work space through the pre-trained deep learning model 20, and predicts the predicted product 46 ), behavior information to be taken by the robot from the 6D pose information of ) can be derived from the previously learned reinforcement learning model 30 .

The controller 130 may control the arm 44 of the robot based on the derived behavior information of the robot. For example, the control unit 130 controls the arm 44 of the robot so that the arm 44 of the robot picks up the product 46 to be placed in the next position information of the work space 42 predicted based on the derived robot behavior information. can be controlled.

Meanwhile, for those skilled in the art, the receiver 100, the deep learning model learning unit 110, the reinforcement learning model learning unit 120, and the control unit 130 may be implemented separately, or one or more of them may be integrated. will fully understand

5 is a flowchart illustrating a method for controlling a robot according to an embodiment of the present invention.

Referring to FIG. 5 , in step S501 , the robot control device 10 may receive a plurality of product images in which products moving according to a preset sequence in the work space are photographed.

In step S503, the robot control device 10 may train the deep learning model based on the plurality of product images so that the deep learning model estimates 6D pose information of the product.

In step S505, the robot control device 10 may train the reinforcement learning model to derive behavior information of the robot based on the 6D pose information of the estimated product.

In step S507, the robot control device 10 may control the arm of the robot based on the behavior information of the robot.

In the above description, steps S501 to S507 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between steps may be changed.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

10: robot control unit
100: receiver
110: deep learning model learning unit
120: reinforcement learning model learning unit
130: control unit

Claims (14)

In the robot control device for controlling the robot,
a receiver configured to receive a plurality of product images in which products moving according to a preset sequence in a work space are photographed;
a deep learning model learning unit for learning the deep learning model so that the deep learning model estimates 6D pose information of the product based on the plurality of product images;
a reinforcement learning model learning unit for learning the reinforcement learning model so that the reinforcement learning model derives behavior information of the robot based on the 6D pose information of the estimated product; and
A control unit for controlling the arm of the robot based on the behavior information of the robot
That comprising a, robot control device.
The method of claim 1,
The plurality of product images include an RGB image and a depth image,
In the deep learning model, a plurality of convolutional neural networks (CNNs) and at least one LSTM (Long Short Term Memory) based neural network are connected to a plurality of network structures that are connected to each other.
3. The method of claim 2,
The plurality of CNNs extract feature data from the plurality of product images,
The LSTM-based neural network is to estimate the 6D pose information of the product based on the extracted feature data, the robot control device.
The method of claim 1,
The deep learning model learning unit is to learn the deep learning model by comparing the 6D pose information of the estimated product and the actual position information of the product, the robot control device.
The method of claim 1,
The reinforcement learning model will include a neural network that determines the behavior information of the robot, Actor (Actor), and a neural network that evaluates the behavioral value of the behavior information, Critic, the robot control device.
6. The method of claim 5,
Based on the control result of the robot in the reinforcement learning model, the reward for the behavior information of the robot is determined, the robot control device.
7. The method of claim 6,
When the robot controls the product according to a preset rule, the reward is determined as a positive compensation value,
When the robot fails to control the product according to the preset rule, the reward is determined as a negative compensation value, the robot control device.
In the method of controlling a robot performed through a robot control device,
Receiving a plurality of product images in which products moving according to a preset sequence in a work space are photographed;
training the deep learning model based on the plurality of product images so that the deep learning model estimates 6D pose information of the product;
training the reinforcement learning model so that the reinforcement learning model derives behavior information of the robot based on the 6D pose information of the estimated product; and
controlling the arm of the robot based on the behavior information of the robot
That comprising a, robot control method.
9. The method of claim 8,
The plurality of product images include an RGB image and a depth image,
In the deep learning model, a plurality of convolutional neural networks (CNNs) and at least one LSTM (Long Short Term Memory) based neural network are connected to a plurality of network structures that are connected to each other.
10. The method of claim 9,
The plurality of CNNs extract feature data from the plurality of product images,
The LSTM-based neural network will estimate 6D pose information of the product based on the extracted feature data.
9. The method of claim 8,
The step of training the deep learning model is
Comprising the step of learning the deep learning model by comparing the 6D pose information of the estimated product and the actual position information of the product, the robot control method.
9. The method of claim 8,
The reinforcement learning model is to include an actor (Actor) which is a neural network that determines the behavioral information of the robot, and a critic, which is a neural network that evaluates the behavioral value of the behavioral information.
13. The method of claim 12,
The method further comprising the step of determining a reward for the behavior information of the robot based on the control result of the robot through a reinforcement learning model, the robot control method.
14. The method of claim 13,
The step of determining the reward
When the robot controls the product according to a preset rule, the reward is determined as a positive compensation value,
When the robot fails to control the product according to the preset rule, determining the reward as a negative compensation value.

Images