RU2745380C1 - Method and system for capturing objects using robotic device - Google Patents

Abstract

FIELD: robotics.SUBSTANCE: invention relates to the field of robotics, and in particular to a method for gripping objects using a robotic device and to a system for implementing said method. The claimed technical result is achieved due to the implementation of the method for capturing an object using a robotic device, containing the stages, at which: a color image and a depth map are obtained; objects to capture on the image are searched for, taking into account the data of the depth map; an object to capture is selected; based on the depth map data, a point cloud of the selected object is formed; point cloud is searched for a given number of key points; arithmetic center is determined between the mentioned key points; the point among the key points closest to the arithmetic center determined; based on the coordinates of key points and the point closest to the arithmetic center, the position of the selected object is determined; based on the data on the position of the object and the data of the model built as a result of machine learning, the orientation and position of the gripping device of the robotic device is determined; then the selected object is captured using a robotic device.EFFECT: improved accuracy of recognizing the position of an object for its capture using a robotic arm.6 cl, 7 dwg

Description

FIELD OF TECHNOLOGY

[0001] The presented technical solution relates generally to the field of robotics, and in particular to a method for gripping objects of variable mass and shape using a robotic arm and to a system for implementing said method.

LEVEL OF TECHNOLOGY

[0002] Robots are widely used in industrial tasks to free people from repetitive work. Techniques for manipulating solid objects are widespread.

[0003] Compared to methods for manipulating solid objects, methods for manipulating deformable objects have additional complexities. For example, in the task of unloading plastic bags from a cart onto a table, it is critical to determine the current shape, location and grip of the bag. These tasks can be easily performed by a human, but can be difficult for a robot.

[0004] At present, various approaches based on algorithms and machine learning models are proposed for the process of training object gripping, in particular using neural networks, with the help of which robotic manipulators are trained to grip objects on the previously trained gripping points. This approach is disclosed in Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation (Peter R. Florence et al., CSAIL, Massachusetts Institute of Technology, 09/07/2018). The known solution proposes to use point clouds of objects in the process of training robotic devices in order to form object descriptors based on photographic and 3D models of such objects for the purpose of further specifying points of capture and interaction with objects at these points.

[0005] Also known is a method and a system for gripping an object using a robotic device, disclosed in patent RU 2700246 C1, publ. 20.09.2019. In the known solution, a robotic device is trained using a machine learning algorithm to recognize and memorize capture points of objects, and the training is performed on data characterizing photographic images of objects and corresponding 3D models of objects in various angles; form at least one gripping point of the object represented on the graphical image of the object; obtaining a photographic image of the object using the camera of the robotic device and determining the angle displaying the formed gripping point of the object; obtaining a three-dimensional cloud of points of the object in perspective using the depth sensor of the robotic device; determine by the obtained cloud of points the location of the point of capture of the object; determining the orientation and position of the gripper of the robotic device at the gripping point by calculating the rotation matrices of the robotic device; the object is gripped by a robotic device based on the calculation of the averaged rotation matrix for a given gripping point.

[0006] The disadvantage of the known solutions is that they are not intended to determine the capture of objects of variable mass and shape. The developed robotic system for capturing objects provides recognition and manipulation of objects that have the following characteristics: variable shape when manipulating with them; weight from several tens of grams to 6 kg or more; movable center of gravity when handling them; breaks easily if the grip is unsuccessful; are located in deep trolleys at different heights.

ESSENCE OF THE TECHNICAL SOLUTION

[0007] The technical problem or technical problem to be solved is to provide a new process of interaction with objects by means of their capture by robotic devices. One of the most important aspects of gripping the above kind of objects is the solution of the problem of positioning the gripper relative to the gripped object inside complex spatial structures, and the gripping algorithm needs to have a complex model that takes into account the probabilities of displacement of the center of mass, gripping of neighboring objects, and deformation of the object.

[0008] The technical result achieved by solving the above technical problem is to improve the accuracy of gripping an object using a robotic arm.

[0009] The claimed technical result is achieved by implementing a method for gripping an object using a robotic device, comprising the steps of:

- get a color image and a depth map;

- search for objects to capture on the image, taking into account the data of the depth map;

- select an object to capture;

- based on the depth map data, a point cloud of the selected object is formed;

- search in the cloud of points for a given number of key points;

- determine the arithmetic center between the mentioned key points;

- determine the point among the key points closest to the arithmetic center;

- based on the coordinates of key points and the point closest to the arithmetic center, determine the position of the selected object;

- based on the data on the position of the object and the data of the model built as a result of machine learning, determine the orientation and position of the gripping device of the robotic device;

- carry out the capture of the selected object using a robotic device.

[0010] In one of the particular embodiments of the method, a step is additionally performed at which, in the event of an unsuccessful grip, a new gripping position of the robotic device is selected.

[0011] In another particular embodiment of the method, the steps are additionally performed, in which:

- segment the image into separate images of objects;

- the resulting images of objects are ranked to determine the order of capture of objects;

and for capturing the object, the first object is selected according to a certain order of capturing the objects.

[0012] In another particular embodiment of the method, the step of searching in a point cloud for a predetermined number of key points includes the steps at which:

- convert the point cloud into a matrix representation;

- calculate the covariance matrix based on the matrix representation;

- search for eigenvectors of the covariance matrix;

- choose the first two eigenvectors of the covariance matrix;

- searches for a given number of key points with minimum and maximum coordinates along the axes, the basis of which is the eigenvectors.

[0013] In another particular embodiment of the method, the step is additionally performed, at which a list of observed parameters is formed, in which data about the previously mentioned key points and the point closest to the arithmetic center are entered to transfer the generated list to the learning algorithm.

[0014] In another preferred embodiment of the claimed solution, a system for gripping an object using a robotic device is provided, comprising:

- a robotic device containing at least one depth sensor and a gripper;

- a computing device connected to the robotic device;

- at least one memory containing machine-readable instructions that, when executed by at least one computing device, execute the method according to any one of claims. 1-5.

BRIEF DESCRIPTION OF DRAWINGS

[0015] The advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0016] FIG. 1 shows a general view of the object gripping system for manipulating objects of variable mass.

[0017] FIG. 2 shows the general algorithm of the above-mentioned system.

[0018] FIG. 3 shows a block diagram of the system control unit.

[0019] FIG. 4 shows an algorithm for making decisions on capturing objects.

[0020] FIG. 5 shows the algorithm of the system in the learning mode.

[0021] FIG. 6 shows the algorithm of the system in operation mode.

[0022] FIG. 7 shows a general view of a computing device.

DETAILED DESCRIPTION

[0023] This technical solution can be implemented on a computer, in the form of an automated information system (AIS) or a computer-readable medium containing instructions for performing the above method.

[0024] The technical solution can be implemented as a distributed computer system that can be installed on a centralized server (set of servers). User access to the system is possible both from the Internet and from the internal network of an enterprise / organization via a mobile communication device on which software with a corresponding graphical user interface is installed, or a personal computer with access to the web version of the system with a corresponding graphical user interface.

[0025] The following will describe the terms and concepts necessary to implement the present technical solution.

[0026] In this solution, the system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of computing operations (actions, instructions).

[0027] By a command processor is meant an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

[0028] An instruction processing device reads and executes machine instructions (programs) from one or more storage devices. The role of data storage devices can be, but are not limited to, hard disks (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

[0029] A program is a sequence of instructions for execution by a computer control device or command processing device. [0030] In accordance with the diagram shown in FIG. 1, the object gripping system 100 for manipulating deformable objects includes a gripping control unit 101: generating a control action on the manipulator 102 and a gripping device 103 (for example, an electromechanical gripping device disclosed in RU 198037 U1 ); determining the geometric parameters of the deformable object in order to move the deformable object from the zone 105 to the target zone 106. The system 100 also includes: a depth sensor 104 located on the capture device 103 and connected to the capture control unit 101; and an optical sensor 107 connected to the gripper control unit 101.

[0031] As the optical sensor 107, any commonly known optical sensor can be used, such as the model OMRON E3Z-D61, which is used to detect the deformable body material inside the gripper based on the amount of light absorbed or reflected by the deformable body.

[0032] The gripping control unit 101 is designed to determine, on the basis of the data received from the sensor 104, the position of the deformable object, its zone and orientation in space, and can be implemented on the basis of at least one computing device equipped with appropriate software for executing assigned to it below functions. As a computing device can be used, for example, a personal computer, laptop, tablet, server, smartphone, server cluster, mainframe, supercomputer, thin client, system on a chip (English "System on a Chip", abbreviated SoC), etc. P. The specific hardware implementation of the computing device depends on the requirements and must provide the necessary computational processing of digital information to ensure the proper functioning of the capture control unit 101 as part of the system 100.

[0033] As a manipulator 102 can be used as a stationary robotic arm for industrial or collaborative purposes, for example, Universal Robotics ™, KUKA ™, Fanuc ™, ABB ™, etc., and robotic manipulators installed on mobile autonomous robotic devices, as limbs, in particular robotic arms. The manipulator 102 is connected to the control unit, can have at least 6 degrees of freedom and is equipped with at least one gripping device 103 for the ability to manipulate deformable objects. As a data transmission channel between the mentioned elements of the system 100, various types of communication can be used, for example, wired and / or wireless, in particular: LAN, WAN, WLAN, Bluetooth, Wi-Fi, GSM / GPRS / LTE / 5G, Ethernet, USB, RS232, RJ45, etc.

[0034] A well-known Red-Green-Blue-Depth (RGB-D) depth camera can be used as the depth sensor 104, which works in conjunction with an RGB camera and is capable of complementing the conventional image with depth information (associated sensor distance) based on image pixels.

[0035] To solve the problem of gripping and manipulating objects, an order 200 for working with deformable objects, shown in FIG. 2. It consists of a reinforcement learning stage 201 and a system operation stage 202. Reinforcement learning is one of the machine learning methods, during which the system under test (agent) learns by interacting with a certain environment. From the point of view of cybernetics, it is one of the types of cybernetic experiment.

[0036] Step 202 consists of creating a description of the system sequence, defining the structure of iterations and episodes of the reinforcement algorithm, determining the control actions of the reinforcement algorithm, and is performed by the system designer 100.

[0037] In accordance with the diagram of FIG. 3, the gripping control unit 101 is composed of: an object gripping determination unit 301; module 302 planning the trajectory of the movement of the manipulator; module 303 data collection from the depth sensor 104; and a capture device control unit 304 103.

[0038] Module 301 can be implemented on the basis of at least one neural network for performing RL (reinforcement learning) algorithms and generating and issuing recommendations for the location and orientation of the capture device 103 for successful capture of the next target, depending on the parameters extracted from the module 303 data collection. These generated recommendations are then transmitted to the module 302 for planning the trajectory of the manipulator movement, taking into account collisions in the working area of the robot for moving objects, which generates control actions that are transmitted to the manipulator control module 304 for moving the manipulator 102 along the trajectory of movement and gripping the object using the gripper 103 The above modules 302, 303 and 304 can be implemented on the basis of the firmware of the gripper control unit 101, equipped with the corresponding software for performing the functions assigned to them below.

[0039] FIG. 4 shows an algorithm 400 for making decisions about capturing objects. In a first step 401, the acquisition module 303 from the depth sensor 104 acquires a color image and a depth map. The depth sensor 104 for data collection can be controlled by the capture control unit 101 by methods known from the prior art, disclosed, for example, in the closest analogue, RU 2700246 C1.

[0040] Next, the color image and the depth map are sent by said module 303 to the object capture determination module 301, which proceeds to step 402. At step 402, using a deep learning neural network, said module 301 searches for objects for capturing in the image using methods known from the prior art. taking into account the data of the depth map, after which the image is segmented into separate images of the mentioned objects. To perform segmentation, any well-known segmentation algorithms can be used, disclosed, for example, in the article "Detectron2: A PyTorch-based modular object detection library" posted on 10/09/2019 on the Internet at: https://ai.facebook.com/ blog / -detectron2-a-pytorch-based-modular-object-detection-library- /. Thereafter, the obtained object images are ranked by said module 301 to determine the order of object capture.

[0041] In the next step 403, said module 301, according to the order of object capture, selects the first object for capturing and, based on the depth map data, forms a point cloud of the selected object, then based on this point cloud, the module 301 calculates the key points of the object (step 404), following the parameters which will be observed and which will be used to determine the position and orientation of the object. To calculate the aforementioned key points, it is proposed to use a new method based on the method of principal components, disclosed in the article Principal component analysis (Jolliffe I.T., Technometrics. - 2003. - T. 45. - No. 3. - P.276), to determine the geometric characteristics of the shape of an object in 3-dimensional space.

[0042] To implement the principal component method, the point cloud module 301 is converted to a matrix representation X, where each row represents a point in the point cloud and the columns represent [x, y, z] components that represent the coordinates of each point in the point cloud, then the covariance matrix is calculated by the formula:

where X is the matrix representation of the point cloud, X 'is the mathematical expectation of the matrix X, n is the number of points in the point cloud, T is the matrix transposition. Then, in the resulting matrix by widely known methods, for example, using mathematical packages, there is a search for 3 eigenvectors, for example, by the method disclosed in the book "A guide to NumPy" (Oliphant T. E., Trelgol Publishing, 2006. - T. 1. - P. 179), the resulting vectors are sorted in decreasing order of matrix eigenvalues. To further process the data, module 301 selects the first two eigenvectors of the covariance matrix, which are vectors in three-dimensional space. Then, the module 301 searches for the number of key points specified by the developer in the software algorithm, for example, four points having minimum and maximum coordinates along the axes, the basis of which is the eigenvectors found in the previous step.

[0043] In the next step, the module 301 searches for an existing midpoint in the point cloud among key points, in particular four points having minimum and maximum coordinates along the axes. To do this, the module 301 determines the arithmetic center between the key points, extracting their coordinates along the planes of the eigenvectors, and sorting the point cloud array in order of proximity of the points to the arithmetic center, the module 301 selects an additional point, among the mentioned key points, which is closest to the mentioned arithmetic center. Further, based on the coordinates of the key points and the point closest to the said arithmetic center, the module 301 determines the position of the object. Further, based on the data on the position of the object, in particular its coordinates, the coordinates of the mentioned key points and the point closest to the arithmetic center, and the data of the model built as a result of machine learning, the module 301 determines the orientation and position of the gripping device 103 of the robotic device, after which generates recommendations for the location and orientation of the capture device 103 for successful capture of the next object for transmission to the module 302 planning the trajectory of the manipulator.

[0044] Additionally, the object capture determination module 301 can be configured to generate a list of observed parameters, into which data about the previously mentioned key points and the point closest to the arithmetic center are entered, for transferring the generated list to the learning algorithm, the purpose of which is to determine the best poses of the gripper in space for gripping an object in the working area, taking into account the deformation and displacement of the center of gravity of the object, as well as geometric collisions in the working area.

[0045] In the training mode, a simulation of a robotic installation and its working environment, as well as a model of objects to be manipulated, is used. The simulator should allow calculating the interactions between objects of the simulated scene, using the so-called soft contact method, and support the ability to simulate deformable or soft objects with filling. The training of deep reinforcement learning algorithms occurs simultaneously in a large number of simulators, in which the positions and orientations of manipulated objects are generated randomly in specified work areas.

[0046] the Algorithm of the system in the learning mode, which is performed by means of neural networks, which can also be placed in the module 301 capture the object, is shown in FIG. 5 At step 501, data is requested from the module 303 for collecting data from the depth sensor 104 and retrieving the list of observed parameters, which are transmitted to the learning algorithm, at the next step 502, the reinforcement learning algorithm proposes the positions and orientations of the robot gripper (coordinates [x, y, z] and quaternion [x, y, z, w]), however, not all of them can be attainable under the conditions of a given position of the robot and the parameters of the kinematic chain of the robot. To cut off unattainable gripping positions at the next step 503, an additional environment in the simulator is used, which allows, from the values of the target gripping position, to determine all possible geometric configurations of the robot's links in space by solving the inverse kinematics problem. For each of these possible configurations, a collision model is calculated and it is checked whether there are collisions within the scene between objects (block 508) in order to reject obviously impossible configurations of the robot arm. If the target pose is not in collision, then an attempt is made to capture the object within the simulator at block 505, which computes the physical interactions of the capture with the manipulated object. Based on the simulation results, the reward function is calculated, the value of which is sent to the reinforcement learning algorithm to correct the control actions 506. Thus, the learning algorithm forms an internal model of the deformable body, taking into account the variable center of gravity of the possibility, the deformation model of both the bag and the environment. The training process ends after a certain number of training iterations.

[0047] The assessment of the quality of learning of the reinforcement learning algorithm is calculated by running several parallel processes in simulators for a number of typical random initial situations in the zones of the manipulation objects. The results of these simulations are saved and their statistical analysis is performed, as a result of which it is found out how many successful attempts to capture objects were for each of the zones where the objects of manipulation were located.

[0048] In the operating mode 600 shown in FIG. 6, the robot is initialized in a predetermined position to obtain data on the location of objects in the working area. Algorithm 400 selects a candidate for a grip and its location is calculated in step 601, the decision system recommends the target position and orientation of the gripper in step 602. Rapid trajectory planning engine 603 is used to move the gripper to a target position, which proposes a trajectory. robot. In order to avoid a collision of the robot with an object, the target posture is adjusted so that at the end of the execution of the trajectory from the gripper to the object, a small distance remains. This distance is covered by the robot in a compliant mode so that when the gripper reaches contact with the object, the movement of the robot stops. Next, the manipulation object is gripped and moved, information about the successful gripping from the gripper sensor 107 is transmitted to the reinforcement learning algorithm at step 604, which, in case of an unsuccessful attempt, recommends a different gripping position of the robotic device. The process continues until the work area is free of all manipulation objects.

[0049] Thus, due to the fact that when determining the position of an object, its key points and the point among the key points closest to their arithmetic center are taken into account, the accuracy of recognizing the position of the object and the accuracy in choosing the orientation and position of the gripping device of the robotic device for gripping are increased. of this object, i.e. the accuracy of gripping the object with the help of a robotic arm is increased.

[0050] In general terms (see Fig. 7), the computing device contains one or more processors (701) united by a common data exchange bus, memory means such as RAM (702) and ROM (703), input / output interfaces (704 ), input / output devices (705), and a device for networking (706).

[0051] The processor (701) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, manufacturers such as: Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™, etc. Under the processor or one of the processors used in the device (700), it is also necessary to take into account the GPU, for example, NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method, and can also be used for training and applying machine learning models in various information systems.

[0052] RAM (702) is a random access memory and is intended for storing machine-readable instructions executed by the processor (701) for performing the necessary operations for logical processing of data. RAM (702) typically contains executable instructions of the operating system and associated software components (applications, software modules, etc.). In this case, the available memory of the graphics card or graphics processor can act as RAM (702).

[0053] ROM (703) is one or more persistent storage devices such as a hard disk drive (HDD), solid state data storage device (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0054] Various types of I / O interfaces (704) are used to organize the operation of the components of the device (700) and to organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific version of the computing device, which can be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0055] To ensure the interaction of the user with the device (700), various means (705) of I / O information are used, for example, a keyboard, display (monitor), touch display, touch pad, joystick, mouse manipulator, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, light indicators, projector, camera, biometric identification (retina scanner, fingerprint scanner, voice recognition module), etc.

[0056] The networking tool (706) provides data transmission via an internal or external computer network, for example, Intranet, Internet, LAN, and the like. One or more means (706) may be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module, and dr.

[0057] Additionally, satellite navigation means can be used as part of the system (700), for example, GPS, GLONASS, BeiDou, Galileo. The specific choice of elements of the device (700) for the implementation of various software and hardware architectural solutions can vary while maintaining the required functionality provided.

[0058] Modifications and improvements to the above-described embodiments of the present technical solution will be clear to those skilled in the art. The foregoing description is provided by way of example only and is not intended to be limiting in any way. Thus, the scope of the present technical solution is limited only by the scope of the attached claims.

Claims (27)

1. A method of capturing objects using a robotic device, performed by at least one computing device, comprising the steps of: - get a color image and a depth map; - search for objects to capture on the image, taking into account the data of the depth map; - select an object to capture; - based on the depth map data, a point cloud of the selected object is formed; - search in the cloud of points for a given number of key points; - determine the arithmetic center between the mentioned key points; - determine the point among the key points closest to the arithmetic center; - based on the coordinates of key points and the point closest to the arithmetic center, determine the position of the selected object; - based on the data on the position of the object and the data of the model built as a result of machine learning, determine the orientation and position of the gripping device of the robotic device; - carry out the capture of the selected object using a robotic device. 2. The method according to claim 1, further comprising the step of selecting a new gripping posture of the robotic device in the event of an unsuccessful grip. 3. The method according to claim 1, characterized in that it further comprises the stages at which: - segment the image into separate images of objects; - the resulting images of objects are ranked to determine the order of capture of objects; and for capturing the object, the first object is selected according to a certain order of capturing the objects. 4. The method according to claim 1, characterized in that the stage of searching in a cloud of points for a predetermined number of key points includes the stages at which: - convert the point cloud into a matrix representation; - calculate the covariance matrix based on the matrix representation; - search for eigenvectors of the covariance matrix; - choose the first two eigenvectors of the covariance matrix; - searches for a specified number of key points with minimum and maximum coordinates along the axes, the basis of which is the eigenvectors. 5. The method according to claim 1, characterized in that it further comprises a stage at which a list of observed parameters is formed, in which data about the previously mentioned key points and the point closest to the arithmetic center are entered to transfer the generated list to the learning algorithm. 6. A system for gripping an object using a robotic device, containing: - a robotic device containing at least one depth sensor and a gripper; - a computing device connected to the robotic device; - at least one memory containing machine-readable instructions that, when executed by at least one computing device, perform the method according to any one of claims. 1-5.

Images