KR20230068530A - cooperative robot monitoring-control system - Google Patents

Abstract

The present invention relates to a collaborative robot monitoring and control system, and more particularly, to a starting point of a cooperative robot operation obtained through 3D vision (3D sensor) technology, RGB camera image information, etc. Intelligent control of collaborative robots is possible based on analysis of the joint movement conditions of collaborative robots that are subject to monitoring and control through a smart teaching pendant that enables real-time monitoring and improves the convenience of users such as workers. It is about a monitoring control system for intelligent control of collaborative robots utilizing process recognition and automatic motion generation algorithms.

Description

Monitoring and control system for collaborative robot intelligent control {cooperative robot monitoring-control system}

The present invention relates to a collaborative robot monitoring and control system, and more particularly, to a starting point of a cooperative robot operation obtained through 3D vision (3D sensor) technology, RGB camera image information, etc. The present invention relates to a monitoring control system for intelligent control of a collaborative robot capable of real-time monitoring of a cooperative robot and capable of controlling or learning the operation of a cooperative robot through a smart teaching pendant with improved user convenience such as a worker.

In addition, the present invention relates to a monitoring control system for intelligent control of a cooperative robot utilizing a process recognition and automatic motion generation algorithm capable of intelligent control of a cooperative robot based on analysis of the motion state of the joint of the cooperative robot to be controlled. it is also

Collaborative robots, which are commonly defined as robots that operate in close proximity to humans, are developing collaborative robots that can work in the same space as humans with advances in information and communication technologies such as IoT, robotics, artificial intelligence technologies, and sensing technologies. is expanding across

These collaborative robots can be directly taught and used by workers as needed, and as a means of assisting workers, etc. for various tasks, they can increase work efficiency and, at the same time, have recently attracted attention as a means to reduce fixed costs in the long term. have and do

However, there is a risk of colliding with a worker, etc., and there are restrictions (the main purpose of reducing the amount of impact) that a collaborative robot is developed and used with a small center that does not have a large amount of impact. Persistent limitations have been pointed out, such as constraints that only work is performed.

In addition, the limitations of automation and control-related systems dependent on SI (System Integration) and the limitations of requiring design and installation of peripheral devices such as JIG in advance for work progress are also pointed out as problems to be continuously improved.

Republic of Korea Patent Publication No. 10-2021-0118380 (published on September 30, 2021)

The present invention, as a technology developed to solve the above-mentioned problems, integrates an AI (artificial intelligence)-based 3D vision system and automatic motion generation technology in which robot control is interlocked to create a collaborative robot technology that recognizes a task and generates motion by itself. is in providing

In order to achieve the object of the present invention described above, information on at least one of a starting point of a robot operation performed by a collaborative robot, a passing point on a moving path, and a destination point is obtained in real time through a 3D sensor, and obtained through the 3D sensor A control box capable of controlling at least one of a position, state, and operation of the collaborative robot based on (vision information) information about at least one type of a starting point, a passing point, and a destination point on a movement path; and a control box and a communication network; connected by wire or wirelessly, and of the cooperative robot in a work environment set according to at least one of acquired information and a pre-learned pattern among the location, state, and operation of the cooperative robot. It is possible to implement a monitoring and control platform for a 3D vision-integrated collaborative robot, including a smart teaching pendant capable of maintaining control of at least one of position, state, and operation, including collision avoidance motion with a worker, etc. A monitoring control system for intelligent control of a collaborative robot, characterized in that it has behavioral intelligence capable of self-recognizing tasks and generating motions, may be provided.

In addition, the control box according to the present invention determines the posture of the whole or a specific part of the cooperative robot on a preset 3D coordinate system based on at least one information among the position, state, and operation of the cooperative robot monitored in real time through the 3D sensor. It can also be provided in the form of a monitoring control system for intelligent control of collaborative robots, characterized in that it further includes a vision process capable of inferring.

In addition, in the monitoring and control system for intelligent control of the collaborative robot according to the present invention, while the cooperative robot performing a task according to direct teaching by an operator or a preset workflow performs a set operation, the control of the cooperative robot An RGB camera capable of real-time monitoring of at least one of the position, posture, and motion of the whole or a specific part; wherein the control box further includes, through the vision process, a real-time image (video) monitored through the RGB camera. It can also be provided in the form of a monitoring and control system for intelligent control of collaborative robots, characterized in that information can be generated as a three-dimensional model through a three-dimensional point cloud base.

In addition, the smart teaching pendant pre-designates the location information (x, y, z position, rotation x, rotation y, and rotation z) of the set end of the collaborative robot, and at the same time, the set end of the collaborative robot When the specified teaching position is reached according to the set workflow, the conditions for proceeding to the next task are compared, and when the condition is satisfied, the operator performs the repetitive operation task with the next task designated by the operator. A robot interface app having an interface capable of directly teaching the robot workflow information; The purpose of the present invention described above through the configuration of the monitoring control system for intelligent control of collaborative robots, characterized in that it includes; and a vision interface app that supports matching with vision information obtainable within the working environment where the collaborative robot is located. it is possible to achieve

In addition, the control box may include: a work environment measurement unit that controls the collaborative robot to obtain information about a working environment in which the cooperative robot operation is performed according to a work type and a workflow of the cooperative robot operation; a work information receiving unit configured to control the collaborative robot to obtain work information related to an operation of the cooperative robot according to a work type and a workflow of the cooperative robot operation; and a robot motion control unit for controlling the robot to perform an operation according to the work type and workflow of the cooperative robot operation based on the obtained information on the working environment and the job information on the operation of the collaborative robot. The working environment measuring unit may operate the cooperative robot according to the work type and workflow by direct teaching in which a worker directly moves the cooperative robot to inform the location and movement of the cooperative robot, and the starting point and movement path of the cooperative robot operation. Obtains information on the work environment including a passing point and a destination on the work, and the work information receiving unit is configured according to the work type and workflow by direct teaching in which a worker directly moves the collaborative robot to inform the location and movement. A monitoring and control system for intelligent control of a collaborative robot, characterized in that, while operating the robot, work information on the operation of the cooperative robot including characteristics of the operation of the cooperative robot generated based on the operation of the cooperative robot is acquired. Achievement of the object of the present invention through configuration may also be considered.

In addition, the work environment measurement unit according to the present invention obtains information about the work environment by a set 3D sensor or RGB camera while the collaborative robot performs an operation according to the work type and workflow by the direct teaching. It is also possible to provide in the form of a monitoring control system for intelligent control of collaborative robots, characterized in that acquired.

According to the above object and solution of the present invention, it is possible to implement a monitoring and control platform for a 3D vision-integrated collaborative robot, and through this, behavioral intelligence that recognizes tasks by itself and generates motion, including collision avoidance motion with a worker, etc. It is possible to provide a collaborative robot technology having a.

However, the effect according to the present invention is not limited by the contents exemplified above, and more diverse effects should be included and interpreted within this specification.

1 is a diagram schematically illustrating the configuration of a monitoring control system for intelligent control of a collaborative robot capable of recognizing a deep learning-based 3D task according to an embodiment of the present invention.
2 is a schematic diagram of providing learning data for at least one collaborative robot based on a cloud and performing intelligent control of a collaborative robot based on work information acquired based on an RGB camera, etc., according to an embodiment of the present invention. It is a drawing showing
3 is a diagram showing a concept of responding to a task by tracking a plurality of work objects in 3D dimension through a monitoring control system for intelligent control of a collaborative robot according to an embodiment of the present invention.
4 shows a conceptual diagram of processing for speeding up algorithm operation based on deep learning.

Hereinafter, the present invention will be described with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and, therefore, is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this is not only "directly connected", but also "indirectly connected" with another member in between. "Including cases where In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, with reference to FIGS. 1 to 3, a monitoring control system 100 for intelligent control of a collaborative robot according to the present invention will be described.

1 is a diagram schematically illustrating the configuration of a monitoring control system 100 for intelligent control of a collaborative robot capable of recognizing a deep learning-based 3D task according to an embodiment of the present invention. 2 is a schematic diagram of providing learning data for at least one collaborative robot based on a cloud according to an embodiment of the present invention and performing intelligent control of a collaborative robot based on work information acquired based on an RGB camera, etc. It is a drawing showing 3 is a diagram showing the concept of responding to a task by tracking a plurality of work objects in 3D dimension through the monitoring control system 100 for intelligent control of a collaborative robot according to an embodiment of the present invention.

The monitoring control system 100 for intelligent control of a collaborative robot according to the present invention includes a control box 110 and a smart teaching pendant 130 connected to an internal and external 3D sensor (vision sensor, etc.) and/or an RGB camera. can include Through this, the monitoring control system 100 for intelligent control of collaborative robots can implement a monitoring and control platform for 3D vision-integrated collaborative robots. intelligence can be provided. In addition, it also has a technical feature capable of tracking and specifying work objects having various 3D shapes and shapes moving at high speed in real time through synchronization of the vision processor 111 and the robot controller 113 (see FIG. 3).

The control box 110 according to the present invention transmits information about at least one of the starting point of the robot operation performed by the cooperative robots 1, 2, and n, the passing point and the destination point on the movement path to the 3D sensor 11 and/or Real-time acquisition through cameras (can be 12, RGB cameras, multiple RGB-D cameras, etc.). In addition, the control box 110 according to the present invention is the starting point of the operation of the cooperative robots 1, 2, and n obtained through the 3D sensor 11 and/or the camera 12 (vision information, etc.), and the passing point on the movement path. and an independent device or PC or server capable of real-time monitoring and control of at least one of the location, state, and operation of the collaborative robot ((1, 2, n) based on information on the type of at least one of the destination points, or an IT capable of communication. It can also be a module driven in the device (see Figs. 2 and 3).

To this end, the control box 110 according to the present invention is based on at least one information of the position, state, and operation of the cooperative robots 1, 2, and n monitored in real time through the 3D sensor 11 on a preset 3D coordinate system. may include a vision process 111 capable of inferring the posture of all or a specific part (eg, an end, tip, elbow, etc. of a robot finger, etc.) of the cooperative robots 1, 2, n of .

The vision process 111 according to the present invention may perform a function of mapping pixels of a real-time image obtained through the 3D sensor 11 and/or the camera 12 to a predetermined coordinate system. In this case, the target is a feature defining a coordinate system (e.g., the X-Y-Z axis arrangement of a series of checkerboards), such as a 3D code inlaid into a feature pattern, or a distinct fiducial point defining the pattern coordinate system. ) may be included. That is, as an example, the vision process 111 according to the present invention acquires characteristics of a work object or work environment to be operated by the cooperative robots 1, 2, and n through a vision sensor or a camera, and pixels of the acquired image. Through mapping to , the system can be calibrated to the target. In this case at least one 3D sensor 11 and/or camera 12 used to acquire images of all or part of the calibration target, the characteristics of the target (eg x and y, z along the plane of the target) (height) and rotation about the z-axis in the x-y plane θ) or other (eg global) coordinate system.

In addition, the vision processor 111 according to the present invention may include a built-in calibration application, through which one or more 3D sensors ( 11) and/or the relative position of the separate camera 12 can be inferred, and the fiducial of the target can also be used to orient the 3D sensor 11 and/or the separate camera 12 relative to the target within the respective field of view. there is.

In addition, the vision process 111 according to the present invention uses the depth and RGB images captured through the 3D sensor 11 and/or the camera 12 to create a point cloud following the coordinate system of the depth camera, respectively, by the 3D sensor 11 and the camera 12. / or may be obtained from the camera 12.

The camera 12 according to the present invention can perform the above-described functions as an integrated unit or a single module with the 3D sensor 11, and together with this, direct teaching of the operator or cooperation to perform tasks according to a preset workflow It may be an RGB camera capable of real-time monitoring at least one of the position, posture, and motion of the whole or a specific part of the collaborative robot 1 while the robot 1 performs the set operation. In this case, the control box 110 transforms real-time image (video, still image, etc.) information monitored (acquired) through the camera 12 through the vision process 111 into a 3D model through a 3D point cloud base. can also be created.

In addition, the 3D sensor and/or camera 12 according to the present invention images all or a part of the previously set cooperative robots 1, 2, n (for example, joints or ends capable of specific rotation, bending, etc.) Based on this, it can be sensed (a kind of kinectic sensor), and the data on the movement or operation of the robot joint acquired through this can also be transmitted to the control box 110 and processed.

As a result, the monitoring control system 100 for intelligent control of collaborative robots according to the present invention utilizes 3D sensor data, RGB camera data, and the above-described robot joint data to perform instance segmentation of overlapping specimens in the image, segmentation, and posture on the 3D coordinate system It also has a technical feature that enables inference. To this end, the vision processor 111 according to the present invention integrates at least one or more AI (artificial intelligence)-based reasoning models that have been previously known, so that it is also possible to implement an algorithm for recognition of a 3D workpiece and work motion. (See also Fig. 4, deep learning algorithm operation processing concept).

As shown in Figure 4, the system 100 according to the present invention, more specifically, the robot controller 130 can accurately recognize a three-dimensional workpiece in real time based on AI (artificial intelligence), such An algorithm for automatically generating information such as the posture, motion, position, etc. of the collaborative robot 1 based on the state and information of the workpiece may also be included.

In addition, as shown in FIG. 2, the system 100 according to the present invention communicates data with a plurality of collaborative robots 1, 2, and n through an AI (artificial intelligence)-based reasoning model that is lightweight and optimized ( For example, the collaborative robot 1 may receive (real-time) a video image of a workpiece or the like through a 3D sensor 11 and/or a camera 12 installed in a set interior or exterior space, and at the same time obtain By transmitting the image recognition result data analyzed based on video image information to the cooperating robots (1, 2, n) in real time, it is possible to support the cobots (1) to quickly and accurately perform tasks according to the workpiece characteristics and workflow. do.

In addition, the system 100 according to the present invention transmits learning data to the cooperative robots 1, 2, n or control box 110 according to a situation in which new work processing or response is required in conjunction with a separate cloud learning server. , it is possible to enhance the ability of the collaborative robot to respond to the workpiece, and at the same time, it is possible to minimize the intervention of the operator. In this case, in the case of a learning model that can be adopted, a method of classification by receiving the result of segmentation with Detectron2 from MaskRCNN can be considered.

In addition, in order to achieve the above object of the present invention, the system 100 according to the present invention provides space shape information of a work space (site) where the collaborative robot 1 is installed and operated, joint data and work of the collaborative robot 1 Label data information of the water (pick & place object) may be stored in advance or may be input through a separate external server.

Also, according to another embodiment according to the present invention, the control box 110 may include a work environment measuring unit, a work information receiving unit, and a robot operation control unit.

In this case, the main function of the work environment measurement unit is to control the cooperating robot 1 to acquire information about the work environment in which the cobot 1 operates according to the work type and workflow of the cooperating robot 1. refers to a module with The working environment measurement unit according to the present invention operates the cooperative robot 1 according to the work type and workflow by direct teaching in which a worker directly moves the cooperative robot 1 to inform the position and movement of the cooperative robot 1. Information about the work environment including the start point of the operation, the passing point and the destination point on the movement route may be obtained.

In addition, the work environment measurement unit obtains information about the work environment by the set 3D sensor 11 or camera 12 while the collaborative robot performs an operation according to the work type and workflow by the direct teaching. may also be obtained.

The work information receiving unit refers to a module whose main function is to acquire work information on the operation of the cooperative robot 1 according to the work type and workflow of the operation of the cooperative robot 1 by controlling the cooperative robot 1. . In this case, the work information receiving unit operates the cooperative robot 1 according to the work type and workflow by direct teaching in which the operator directly moves the cooperative robot 1 to inform the location and movement, and based on the operation of the cooperative robot A function of monitoring information about characteristics of the generated collaborative robot motion may be additionally included.

The robot operation control unit causes the collaborative robot to perform an operation according to the work type and workflow of the operation of the cooperative robot 1 based on the acquired information on the working environment and the operation information on the operation of the cooperative robot 1. Refers to a module that performs a function of controlling the robot 1.

The smart teaching pendant 130 according to the present invention is wired and/or wirelessly connected to the control box 110 through the communication network 150, and at least one of the position, state, and operation of the collaborative robot 1 A module capable of maintaining control of at least one of the position, state, and operation of the collaborative robot 1 within the set work environment according to the acquired information and pre-learned patterns of Alternatively, it may be provided in the form of a module (program or algorithm) that can be driven inside an IT device such as a PC or tablet having a display screen.

In addition, the smart teaching pendant 130 according to the present invention designates in advance the location information (x, y, z position and rotation x, rotation y, rotation z) of the set end of at least one collaborative robot 1, and at the same time cooperates When the set end of the robot 1 reaches the designated teaching position according to the set workflow, the condition for proceeding to the next job is compared, and when the condition is satisfied, the operator can perform the repetitive operation with the next job designated by the operator. can be monitored and controlled. In addition, along with (or separately), a robot interface app (App) having an interface capable of directly teaching workflow information to the collaborative robot 1 through the smart teaching pendant 130, input and output is possible through a set display screen. ) and a vision interface app (App) that supports matching with vision information (obtained through the 3D sensor 11 and/or camera 12) obtainable within the work environment where the collaborative robot 1 is located. there is.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

1: Collaborative robot (n)
100: monitoring control system for collaborative robot intelligent control
110: control box
111: vision process
113: robot controller
130: smart teaching pendant
150 wired and/or wireless network

Claims (6)

Information on at least one of a starting point of a robot operation performed by the collaborative robot, a passing point, and a destination point on a movement path is obtained in real time through a 3D sensor; a control box capable of controlling at least one of the position, state, and operation of the collaborative robot based on information about at least one type of a starting point, a passing point, and a destination point on a movement path; and
It is connected to the control box by wire or wirelessly through a communication network, and the position of the cooperative robot in a work environment set according to the pre-learned pattern and at least one information obtained from among the position, state, and operation of the cooperative robot. , a smart teaching pendant capable of maintaining control of at least one of a state and an operation; including,
It is possible to implement a monitoring control platform for 3D vision-integrated collaborative robots, through which it has behavioral intelligence that recognizes tasks, including collision avoidance motions with workers, etc. and generates motions. Monitoring control for intelligent control of collaborative robots. system.
The method of claim 1,
The control box,
A vision process capable of inferring the posture of the whole or a specific part of the collaborative robot on a preset 3D coordinate system based on at least one of the position, state, and motion of the cooperative robot monitored in real time through the 3D sensor. A monitoring control system for intelligent control of collaborative robots.
The method of claim 2,
The monitoring control system for intelligent control of the collaborative robot,
At least one of the position, posture, and motion of the whole or a specific part of the collaborative robot is monitored in real time while the collaborative robot performing a task according to a worker's direct teaching or a preset workflow is performing a set operation. An RGB camera that can be further included,
The control box,
Characterized in that, through the vision process, real-time image (video) information monitored through the RGB camera can be generated as a 3D model through a 3D point cloud base.
Monitoring and control system for intelligent control of collaborative robots.
The method of claim 1,
The smart teaching pendant,
When the position information (x, y, z position, rotation x, rotation y, and z) of the set end of the collaborative robot is designated in advance, and the set end of the collaborative robot reaches the designated teaching position according to the set workflow Compare the conditions for proceeding to the next task, and when the conditions are satisfied, allow the operator to perform the repetitive action task as the next task specified,
A robot interface app having an interface through which a worker can directly teach workflow information to the collaborative robot; and
Characterized in that it includes; a vision interface app that supports matching with vision information obtainable within the work environment where the collaborative robot is located.
Monitoring and control system for intelligent control of collaborative robots.
The method of claim 1,
The control box,
a work environment measuring unit controlling the collaborative robot to obtain information about a working environment in which the cooperative robot operation is performed according to a work type and a workflow of the cooperative robot operation;
a work information receiving unit configured to control the collaborative robot to obtain work information related to an operation of the cooperative robot according to a work type and a workflow of the cooperative robot operation; and
a robot operation control unit controlling the collaborative robot to perform an operation according to the work type and workflow of the cooperative robot operation based on the obtained information on the work environment and the job information on the operation of the collaborative robot; Including,
The working environment measurement unit,
The operator directly moves the collaborative robot and operates the cooperative robot according to the work type and workflow by direct teaching to inform the location and movement, and includes a starting point of the cooperative robot operation, a passing point and a destination point on a movement path. obtain information about the work environment;
The work information receiving unit,
Including characteristics of the cooperative robot operation generated based on the operation of the cooperative robot while operating the robot according to the work type and workflow by direct teaching in which the operator directly moves the cooperative robot to inform the position and movement Characterized in that obtaining work information about the cooperative robot operation,
Monitoring and control system for intelligent control of collaborative robots.
The method of claim 5,
The working environment measurement unit,
Characterized in that information on the work environment is acquired by a set 3D sensor or RGB camera while the collaborative robot performs an operation according to the work type and workflow by the direct teaching,
Monitoring and control system for intelligent control of collaborative robots.

Images