KR102452315B1 - Apparatus and method of robot control through vision recognition using deep learning and marker - Google Patents

Abstract

The present invention relates to an apparatus and method for controlling a robot through vision recognition using deep learning and a marker, and to a gripping target provided at a specific location using a robot in an automated production process to be accommodated at another location When moving to an object, the coordinates are extracted by recognizing the marker attached to the gripping target object, the target object is recognized and the coordinates are extracted through the deep learning model created in advance, and the extracted gripping target By precisely controlling the gripping and dropping of the robot based on the information on the coordinates of the object and the target object, the gripping target can be accurately moved to the target object through deep learning and vision recognition using a marker. It relates to a device capable of and a method therefor.

Description

Robot control device and method through vision recognition using deep learning and markers

The present invention relates to an apparatus and method for controlling a robot through vision recognition using deep learning and a marker, and more particularly, to a gripping target provided at a specific location using a robot in an automated production process at another location When moving to the received target object, the coordinates are extracted by recognizing the marker attached to the gripping target, the coordinates are extracted by recognizing the target object through the deep learning learning model created in advance, and the By precisely controlling the gripping and dropping of the robot based on the extracted information on the coordinates of the gripping target and the target object, the gripping target is moved to the target object through deep learning and vision recognition using a marker. It relates to an apparatus and method capable of accurately moving the

The 4th industrial revolution refers to a change in the industrial environment that maximizes intelligence, automation and connectivity through the convergence of information and communication technology with existing industrial fields. etc. are known as the main technological drivers.

Among them, the robot field is seeking new products and markets based on the development of artificial intelligence technology and information and communication technology. Based on this, the public and market interest in robots is growing.

When the robot is applied to logistics or delivery, the robot is moved to a desired position and then the robot is gripped through the robot arm to perform the operation of loading, collecting, sorting, and packaging. In this case, the role of computer vision is very important.

The computer vision is a field of artificial intelligence that uses a computer to reproduce the general human visual recognition ability. It is used to check posture.

However, when moving the robot to a desired position and then gripping a predetermined object through the robot arm, or dropping the gripped object to another object provided at a specific location to perform an operation such as combining, If the position or posture of an object is not accurately recognized through a camera or sensor, the gripping or dropping operation becomes inaccurate.

Therefore, in the present invention, the coordinates are extracted by recognizing the marker attached to the gripping target provided at a specific location, and the coordinates are extracted by recognizing the target object provided at another location through the deep learning learning model created in advance. , A method to increase the accuracy when transferring the gripping object to the accommodated object by precisely controlling the gripping and dropping of the robot based on the extracted coordinate information of the gripping object and the accommodated object would like to present

Next, the prior art existing in the technical field of the present invention will be briefly described, and then the technical matters that the present invention intends to achieve differently from the prior art will be described.

First, Korean Patent Application Laid-Open No. 2020-0069713 (2020.06.17.) relates to a method of generating a work path of a welding robot, attaching a marker to an arbitrary position on each side of the work object, It relates to a method of generating a work path of a welding robot that takes a picture and creates a work path through a teaching manipulator and a photographed marker and vision recognition.

That is, the prior art describes a method of generating a working path of a welding robot intended to generate a working path of the welding robot by recognizing the shape of the work object, extracting the direction of the welding line and the length of the welding line.

However, in the present invention, the coordinates of the gripping target extracted through the marker attached to the gripping target and the position and posture of the target object estimated using the deep learning learning model created in advance. Since the gripping and dropping of the robot is precisely controlled based on information on the coordinates of the target object, there is a significant difference between the prior art and the present invention in configuration.

In addition, Korea Patent Publication No. 2018-0114698 (2018.10.19.) relates to an efficient learning model and learning technique for generating the work trajectory of a multi-joint robot, and completes the task based on the cognitive information processed by the robot's sensing data. It is about a technique for efficiently generating robot joint trajectories for

That is, the prior art proposes a learning model for generating the trajectory of the articulated robot, reduces the amount of computation to ensure real-time, and provides a learning algorithm that can approximate the optimal trajectory for completing the task A system and method thereof are described.

On the other hand, in the present invention, the gripping and dropping of the robot is precisely performed based on the coordinates of the gripping target extracted using a marker and information about the coordinates of the target object extracted using the deep learning learning model created in advance. Therefore, the difference in technical configuration between the prior art and the present invention is clear.

The present invention was created to solve the above problems, and through deep learning and vision recognition using a marker, when moving a gripping target provided at a specific location to a receiving target provided at another location, An object of the present invention is to provide an apparatus and method capable of precisely controlling gripping and dropping of a robot.

In addition, the present invention provides an apparatus and method for recognizing a marker attached to the gripping target from an image captured by a camera provided in the robot, and extracting the coordinates of the gripping target through the recognized marker. provide for a different purpose.

In addition, the present invention generates a deep learning learning model by learning a dataset photographed from various angles with respect to the target object, and inputting an image photographed with a camera provided in the robot into the created deep learning learning model. Another object of the present invention is to provide an apparatus and method capable of extracting the position and posture of an accommodation target object, and extracting coordinates of the accommodation target object through the extracted location and posture of the accommodation target object.

In addition, the present invention provides the target object extracted through the coordinates of the gripping target extracted through the marker attached to the gripping target and the position and posture of the target target estimated using the deep learning learning model created in advance. Another object of the present invention is to provide an apparatus and method that can precisely control the robot's gripping and dropping based on the information on the coordinates of the object to be accommodated.

According to an embodiment of the present invention, there is provided an apparatus for controlling a robot through deep learning and vision recognition using a marker, comprising: a marker recognition unit for recognizing a marker from an image taken at a marker recognition position of a gripping object; A target object recognition unit for recognizing the target object by applying the image taken at the recognition position of the target object to a preset deep learning learning model; and a robot arm control unit controlling the gripping target object to be gripped and dropped to the receiving target object based on the recognition of the marker and the recognition of the target object; including, the marker and the target object It is characterized by recognizing through vision recognition.

In addition, the marker is characterized by using an ArUco marker of an n*n two-dimensional pattern combining white cells and black cells, or a chArUco marker in which a check board and four ArUco markers are arranged in a diamond shape.

In addition, the robot control device, a learning unit for generating a deep learning learning model for estimating the position and posture of the target object by learning the dataset obtained by photographing the target object from a plurality of angles; Further comprising, the dataset includes an image obtained by photographing the object to be accommodated from at least four different angles to include the object to be accommodated, and the three-dimensional modeling after three-dimensional modeling of the captured image An image is generated based on a mask that is binarized by back-projecting an image onto the captured image, and the learning unit calculates the mapping relationship between the image captured by the target object of the generated dataset and the mask that has been binarized, Mask R- It is characterized in that it further comprises generating a deep learning learning model by learning with the CNN model.

In addition, the marker recognition unit, at a marker recognition position for recognizing the marker attached to the gripping target object, multiplies the transformation matrix from the base of the robot to the recognized marker and the transformation matrix from the base of the robot to the robot arm, obtaining a transformation matrix from the marker to the robot arm; obtaining a transformation matrix from the robot arm to the gripping object with reference to X, Y, and Z-axis distance information from the robot arm to the gripping object based on the coordinate system of the robot arm; By multiplying the transformation matrix from the base of the robot to the recognized marker, which is changed as the robot moves to the execution position for gripping the gripping target object, and the transformation matrix from the marker obtained at the marker recognition position to the robot arm, obtaining a transformation matrix from the base of the robot to the robot arm; The transformation matrix from the base of the robot to the robot arm obtained at the execution position is multiplied by the transformation matrix from the robot arm to the gripping object obtained from the marker recognition position, and finally the base of the robot to the gripping target object extracting the coordinates of the object to be gripped by obtaining a transformation matrix into It is characterized in that the robot arm can be moved to the gripping target object even if the positions of the base and the marker of the robot are changed.

In addition, the accommodated object recognition unit, at the recognition position of the accommodated object, multiplies the transformation matrix from the base of the robot to the accommodated object and the transformation matrix from the base of the robot to the robot arm to multiply the received object to obtain a transformation matrix of the robot arm; extracting a rotation matrix from the transformation matrix of the obtained target object to the robot arm to obtain a transformation matrix from the new target object to the robot arm from which a position vector is deleted; Multiplying the obtained transformation matrix from the new target object to the robot arm and the obtained transformation matrix from the base of the robot to the target object to obtain a final transformation matrix from the base of the robot to the target object by obtaining, extracting the coordinates of the target object; Through the obtained final transformation matrix from the base of the robot to the target object, it is characterized in that the robot arm does not follow the coordinate axis of the target object.

In addition, the robot control method through deep learning and vision recognition using a marker according to an embodiment of the present invention, in the robot control device, a marker recognition step of recognizing a marker from an image taken at a marker recognition position of a gripping object ; an accommodation target object recognition step of recognizing an accommodation target object by applying, in the robot control device, an image captured at a recognition position of the accommodation target object to a preset deep learning learning model; and a robot arm control step of controlling, in the robot control device, to grip the gripping object and drop it to the accommodated object based on the recognition of the marker and the recognition of the accommodated object; It is characterized in that the marker and the target object are recognized through vision recognition.

In addition, the robot control method, in the robot control device, by learning a dataset photographed from a plurality of angles with respect to the target object to generate a deep learning learning model for estimating the position and posture of the target object Learning step; further comprising, wherein the dataset is obtained by three-dimensional modeling of an image obtained by photographing the object to be accommodated from at least four different angles to include the object to be accommodated, and the captured image The three-dimensional modeled image is back-projected onto the captured image and generated based on a binarized mask, and the learning step is a mapping of an image captured by the target object of the generated dataset and the binarized mask. It is characterized in that it further comprises generating a deep learning learning model by learning the relationship with the Mask R-CNN model.

In addition, in the marker recognition step, at a marker recognition position for recognizing a marker attached to the gripping target object, a transformation matrix from the base of the robot to the recognized marker is multiplied by a transformation matrix from the base of the robot to a robot arm. obtaining a transformation matrix from the marker to the robot arm; obtaining a transformation matrix from the robot arm to the gripping object with reference to X, Y, and Z-axis distance information from the robot arm to the gripping object based on the coordinate system of the robot arm; By multiplying the transformation matrix from the base of the robot to the recognized marker, which is changed as the robot moves to the execution position for gripping the gripping target object, and the transformation matrix from the marker obtained at the marker recognition position to the robot arm, obtaining a transformation matrix from the final robot base to the robot arm; and the transformation matrix from the base of the robot to the robot arm obtained at the execution position multiplied by the transformation matrix from the robot arm to the gripping target object obtained from the marker recognition position to finally obtain the gripping target at the base of the robot. Obtaining a transformation matrix into an object; by performing, the coordinates of the gripping object are extracted, and even if the positions of the base and the marker of the robot are changed, the robot arm can be moved to the gripping object characterized by having

In addition, the step of recognizing the target object may include multiplying the transformation matrix from the base of the robot to the target object by the transformation matrix from the base of the robot to the robot arm at the recognition position of the target object to obtain the target object. obtaining a transformation matrix from the object to the robot arm; extracting a rotation matrix from the obtained transformation matrix of the received target object to the robot arm, and obtaining a transformation matrix from the new received target object to the robot arm from which a position vector is deleted; and a final transformation matrix from the base of the robot to the target object by multiplying the obtained transformation matrix from the new target object to the robot arm and the obtained transformation matrix from the base of the robot to the target object. By performing a; to extract the coordinates of the accommodated object, and through a transformation matrix from the obtained final robot base to the accommodated object, the robot arm is It is characterized in that it does not follow the coordinate axis of .

As described above, according to the robot control apparatus and method through deep learning and vision recognition using markers of the present invention, the coordinates are extracted by recognizing the marker attached to the gripping target provided at a specific position, and the Through the deep learning learning model, the coordinates are extracted by recognizing the target object provided in another location, and the gripping and dropping of the robot is precisely performed based on the extracted coordinates of the gripping object and the target object. By controlling, there is an effect of accurately moving the gripping target to the receiving target through deep learning and vision recognition using a marker.

1 is a view for explaining the overall configuration of a robot control device through deep learning and vision recognition using a marker according to an embodiment of the present invention.
2 is a diagram showing in more detail the configuration of a robot control apparatus according to an embodiment of the present invention.
3 and 4 are views each showing an example of a marker applied to the present invention.
5 is a view for explaining in detail the recognition process of the ArUco marker applied to the present invention.
6 is a view for explaining in detail the recognition process of the chArUco marker applied to the present invention.
7 is a diagram for explaining the process of the Mask R-CNN model applied to the present invention.
8 is a diagram for explaining a process of generating a dataset used for learning using the Mask R-CNN model applied to the present invention.
9 is a flowchart illustrating in detail an operation process of a robot control method through deep learning and vision recognition using a marker according to an embodiment of the present invention.
10 is a flowchart illustrating in detail an operation process for performing gripping of a robot control method according to an embodiment of the present invention.
11 is a flowchart illustrating in detail an operation process for performing a drop in a method for controlling a robot according to an embodiment of the present invention.
12 and 13 are diagrams for explaining a process of extracting coordinates of a gripping target object through marker recognition applied to the present invention.
14 is a view for explaining a process of extracting coordinates of an accommodated object through recognition of the accommodated object applied to the present invention.

Hereinafter, a preferred embodiment of a robot control apparatus and method through deep learning and vision recognition using a marker of the present invention will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements. In addition, specific structural or functional descriptions of the embodiments of the present invention are only exemplified for the purpose of describing the embodiments according to the present invention, and unless otherwise defined, all terms used herein, including technical or scientific terms They have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. It is preferable not to

1 is a view for explaining the overall configuration of a robot control device through deep learning and vision recognition using a marker according to an embodiment of the present invention.

As shown in FIG. 1 , the present invention provides a robot control device 100 , a robot 200 , a gripping target object 300 provided with a marker 310 at a predetermined position, a receiving target object 400 , and a database. (500) and the like.

The robot control device 100 grips the gripping target object 300 located in a specific workspace through the robot 200 through vision recognition using a deep learning learning model and a marker in various production processes. and control to drop the gripping target object 300 to the receiving target object 400 located in another work space.

At this time, the robot control device 100 recognizes the marker 310 attached to the gripping target object 300 from the image captured by the camera provided in the robot 200, and the direction of the marker 310 or The coordinates of the gripping target object 300 may be extracted through the position.

In addition, the robot control device 100 learns a dataset photographed from various angles with respect to the target object 400 to generate a deep learning learning model, stores it in the database 500 and manages it, and the robot ( 200), the position and posture of the target object 400 are estimated by inputting the image taken with the camera provided in the generated deep learning learning model, and the coordinates of the target object can be extracted based on this. have. In this case, the deep learning learning model may be continuously updated and managed.

That is, the robot control device 100 determines the position and posture of the target object 400 estimated using the coordinates of the gripping target 300 extracted through the marker 310 and the deep learning learning model. It is to precisely control the gripping and dropping of the robot 200 based on the information on the coordinates confirmed through it.

Specifically, the robot control device 100 controls the driving of the robot 200 to move the robot arm to the workspace #1 where the gripping target object 300 is provided (①), and the camera The specific coordinates are confirmed by recognizing the marker 310 provided at a specific position of the gripping target 300 through the image taken with the .

In addition, the robot control device 100 grips the gripping target 300 by moving the robot 200 to the coordinates of the gripping target 300 confirmed according to the recognition of the marker 310 . control to do so (③).

Next, the robot control device 100 moves the gripping target object 300 gripped by the robot 200 to the workspace #2 provided with the accommodated target object 400 (④) , the image taken with the camera is applied to the previously generated deep learning learning model to confirm the specific coordinates of the target object 400 (⑤).

In addition, the robot control device 100 moves the robot 200 onto the accommodated object 400 and grips the gripping object 300 based on the identified coordinates of the accommodated object 400 . ) is controlled to drop on the target object 400 (⑥).

2 is a diagram showing in more detail the configuration of a robot control apparatus according to an embodiment of the present invention.

As shown in FIG. 2 , the robot control apparatus 100 according to an embodiment of the present invention includes a learning unit 110 , an image receiving unit 120 , a marker recognition unit 130 , and a target object recognition unit 140 . ), a robot arm control unit 150, a memory 160, and the like.

The learning unit 110 learns a dataset obtained by photographing from a plurality of angles with respect to the received target object 400 , and deep learning for estimating the position and posture of the received target object 400 . A model is generated, and the generated deep learning learning model is stored and managed in the database 500 . In this case, the deep learning model creation process or the setting of the dataset will be described in detail with reference to FIGS. 7 and 8 .

The image receiving unit 120 receives the image taken from the camera provided in the robot 200, pre-processes the received image to the marker recognition unit 130 and the target object recognition unit 140. transmit

The marker recognition unit 130 performs a function of recognizing the marker 310 from the image taken at the marker recognition position of the gripping target object 300 .

That is, the marker recognition unit 130 identifies the marker 310 attached to the gripping target 300 from the image provided from the image receiving unit 120 , and determines the direction or rotation of the identified marker 310 . Through this, the coordinates of the gripping target object 300 are recognized. In this case, the process of recognizing the coordinates of the gripping target object 300 will be described in detail with reference to FIGS. 12 and 13 .

The received target object recognition unit 140 applies the image captured at the recognition position of the received target object to the deep learning learning model created in advance through the learning unit 110 to the received target object 400 . It performs the function of recognizing

That is, the target object recognition unit 140 inputs the image received from the image receiving unit 120 into the deep learning learning model, estimates the direction and posture with the highest probability, and calculates the estimated direction and posture. Through this, the coordinates of the gripping target object 300 are recognized. In this case, the process of recognizing the coordinates of the target object 400 will be described in detail with reference to FIG. 14 .

The robot arm control unit 150 grips the gripping target object 300 based on the recognition of the marker 310 and the recognition of the accommodated object 400 to be used as the accommodated object 400 . control to drop.

That is, the robot arm control unit 150 moves the robot arm of the robot 200 based on the information on the coordinates of the gripping target object 300 confirmed through vision recognition by the marker recognition unit 130 to move the robot arm. Control to grip the ripping target object 300, and based on the information on the coordinates of the target object 400 recognized by the target object recognition unit 140 through vision recognition, the robot 200 It controls to drop the gripping target object 300 to the target object 400 by moving the robot arm.

The memory 160 stores various operation programs used in the robot control device 100 .

Meanwhile, in the present invention, various types of markers may be used to accurately recognize the marker 310 provided in the gripping target 300 . This will be described in detail with reference to FIGS. 3 to 6 as follows.

3 and 4 are views each showing an example of a marker applied to the present invention.

3 shows an example of an ArUco marker, and is composed of a two-dimensional pattern of n*n size and a black border region surrounding it.

The black border area is formed to quickly recognize the marker, and the two-dimensional pattern inside expresses the unique ID of the marker by combining the white cell and the black cell, and is used to identify the marker.

4 shows an example of a chArUco marker, in which a check board and four ArUco markers are arranged in a diamond shape.

The chArUco marker is easy to detect because the ArUco marker is located between the checkboards with clear color classification, and a 3D axis is generated at the center of the checkboard by calculating the relative positions of the four ArUco markers, and based on the generated 3D axis to perform posture extraction.

5 is a view for explaining in detail the recognition process of the ArUco marker applied to the present invention.

As shown in FIG. 5 , in order to detect the ArUco marker, an image captured by a camera is converted to gray scale, binarized, and an outline is extracted.

Next, in order to analyze the ID of the ArUco marker, the obliquely photographed image is converted into a shape viewed from the front, and then the number of white pixels excluding the black pixel is counted to extract the ID of the ArUco marker.

In addition, a three-dimensional coordinate axis is generated at the center of the ArUco marker through the four vertices of each corner of the ArUco marker and a unique ID.

Since the 3D coordinate axis of the ArUco marker remains constant even when the vertex rotates, it is possible to check the movement vector and rotation from the camera to the marker using this.

On the other hand, the ArUco marker has the advantage of fast detection and small size, but when the edge of the edge is blurred by the surrounding environment such as brightness, the camera does not recognize it, and when a single ArUco marker is used, the axis shakes due to noise. There is a disadvantage in that it is difficult to establish an accurate three-dimensional coordinate axis.

6 is a view for explaining in detail the recognition process of the chArUco marker applied to the present invention.

As shown in FIG. 6 , the chArUco marker is recognized from the image captured by the camera, and four ArUco markers and the center of the checkboard are extracted. Accordingly, a three-dimensional coordinate axis is created at the center of the checkboard by calculating the relative positions of the four ArUco markers, and using this, the movement vector and rotation from the camera to the marker can be checked.

On the other hand, the chArUco marker is easier to detect compared to the ArUco marker because the ArUco marker is located between the checkboards with clear color distinction, and is not significantly affected by the surrounding environment such as brightness, so in the present invention, the chArUco marker It is preferable to use

Next, in relation to the creation of the deep learning learning model and the setting of the dataset, it will be described in more detail with reference to FIGS. 7 and 8 .

7 is a diagram for explaining the process of the Mask R-CNN model applied to the present invention.

As shown in FIG. 7 , in the present invention, ResNet of the Mask R-CNN model is used as a deep learning learning model for recognizing the target object 400 . The Mask R-CNN model is a form in which FPN (Feature Pyramid Network) and RoI (Region of Interest) align are added to the Faster R-CNN model.

The FPN generates a feature map step by step while passing through the layers, and performs detection by combining features while descending from the uppermost layer. function to prevent

In addition, the RoI align uses a linear interpolation method for one pixel to make segmentation of the position information clear, so that the object detection in the form of a bounding box in the past can be distinguished more clearly in the form of a mask. do.

Meanwhile, although the present invention has been described as an example of generating a deep learning learning model for recognizing a target object through a Mask R-CNN model, it is not limited thereto, and various other models can be applied.

8 is a diagram for explaining a process of generating a dataset used for learning using the Mask R-CNN model applied to the present invention.

As shown in FIG. 8 , a dataset applied to the Mask R-CNN model in order to generate a deep learning learning model for recognizing a target object is generated as follows.

In order to generate the data set, the actual target object must be photographed at various angles.

In order to set the detection range as described above, in the present invention, a shooting range is selected to include the target object at at least four different angles, and the target object is photographed at the selected at least four different angles. (Example: 100 sheets).

In addition, after three-dimensional modeling of the captured image, the three-dimensional modeled image is back-projected onto the captured image to be subjected to a binarization mask, and the image of the target object photographed from different angles and the data as a binarization mask, respectively. create a set

Accordingly, the robot control device 100 generates a deep learning learning model by learning the mapping relationship between the captured image of the generated dataset and the binarized mask through the Mask R-CNN model.

Next, an embodiment of a robot control method through deep learning and vision recognition using a marker according to the present invention configured as described above will be described in detail with reference to FIGS. 9 to 14 . In this case, the order of each step according to the method of the present invention may be changed by the environment of use or by those skilled in the art.

9 is a flowchart illustrating in detail an operation process of a robot control method through deep learning and vision recognition using a marker according to an embodiment of the present invention.

As shown in FIG. 9 , the robot control apparatus 100 according to an embodiment of the present invention learns a data set photographed from a plurality of angles with respect to the target object to determine the position and posture of the target object. A learning step of generating a deep learning learning model for estimating is performed (S100).

In this case, the dataset includes an image obtained by photographing the object to be accommodated from at least four different angles to include the object to be accommodated, and the three-dimensional modeled image after three-dimensional modeling of the captured image. It is generated based on a mask that has been binarized by back-projecting onto the captured image, and the robot control device 100 calculates the mapping relationship between the image of the target object of the generated data set and the binarized mask in Mask R. -Create a deep learning learning model by learning through a CNN model.

After generating the deep learning learning model for recognizing the target object through step S100, the robot control device 100 starts the operation using the robot 200 in the production process (S200).

Subsequently, the robot control device 100 recognizes a marker 310 provided at a predetermined position of the gripping target 300 from an image captured at the marker recognizing position of the gripping target 300 in a marker recognition step. and driving the robot arm of the robot 200 based on the recognized coordinates of the marker 310 to grip the gripping target object 300 (S300). That is, the robot control device 100 controls to grip the gripping target object 300 based on the recognition of the marker.

At this time, the process of gripping the gripping target object 300 in the robot control device 100 through the step S300 will be described in more detail with reference to FIG. 10 .

10 is a flowchart illustrating in detail an operation process for performing gripping of a robot control method according to an embodiment of the present invention.

As shown in FIG. 10 , when the operation is executed, the robot control device 100 controls the movement of the robot 200 to the marker recognition position (S301), and the gripping target object 300 at the corresponding marker recognition position. ), it is determined whether a specific marker 310 attached to it is detected (S302).

When a marker is detected as a result of the determination in step S302, the robot control device 100 grips the gripping target object 300 by moving the robot arm of the robot 200 to the coordinates of the detected marker (S303). ).

In addition, the robot control apparatus 100 determines whether an error occurs in the gripping execution process in step S303 (S304), and if no error occurs as a result of the determination in step S304, in step S400, dropping the gripped object If an error occurs as a result of the determination in step S304, the robot 200 is reset (S305), and the robot arm is moved to the marker recognition position to perform the step S301 again to find the marker.

However, if the marker is not detected as a result of determination in step S302, the robot control device 100 rotates the camera provided in the robot 200 (S306), and then finds the marker again (S307), and according to the result It is determined whether a marker is detected (S308).

If the marker is detected as a result of the determination in step S308, the step S303 for the gripping operation is performed, and if the marker is not detected as a result of the determination in step S308, the variable is increased according to the position change and the number of movements of the robot arm (S309) .

Subsequently, it is determined whether the variable reaches a preset specific number (eg, 9) (S309), and if the variable reaches a preset specific number as a result of the determination in step S309, the z-axis value of the robot arm is corrected ( S310), move the robot arm to the marker recognition position and perform the step S301 to find the marker again, and as a result of the determination in step S309, if the variable does not reach a preset specific number, the camera is rotated to find the marker S306 Do the steps again.

Referring back to FIG. 9 , after the marker recognition and the gripping operation are performed in step S300 , the robot control device 100 avoids the robot 200 gripping the gripping target object 300 . After moving to the recognition position of the receiving target object 400, the image taken at the recognition position of the receiving target object 400 is applied to the deep learning learning model generated in step S100 to apply the received target object 400 ) is performed, and the robot arm of the robot 200 is driven based on the recognized coordinates of the target object 400 to drive the gripping target object 300 ) to the target object 400 to be accommodated (S400). That is, the robot control device 100 controls to drop the gripped gripping target object 300 to the receiving target object 400 based on the recognition of the receiving target object 400 .

At this time, the process of dropping the gripping target object 300 to the receiving target object 400 in the robot control device 100 through the step S400 will be described in more detail with reference to FIG. 11 as follows.

11 is a flowchart illustrating in detail an operation process for performing a drop in a method for controlling a robot according to an embodiment of the present invention.

As shown in FIG. 11 , the robot control device 100 moves the gripping target 300 gripped through the step S300 to the drop position, and the robot 200 moves to the target object recognition position. control the movement of (S401), and determine whether the target object 400 is detected at the object recognition position (S402).

If the target object 400 is detected as a result of the determination in step S402, the robot control device 100 moves the robot arm of the robot 200 to the coordinates of the detected target object 400 for gripping. One of the gripping target objects 300 is dropped onto the receiving target object 400 (S403).

In addition, the robot control device 100 determines whether an error occurs in the drop execution process in step S403 (S404), and if an error does not occur as a result of the determination in step S404, the drop operation is terminated and the next step S500 is performed, , if an error occurs as a result of the determination in step S404, the robot 200 is reset (S405), and the robot arm is moved to the object recognition position to perform the step S401 again to find the target object 400 to be accommodated.

However, if the marker is not detected as a result of the determination in step S402, the robot control device 100 moves the robot arm of the robot 200, increases the variable according to the position change and the number of movements of the robot arm (S406) , it is determined whether an object is detected (S407).

If the object is detected as a result of the determination in step S407, the step S403 for the drop operation is performed, and if the object is not detected as a result of the determination in step S407, a variable is set in advance according to the position change and the number of movements of the robot arm ( Yes: it is determined whether 9) is reached (S408).

As a result of the determination of step S408, if the variable reaches a preset specific number, the z-axis value of the robot arm is corrected (S409), and then the step S401 of recognizing the object by moving the robot arm to the object recognition position is performed again. , if it is determined in step S408 that the variable does not reach a predetermined specific number, the robot arm is moved and the step S406 of increasing the variable according to the position change and the number of movements of the robot arm is performed again.

Referring back to FIG. 9 , the marker 310 and the target object 400 are recognized through vision recognition, and the gripping target object 300 is gripped and dropped to the target object 400 . After that, the robot control device 100 determines whether all operations are finished (S500), and repeats steps S200 to S400 until all operations are finished.

Meanwhile, an operation process for extracting coordinates of the gripping target object 300 when gripping is performed in step S300 will be described in more detail with reference to FIGS. 12 and 13 as follows.

12 and 13 are diagrams for explaining a process of extracting coordinates of a gripping target object through marker recognition applied to the present invention.

As shown in FIG. 12 , the robot control device 100 recognizes the marker recognition position at the base of the robot at the marker recognition position for recognizing the marker 310 attached to the gripping target object 300 . A transformation matrix ( ^R T _M ) to one marker and a transformation matrix ( ^R T _T ) from the base of the robot to the robot arm are obtained, respectively.

Then, the robot control device 100 obtains a transformation matrix ( ^M T _T ) from the marker to the robot arm as in Equation 1. That is, by multiplying the transformation matrix from the base of the robot to the marker recognized at the marker recognition position ( ^R T _M ) and the transformation matrix from the base of the robot to the robot arm ( ^R T _T ), the transformation matrix from the marker to the robot arm ( ^M T ) _T ) is to be found.

[Equation 1]

At this time, the position from the robot arm of the robot to the marker is fixed. For example, in the robot arm of the robot, if the marker is set to a position moved 0.07 m in the X axis, 0.26 m in the Y axis, and 0.05 m in the Z axis, the transformation matrix from the robot arm of the robot to the gripping target object ( ^T T _C ) is obtained as in Equation (2). That is, the transformation matrix from the robot arm to the gripping object 300 can be obtained by referring to the X, Y, and Z axis distance information from the robot arm to the gripping object 300 based on the coordinate system of the robot arm. have. In this case, the numerical value for the distance may be changed according to the position of the gripping target object.

[Equation 2]

In this way, after obtaining the axis angle of the robot arm at the marker recognition position, the transformation matrix from the marker to the robot arm ( ^M T _T ), and the transformation matrix from the robot arm to the gripping target object ( ^T T _C ), control the robot The device 100 moves the robot arm of the robot to an execution position for performing gripping.

At this time, when the robot 200 moves to a position for gripping the gripping target object 300 , the position of the robot 200 may be changed from the marker recognition position, so that the robot control device 100 ), as shown in FIG. 13, finds a marker at the changed execution position, and then obtains a transformation matrix ( ^R T _M `) from the changed base of the robot to the recognized marker.

Next, the robot control device 100 converts a transformation matrix from the base of the robot to the recognized marker as in Equation 3 ( ^R T _M `) and a transformation matrix from the marker obtained at the marker recognition position to the robot arm. Multiply by ( <sup>M</sup> T <sub>T</sub> ) to obtain the final transformation matrix ( <sup>R</sup> T <sub>T</sub> ` ) from the base of the robot to the robot arm.

[Equation 3]

In addition, the robot control device 100 converts the transformation matrix ( ^R T _T `) from the base of the robot to the robot arm obtained from the execution position as in Equation 4 and the robot arm obtained from the marker recognition position to the gripping target object. The transformation matrix ( ^R T _C ) from the base of the robot to the gripping target is obtained by multiplying the transformation matrix ( ^T T _C ).

[Equation 4]

Accordingly, the coordinates of the gripping object 300 are extracted, and even if the positions of the base and the marker of the robot 200 are changed, the robot arm can be moved to the gripping object 300 .

On the other hand, the operation process for extracting the coordinates of the target object 400 to which the gripping target object 300 is dropped through the step S400 will be described in more detail with reference to FIG. 14 .

14 is a view for explaining a process of extracting coordinates of an accommodated object through recognition of the accommodated object applied to the present invention.

As shown in 14, the robot control device 100 at the recognition position of the target object, the transformation matrix from the base of the robot to the target object ( ^R T _O ) and the robot arm from the base of the robot to the robot arm. Each of the transformation matrices ( ^R T _T ) is obtained.

Accordingly, the robot control device 100 converts the transformation matrix from the base of the robot to the target object ( ^R T _O ) and the transformation matrix from the base of the robot to the robot arm ( ^R T _T ) as in Equation 5. Multiply, to obtain the transformation matrix ( ^OT _T ) from the target object to the robot arm.

[Equation 5]

In addition, the robot control device 100 extracts only the rotation matrix from the transformation matrix ( ^OT _T ) from the target object to the robot arm as in Equation 6 and deletes the position vector from the target object to the robot arm. Create a transformation matrix ( ^O T _T `).

[Equation 6]

In addition, the robot control device 100 is a transformation matrix from the new target object to the robot arm ( ^OT _T `) and the transformation matrix from the base of the robot to the target object ( <sup>R</sup> T ) generated as in Equation 7 <sub>O</sub> ) to obtain a transformation matrix ( <sup>R</sup> T <sub>O</sub> ` ) from the final robot base to the target object.

[Equation 7]

Accordingly, the coordinates of the target object 400 are extracted, and through the transformation matrix ( ^R T _O `) from the obtained final robot base to the target object, the robot arm is moved to the target object. You can avoid following the coordinate axes of .

That is, when the target object 400 is moved by converting it into the coordinate axis of the robot arm, the transformation matrix from the obtained final robot base to the target object can be maintained so that the robot arm can maintain a vertical view. Through ( ^R T _O ` ), the robot arm does not follow the coordinate axis of the target object 400 .

As such, the present invention extracts coordinates by recognizing a marker attached to a gripping target provided at a specific location, and extracts coordinates by recognizing a target object provided at another location through a deep learning learning model created in advance. and precisely control the gripping and dropping of the robot based on the extracted coordinates of the gripping target and the receiving target, so that the gripping target can be accurately moved to the receiving target. .

As described above, the present invention has been described with reference to the embodiments shown in the drawings, which are merely exemplary, and those of ordinary skill in the art can make various modifications and equivalent other embodiments therefrom. You will understand that it is possible. Therefore, the technical protection scope of the present invention should be judged by the following claims.

100: robot control device 110: learning unit
120: image receiving unit 130: recognition unit
131: marker recognition unit 132: target object recognition unit
140: coordinate extraction unit 141: gripping target object coordinate extraction unit
142: target object coordinate extraction unit 150: robot arm control unit
160: memory 200: robot
300: gripping target object 310: marker
400: target object 500: database

Claims (7)

a learning unit for generating a deep learning learning model for estimating the position and posture of the target object by learning a dataset obtained by photographing the target object from a plurality of angles;
a marker recognition unit for recognizing a marker from an image captured at a marker recognizing position of the gripping target;
a target object recognition unit for recognizing the target object by applying the image taken at the recognition position of the target object to the generated deep learning learning model; and
A robot arm control unit that grips the gripping target object based on recognition of the marker and the target object and controls to drop the gripping target object to the target object;
The dataset includes an image obtained by photographing the object to be accommodated from at least four different angles to include the object to be accommodated, and the three-dimensional modeled image after three-dimensional modeling of the captured image. It is created based on a mask that has been binarized by back-projecting the captured image.
A robot control device through vision recognition using deep learning and a marker, characterized in that the marker and the target object are recognized through vision recognition. The method according to claim 1,
The marker is
ArUco markers in an n*n two-dimensional pattern with a combination of white and black cells, or
A robot control device through deep learning and vision recognition using markers, characterized in that a checkboard and 4 ArUco markers are arranged in a diamond shape using chArUco markers. The method according to claim 1,
The learning unit,
Vision using deep learning and markers, characterized in that it further comprises generating a deep learning learning model by learning the image obtained by photographing the target object of the generated dataset and the mask subjected to binarization with the Mask R-CNN model. Robot control through recognition. The method according to claim 1,
The marker recognition unit,
obtaining a transformation matrix from the marker to the robot arm by multiplying the transformation matrix from the base of the robot to the recognized marker at the marker recognition position and the transformation matrix from the base of the robot to the robot arm;
obtaining a transformation matrix from the robot arm to the gripping object with reference to X, Y, and Z-axis distance information from the robot arm to the gripping object based on the coordinate system of the robot arm;
By multiplying the transformation matrix from the base of the robot to the recognized marker, which is changed as the robot moves to the execution position for gripping the gripping target object, and the transformation matrix from the marker obtained at the marker recognition position to the robot arm, obtaining a transformation matrix from the base of the robot to the robot arm;
The transformation matrix from the base of the robot to the robot arm obtained at the execution position is multiplied by the transformation matrix from the robot arm to the gripping object obtained from the marker recognition position, and finally the base of the robot to the gripping target object extracting the coordinates of the object to be gripped by obtaining a transformation matrix into
A robot control device through deep learning and vision recognition using a marker, characterized in that it is possible to move the robot arm to the gripping target object even if the positions of the base and the marker of the robot are changed. The method according to claim 1,
The receiving target object recognition unit,
At the recognition position of the target object, the transformation matrix from the target object to the robot arm is obtained by multiplying the transformation matrix from the base of the robot to the target object and the transformation matrix from the base of the robot to the robot arm. seek;
extracting a rotation matrix from the transformation matrix of the obtained target object to the robot arm to obtain a transformation matrix from the new target object to the robot arm from which a position vector is deleted;
By multiplying the obtained transformation matrix from the new target object to the robot arm and the obtained transformation matrix from the base of the robot to the target object, a final transformation matrix from the base of the robot to the target object extracting the coordinates of the target object by obtaining
Vision recognition using deep learning and markers, characterized in that the robot arm does not follow the coordinate axis of the target object through the obtained final transformation matrix from the base of the robot to the target object through the robot control system. A learning step of generating, in the robot control device, a deep learning learning model for estimating the position and posture of the accommodated object by learning a dataset obtained by photographing the object from a plurality of angles;
a marker recognition step of recognizing a marker from an image taken at a marker recognition position of the gripping target;
an accommodation target object recognition step of recognizing the accommodation target object by applying the image taken at the recognition position of the accommodation target object to the generated deep learning learning model; and
A robot arm control step of controlling to grip the gripping target object and drop it to the target object based on the recognition of the marker and the target object;
The dataset includes an image obtained by photographing the object to be accommodated from at least four different angles to include the object to be accommodated, and the three-dimensional modeled image after three-dimensional modeling of the captured image. It is created based on a mask that has been binarized by back-projecting the captured image.
A robot control method through vision recognition using deep learning and a marker, characterized in that the marker and the target object are recognized through vision recognition. 7. The method of claim 6,
The learning step is
Vision using deep learning and markers, characterized in that it further comprises generating a deep learning learning model by learning the image obtained by photographing the target object of the generated dataset and the mask subjected to binarization with the Mask R-CNN model. Robot control method through recognition.

Images