KR20220125876A - Apparatus and method for generating training data set using robot arm - Google Patents

Abstract

Disclosed are a device and method for generating learning data using a robot arm. The device for generating learning data using a robot arm according to one aspect of the present invention comprises: a rotating body for implementing various angles of an object; a robot arm having a multi-joint structure including a plurality of links and a plurality of joints, and rotating each link at a predetermined angle while rotating each joint according to a control signal; a sensor unit mounted on the end link of the robot arm to sense the object; and a controlling device generating the control signal controlling the driving of at least one of the robot arm and the rotating body, obtaining image data from the sensor unit moving and rotating according to the control signal, and generating a bounding box for the object from the image data by using a reference 3D model for the object to generate learning data for the object. According to one embodiment of the present invention, the device and method for generating learning data using a robot arm can generate accurate learning data at a high speed by automatically generating a learning data set for the object using the robot arm.

Description

Apparatus and method for generating a learning dataset using a robot arm {APPARATUS AND METHOD FOR GENERATING TRAINING DATA SET USING ROBOT ARM}

The present invention relates to an apparatus and method for generating a learning data set using a robot arm, and more particularly, to an apparatus and a method for generating a learning data set using a robot arm to automatically generate learning data for an object using the robot arm; it's about how

Machine learning, which is a field of artificial intelligence, is a field that allows people to acquire knowledge by providing data and learning as a human would. To do this, it is necessary to label a lot of data and configure the dataset for training.

This can be seen as being connected with the performance of artificial intelligence according to the amount and accuracy of the learned pattern by classifying the data for learning and configuring the dataset to learn.

In the past, data was augmented to compose a dataset, or data was manually classified (labeled).

The data increase method is a method of expanding the dataset by inverting the top or left or right side of the original image or cutting out some areas.

Since these tasks are performed manually, it takes a lot of time to prepare in advance for learning, which has the disadvantage of lowering the effectiveness of services incorporating actual artificial intelligence.

As a prior art related to the present invention, there is Republic of Korea Patent No. 10-2175531 (announced on November 6, 2020).

The present invention has been devised to improve the above problems, and an object of the present invention is to provide an apparatus and method for generating a learning data set using a robot arm so that learning data for an object can be automatically generated using the robot arm. will provide

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and another problem(s) not mentioned will be clearly understood by those skilled in the art from the following description.

Learning data generating apparatus using a robot arm according to an aspect of the present invention is made of a multi-joint structure including a rotating body for implementing various angles of an object, a plurality of links and a plurality of joints, and each A robot arm that rotates each link at a predetermined angle while rotating a joint part, a sensor unit mounted on a terminal link of the robot arm to detect the object, a control for controlling the driving of at least one of the robot arm and the rotating body Generates a signal, obtains image data from the sensor unit that moves and rotates according to the control signal, and generates a bounding box for the object from the image data using a reference 3D model for the object and a control device for generating learning data for the object by doing so.

In the present invention, the robot arm may be a 6-axis articulated robot arm.

In the present invention, the robot arm is made of a multi-joint structure including a body portion, a plurality of links and a plurality of joints on the upper portion of the body portion, and each joint portion rotates according to the control signal and rotates each link at a predetermined angle. A robot arm that rotates, installed inside the body, converts the position of each link and joint according to the driving of the robot arm into spatial coordinate values, and the image data obtained through the sensor unit and the image data obtained at the time of acquisition It may include a control unit for transmitting the spatial coordinate value of the robot arm to the control device.

In the present invention, the control device includes a communication unit, a storage unit that stores the spatial coordinate values of the robot arm from which the reference 3D model for the object is obtained as an initial value, and a control signal for driving at least one of the robot arm and the rotating body transmits through the communication unit, receives image data obtained through the sensor unit from the robot arm and spatial coordinate values of the robot arm unit, detects the object from the image data using the reference 3D model, and the object It may include a control unit for generating a bounding box for .

In the present invention, the control unit compares the spatial coordinate value of the robot arm received from the robot arm with the initial value to determine the x-axis, y-axis, z-axis, pitch, roll, and yaw. Calculating at least one change value from among, applying the change value to the reference 3D model to generate a changed 3D model, extracting an outline of the changed 3D model, and applying the extracted outline to the detected object You can create a segmentation bounding box.

In the present invention, the control unit compares the spatial coordinate value of the robot arm received from the robot arm with the initial value to determine the x-axis, y-axis, z-axis, pitch, roll, and yaw. Calculating at least one changed value among, generating a changed 3D model by applying the changed value to the reference 3D model, the X-axis value and Y-axis value furthest from the center of the changed 3D model, and the closest X-axis value and Y-axis value A bounding box can be created by applying four points that meet by drawing a straight line, respectively, to the detected object.

In the present invention, the rotating body may rotate in the horizontal direction according to the control signal.

In a method for generating learning data using a robot arm according to an aspect of the present invention, the control device transmits a control signal for controlling the driving of the robot arm to the robot arm, the robot arm is driven according to the control signal, and the sensor acquiring image data including an object through a unit, and transmitting the obtained image data to the control device; and generating learning data for the object by generating a bounding box.

In the present invention, the step of generating the learning data includes: the control device receiving the image data from the robot arm; Comparing with the x-axis, the y-axis, the z-axis, the pitch (pitch), roll (roll), and the step of calculating at least one change value of yaw (yaw), the control device, the reference 3D model based on the change value generating a modified 3D model by applying to , and detecting, by the control device, the object from the image data using the changed 3D model, and generating a bounding box for the object.

In the present invention, in the generating of the bounding box, the control device may generate a segmentation bounding box by extracting an outline of the changed 3D model and applying the extracted outline to the detected object.

In the present invention, in the step of generating the bounding box, the control device draws the X-axis value and the Y-axis value, the closest X-axis value and the Y-axis value from the center of the changed 3D model, respectively, with straight lines to meet the four points. A bounding box can be created by applying it to the detected object.

The apparatus and method for generating a learning data set using a robot arm according to an embodiment of the present invention can generate accurate learning data at a high speed by automatically generating a learning data set for an object using the robot arm. Through this, it is possible to expand the learning efficiency of artificial intelligence machine learning as well as the utility of services incorporating artificial intelligence.

On the other hand, the effects of the present invention are not limited to the above-mentioned effects, and various effects may be included within the range apparent to those skilled in the art from the description below.

1 is a view showing an apparatus for generating learning data using a robot arm according to an embodiment of the present invention.
FIG. 2 is a block diagram schematically showing the configuration of the control device shown in FIG. 1 .
3 is an exemplary diagram for explaining a reference 3D model according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a method of detecting an object from image data using a reference 3D model according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a method of generating a segmentation bounding box using a reference 3D model according to an embodiment of the present invention.
6 is an exemplary diagram for explaining a method of generating a rectangular bounding box using a reference 3D model according to an embodiment of the present invention.

Hereinafter, an apparatus and method for generating a learning data set using a robot arm according to an embodiment of the present invention will be described with reference to the accompanying drawings. In this process, the thickness of the lines or the size of the components shown in the drawings may be exaggerated for clarity and convenience of explanation.

In addition, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, definitions of these terms should be made based on the content throughout this specification.

Further, implementations described herein may be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementations of the discussed features may also be implemented in other forms (eg, as an apparatus or program). The apparatus may be implemented in suitable hardware, software and firmware, and the like. A method may be implemented in an apparatus such as, for example, a processor, which generally refers to a computer, a microprocessor, a processing device, including an integrated circuit or programmable logic device, or the like. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants ("PDA") and other devices that facilitate communication of information between end-users.

1 is a view showing a learning data generating apparatus using a robot arm according to an embodiment of the present invention, Figure 2 is a block diagram schematically showing the configuration of the control device shown in Figure 1, Figure 3 is an embodiment of the present invention An exemplary diagram for explaining a reference 3D model according to an example, FIG. 4 is an exemplary diagram for explaining a method of detecting an object from image data using a reference 3D model according to an embodiment of the present invention, FIG. 5 is the present invention 6 is an exemplary diagram for explaining a method of generating a segmentation bounding box using a reference 3D model according to an embodiment of the present invention. It is an example diagram for explaining.

Referring to FIG. 1 , an apparatus for generating a learning data set using a robot arm according to an embodiment of the present invention is a device capable of generating learning data for an object 10, and a rotating body on which the object 10 is mounted ( 100 ), the sensor unit 300 may include a mounted robot arm 200 , and a control device 400 .

The rotating body 100 is configured to implement various angles of the object 10 , and may be rotated in a horizontal direction according to a control signal from the control device 400 . That is, the rotating body 100 is a frame structure that is rotatably mounted in the horizontal direction according to a control signal, and can rotate the object 10 by 360 degrees in the horizontal direction. Although not shown, the rotating body 100 is provided with a driving means such as an electric motor for providing a driving force necessary for horizontal rotation while rotating according to a control signal of the control device 400 . As such a driving means, a stepping motor or a servo motor capable of precisely adjusting a rotation angle and rotation speed may be used.

The rotating body 100 may allow the object 10 to be sensed in various directions.

The robot arm 200 is a robot device that implements a conventional manipulator function while acquiring image data through the sensor unit 300 , and has a multi-joint structure including a plurality of links 220 and a plurality of joint parts 230 . have. The robot arm 200 may be implemented as a 6-axis articulated robot arm 200 .

The robot arm 200 may include a body part 210, robot arm parts 220 and 230, and a control unit (not shown). In addition, the robot arm 200 may further include a communication unit (not shown) for communicating with the control device (400).

The body part 210 may be configured to fix the robot arm 200 .

The robot arm parts 220 and 230 may have a multi-joint structure including a plurality of links 220 and a plurality of joint parts 230 on the upper portion of the body part 210 . Each joint 230 rotates each link 220 at a predetermined angle while rotating according to the control signal of the control device 400, and rotates according to the control signal at the end of the longitudinal link disposed at the top to rotate the object 10. A sensor unit 300 for photographing (sensing) may be mounted.

Here, the robot arm parts 220 and 230 may have a link structure of two or more joints according to the required manipulator function, and the joint part 230 is provided to enable vertical rotation (tilting, tilting) in addition to horizontal rotation (panning) operation. it is preferable

In addition, although not shown, each joint part 230 is provided with a driving means such as an electric motor for providing a driving force necessary for rotational driving while rotating according to a control signal of the controller 160 , and the driving means includes a stepping motor or A servomotor may be used, and an encoder may be disposed on one side for precise control of the rotation angle.

The control unit is a microcontroller that centrally controls the robot arm 200 , and may drive and control the robot arm units 220 and 230 according to a control signal from the control device 400 . Here, the control signal may be a signal for changing the posture of the robot arm.

The control unit is installed inside the body unit 210, converts the position (posture) of each link 220 and the joint unit 230 according to the driving of the robot arm units 220 and 230 into spatial coordinate values, and the sensor unit 300 The obtained image data and spatial coordinate values of the robot arms 220 and 230 at the time of acquiring the image data may be transmitted to the control device 400 . Here, the spatial coordinate value may include an x-axis, a y-axis, a z-axis, a pitch, a roll, and a yaw value.

The control unit drives and controls the joint unit 230 according to a control signal from the control device 400 , so that the image data acquisition operation through the sensor unit 300 can be performed more easily and precisely.

The sensor unit 300 may detect the object 10 and transmit the sensed data to the control device 400 . Here, the object 10 may be an object for generating learning data.

The sensor unit 300 may include an image sensor (not shown), a lidar sensor (not shown), and a radar sensor (not shown).

The image sensor may photograph the object 10 and transmit the photographed image data to the control device 400 . The image sensor may include, for example, a camera or other device capable of receiving image data. The image sensor may process image data such as still images or moving images, and may output temporally continuous images including the same object 10 . The lidar sensor emits light and measures the light reflected by the object 10 to detect a distance from the object 10 and various physical properties, and can model this as a 3D image.

Learning data for the object 10 can be more accurately generated by using the lidar sensor rather than using only the image sensor.

The robot arm 200 equipped with the sensor unit 300 configured as described above acquires image data including the object 10 through the sensor unit 300 while driving according to a control signal from the control device 400 and , the obtained image data may be transmitted to the control device 400 .

The control device 400 generates a control signal for controlling the driving of at least one of the robot arm 200 and the rotating body 100, and acquires image data from the sensor unit 300 that moves and rotates according to the control signal In addition, learning data for the object 10 may be generated by generating a bounding box for the object 10 from the image data using the reference 3D model for the object 10 .

In order to generate learning data for the object 10 , it is necessary to photograph the object 10 at various angles. Accordingly, the control device 400 controls the driving of the robot arm 200 to move the sensor unit 300 , and controls the driving of the rotating body 100 to control the rotating body 100 to be mounted on the upper surface of the object 10 . After rotating , learning data for the object 10 may be generated by inputting a bounding box into the area of the real object 10 in the image data obtained through the sensor unit 300 .

The control device 400 may include a storage unit 410 , a communication unit 420 , and a control unit 430 as shown in FIG. 2 .

The communication unit 420 is configured for communication with the rotating body 100 and the robot arm 200 , and transmits a control signal to the rotating body 100 and the robot arm 200 or image data and It is possible to receive spatial coordinate values. The communication unit 420 may be implemented in various forms, such as a short-range communication module, a wireless communication module, a mobile communication module, and a wired communication module.

The storage unit 410 may store the spatial coordinate values of the robot arms 220 and 230 from which the reference 3D model of the object 10 is obtained as an initial value. Here, the initial value is the spatial coordinate values (x-axis, y-axis, z-axis, pitch) of the robot arm units 220 and 230 when acquiring the object 10 having the same shape as the reference 3D model using the robot arm 200 ), roll, and yaw values).

Also, the storage unit 410 may store a scenario for movement and rotation of the rotating body 100 and the robot arm 200 . In this case, the rotating body 100 and the robot arm 200 can move and rotate by themselves according to the scenario stored in the storage unit 410 , and the control device 400 uses the image data from the robot arm 200 . Thus, learning data for the object 10 may be automatically generated.

Also, the storage unit 410 may store a reference 3D model of the object 10 . For example, in the case of image data as shown in (a) of FIG. 3 , the storage unit 410 may store a reference 3D model as shown in (b) of FIG. 3 . The reference 3D model may be 3D scan modeling data.

The control unit 430 may generate a control signal for controlling the driving of at least one of the rotating body 100 and the robot arm 200 to control the driving of at least one of the rotating body 100 and the robot arm 200 . have.

The controller 430 may store in advance a driving scenario of the rotating body 100 and the robot arm 200 and control the driving of the rotating body 100 and the robot arm 200 according to the driving scenario. Here, the driving scenario may be set in advance by an administrator for movement and movement paths.

The control unit 430 receives image data from the sensor unit 300 according to the driving of the rotating body 100 and the robot arm 200, detects the object 10 from the image data using the reference 3D model, and the object By creating a bounding box for (10), it is possible to generate training data.

When the robot arm 200 is driven under the control of the control unit 430 to move and rotate the sensor unit 300, the control unit 430 sets the spatial coordinate values of the robot arms 220 and 230 according to the control signal to the initial value and By comparison, a change value of at least one of an x-axis, a y-axis, a z-axis, a pitch, a roll, and a yaw may be calculated. Here, the spatial coordinate values of the robot arm units 220 and 230 include at least one value of the x axis, y axis, z axis, pitch, roll, and yaw of the robot arm units 220 and 230 . can

When the changed value is calculated, the controller 430 may generate a changed 3D model by applying the changed value to the reference 3D model, extract an outline of the changed 3D model, and apply the extracted outline to the object 10 detected from the image data. ) to create a segmentation bounding box.

When the rotating body 100 rotates under the control of the control unit 430 and the robot arm 200 is driven, the observation point (viewing angle) of the sensor unit 300 for the object 10 changes, so the control unit 430 changes A viewpoint (viewing angle) can be applied to the reference 3D model. Then, the shape of the reference 3D model is changed, and the controller 430 may recognize which area is the object 10 in the image data from the sensor unit 300 using the changed 3D model.

For example, a case in which the reference 3D model is changed as shown in FIG. 4A due to the operation of the actuator will be described. In this case, the controller may detect the object A corresponding to the 3D model of (a) from the image data of (b) of FIG. 4 .

The controller 430 may generate a segmentation bounding box by extracting the changed outline of the 3D model and applying the extracted outline to the detected object 10 . For example, when the outline B of the 3D model as shown in (a) of FIG. 5 is extracted, the controller 430 converts the outline of the 3D model to the outline B of the object 10 as shown in FIG. ') can be applied. This is because the controller 430 applies the moving distance of the movable body housing, the moving angle, and the height change value of the sensor unit 300 to the reference 3D model, so the changed 3D model is the size and shape of the object 10 in the image data. may be the same. Accordingly, the outline of the 3D model may be applied as the outline of the object 10 .

In addition, the control unit 430 compares the spatial coordinate values of the robot arms 220 and 230 according to the control signal with the initial values, and compares the x-axis, y-axis, z-axis, pitch, roll, and yaw. Calculate at least one change value among A bounding box can be created by applying four points drawn to meet to the detected object 10 .

For example, the largest X-axis, the smallest X-axis, the largest y-axis, and the largest y-axis are drawn as straight lines from the center point of the 3D model as shown in FIG. When applied to the image data from the unit 300, a bounding box as shown in (b) of FIG. 6 can be generated.

As described above, the controller 430 creates a bounding box on the object 10 in the image data, and specifies a class (eg, sofa, picture frame, book, carpet, curtain) for specifying the object 10 in the bounding box. etc.) can be stored in the training data DB (not shown) by labeling the image data.

Meanwhile, in the embodiment of the present invention, the control device 400 has been described as a device existing outside the robot arm 200 , but the control device 400 may be a device existing inside the robot arm 200 .

In addition, in the embodiment of the present invention, it has been described that the spatial coordinate values of the robot arm units 220 and 230 are calculated by the robot arm 200 , but the spatial coordinate values of the robot arm units 220 and 230 may be calculated by the control device 400 . .

As described above, it can be applied to various machine learning using the automatically generated labeled learning data.

As described above, the apparatus and method for generating a learning data set using a robot arm according to an embodiment of the present invention automatically generate a learning data set for the object 10 using the robot arm, thereby providing an accurate and fast You can create training data. Through this, it is possible to expand the learning efficiency of artificial intelligence machine learning as well as the utility of services incorporating artificial intelligence.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and those of ordinary skill in the art to which various modifications and equivalent other embodiments are possible. will understand

Therefore, the true technical protection scope of the present invention should be defined by the following claims.

10: object
100: rotating body
200: robot arm
210: body part
220 : link
230: joint
300: sensor unit
400: control device
410: storage
420: communication department
430: control unit

Claims (11)

Rotating body for implementing various angles of the object;
a robot arm having a multi-joint structure including a plurality of links and a plurality of joints, and rotating each link at a predetermined angle while rotating each joint according to a control signal;
a sensor unit mounted on the end link of the robot arm to detect the object; and
Generates a control signal for controlling the driving of at least one of the robot arm and the rotating body, acquires image data from the sensor unit that moves and rotates according to the control signal, and uses a reference 3D model for the object A control device for generating learning data for the object by generating a bounding box for the object from the image data
Including, learning data generating device using a robot arm.
According to claim 1,
The robot arm,
Learning data generating device using a robot arm, characterized in that it is a 6-axis articulated robot arm.
According to claim 1,
The robot arm,
body part;
a robot arm part having a multi-joint structure including a plurality of links and a plurality of joint parts on the upper portion of the body part, and each joint part rotates according to the control signal and rotates each link at a predetermined angle; and
It is installed inside the body, converts the positions of the links and joints according to the driving of the robot arm into spatial coordinate values, and the image data obtained through the sensor unit and the spatial coordinates of the robot arm at the time of acquiring the image data Learning data generating apparatus using a robot arm, characterized in that it comprises a control unit for transmitting a value to the control device.
According to claim 1,
The control device is
communication department;
a storage unit for storing the spatial coordinate values of the robot arm from which the reference 3D model of the object is obtained as an initial value; and
Transmitting a control signal for driving at least one of the robot arm and the rotating body through the communication unit, receiving image data obtained through the sensor unit from the robot arm and spatial coordinate values of the robot arm unit, and the reference 3D model and a control unit for detecting the object from the image data by using and generating a bounding box for the object.
5. The method of claim 4,
The control unit is
By comparing the spatial coordinate value of the robot arm received from the robot arm with the initial value, the change value of at least one of the x-axis, y-axis, z-axis, pitch, roll, and yaw Calculating, applying the changed value to the reference 3D model to generate a changed 3D model, extracting an outline of the changed 3D model, and applying the extracted outline to the detected object to generate a segmentation bounding box Learning data generating device using a robot arm characterized in that.
5. The method of claim 4,
The control unit is
By comparing the spatial coordinate value of the robot arm received from the robot arm with the initial value, the change value of at least one of the x-axis, y-axis, z-axis, pitch, roll, and yaw Calculate, apply the changed value to the reference 3D model to generate a changed 3D model, and draw the X-axis and Y-axis values, the closest X-axis values and the Y-axis values from the center of the changed 3D model with straight lines, respectively, and meet 4 Learning data generating apparatus using a robot arm, characterized in that by applying the dots to the detected object to generate a bounding box.
According to claim 1,
The rotating body is
Learning data generating apparatus using a robot arm, characterized in that it rotates in the horizontal direction according to the control signal.
transmitting, by a control device, a control signal for controlling the driving of the robot arm to the robot arm;
acquiring image data including an object through a sensor unit while driving the robot arm according to the control signal, and transmitting the obtained image data to the control device; and
generating, by the control device, learning data for the object by generating a bounding box for the object from the image data using a reference 3D model for the object
Including, a method of generating learning data using a robot arm.
9. The method of claim 8,
The step of generating the learning data is,
receiving, by the control device, the image data from the robot arm;
The control device compares the spatial coordinate value of the robot arm at the time of acquiring the image data with an initial value stored in x-axis, y-axis, z-axis, pitch, roll, and yaw. calculating at least one change value;
generating, by the controller, a changed 3D model by applying the changed value to a reference 3D model; and
and detecting, by the control device, the object from the image data using the changed 3D model, and generating a bounding box for the object.
10. The method of claim 9,
In the step of creating the bounding box,
The control device extracts the outline of the changed 3D model, and generates a segmentation bounding box by applying the extracted outline to the detected object.
10. The method of claim 9,
In the step of creating the bounding box,
The control device draws the X-axis value and Y-axis value, the closest X-axis value and the Y-axis value from the center of the changed 3D model in straight lines, respectively, and applies four points that meet to the detected object to generate a bounding box. A method of generating learning data using a robot arm characterized by the

Images