KR20200059465A - Method and apparatus for controlling a drawing robot based on image - Google Patents

Abstract

Disclosed are a method and an apparatus for controlling a drawing robot based on an image. The method for controlling a drawing robot based on an image comprises: a step of collecting three-dimensional motion data generated by tracking a motion of a bot by drawing of a person; a step of converting the collected three-dimensional motion data into a robot control signal; a step of transmitting the converted robot control signal to a drawing robot to control drawing of the drawing robot; a step of using an external camera to acquire robot drawing image data resulting from monitoring the drawing robot; a step of analyzing the acquired robot drawing image data to generate three-dimensional motion correction data of the bot; and a step of converting the generated three-dimensional motion correction data into a robot control correction signal to transmit the robot control correction signal to the drawing robot.

Description

METHOD AND APPARATUS FOR CONTROLLING A DRAWING ROBOT BASED ON IMAGE}

The present invention relates to a method and apparatus for controlling a drawing robot based on an image, and more particularly, to a technique capable of correcting a precise movement of a brush by monitoring a drawing robot using a brush based on an image.

With the development of robot technology that performs various motions using artificial joints and motors, research to describe the human behavior in the same way continues. As one of these studies, there is a technique in which a robot arm draws the same picture by using two-dimensional coordinate information on a person-drawn picture.

Most of the image drawing techniques using the robot arm use a tool with a hard nib, such as a pencil, ballpoint pen, or signature pen, so that the drawing result is similar to the result of human work.

However, in the case of a robot technology that mimics a person drawing an image using a calligraphy brush, there is a problem in that it is difficult to accurately mimic because it exhibits various results depending on the type, position, and posture of a pigment or brush.

In particular, since the brush can be drawn in various patterns according to tilt or rotation, there is a limit to reproducing a person's drawing result by simply using 3D spatial information.

An object of the present invention for solving the above problems is to provide a method for controlling a drawing robot based on an image.

Another object of the present invention for solving the above problems is to provide an apparatus for controlling a drawing robot based on an image.

One aspect of the present invention for achieving the above object, provides a method for controlling the drawing robot based on the image.

The method of controlling a drawing robot based on an image includes collecting motion generated 3D motion data by tracking the movement of a brush by drawing, and converting the collected 3D motion data into a robot control signal. , Controlling drawing of the drawing robot by transmitting the converted robot control signal to the drawing robot, obtaining robot drawing image data monitoring the drawing robot using an external camera, and obtaining the obtained robot drawing image data The analysis may include generating 3D motion correction data of the brush and converting the generated 3D motion correction data into a robot control correction signal and transmitting it to the drawing robot.

The 3D motion data may include pitch, yaw, roll information indicating at least one of rotation, inclination, and direction of the brush, and 3D spatial coordinate information of the brush.

The generating of the 3D motion correction data may include performing color normalization on the robot drawing image data and reference image data, and comparing the color standardized robot drawing image data and the reference image data to each other to correct the correction region and the And generating coloring data for the correction area.

The reference image data may be image data obtained by monitoring the drawing of the person using the external camera.

In the step of performing the color standardization, comparing the standard color information for a sample collected in advance with the camera color information for the sample, and loading the generated color space conversion data, and the robot using the imported color space conversion data And standardizing the color space of the drawing image data and the reference image data.

The generating of the coloring data may include obtaining standard color data for the correction area, and performing color combination simulation based on the obtained standard color data to generate the coloring data.

In the generating of the 3D motion correction data, after the step of generating the coloring data, selecting a brush type for the correction area based on the brush pattern information collected in advance and the selected brush type and the brush The method may further include generating an optimal path of the brush for drawing the correction region based on the pattern information.

In the step of generating the 3D motion correction data, after the step of generating the optimal path of the brush, the step of calculating the drawing time according to the optimal path and the calculated drawing time exceeds a preset threshold, drawing The method may further include generating additional motion data for preparation.

Another aspect of the present invention for achieving the other object, to provide an apparatus for controlling the drawing robot based on the image.

An apparatus for controlling a drawing robot based on an image may include at least one processor, and a memory in which instructions instructing the at least one processor to perform at least one step are stored. have.

The at least one step includes: collecting the 3D motion data generated by tracking the movement of a brush by drawing of a person, converting the collected 3D motion data into a robot control signal, and converting the robot Controlling the drawing of the drawing robot by transmitting a control signal to the drawing robot, obtaining robot drawing image data monitoring the drawing robot using an external camera, analyzing the acquired robot drawing image data, and drawing The method may include generating 3D motion correction data and converting the generated 3D motion correction data into a robot control correction signal and transmitting the generated 3D motion correction data to the drawing robot.

The 3D motion data may include pitch, yaw, roll information indicating at least one of rotation, inclination, and direction of the brush, and 3D spatial coordinate information of the brush.

The generating of the 3D motion correction data may include performing color normalization on the robot drawing image data and reference image data, and comparing the color standardized robot drawing image data and the reference image data to each other to correct the correction region and the And generating coloring data for the correction area.

The reference image data may be image data obtained by monitoring the drawing of the person using the external camera.

In the step of performing the color standardization, comparing the standard color information for a sample collected in advance with the camera color information for the sample, and loading the generated color space conversion data, and the robot using the imported color space conversion data And standardizing the color space of the drawing image data and the reference image data.

The generating of the coloring data may include obtaining standard color data for the correction area, and performing color combination simulation based on the obtained standard color data to generate the coloring data.

In the generating of the 3D motion correction data, after the step of generating the coloring data, selecting a brush type for the correction area based on the brush pattern information collected in advance and the selected brush type and the brush The method may further include generating an optimal path of the brush for drawing the correction region based on the pattern information.

In the step of generating the 3D motion correction data, after the step of generating the optimal path of the brush, the step of calculating the drawing time according to the optimal path and the calculated drawing time exceeds a preset threshold, drawing The method may further include generating additional motion data for preparation.

When using the method and apparatus for controlling the drawing robot based on the image according to the present invention as described above, it is possible to perform sophisticated drawing control in consideration of posture as well as three-dimensional coordinates of the brush.

In addition, there is an advantage in that correction is performed to correct errors in consideration of various variables that occur as the drawing robot performs drawing using a brush.

In addition, standardized control may be performed in consideration of camera characteristics that may occur as a result of image-based control.

1 is a conceptual diagram of a method and apparatus for controlling a drawing robot based on an image according to an embodiment of the present invention.
2 is a flowchart illustrating a method for controlling a drawing robot based on an image according to an embodiment of the present invention.
3 is a flow chart embodying the steps of generating 3D motion correction data according to an embodiment of the present invention.
4 is a conceptual diagram illustrating a color standardization method according to an embodiment of the present invention.
5 is a conceptual diagram for explaining a method of simulating a color combination for coloring a correction area according to an embodiment of the present invention.
6 is a hardware configuration diagram of an apparatus for controlling a drawing robot based on an image according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but there may be other components in between. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a method and apparatus for controlling a drawing robot based on an image according to an embodiment of the present invention.

Referring to FIG. 1, in a method of controlling a drawing robot based on an image according to an embodiment of the present invention, first, a scene in which a person 40 is drawing and / or a result thereof is photographed with the camera 20 as a reference An image (or reference image) may be generated, and at the same time, 3D motion data may be generated by tracking the movement of a brush used by the person 40.

At this time, the 3D motion data is not limited to tracking 2D or 3D spatial coordinates (x, y, z coordinates) used by a general pen or pencil, but at least one of rotation, tilt, and direction of the brush (or It may include pitch, yaw, and roll information indicating a brush posture).

Here, the 3D motion data of the generated brush may be transmitted to the apparatus 10 for controlling the drawing robot based on the image according to an embodiment of the present invention, and the reference image also controls the drawing robot based on the image from the camera. It can be sent to the device 10.

The apparatus 10 for controlling the drawing robot based on the image generates a robot control signal capable of directly controlling the drawing robot's gear, motor, and the like using the 3D motion data of the previously collected brush, and the generated robot control signal Can be transferred to the drawing robot 30.

The drawing robot 30 may perform an operation of drawing a picture by imitating the drawing scene of the person 40 according to the robot control signal transmitted from the apparatus 10 for controlling the drawing robot based on the image. At this time, the scene where the drawing robot 30 draws a picture and / or the intermediate result may be photographed by the camera 20 to generate a robot drawing image. The robot drawing image generated here is transmitted back to the device 10 for controlling the drawing robot, and the device 10 for controlling the drawing robot can generate a robot control signal that corrects an operation error of the drawing robot 30. . The generated robot control signal is transmitted to the drawing robot 30 again to correct the operation of the drawing robot.

In FIG. 1, although the apparatus 10, the camera 20, and the drawing robot 30 for controlling the drawing robot based on the image are respectively shown individually, some of them may be implemented in combination with each other, and in such a case, the present invention It should be interpreted as being included in the scope of.

Hereinafter, each individual operation will be described in more detail.

2 is a flowchart illustrating a method for controlling a drawing robot based on an image according to an embodiment of the present invention.

Referring to FIG. 2, a method of controlling a drawing robot based on an image includes collecting 3D motion data generated by tracking the movement of a brush by drawing of a person (S100), and collecting 3D motion Converting data into a robot control signal (S110), transmitting the converted robot control signal to a drawing robot to control the drawing of the drawing robot (S120), a robot monitoring the drawing robot using an external camera Obtaining the drawing image data (S130), analyzing the obtained robot drawing image data to generate 3D motion correction data of the brush (S140) and converting the generated 3D motion correction data into a robot control correction signal It may include the step of transmitting to the drawing robot (S150).

The step of collecting 3D motion data (S100) may collect 3D motion data indicating changes in the position, direction, posture, and position of the brush over time by tracking the movement of the brush using motion tracking equipment. .

On the other hand, in order for the drawing robot to reproduce the same movement as a person with a brush by using the three-dimensional motion data, it must be converted into a control signal for controlling the motor, gear, and the like of the robot joint. Therefore, the step (S110) of converting to a robot control signal may be performed, and the signal conversion may be applied differently according to the type of drawing robot, command support method, or the like. Also, 3D position information of the canvas may be reflected in the signal conversion process. That is, a process of moving the motion plane of the 3D motion data to the plane according to the 3D location information of the canvas may be added.

In the step of controlling the drawing of the drawing robot (S120), it goes beyond simply controlling the movement of the drawing robot according to the 3D motion data, detecting the remaining amount of paint on the brush or detecting the remaining amount of paint on the drawn plane, etc. In the manner of recognizing the lack of paint on the brush, the motion control of adding the paint to the brush may be performed, and the motion control of straightening the brush by detecting whether the brush is bent may be performed.

Hereinafter, the step (S140) of generating 3D motion correction data will be described in detail.

3 is a flow chart embodying the steps of generating 3D motion correction data according to an embodiment of the present invention. 4 is a conceptual diagram illustrating a color standardization method according to an embodiment of the present invention. 5 is a conceptual diagram for explaining a method of simulating a color combination for coloring a correction area according to an embodiment of the present invention.

Referring to FIG. 3, the step (S140) of generating the 3D motion correction data of FIG. 2 is a step of performing color standardization on the robot drawing image data and reference image data (S141), and the color standardized robot drawing image data And comparing reference image data with each other to generate a correction area and coloring data for the correction area (S142), and selecting a brush type for the correction area based on previously collected brush pattern information (S143) and And generating an optimal path of the brush for drawing the correction area based on the selected brush type and the brush pattern information (S144).

Here, the reference image data may be image data obtained by monitoring a person's drawing using an external camera, as described in FIG. 1 above. Therefore, in the present invention, the robot drawing obtained by monitoring the drawing scene and / or the intermediate result of the drawing robot by using the image data obtained by monitoring the human drawing (including the drawing scene and the drawn intermediate result) as reference image data By comparing the image data with each other, an area requiring correction can be determined, and coloring data to be colored with respect to the determined correction area can be generated.

In addition, in the step of generating coloring data (S142), when comparing the robot drawing image data with the reference image data, for each image data in order to correct distortion caused by a difference in surrounding environment such as a lens, lighting, canvas, etc. Geometric correction can be performed.

In addition, in the step of generating the coloring data (S142), when comparing the robot drawing image data and the reference image data, the colored region and the line portion of the intermediate result are separated, and the size and shape of each separated portion (or region) are determined. Geometrical analysis, and comparison of colors can be performed.

On the other hand, in general, RGB (Red, Green, Blue) color information of an image acquired by the camera may be expressed differently depending on the lighting and the settings of the camera, even if the colors are the same. For this reason, when a color appearing in an image is used as a representative color of an object, a problem that a recognition rate is greatly reduced may occur. Therefore, according to an embodiment of the present invention, by standardizing the color of an image acquired by the camera, it is possible to minimize the influence of the surrounding environment such as characteristics of the camera or lighting and ensure uniform reproduction quality.

Specifically, referring to FIG. 4, an apparatus for controlling a drawing robot based on an image according to an embodiment of the present invention may include a color standardization module (software or hardware module, 40). Here, the color standardization module 40 may convert color data (or RGB data, 41) of the camera into color data 43 according to the standard color space using the color space conversion data 42. Here, an example of the standard color space may be CIELAB.

According to an embodiment of the present invention, color information 42a in a standard color space may be collected in advance for various samples including paint and canvas (paper, Korean paper, etc.) and stored in a database. In addition, RGB color information 42b may be collected and stored in a database through camera shooting for the same sample and the same lighting environment as the color information 42a collected in the standard color space.

The color space conversion data 42 according to an embodiment of the present invention represents data representing a relationship and / or a conversion rule between standard color information 42a and camera color information 42b for various samples collected above. Can mean

Therefore, in the step of performing color standardization according to FIG. 3 (S141), comparing the standard color information for the sample collected in advance with the camera color information for the sample, and loading the generated color space conversion data and the imported color space And standardizing a color space of the robot drawing image data and the reference image data using transformed data.

On the other hand, in the step of generating coloring data (S142), in order to generate coloring data to be colored for the correction region, a combination of paints capable of minimizing the color difference between the correction region of the reference image data and the correction region of the robot drawing image data May need to be selected.

To solve this problem, an apparatus for controlling a drawing robot based on an image according to an embodiment of the present invention may include a color combination simulation module 50.

Specifically, referring to FIG. 5, the color combination simulation module 50 previously learns standard color information 52 for individual paints and canvases and / or standard color information 53 for combinations of several paints and canvases, Using the learned information, it is possible to simulate color information according to a combination of virtual paint and canvas.

That is, the color combination simulation module 50 simulates the process of coloring according to the virtual color / canvas combination on the standard color data 51 for the correction area of the robot drawing image data, and the simulated color result is the reference image data. It is possible to derive the optimal coloring data 54 that matches or is most similar to the correction region.

When the contents according to FIG. 5 are summed up, the step of generating the coloring data according to FIG. 3 (S142) includes: obtaining standard color data for a correction area, and performing color combination simulation based on the obtained standard color data. It may include the step of generating the coloring data.

In the step of selecting the brush type (S143), the brush pattern information is a variety of patterns (thickness, start and end of the line) that can be displayed as a result when painting with various poses, directions, angles, etc. Shape, etc.). Such brush pattern information can be obtained by matching and collecting a person's drawing motion in advance.

In step S144 of generating the optimal path of the brush, the optimal path of the brush may vary depending on the color of the paint, canvas, and brush, so the coloring data generated in step S142 and the brush and canvas of the drawing robot can be used. May need to be considered.

In addition, if the process of drawing with a brush is lengthened, it may be necessary to re-paint the brush or to straighten the bent brush again. Therefore, after the step of generating the optimal path of the brush (S144), if the step of calculating the drawing time according to the optimal path and the calculated drawing time exceeds a preset threshold, additional motion data for drawing preparation is added. It may further include the step of generating.

Here, the additional motion data for preparing the drawing may be data representing an operation of applying paint to a brush or an operation of straightening a bent brush.

6 is a hardware configuration diagram of an apparatus for controlling a drawing robot based on an image according to an embodiment of the present invention.

Referring to FIG. 6, an apparatus 100 for controlling a drawing robot based on an image may include at least one processor 110 and instructions to instruct the at least one processor to perform at least one step. Can be.

The at least one step includes: collecting the 3D motion data generated by tracking the movement of a brush by drawing of a person, converting the collected 3D motion data into a robot control signal, and converting the robot Controlling the drawing of the drawing robot by transmitting a control signal to the drawing robot, obtaining robot drawing image data monitoring the drawing robot using an external camera, analyzing acquired robot drawing image data, and drawing The method may include generating 3D motion correction data and converting the generated 3D motion correction data into a robot control correction signal and transmitting the generated 3D motion correction data to the drawing robot.

The 3D motion data may include pitch, yaw, roll information indicating at least one of rotation, inclination, and direction of the brush, and 3D spatial coordinate information of the brush.

The generating of the 3D motion correction data may include performing color normalization on the robot drawing image data and reference image data, and comparing the color standardized robot drawing image data and the reference image data to each other to correct the correction region and the And generating coloring data for the correction area.

The reference image data may be image data obtained by monitoring the drawing of the person using the external camera.

In the step of performing the color standardization, comparing the standard color information for a sample collected in advance with the camera color information for the sample, and loading the generated color space conversion data, and the robot using the imported color space conversion data And standardizing the color space of the drawing image data and the reference image data.

The generating of the coloring data may include obtaining standard color data for the correction area, and performing color combination simulation based on the obtained standard color data to generate the coloring data.

In the generating of the 3D motion correction data, after the step of generating the coloring data, selecting a brush type for the correction area based on the brush pattern information collected in advance and the selected brush type and the brush The method may further include generating an optimal path of the brush for drawing the correction region based on the pattern information.

In the step of generating the 3D motion correction data, after the step of generating the optimal path of the brush, the step of calculating the drawing time according to the optimal path and the calculated drawing time exceeds a preset threshold, drawing The method may further include generating additional motion data for preparation.

For example, an apparatus 100 for controlling a drawing robot based on an image, for example, a communicable desktop computer, a laptop computer, a notebook, a smart phone, and a tablet PC ), Mobile phone, smart watch, smart glass, e-book reader, portable multimedia player (PMP), portable game machine, navigation device, digital camera , Digital multimedia broadcasting (DMB) player, digital audio recorder, digital audio player, digital video recorder, digital video player, PDA (Personal Digital Assistant) ).

The methods according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. Computer-readable media may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention or may be known and usable by those skilled in computer software.

Examples of computer-readable media may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include machine language codes such as those produced by a compiler, as well as high-level language codes that can be executed by a computer using an interpreter or the like. The above-described hardware device may be configured to operate with at least one software module to perform the operation of the present invention, and vice versa.

In addition, the above-described method or apparatus may be implemented by combining all or part of its configuration or function, or may be implemented separately.

Although described above with reference to preferred embodiments of the present invention, those skilled in the art variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You can understand that you can.

Claims (1)

Collecting 3D motion data generated by tracking the movement of a brush by drawing of a person;
Converting the collected 3D motion data into a robot control signal;
Controlling the drawing of the drawing robot by transmitting the converted robot control signal to the drawing robot;
Obtaining robot drawing image data monitoring the drawing robot using an external camera;
Analyzing the acquired robot drawing image data and generating 3D motion correction data of the brush; And
A method of controlling a drawing robot based on an image, comprising converting the generated 3D motion correction data into a robot control correction signal and transmitting it to the drawing robot.

Images