KR20220092288A - Control system and control method for manipulator - Google Patents

Abstract

Disclosed are a system and a method for controlling a manipulator. The present invention relates to the system for controlling a manipulator performing vision-based recognition with respect to a target, which comprises: a first image obtaining unit spaced from the manipulator for vision-based recognition to capture the target; a second image obtaining unit installed in the manipulator for the vision-based recognition to capture the target; and a control unit learning a characteristic of the target from the image received from the first image obtaining unit and the second image obtaining unit and controlling the manipulator based on a learning result.

Description

CONTROL SYSTEM AND CONTROL METHOD FOR MANIPULATOR

The present invention relates to a manipulator control system and a control system. More particularly, it relates to a system and method for controlling a manipulator using a hand-in-eye method and a hand-to-eye method camera.

Following the industrial robot replacing the conventional simple labor, interest in intelligent robots is increasing due to the rapid development of artificial intelligence technology. As the field of application of intelligent robots moved away from manufacturing and entered the home, medical, and automobile fields, intelligent robots using artificial intelligence appeared with the development of the 4th industry.

As the utilization of robots in industry increases, the development of manipulators that can function as arms of robots is being developed. A manipulator has a function similar to that of a human upper extremity, and “grasses” (referring to grasping, adsorbing, and holding) an object by means of a mechanical hand corresponding to the tip of the manipulator, thereby creating a spatial effect. It refers to a work that can be carried out by moving it to a pole or painting and welding with a tool such as a spray gun or a welding torch attached to the tip.

Artificial intelligence refers to a technological field whose purpose is to realize the ability to solve problems based on human experience and knowledge with electronic technology. Recently, along with the development of the 4th industry, artificial intelligence technology is developing, and in particular, remarkable development in the field of machine learning has recently been attracting attention.

As such, technologies related to manipulators that can automatically realize specific motions by applying machine learning technology to manipulators widely used in industry are emerging. In particular, by utilizing the vision recognition function, the manipulator may recognize a target and perform a predetermined operation on the recognized target. However, conventionally, vision recognition has been mainly performed with a single camera, such as a camera for acquiring depth information or a camera capable of acquiring an RGB image, but this vision recognition technology has a limitation in the accuracy of its operation.

In particular, the conventional Hand-to-Eye method has a limitation that the observable area is fixed, and the conventional Hand-in-Eye method vision recognition is highly dependent on FPS (Frame Per Second), so the efficiency of vision recognition is This lowering problem existed.

Republic of Korea Patent Publication No. 10-2019-0075416

An object of the present invention is to provide a manipulator control system and control method for recognizing a target by vision using two image acquisition units and controlling the manipulator based on the vision recognition.

Another object of the present invention is to provide a manipulator control system and a control method for recognizing a target using the Hand-to-Eye method and the Hand-in-Eye method and controlling the manipulator based on the recognition method.

In order to solve the above problems, the present invention provides a manipulator control system for vision recognition of a target, a first image acquisition unit spaced apart from the manipulator to photograph the target for vision recognition, A second image acquisition unit installed in the manipulator to photograph the target, and the first image acquisition unit and the second image acquisition unit learn the characteristics of the target from the images received, and use the manipulator based on the learning result It may include a control unit to control.

Also, the controller may receive a first image from the first image acquisition unit, and acquire depth information of the target from the first image through a hand-to-eye method.

In addition, the controller may receive a second image from the second image acquisition unit, and acquire 2D information of the target from the second image through a Hand-in-Eye method.

Also, the two-dimensional information may include a rotation angle and a rotation direction of the manipulator.

In addition, in order to solve the above problem, the present invention provides a manipulator control method for vision recognition of a target through a first image acquisition unit spaced apart from a manipulator and a second image acquisition unit installed in the manipulator, the first image acquisition Receiving a first image obtained by photographing the target from a unit, receiving a second image obtained by photographing the target from the second image obtaining unit, ROI (Region of Interest) based on depth information from the first image acquiring an image, acquiring two-dimensional information from the second image, learning and recognizing a characteristic of the target based on the ROI image and the two-dimensional information, and based on the recognition result, the manipulator may include controlling an operation with respect to the target.

In addition, the step of obtaining an ROI (Region of Interest) image based on depth information from the first image includes: thresholding the first image; after the thresholding, the target is selected based on a deep learning algorithm The method may include detecting, obtaining a depth image of the detected target in the first image, and obtaining the ROI image based on the depth image.

In addition, the step of obtaining the two-dimensional information from the second image may include: thresholding the second image; detecting a contour of the target based on a deep learning algorithm after the thresholding; and obtaining a rotation angle of the target based on the outline.

In addition, the step of controlling the operation of the manipulator with respect to the target may include obtaining a rotation direction of the manipulator based on the rotation angle and controlling the operation of the manipulator based on the rotation direction. have.

The present invention has the effect of providing a manipulator control system and control method that overcomes the problems of the Hand-to-Eye method and the Hand-in-Eye method using two image acquisition units.

In addition, the present invention provides a manipulator control system and control method for automatically performing a predetermined operation on a target through vision recognition.

In addition, the present invention has the effect of providing more stable vision recognition by controlling the manipulator using an image acquisition unit including a lens that is not contaminated due to the characteristics of the manipulator mainly used in industrial sites.

Effects that can be obtained according to the present invention are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which this specification belongs from the description below. will be.

1 is a view showing a manipulator control system according to the present invention.
2 is a view showing a control unit according to the present invention.
3 is a view showing an image acquisition unit according to the present invention.
4 (a) and 4 (b) are views showing a first image and a second image according to the present invention.
5 is a diagram illustrating a manipulator control method according to the present invention.
6 is a diagram illustrating a process of acquiring an ROI image based on a first image according to the present invention.
7 is a diagram illustrating a process of acquiring two-dimensional information based on a second image according to the present invention.
8(a) and 8(b) are diagrams illustrating a process of obtaining a rotation angle based on a second image according to the present invention.
9 is a diagram illustrating a process of controlling the operation of the manipulator according to the present invention.
BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help the understanding of the present specification, provide embodiments of the present specification, and together with the detailed description, explain the technical features of the present specification.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numbers regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted.

In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, a manipulator control system and a control method according to a preferred embodiment of the present invention will be described in detail based on the above description.

1 is a view showing a manipulator control system according to the present invention.

Referring to FIG. 1 , the control system according to the present invention may be a manipulator 11 control system for vision-recognizing a target 10 . The control system according to the present invention may include a first image acquisition unit 110 , a second image acquisition unit 120 , and a controller 130 . The manipulator 11 may be included in the control system according to the present invention. The target 10 is a target for vision recognition, and may be a target object for pick and place through the manipulator 11 .

The first image acquisition unit 110 may be a camera configured to photograph the target 10 while being spaced apart from the manipulator 11 for vision recognition. The second image acquisition unit 120 is installed on the manipulator 11 for vision recognition to photograph the target 10 and may be a camera. The image acquired by the first image acquiring unit 110 may be a first image, and the image acquired by the second image acquiring unit 120 may be a second image.

The controller 130 may learn the characteristics of the target 10 from the images received by the first image acquisition unit 110 and the second image acquisition unit 120 . Also, the controller 130 may control the manipulator 11 based on the learning result. That is, the controller 130 may receive the first image and the second image, and learn the characteristics of the target 10 therefrom. The characteristics of the target 10 may be visual and morphological characteristics.

The controller 130 may acquire depth information of the target 10 from the first image through a hand-to-eye method. The depth information may mean 3D information. That is, the controller 130 may obtain 3D information about the target 10 from the first image. The controller 130 may acquire depth information of the first image by using an algorithm such as YOLOv3. The controller 130 may acquire a region of interest (ROI) image based on the depth information.

The controller 130 may receive the second image from the second image acquisition unit 120 and acquire 2D information of the target 10 from the second image through the Hand-in-Eye method. In this case, the two-dimensional information may include a rotation angle and a rotation direction of the manipulator 11 .

2 is a view showing a control unit according to the present invention.

Referring to FIG. 2 , the control unit 130 may include a processor 131 , a memory 132 , and a communication unit 133 . The processor 131 is a component capable of performing calculations and controlling other devices. Mainly, it may mean a central processing unit (CPU), an application processor (AP), a graphics processing unit (GPU), or the like. In addition, the CPU, AP, or GPU may include one or more cores therein, and the CPU, AP, or GPU may operate using an operating voltage and a clock signal. However, a CPU or AP may consist of a few cores optimized for serial processing, whereas a GPU may consist of thousands of smaller and more efficient cores designed for parallel processing.

The memory 132 stores data supporting various functions of the controller 130 . The memory 132 may store a plurality of application programs (or applications) and commands driven by the controller 130 . At least some of these application programs may be downloaded from an external server through wireless communication.

The memory 132 is a flash memory type, a hard disk type, a solid state disk type (SSD), a silicon disk drive type (SDD), and a multimedia card micro type. ), card-type memory (such as SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read (EEPROM) -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. Also, the memory 132 may include a web storage that performs a storage function on the Internet.

The communication unit 133 transmits/receives information to and from a base station or a vehicle including a communication function through an antenna. The communication unit 133 may include a modulator, a demodulator, a signal processor, and the like.

Wireless communication may refer to communication using a communication facility installed by telecommunication companies and a wireless communication network using a frequency of the communication facility. At this time, the communication unit 133, CDMA (code division multiple access), FDMA (frequency division multiple access), TDMA (time division multiple access), OFDMA (orthogonal frequency division multiple access), SC-FDMA (single carrier frequency division multiple) access), and the like, may be used in various wireless communication systems, and the communication unit 133 may also be used in 3rd generation partnership project (3GPP) long term evolution (LTE) and the like. In addition, not only 5G communication, which is currently commercialized, but also 6G, which is scheduled to be commercialized in the future, may be used. However, in the present specification, a pre-installed communication network may be utilized without being limited by such a wireless communication method.

3 is a view showing an image acquisition unit according to the present invention.

Referring to FIG. 3 , the image acquisition unit of the present invention may refer to the first image acquisition unit 110 and the second image acquisition unit 120 . For convenience of description, the first image acquisition unit 110 is described, but the second image acquisition unit 120 may also have the same configuration.

The first image acquisition unit 110 may include a camera module 111 as a camera. The camera module 111 may generate a video image from the optical image. The camera module 111 may include a housing including a through hole in a sidewall, a lens 113 installed in the through hole, and a driving unit 112 for driving the lens 113 . The through hole may be formed in a size corresponding to the diameter of the lens 113 . The lens 113 may be inserted into the through hole.

The driving unit 112 may be configured to control the lens 113 to move forward or backward. The lens 113 and the driving unit 112 may be connected in a conventionally known manner, and the lens 113 may be controlled by the driving unit 112 in a conventionally known manner.

In order to obtain various image images, the lens 113 needs to be exposed to the outside of the housing constituting the camera module 111 or the image acquisition unit 110 .

Preferably, the lens 113 is a siloxane-based compound represented by the following [Formula 1] on its surface; It may be coated with a coating composition containing an organic solvent, inorganic particles and a dispersing agent.

[Formula 1]

(Where n is an integer from 1 to 100.)

When the lens 1321 is coated with the coating composition, it can exhibit excellent water repellency and stain resistance, so even if the lens 113 installed outside the vehicle is exposed to a polluted environment for a long time, images or images that can be used as road information are collected. can do.

The inorganic particles may be selected from the group consisting of silica, alumina, and mixtures thereof. The average diameter of the inorganic particles is 70 to 100 μm, but is not limited to the above example. After the inorganic particles are formed as a coating layer 114 on the surface of the lens 113 , physical strength may be improved, and the viscosity may be maintained within a certain range to increase moldability.

The organic solvent is selected from the group consisting of methyl ethyl ketone (MEK), toluene, and mixtures thereof, and preferably methyl ethyl ketone may be used, but is not limited thereto.

As the dispersant, a polyester-based dispersant may be used, and specifically, as a polyester-based dispersion stabilizer composed of a copolymer of 2-methoxypropyl acetate and 1-methoxy-2-propyl acetate, TEGO-Disperse 670 (manufacturer : EVONIK), but is not limited to the above examples and any dispersant obvious to those skilled in the art can be used without limitation.

The coating composition may further include a stabilizer as other additives, and the stabilizer may include a UV absorber, an antioxidant, and the like, but is not limited to the above examples and may be used without limitation.

For forming the coating layer 114, the coating composition may include a siloxane-based compound represented by Chemical Formula 1; organic solvents, inorganic particles and dispersants.

The coating composition may include 40 to 60 parts by weight of the siloxane-based compound represented by Formula 1, 20 to 40 parts by weight of inorganic particles, and 5 to 15 parts by weight of a dispersant based on 100 parts by weight of the organic solvent. In the case of the above range, a synergistic effect is expressed to the extent that the water repellency effect due to the interaction of each component has a critical significance, and when it is out of the above range, the synergistic effect is rapidly reduced or almost absent.

More preferably, the viscosity of the coating composition is 1500 to 1800 cP, and when the viscosity is less than 1500 cP, when applied to the surface of the lens 113, there is a problem that the formation of the coating layer 114 is not easy because it flows down, and exceeds 1800 cP In this case, there is a problem in that the formation of a uniform coating layer 114 is not easy.

[Preparation Example 1: Preparation of coating layer]

1. Preparation of coating composition

A coating composition was prepared by mixing methyl ethyl ketone with the siloxane-based compound represented by the above [Formula 1], inorganic particles and a dispersant:

A more specific composition of the antistatic composition is shown in Table 1 below.

TX1 TX2 TX3 TX4 TX5 organic solvent 100 100 100 100 100 polysiloxane 30 40 50 60 70 inorganic particles 10 20 30 40 50 dispersant One 5 10 15 20

(unit parts by weight) 2. Preparation of coating layer

The coating composition of DX1 to DX5 was applied to one surface of the lens 113 and cured to form a coating layer 114 .

[Experimental example]

1. Evaluation of surface appearance

Due to the difference in viscosity of the coating composition, after the coating layer 114 was prepared, sensory evaluation was performed as to whether a uniform surface was formed. Whether or not a uniform coating layer 114 was formed was evaluated, and evaluation was performed according to the following criteria.

○: uniform coating layer formation

×: Formation of a non-uniform coating layer

TX1 TX2 TX3 TX4 TX5 sensory evaluation Х ○ ○ ○ Х

When the coating layer 114 is formed, if the viscosity is less than a certain level, a flow occurs on the surface of the lens 113 , and after the curing process, it is difficult to form a uniform coating layer 114 . Accordingly, there may be a problem in that the production yield is lowered. In addition, even when the viscosity is too high, it is difficult to uniformly apply the composition to form a uniform coating layer 114 .

2. Measurement of water repellency angle

After forming the coating layer 114 on the surface of the lens 113, the results of measuring the water repellency angle are shown in Table 3 below.

) advancing contact angle (

) stop contact angle (

) receding contact angle (

) TX1 117.1±2.9 112.1±4.1 < 10 TX2 132.4±1.5 131.5±2.7 141.7±3.4 TX3 138.9±3.0 138.9±2.7 139.8±3.7 TX4 136.9±2.0 135.6±2.6 140.4±3.4 TX5 116.9±0.7 115.4±3.0 < 10

As shown in Table 3, after forming the coating layer 114 using the coating composition of TX1 to TX5, the result of measuring the contact angle was confirmed. For TX1 and TX5, the receding contact angle was measured to be less than 10 degrees. That is, when it is out of the optimal range for preparing the coating composition, it was confirmed that the phenomenon of pinning water droplets occurs. On the other hand, it was confirmed that the peening phenomenon did not occur in TX2 to 4, thereby exhibiting an excellent waterproof effect.

3. Pollution resistance evaluation

The lens 113 having the coating layer 114 according to the embodiment formed on the outside of the facility was attached to the model camera, and exposed to the driving environment on a general road for 4 days. As the comparative example (Con), the same lens 113 on which the coating layer 114 was not formed was used, and in each embodiment, the model camera was attached to the same position of the vehicle.

Then, the degree of contamination of the lens 113 before and after the experiment was evaluated as related, and for objective comparison, the result was evaluated as an index of 1 to 10 in comparison with the comparative example in which the coating layer 114 was not formed. shown in In the following index, the lower the number, the better the stain resistance.

Con TX1 TX2 TX3 TX4 TX5 stain resistance 10 7 3 3 3 8

(Unit: Index) Referring to Table 4, when the coating layer 114 is formed on the lens 113, the lidar sensor or camera is installed on the outside of the vehicle and the lens 113 is exposed to the outside, but the high pollution resistance It can be seen that image data can be collected in a form that is easy to analyze over a long period of time. In particular, in the case of TX2 to TX4, it can be seen that the stain resistance by the coating layer 114 is very excellent.

4(a) and 4(b) are views showing a first image and a second image according to the present invention.

The first image is captured by the fixed first image acquisition unit 110, and the Field of View may be fixed. The second image is captured by the second image acquisition unit 120 installed in the manipulator 11 , and the Field of View is not fixed but can be continuously moved according to the operation of the manipulator 11 .

5 is a diagram illustrating a manipulator control method according to the present invention.

The subject performing the method for controlling the manipulator 11 according to FIG. 5 may be the controller 130 or the processor 131 included in the controller 130 . Also, in the description of the control method according to the present invention, content identical to or overlapping with the above description may be omitted.

Referring to FIG. 5 , the control method according to the present invention may start with generating and transmitting operation commands for the first image acquiring unit 110 and the second image acquiring unit 120 ( S1100 ).

The control method according to the present invention includes receiving a first image obtained by photographing the target 10 from the first image obtaining unit 110 ( S1210 ), and photographing the target 10 from the second image obtaining unit 120 . Receiving a second image (S1220), obtaining an ROI (Region of Interest) image based on depth information from the first image (S1310), and obtaining two-dimensional information from the second image (S1320) may include more.

In addition, the control method according to the present invention includes the step of recognizing and learning the characteristics of the target 10 based on the ROI image and the two-dimensional information (S1400) and based on the recognition result, the manipulator 11 controls the target 10 It may further include the step of controlling to perform the operation (S1500).

Receiving the first image of the target 10 from the first image obtaining unit 110 (S1210) may be a step of receiving the first image captured by the hand-to-eye method. Receiving the second image of the target 10 from the second image acquisition unit 120 (S1220) may be a step of receiving the second image captured by the hand-in-eye method. In the step of obtaining a region of interest (ROI) image based on depth information from the first image ( S1310 ), an ROI image including the target 10 may be obtained based on the depth information. The controller 130 may acquire a depth image based on the depth information of the first image, and may acquire an ROI image from among the depth images. The algorithm for acquiring the ROI image may be a YOLO algorithm. The YOLO algorithm is an algorithm for object detection, and may be an algorithm for finding an object in an image. The YOLO algorithm may detect the target 10 based on the depth information and obtain an ROI image for the target 10 . This will be described later with reference to FIG. 6 .

The step of obtaining the two-dimensional information from the second image ( S1320 ) may be a step of obtaining the two-dimensional information based on an image box of the second image and the outline of the target 10 . This will be described later with reference to FIG. 7 .

6 is a diagram illustrating a process of acquiring an ROI image based on a first image according to the present invention.

According to FIG. 6 , the step of obtaining an ROI (Region of Interest) image based on depth information from the first image (S1310) includes the step of obtaining a first image (original image, Original Image) (S1311), the first image The step of thresholding (S1312), the step of recognizing the target 10 by object detection for the thresholded first image (S1313), the depth of the recognition target 10 ( Depth) obtaining an image (S1314) and obtaining an ROI image for the target 10 from the depth image (S1315). Thresholding may be a method of segmenting an image by using a difference between the pixel intensity of the target 10 and the background intensity for object recognition. Mainly, thresholding can be performed with Opencv.

7 is a view showing a process of acquiring two-dimensional information based on a second image according to the present invention, and FIGS. 8 (a) and 8 (b) are the rotation angles based on the second image according to the present invention. It is a diagram showing the process of obtaining.

According to FIG. 7 , the step of obtaining two-dimensional information from the second image ( S1320 ) includes the step of thresholding the second image ( S1321 ) and the deep learning algorithm after the thresholding of the target 10 . It may include detecting a contour (S1322) and obtaining a rotation angle of the target 10 based on the contour (S1323).

8 (a) and 8 (b), the step of obtaining the rotation angle of the target 10 based on the outline (S1323) is a step of obtaining a reference line based on the image box for the second image, a plurality of It may include obtaining a representative contour among the contours of , and obtaining a two-dimensional angle between the representative contour and the reference line. In this case, the two-dimensional angle may be a rotation angle of the target 10 with respect to the reference line.

In this case, the representative outline may be one arbitrarily selected from among a plurality of outlines. In addition, the representative outline may be one selected according to a predetermined criterion among a plurality of outlines. As an example, the representative contour may be a contour having the longest length among the plurality of contours.

는 하기의 수학식 1을 기초로 생성될 수 있으며, 상술한 회전 방향은 회전 각도

가 0이 되는 방향일 수 있다. At this time, the rotation angle of the target 10

may be generated based on Equation 1 below, and the above-described rotation direction is a rotation angle

It may be a direction in which is 0.

[Equation 1]

(only,

: YOLO에서 사용되는 Bounding Box 좌표,

: Bounding Box coordinates used in YOLO,

: 회전각도, 단위는 °)

: Rotation angle, unit is °)

는 하기의 수학식 2를 예시로 할 수 있다. Further, the angle of rotation of the target 10 according to FIGS. 8 ( a ) and 8 ( b ).

may be exemplified by Equation 2 below.

[Equation 2]

9 is a diagram illustrating a process of controlling the operation of the manipulator according to the present invention.

According to FIG. 9 , the step of controlling the manipulator to perform an operation on the target 10 based on the recognition result ( S1500 ) includes obtaining the rotation direction of the manipulator based on the rotation angle ( S1510 ) and the rotation direction It may include controlling the operation of the manipulator based on ( S1520 ).

가 0°가 되는 방향일 수 있다. 이때, 도출되는 회전 방향은 2개의 방향일 수 있으며, 도 8에 따르면, 제어부(130)는 제1 방향 및 제2 방향 중 보다 작은 각도만큼 움직이는 방향을 회전 방향으로 선택할 수 있다. The step of obtaining the rotation direction of the manipulator based on the rotation angle (S1510) is the rotation angle.

It may be a direction in which 0° becomes. In this case, the derived rotation direction may be two directions, and according to FIG. 8 , the controller 130 may select a direction moving by a smaller angle among the first direction and the second direction as the rotation direction.

The controller 130 may rotate the manipulator so that the reference line of the second image and the representative outline coincide. Through this process, the controller 130 may control the operation of the manipulator to move the manipulator to a position for performing a predetermined operation with respect to the target 10 .

In addition, the control system according to the present invention may include the control unit 130 and the manipulator having the following specifications.

The present invention described above can be modeled as computer readable code on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. It also includes modeling in the form of a carrier wave (eg, transmission over the Internet). Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of this specification should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of equivalents of this specification are included in the scope of this specification.

Any or other embodiments of the present invention described above are not mutually exclusive or distinct. Any of the above-described embodiments or other embodiments of the present invention may be combined or combined with each configuration or function.

The above detailed description should not be construed as restrictive in all respects and should be considered as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

10: target
11: Manipulator
110: first image acquisition unit
120: second image acquisition unit
130: control unit

Claims (8)

A manipulator control system for vision recognition of a target, comprising:
a first image acquisition unit spaced apart from the manipulator to capture the target in order to recognize the vision;
a second image acquisition unit installed on the manipulator to capture the target for the vision recognition; and
and a control unit configured to learn the characteristics of the target from the images received from the first image acquisition unit and the second image acquisition unit, and control the manipulator based on the learning result.
According to claim 1,
The control unit is
receiving a first image from the first image acquisition unit, and acquiring depth information of the target from the first image through a hand-to-eye method.
According to claim 1,
The control unit is
receiving a second image from the second image acquiring unit, and acquiring two-dimensional information of the target from the second image through a Hand-in-Eye method.
4. The method of claim 3,
The two-dimensional information,
and a rotation angle and a rotation direction of the manipulator.
A method for controlling a manipulator for vision-recognizing a target through a first image acquisition unit spaced apart from a manipulator and a second image acquisition unit installed in the manipulator, the method comprising:
receiving a first image of the target from the first image acquisition unit;
receiving a second image of the target from the second image acquisition unit;
obtaining an ROI (Region of Interest) image based on depth information from the first image;
obtaining two-dimensional information from the second image;
learning and recognizing the characteristics of the target based on the ROI image and the two-dimensional information; and
and controlling, by the manipulator, an operation of the target based on the recognition result.
6. The method of claim 5,
The step of obtaining an ROI (Region of Interest) image based on depth information from the first image includes:
thresholding the first image;
detecting the target based on a deep learning algorithm after the thresholding;
acquiring a depth image of the detected target in the first image; and
Acquiring the ROI image based on the depth image; will include, a manipulator control method.
6. The method of claim 5,
The step of obtaining the two-dimensional information from the second image,
thresholding the second image;
detecting a contour of the target based on a deep learning algorithm after the thresholding; and
Acquiring the rotation angle of the target based on the outline; will include a manipulator control method.
8. The method of claim 7,
The step of the manipulator controlling the operation of the target comprises:
obtaining a rotation direction of the manipulator based on the rotation angle; and
and controlling the operation of the manipulator based on the rotation direction.

Images