KR102109696B1 - Robot hand for grasping unknown object and control method thereof - Google Patents

Abstract

The present invention relates to a robot hand and a control method thereof, and the present invention relates to a robot hand and a control method that can be handled by using a visual and tactile sensor without detailed modeling or programming work on an object. A robot hand, which is an aspect of the present invention, includes a camera that acquires an image of at least one first object; A control unit for determining at least one of a position coordinate, a size, and a center of the first object using the acquired image; And a hand for gripping the first object using the determined factor when at least one factor of the position coordinate, size, and center is determined, wherein the hand grips the first object. In case of failure, the control unit re-determines at least one of the position coordinates, size, and center of the first object based on the failure event, and the hand grips the first object again using the determined factor. Can be.

Description

A robot hand gripping an object without information and its control method {ROBOT HAND FOR GRASPING UNKNOWN OBJECT AND CONTROL METHOD THEREOF}

The present invention relates to a robot hand and a control method thereof, and the present invention relates to a robot hand and a control method that can be handled by using a visual and tactile sensor without detailed modeling or programming work on an object.

The robot hand means a machine hand that restrains or moves an object by a plurality of fingers, whereas the robot hand makes a general positioning within a wide working area, whereas the robot hand is a limited area. It is used for fine manipulations and grasping of objects.

In addition, the application of artificial intelligence and robot technology is spreading in various fields, and can be used as a core technology for breakthrough of handling and assembly of parts in manufacturing environments.

However, most of the operations performed in the actual manufacturing environment based on the above-mentioned robot hand are handling operations for holding and assembling parts, and most of the handling operations are performed by the number of human processes except for simple repetitive operations in a standardized environment. There is a problem that it is performed.

Therefore, in order to solve the above problems and increase production efficiency in response to changes in the manufacturing environment, it is urgent to develop a gripping / assembly technology equipped with artificial intelligence.

(1) Korea Patent Office Registration No. 10-1323217 (2) Republic of Korea Patent Office Registration No. 10-1048762

The present invention relates to a robot hand and a control method thereof, and the present invention is to provide a user with a robot hand and a control method thereof that can be handled using a visual and tactile sensor without detailed modeling or programming work on an object.

In addition, the present invention provides a handling technique that attempts to grasping by grasping the approximate location and shape of an object through a vision, learning a failed event according to an algorithm when performing task execution, and attempting regrasping. In addition, an object of the present invention is to provide a technique for further improving the efficiency of a learning algorithm by using a tactile sensor.

In addition, the present invention overcomes weaknesses such as long learning time by developing a previously proposed model-free method and a model-based method to develop a gripping and manipulation method through conventional machine learning, and has model uncertainty. The aim is to provide a handling technique that can be utilized even in situations.

In addition, an object of the present invention is to provide a handling technique capable of classifying and learning an object by measuring shear force, conductivity of an object, as well as measuring a normal force that is commonly used in the past by additionally utilizing a tactile sensor.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clearly understood by a person having ordinary knowledge in the technical field to which the present invention belongs from the following description. Will be understandable.

A robot hand, which is an aspect of the present invention for achieving the above technical problem, includes: a camera acquiring an image of at least one first object; A control unit for determining at least one of a position coordinate, a size, and a center of the first object using the acquired image; And a hand for gripping the first object using the determined factor when at least one factor of the position coordinate, size, and center is determined, wherein the hand grips the first object. In case of failure, the control unit may re-determine at least one of the coordinates, size, and center of the first object based on the failure event, and the hand may grip the first object again using the determined factor. have.

In addition, the controller may control the hand to hold the first object using only the determined factor in the absence of prior information on the first object.

In addition, when the hand re-fails gripping of the first object, the controller determines again at least one of the position coordinate, size, and center of the first object based on the re-failed event, and the hand The operation of controlling to hold the first object again may be repeated using the determined factor.

In addition, when the hand grips the first object again using the re-determined factor, the control unit determines the gripping stability of the first object gripped by the hand, and the gripping stability is less than or equal to a preset criterion. In this case, the hand may be controlled to release the first object.

In addition, when the hand is in contact with the first object for the gripping, a tactile sensor that senses at least one tactile factor of the normal force, shear force, and conductivity of the first object. ; When the hand holds the first object again using the re-determined factor, at least one tactile factor sensed by the tactile sensor may be used together.

Further, the hand includes a plurality of fingers, and each of the plurality of fingers includes a distance sensor that senses a distance from the first object, and when gripping the first object, the plurality of distance sensors Based on the obtained information, each of the plurality of fingers may move close to the first object at the same speed.

On the other hand, another aspect of the present invention for achieving the above technical problem is a control method of a robot hand, the camera comprising: a first step of acquiring an image of at least one first object; A second step in which a control unit determines at least one of a position coordinate, a size, and a center of the first object using the acquired image; A third step in which a hand grips the first object using the determined factor when at least one factor of the location coordinate, size, and center is determined; A fourth step in which the hand fails to grip the first object; A fifth step in which the controller again determines at least one of a position coordinate, a size, and a center of the first object based on the failure event; And a sixth step in which the hand grips the first object again using the determined factor.

In addition, in the third step, in the absence of prior information on the first object, the hand may hold the first object using only the determined factor.

Further, in the sixth step, when the hand re-fails gripping of the first object, the first to sixth steps may be repeatedly performed.

In addition, after the sixth step, a seventh step of determining the gripping stability of the first object gripped by the hand; And an eighth step in which the hand releases the first object when the gripping stability is below a preset criterion.

Further, in the third step, when the hand is in contact with the first object for the gripping, among the normal force, shear force and conductivity of the first object through a tactile sensor When detecting at least one tactile factor, and in the sixth step, when the hand grips the first object again using the determined factor, at least one tactile factor sensed by the tactile sensor may be used together. have.

The present invention relates to a robot hand and a control method thereof, and the present invention can provide a user with a robot hand and a control method thereof that can be handled by using a visual and tactile sensor without detailed modeling or programming work on an object.

According to the present invention, the existing proposed model-free method and model-based method are developed together to develop gripping and manipulation methods through machine learning to overcome weaknesses such as long learning time and can be utilized in situations where model uncertainty exists. So it can be a complementary development.

In addition, according to the present invention, a conventional tactile sensor capable of classifying an object by developing normal force measurement as well as shear force and measuring the conductivity of an object and measuring the conductivity of the object is developed and utilizes this tactile information like a human. It is possible.

In addition, the present invention provides real-time position / position / state recognition technology of parts using visual and tactile information, optimal gripping type reasoning technology for stable gripping of parts, and intelligent gripping technology based on recognition information and experience (single gripper / hand use) , 30 or more objects), position / direction manipulation (In-Hand) technology of parts using visual and tactile information, and experience-based assembly strategy learning technology for various parts. It is possible to solve the current problem being performed by the number of steps.

On the other hand, the effects that can be obtained in the present invention are not limited to the above-mentioned effects, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description. Will be able to.

1 is a block diagram illustrating the configuration of a robot hand system proposed by the present invention.
2 is a view for explaining the configuration of the robot hand proposed by the present invention.
Figure 3 shows a specific example of gripping the object according to the vision, control and machine learning of the present invention.
4 is a block diagram illustrating the configuration of a robot hand system according to the present invention for holding an object based on a vision system.
5 is a flowchart illustrating a method of learning a failed event according to an algorithm when grasping is attempted by grasping the approximate location, shape, etc. of an object according to the present invention through a vision, and when the task execution fails.
6 is a block diagram illustrating the configuration of a robot hand system according to the present invention for gripping an object by additionally using a tactile sensor in addition to the vision system.
7A and 7B illustrate an example of a specific shape of a robot gripping using vision information and tactile information together with respect to the present invention.
8 is a flowchart illustrating the operation of the robot gripping using vision information and tactile information together with respect to the present invention.
9 shows an example of a small-sized model and a three-dimensional position fitting used for the gripping of the present invention.
FIG. 10 specifically shows an example of applying to a grip by fitting to a three-dimensional position according to the present invention.
11 is a flowchart illustrating an overall operation of a robot gripping using vision information and tactile information together with respect to the present invention.
12 is a diagram illustrating an example of a flowchart illustrating object gripping using learning in relation to the present invention.
13 illustrates an example in which the gripping method changes according to a change in the direction and position of an object in relation to the present invention.
14 is a view for explaining the need for re-gripping according to the present invention.
15 shows an example of a re- gripping learning model according to the present invention.
16 is a view for explaining a method of performing re-gripping according to the probability of successful gripping of the present invention.
17 is a diagram for explaining the performance of a task to which re-phage is applied in connection with the present invention.
18 is a flowchart illustrating a method capable of accomplishing a specific task using regrasping when manipulation in accordance with a given purpose cannot be performed in the form of grasping in relation to the present invention.

Prior to the detailed description of the present invention, the configuration of the robot hand system applied to the present invention will be described with reference to the drawings.

1 is a block diagram illustrating the configuration of a robot hand system proposed by the present invention.

Referring to FIG. 1, the robot hand system 100 includes a wireless communication unit 110, an audio / video (A / V) input unit 120, a user input unit 130, a sensing unit 140, and an output unit 150, It may include a memory 160, an interface unit 170, a control unit 180, a power supply unit 190 and a robot hand 200.

However, since the components illustrated in FIG. 1 are not essential, a robot hand system having more or fewer components may be implemented.

Hereinafter, the components will be described in turn.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the robot hand system and the wireless communication system or between a device and a network in which the device is located.

For example, the wireless communication unit 110 may include a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115.

The mobile communication module 112 transmits and receives wireless signals to and from at least one of a base station, an external device, and a server on a mobile communication network.

It may include various types of data according to the transmission and reception of text / multimedia messages.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be built in or external to the robot hand system. Wireless Internet technology (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) may be used.

The short-range communication module 114 refers to a module for short-range communication. Short-range communication technology includes Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Wireless Fidelity (Wi-Fi), etc. Can be used.

The location information module 115 is a module for acquiring the location of the robot hand system, and a representative example thereof is a Global Position System (GPS) module.

Referring to FIG. 1, the A / V (Audio / Video) input unit 120 is for inputting an audio signal or a video signal, which may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving pictures obtained by an image sensor in a shooting mode. The processed image frame may be displayed on the display unit 151.

The image frames processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110.

Two or more cameras 121 may be provided according to the use environment.

The microphone 122 receives an external sound signal by a microphone in a recording mode, a voice recognition mode, etc., and processes it as electrical voice data. The processed voice data may be converted and output in a form that can be transmitted to the mobile communication base station through the mobile communication module 112. Various noise reduction algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone 122.

The user input unit 130 generates input data for the user to control the operation of the robot hand system. The user input unit 130 may be configured with a key pad dome switch, a touch pad (static pressure / blackout), a jog wheel, a jog switch, or the like.

The sensing unit 140 detects the current state of the robot hand system, such as the open / closed state of the robot hand system, the position of the robot hand system, the presence or absence of user contact, the orientation of the robot hand system, the acceleration / deceleration of the robot hand system, and the like. Generates a sensing signal to control the operation of.

The sensing unit 140 may sense whether the power supply unit 190 is supplied with power, whether the interface unit 170 is coupled with external devices, or the like.

Meanwhile, the sensing unit 140 may include a proximity sensor (not shown).

The output unit 150 is for generating output related to vision, hearing, or tactile sense, which includes a display unit 151, an audio output module 152, an alarm unit 153, a haptic module 154, and a projector module ( 155).

The display unit 151 displays (outputs) information processed by the robot hand system.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible) display) and a 3D display.

Some of these displays may be of a transparent type or a light transmissive type so that the outside can be seen through them. This may be referred to as a transparent display, and a typical example of the transparent display is TOLED (Transparant OLED). The rear structure of the display unit 151 may also be configured as a light transmissive structure.

Two or more display units 151 may be present depending on the implementation form of the robot hand system. For example, in the robot hand system, a plurality of display units may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces.

When the display unit 151 and a sensor that senses a touch operation (hereinafter referred to as a “touch sensor”) form a mutual layer structure (hereinafter, referred to as a “touch screen”), the display unit 151 may be used in addition to an output device. It can also be used as an input device. The touch sensor may have a form of, for example, a touch film, a touch sheet, or a touch pad.

The touch sensor may be configured to convert changes in pressure applied to a specific portion of the display unit 151 or capacitance generated in a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only the touched position and area, but also the pressure at the time of touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and then transmits corresponding data to the controller 180. Accordingly, the control unit 180 can know which area of the display unit 151 has been touched, and the like.

The proximity sensor (not shown) may be disposed in an inner area of the robot hand system wrapped by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object present in the vicinity without mechanical contact using electromagnetic force or infrared rays. Proximity sensors have a longer lifespan and higher utilization than contact sensors.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, it is configured to detect the proximity of the pointer due to a change in electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of description, an act of causing the pointer to be recognized as being located on the touch screen without being touched by the pointer on the touch screen is referred to as a “proximity touch”, and the touch The act of actually touching the pointer on the screen is referred to as "contact touch". The location on the touch screen that is a proximity touch with a pointer refers to a location where the pointer corresponds vertically to the touch screen when the pointer is touched close.

The proximity sensor detects a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

In addition, the robot hand system 100 according to the present invention may include a gyro sensor 141.

The gyro sensor 141 is a sensor that measures a change in azimuth of an object using a property that maintains a predetermined direction that is always initially set with high accuracy regardless of the rotation of the earth, and the gyroscope has a mechanical method and optical type using light. have.

In addition, the robot hand system 100 according to the present invention may include an acceleration sensor 142.

The acceleration sensor 142 processes output signals and measures dynamic forces such as acceleration, vibration, and impact of an object.

When the acceleration sensor 142 is largely classified as a detection method, there are inertial, gyro, and silicon semiconductor types, and an accelerometer or an inclinometer may also be considered as one type of acceleration sensor.

In addition, the robot hand system 100 according to the present invention may include a pressure sensor 143.

The pressure sensor 143 refers to a device and element that detects the pressure of a liquid or gas and converts it into an electrical signal that is easy to use for measurement or control.

As the principle of measurement, there are very many types, such as displacement and strain, and using thermal conductivity of molecular density. Recently, strain gauge type pressure sensors made of silicon have been developed and used for precise pressure measurement. Integrated pressure sensors have also been developed in which integrated circuits are made on the same substrate to process signals.

In addition, the robot hand system 100 according to the present invention may include a tactile sensor 144.

The tactile sensor (144, Tactile Sensor) is a pressure sensor intended to artificially realize human tactile sense in a robot, and is largely divided into a contact sensor, a pressure sensor, a slip sensor, and a temperature sensor. It is necessary to implement a human advanced tactile system. Technology.

A tactile sensor array is a sensor for obtaining two-dimensional information by arranging several to tens of contact angle sensors 144 or pressure sensors in a flat shape, and can also be used to detect shape or motion.

Many of these sensors constitute an array-type sensor by dividing either of a conductive rubber, a piezoelectric polymer, or electrodes on both sides of a pressure-sensitive polymer, and the two-dimensional pressure distribution is detected from the resistance change or voltage output between the counter electrodes.

In particular, the tactile sensor 144 according to the present invention can detect at least one of a normal force, a shear force, and conductivity of an object that the robot hand 200 contacts.

Meanwhile, the audio output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a recording mode, a voice recognition mode, a broadcast reception mode, or the like. The sound output module 152 also outputs sound signals related to functions performed in the robot hand system. The sound output module 152 may include a receiver, a speaker, and a buzzer.

The alarm unit 153 outputs a signal for notifying the occurrence of an event in the robot hand system.

The alarm unit 153 may output a signal for notifying the occurrence of an event by vibrating in a form other than a video signal or an audio signal, for example, vibration.

The video signal or audio signal may also be output through the display unit 151 or the audio output module 152, so that they 151 and 152 may be classified as part of the alarm unit 153.

The haptic module 154 generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic module 154 is vibration. The intensity and pattern of vibration generated by the haptic module 154 can be controlled.

For example, different vibrations may be synthesized and output or sequentially output.

In addition to vibration, the haptic module 154 is used for stimulation such as pin arrangement that vertically moves with respect to the contact skin surface, jet or suction power of air through a jet or intake, grazing on the skin surface, contact with an electrode, and electrostatic force. Various tactile effects can be generated, such as an effect and an effect of reproducing a feeling of cold and warm using an element capable of absorbing heat or generating heat.

The haptic module 154 may not only deliver the tactile effect through direct contact, but may also be implemented so that the user can feel the tactile effect through muscle sensations such as fingers or arms. Two or more haptic modules 154 may be provided according to a configuration aspect of the robot hand system.

The projector module 155 is a component for performing an image project function using a robot hand system, which is the same as or at least the image displayed on the display unit 151 according to the control signal of the controller 180 Some of the other images can be displayed on an external screen or wall.

Specifically, the projector module 155 generates a light source (not shown) that generates light (for example, laser light) for outputting an image to the outside, and generates an image to be output to the outside using light generated by the light source. It may include an image generating means (not shown), and a lens (not shown) for expanding and outputting the image at a certain focal length to the outside. In addition, the projector module 155 may include a device (not shown) that can adjust the image projection direction by mechanically moving the lens or the entire module.

The projector module 155 may be divided into a cathode ray tube (CRT) module, a liquid crystal display (LCD) module, and a digital light processing (DLP) module depending on the type of device of the display means. In particular, the DLP module may be advantageous for miniaturization of the projector module 151 by expanding and projecting an image generated by reflecting light generated from a light source to a DMD (Digital Micromirror Device) chip.

Preferably, the projector module 155 may be provided in the longitudinal direction on the side, front or rear of the robot hand system. Of course, the projector module 155 can be provided in any position of the robot hand system, if necessary.

Meanwhile, the memory 160 may store programs for processing and control of the controller 180 and temporarily store input / output data (eg, messages, audio, still images, videos, etc.). It can also perform the function. The frequency of use for each of the data may also be stored in the memory unit 160. In addition, the memory unit 160 may store data related to various patterns of vibration and sound output when a touch is input on the touch screen.

The memory 160 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), RAM (Random Access Memory, RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic It may include a storage medium of at least one type of disk, optical disk. The robot hand system may operate in connection with a web storage that performs a storage function of the memory 160 on the Internet.

The interface unit 170 serves as a passage with all external devices connected to the robot hand system. The interface unit 170 receives data from an external device, receives power, and transmits data to each component in the robot hand system, or allows data in the robot hand system to be transmitted to the external device. For example, wired / wireless headset port, external charger port, wired / wireless data port, memory card port, port for connecting devices equipped with an identification module, audio input / output (I / O) port, A video input / output (I / O) port, an earphone port, and the like may be included in the interface unit 170.

The identification module is a chip that stores various information for authenticating the usage rights of the robot hand system, a user identification module (UIM), a subscriber identification module (SIM), and a universal user authentication module (Universal Subscriber) Identity Module, USIM). The device provided with the identification module (hereinafter referred to as 'identification device') may be manufactured in a smart card format. Therefore, the identification device can be connected to the robot hand system through the port.

When the robot hand system is connected to an external cradle, the interface unit becomes a passage through which power from the cradle is supplied to the robot hand system, or various command signals input from the cradle by a user are transmitted to the mobile device. It can be a passage. Various command signals or power inputted from the cradle may be operated as signals for recognizing that the mobile device is correctly mounted on the cradle.

The controller 180 controls the overall operation of the robot hand system.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power required for the operation of each component.

The various embodiments described herein can be implemented in a computer- or similar device-readable recording medium using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented by the controller 180 itself.

According to a software implementation, embodiments such as procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code can be implemented in a software application written in an appropriate programming language. The software code may be stored in the memory 160 and executed by the controller 180.

In addition, the system 100 according to the present invention may include a robot hand 200.

The robot hand 200 largely includes a palm portion 210 and a finger portion 200.

The finger part 200 may be configured to have a structure having at least one node, and it is also possible to move at the same speed with respect to an object through a distance sensor (not shown).

The object according to the present invention may be an object in which information is not present at all, an object in which information received from the outside exists, or an object in which information in which information is accumulated through a failure event exists. .

A state in which information about an object according to the present invention exists includes not only a state in which complete information about the object is built, but also a state in which information is accumulated to some extent through failure events and performance.

On one side, the robot hand 200 may be provided in the system 100 in plural, and performing a task having a specific order and coupling direction as well as a simple gripping operation through the plurality of robot hands 200. It is possible.

Based on the configuration of the system 100 of the present invention described above, a specific technique to be proposed in the present specification will be described with reference to the drawings.

Embodiment 1-Technology for handling unknown objects using visual sensors

Most of the operations performed in the actual manufacturing environment based on the conventional robot hand are handling operations for holding and assembling parts, and most of the handling operations are performed by a number of human processes except simple repeating operations in a standardized environment. There was a problem.

Therefore, the present invention seeks to solve the above problems and propose a phage / assembly technology equipped with artificial intelligence to increase production efficiency in response to changes in the manufacturing environment.

That is, the present invention attempts to grasping by grasping the approximate location, shape, etc. of an object through vision based on the above-described system 100 configuration, and in case of failing to perform the task, learn the failed event according to the algorithm In addition, it will provide a handling technique that attempts to perform regrasping, and additionally, a technique that increases the efficiency of the learning algorithm by using a tactile sensor.

In addition, the present invention overcomes weaknesses such as long learning time by developing a previously proposed model-free method and a model-based method to develop a gripping and manipulation method through conventional machine learning, and has model uncertainty. The aim is to provide a handling technique that can be utilized even in situations.

2 is a view for explaining the configuration of the robot hand proposed by the present invention.

Referring to FIG. 2, a robot hand 200 for gripping an object 10 is shown.

In addition, a plurality of cameras 121 are arranged in the system 1 according to the present invention.

Here, the controller 180 may perform an operation of gripping the object 10 by controlling the robot hand 200 in a situation where there is no information on the object 10.

That is, in the present invention, since there is no acquired information or learned information about an object, the robot is based on the vision system through a plurality of cameras 121 to identify the location and shape of the object, and based on the identified information The object 10 may be gripped by controlling the hand 200.

Specifically, in this case, the controller 180 determines at least one of the coordinates, size, and center of the object 10 using the acquired image, and the robot hand 200 is the object 10 directly based on the identified information. It can be controlled to grip.

In this case, the controller 180 may allow the robot hand 200 to grip the object 10 based on power grasping rather than precise grasping.

Since the robot hand 200 does not grip the object 10 based on the accurate information, an event of releasing the object 10 may not occur or the object 10 may be released after the object 10 is not gripped. .

At this time, in the present invention, a machine learning algorithm for a failed event is applied.

That is, when the robot hand 200 fails to grip the object 10, the controller 180 determines again at least one of the coordinates, size, and center of the object 200 based on the failure event.

Thereafter, the robot hand 200 attempts to grip the object 10 again using the determined factor.

Such a machine learning algorithm may be repeatedly performed when the object 10 cannot be gripped, and, in the end, uses much less trial, computation, and time than the method of gripping the object 10 after obtaining specific information. By doing so, the object 10 can be gripped.

Figure 3 shows a specific example of gripping the object according to the vision, control and machine learning of the present invention.

Referring to FIG. 3, the robot hand 200 according to the present invention includes one palm portion 210 and three finger portions 220, and each finger portion 220 is composed of a plurality of nodes.

Referring to (a) of FIG. 3, the robot hand 200 according to the present invention can grip the object 10 through an operation of opening and closing a plurality of finger parts 220 based on the palm part 210. have.

That is, based on the image obtained through the camera 121, the controller 180 determines at least one of the coordinates, size and center of the object 10, and the robot hand 200 is immediately based on the identified information. Control to hold the object (10).

At this time, when the robot hand 200 does not grip the object 10 or an event to release the object 10 occurs after gripping, the controller 180 coordinates the object 200 based on the failure event , Re-determining at least one of the size and center, and the robot hand 200 attempts to grip the object 10 again using the determined factor.

According to the machine learning algorithm, the appearance of the robot hand 200 gripping the object 10 may be changed.

That is, as shown in (a) of FIG. 3, an attempt can be made to grip on the right side of the object 10, and an attempt can be made to grip on the left side, as shown in (b), shown in (c) It is also possible to grasp through the other finger portion 220 from the left side while supporting the lower end of the object 10 as shown.

4 is a block diagram illustrating the configuration of a robot hand system according to the present invention for holding an object based on a vision system.

Referring to FIG. 4, a tactile sensor 144 may be additionally used in addition to the configuration using the vision system of the system 100, which will be described later in detail with reference to the drawings.

In addition, in gripping the object 10, the controller 180 may determine the grasping safety of the unknown object 10 currently held by using vision information (tactile sensor information can be used together).

Specifically, the controller 180 may check the safety by measuring the required force / torque as a whole while moving the gripping object 10.

In addition, the controller 180 may estimate the force / torque required in the current grasping form in consideration of the expected physical properties of the object 10.

At this time, when it is determined that the grasping safety of the robot hand 200 is lower than a preset value, the controller 180 controls the robot hand 200 to release the object 10 and re-grasp based on the learned information. It might be.

It is also possible to grasp the optimal conditions with vision information and tactile information for regrasping, and to grasp objects according to the purpose of holding and manipulating objects during the regrasing process.

5 is a flowchart illustrating a method of learning a failed event according to an algorithm when grasping is attempted by grasping the approximate location, shape, etc. of an object according to the present invention through vision.

Referring to FIG. 5, first, a step S10 in which the camera 121 acquires an image of at least one first object 10 is performed.

Here, the present invention does not have any prior information on the first object 10, and attempts to grip the first object 10 based on only the information obtained through the camera 121.

Thereafter, the control unit 180 determines at least one of coordinates, size, and center of the first object 10 using the obtained image (S11).

In step S11, the controller 180 does not acquire all information about the coordinates, size, and center through the image, but when at least one factor among coordinates, size, and center is determined, the hand is determined using the determined factor ( 200) is controlled to grip the first object (10) (S12).

In step S12, since the robot hand 200 does not have accurate information, gripping of the first object 10 may fail (S13).

For example, the degree to which the first object 10 does not exist in the place where the robot hand 200 has moved or the finger 220 of the robot hand 200 is opened is less than the width of the first object 10. An event in which the first object 10 cannot be gripped may be generated due to factors such as the robot hand 200 moving toward a center different from the center of the expected one object 10.

At this time, the controller 180 again determines at least one of the coordinates, size, and center of the first object based on the failure event (S14), and the robot hand 200 uses the determined factor again to first object 10 Will be gripped again (S15).

Until the object 10 is successfully grasped, steps S13 to S15 according to the present invention may be repeatedly performed.

In addition, even if the object 10 gripping is successful, as described above, if it is determined that the gripping is unstable through the stability test, the object 10 gripping may be released and re- gripped.

Accordingly, according to the present invention, the existing proposed model-free method and model-based method are developed together to develop gripping and manipulation methods through machine learning, and overcome weaknesses such as long learning time and utilize them in situations where model uncertainty exists. So that it can be a complementary development.

Embodiment 2-Technology for handling objects using camera and tactile sensor together

In the present invention, in order to increase the efficiency of machine learning described in the first embodiment, a technique in which the robot hand 200 attempts to grip using a tactile sensor 144 in addition to the camera 121 is proposed.

That is, the present invention overcomes weaknesses such as long learning time by developing a previously proposed model-free method and a model-based method for the development of a gripping and manipulation method through conventional machine learning, and has model uncertainty. The aim is to provide a handling technique that can be utilized even in situations.

In addition, in the second embodiment, a tactile sensor 144 is additionally used to provide a handling technique capable of classifying and learning an object by measuring shear force, conductivity of the object, etc. It is aimed at.

6 is a block diagram illustrating the configuration of a robot hand system according to the present invention for gripping an object by additionally using a tactile sensor in addition to the vision system.

The configuration shown in FIG. 6 is the same as that shown in FIG. 2 described above, but additionally uses a tactile sensor 144.

The tactile sensor 144 according to the present invention may sense at least one of a normal force, shear force, and conductivity of the object 10 held by the robot hand 200.

Referring to FIG. 6, a plurality of cameras 121 are disposed in the system 1 according to the present invention, and a plurality of tactile sensors 144 are provided on at least a portion of the finger unit 220 of the robot hand 200. Can be.

In the present invention, since there is no acquired information or learned information about an object, a robot hand is obtained based on the vision system through a plurality of cameras 121 to grasp the approximate location and shape of the object, and based on the identified information. The object 10 can be gripped by controlling the 200, and specifically, the controller 180 determines at least one of the coordinates, size, and center of the object 10 using the obtained image, and displays the identified information. As a basis, the robot hand 200 can be controlled to grip the object 10 immediately.

Since the robot hand 200 does not grip the object 10 based on the accurate information, an event of releasing the object 10 may not occur after the object 10 cannot be gripped or gripped. , In the present invention, a machine learning algorithm for a failed event is applied.

That is, when the robot hand 200 fails to grip the object 10, the controller 180 determines again at least one of the coordinates, size, and center of the object 200 based on the failure event.

In this case, unlike the one described in FIG. 2, in FIG. 6, when the finger 220 of the robot hand 200 contacts the object 10 for gripping, the object 10 obtained through the tactile sensor 144 At least one of the normal force, shear force, and conductivity of is additionally utilized for re-gripping.

That is, since the robot hand 200 uses the determined factor and at least one tactile factor sensed by the tactile sensor 144 again to grip the object 10 again, much fewer attempts and computations than the vision system alone , It is possible to grip the object 10 using time.

7A and 7B illustrate an example of a specific shape of a robot gripping using vision information and tactile information together with respect to the present invention.

Referring to FIG. 7A (a), the robot hand 200 according to the present invention includes one palm portion 210 and two finger portions 220, and each finger portion 220 is a plurality of nodes. It is composed.

Robot hand 200 according to the present invention, as shown in (a) to (d) of Figure 7a, the object through the operation of opening and closing a plurality of finger parts 220 relative to the palm portion 210 ( 10) can be gripped.

That is, based on the image obtained through the camera 121, the controller 180 determines at least one of the coordinates, size and center of the object 10, and the robot hand 200 is immediately based on the identified information. Control to hold the object (10).

At this time, when the robot hand 200 in FIG. 7A (d) fails to grip the object 10 or an event to release the object 10 occurs after gripping, the controller 180 based on the failure event At least one of the coordinates, size, and center of the raw object 200 is determined again, and the robot hand 200 attempts to grip the object 10 again using the determined factor. When the finger 220 is in contact with the object 10 for gripping, among the normal force, shear force, and conductivity of the object 10 obtained through the tactile sensor 144 At least one is further utilized for re- gripping.

The robot hand 200 gripping the object 10 according to the machine learning algorithm may be used to perform a specific task in addition to the simple gripping.

That is, as shown in FIG. 7B, two robot hands 200 are provided, and the first robot hand 200a includes a first palm portion 210a and a first finger portion 220a, and a second The robot hand 200b may include a first palm portion 210b and a first finger portion 220b.

In addition, in FIG. 7B, the first robot hand 200a and the second robot hand 200b are used to perform an operation of inserting the second object 10b into the first object 10a.

Since it is necessary to perform an operation of inserting the second object 10b into the first object 10a rather than a simple gripping operation, the two robot hands 200 may continuously fail to grip, and the vision system described above may The following learning algorithm is used repeatedly.

Furthermore, at least one of the normal force, shear force, and conductivity of the object 10 obtained through the tactile sensor 144 is further utilized for re-gripping.

For example, as in the case of manipulating the water bottle 10 containing only half of the water, grasping & manipulating an object while adjusting force according to the disturbance of an object (effect of changing the center of gravity) using tactile information.

Also, in the present invention, the shape of the hand 200 holding the object 10 may be calculated and applied in advance using the learned information.

In addition, as described above, in order to perform a specific task (task), an algorithm for changing the method of holding the object 10, the direction, and the like for the purpose of holding and manipulating the object 10 may be applied.

The tactile information used in the second embodiment includes information such as normal force, shear force, and electrical conductivity, and additionally, information that can be caught while in contact with the external environment (such as the bottom surface) when holding the small object 10 It can also be obtained additionally.

Therefore, precise assembly work may be possible by measuring precise force (including shear force) by using vision information and tactile information together.

8 is a flowchart illustrating the operation of the robot gripping using vision information and tactile information together with respect to the present invention.

Referring to FIG. 8, first, a step S20 in which the camera 121 acquires an image of at least one first object 10 is performed.

Thereafter, the controller 180 determines (S21) at least one factor among coordinates, size, and center of the first object using the obtained image (S21), and the hand 200 is the first using the determined factor. The object 10 is gripped (S22).

At this time, unlike the method in Figure 5, in the second embodiment, when the hand 200 is in contact with the first object 10 for the gripping, the normal force, shear force of the first object 10 A step S22 in which the tactile sensor 144 senses at least one of (shear force) and conductivity is performed.

After step S22, when the robot hand 200 fails to grip the first object 10 due to insufficient information (S23), the controller 180 based on the failure event, the coordinate, size, and center of the first object At least one of the factors is determined again (S24).

Based on the step S24, when the hand 200 wants to hold the first object again, unlike the method in FIG. 5, the second embodiment uses the determined factor and at least one tactile factor detected by the tactile sensor again. The first object is gripped again (S25).

Therefore, according to the present invention, a conventional tactile sensor capable of classifying an object by developing normal force measurement as well as shear force and measuring the conductivity of an object and measuring the conductivity of the object can be developed, and such tactile information can be utilized like a human. Do.

In addition, real-time position / posture / status recognition technology of parts using visual and tactile information, optimum gripping type reasoning technology for stable gripping of parts, and intelligent gripping technology based on recognition information and experience (single gripper / hand use, 30 types) Ideal object), position / direction manipulation (In-Hand) of parts using visual and tactile information, and learning strategy of assembling various parts based on experience. Can solve the current problem being performed by the.

Third embodiment-a method of more accurately gripping an object based on learned information

In the third embodiment according to the present invention, when information about the object 10 is learned based on the above-described first and second embodiments, an additional method for gripping the object 10 faster and more accurately To suggest.

Additional methods applied in the third embodiment are as follows.

1) How to solve the difference between the pre-stored reference image and the input image through an error function

2) How to determine the small-sized model by determining the two-dimensional point in determining the gripping part, and fitting it to the three-dimensional position to apply it to the phage

First, a description will be given of a method of solving a difference between a reference image and an input image stored in advance through an error function.

In the first method, the system 100 may store information related to the object 10 in advance through the memory 160 or receive it from the outside through the wireless communication unit 110.

In addition, information about the object 10 may be obtained while attempting to grip the object 10 through the camera 121 or the tactile sensor 144.

At this time, the controller 180 compares information about the object 10 previously stored or received from the outside with information obtained through the camera 121, and determines the degree of error, and uses or obtained the previously stored information. And a method of modifying information.

At this time, the control unit 180 is an error function for comparing information about the object 10 previously stored or received from the outside with information obtained through the camera 121, etc., as follows.

Here, the cover error is εcover (u, v, x, y) and the range error is range (u, v, x, y, z). These error terms are evaluated for each pixel in the coordinates (u, v) of the input range image. The pixel translation value (x, y, z) of the reference range image determines its position with respect to the input range image.

The function is summed across all image pixels (u, v) using the weight λ (eg λ = 10). The standardization coefficients Ncover and Nrange separate errors from object and image size. The error can be minimized if the image R is aligned with a possibly partially shielded object in the input image I.

Next, a description will be given of a method of determining a small-sized model by determining a two-dimensional point in determining a gripping part, and fitting it to a three-dimensional position to apply to the grip.

9 shows an example of a small-sized model and a 3D position fitting used in the grip of the present invention, and FIG. 10 specifically shows an example of fitting to a grip by fitting to a 3D position according to the present invention.

Referring to FIG. 9, the control unit 180 accepts a designation of a “recognition area” and a range to be performed in a captured image through an input device. Here, the “recognition area” is an area set by the control unit 180 into the captured image in order to designate a part to be gripped by the hand 200. The recognition area is DT that can be used as a target area for distance measurement.

Referring to (a) of FIG. 9, the recognition areas 11, 12, and 13 representing the area that the control unit 180 wants to grip in the object 10 are illustrated.

The recognition area 11 designates a body part of the cup, the recognition area 12 designates a border portion of the cup opening, and the recognition area 13 designates a handle part.

Also, as illustrated in FIG. 9B, an icon list 20 representing a two-dimensional figure associated with the corresponding recognition area 13 may be further displayed on the display unit 151.

The icons 21 to 24 shown in Fig. 9B show examples of small-sized models, respectively.

That is, the icon 41 is a square column model, the icon 42 is a flat panel model, the icon 43 is a cylindrical model, and the icon 43 is a truncated cone model.

Each miniature model has shape parameters to define shape and size, and placement parameters to define position and posture.

In addition, the type of the miniature model is not limited to that shown in FIG. 9 (b), and various shapes such as a secondary ellipsoid, L-shaped footnote, and C-shaped cylinder may be further used.

9 (c), it is a conceptual diagram showing the acquisition of the three-dimensional position data 30 of the work space corresponding to the recognition areas 11, 12, and 13.

The 3D position data 30 in FIG. 9C shows the depth seen from the robot 200 of the work space together with the recognition area.

In addition, referring to Fig. 9 (d), the small-sized model (specifically, the cylindrical model 40) is superimposed and shows the fitting result when the user selects the cylindrical model.

In addition, referring to FIG. 10, the data structure is a specific example and shows the data content of the gripping pattern applicable to the cylindrical model 40.

When the types of the robot hand 200 are different, the executable gripping patterns are different, and the gripping patterns applicable to the same compact model may also be different.

That is, in Fig. 10, the hand 200 is a gripping pattern applicable when the hand type is a "flat 3 joint 2 finger hand" and the compact model is a cylindrical model, and 4 gripping patterns are recorded together with the application conditions.

Specifically, as the four gripping patterns, a side scissors grip, a cross-section scissors grip, a gripping grip, and an edge face scissors grip can be used.

Therefore, in the present invention, based on the acquired image, the controller 180 determines a 2D model corresponding to a point to be gripped by the robot hand 200, and determines the 2D model as a 3D model through depth information. The robot hand 200 may grip the object 10 by determining an optimized posture for gripping the determined 3D model.

Through this, the robot hand 200 can quickly grip the object 10 without obtaining additional information based on the acquired information.

4th Example  - vision  A method of performing a task through an operation of grasping based on information and tactile information

In addition, in the present invention, a description will be given of a method in which the robot hand 200 grips the object 10 and performs a specific task based on the first to third embodiments described above.

11 is a flowchart illustrating an overall operation of a robot gripping using vision information and tactile information in connection with the present invention, and FIG. 12 shows an example of a flowchart illustrating object gripping using learning in connection with the present invention. Did.

Referring to FIG. 11, based on the information obtained from the tactile sensor 144 and the non-metallic sensor 145 and the information obtained through the camera 122, the control unit 180 displays properties such as physical properties of the object 10. Predict and attempt to grip the object 10 based on this.

At this time, the gripping on the object may fail, and the controller 180 adjusts a factor for re- gripping based on the information obtained through the camera 122 whenever a failure event occurs, and the tactile sensor 144 The central tactile process (S30) is performed based on the information obtained through the information.

That is, in step S30, the control signal is fed back, pressure, rough information is provided, or property information about the object 10 is transmitted to the controller 180.

At this time, the control unit 180 learns information about the object 10 based on the information obtained through the camera and the information obtained through the step S30 (S31), and based on the learned information to the object 10 The gripping condition for the reset is set (S32), and a specific task can be performed by repeatedly gripping the object 10 based on the reflected reset condition.

Referring to FIG. 12, a process 1100 of learning based on information obtained through a tactile sensor 144 and a process of learning based on information acquired through a camera 122 are illustrated, and such a view is illustrated. A process 1200 of a robot hand performing a gripping operation based on input information and tactile input information is illustrated.

Fifth embodiment-Method of performing a task through a re-grasp operation

On the other hand, in order for the robot hand 200 to perform a specific task, it may be necessary to use regrasping when manipulation cannot be performed for a given purpose in the current grasping form.

Therefore, in the present invention, the purpose of the present invention is to propose a technique capable of accomplishing a specific task using regrasping when the robot hand cannot perform manipulation for a given purpose in a current grasping form in order to perform a specific task. Is done.

In other words, the present invention is an optimal algorithm required for the two-arm robot hand 200 to assemble an object, and if the assembly operation cannot be performed while the object is currently being held, place one or both objects and re-assemble to enable the assembly operation. It aims to propose.

In addition, according to the present invention, the tactile sensor 144 is additionally used in the re-gripping operation, as well as the normal force measurement which is commonly used, and the shearing force, the conductivity of the target object, and the like, and the handling technique capable of classifying and learning the object. It is aimed at providing.

13 illustrates an example in which the gripping method changes according to a change in the direction and position of an object in relation to the present invention.

Referring to (a) of FIG. 13, the position or direction of the object 10 may be changed by an external force or an action to be gripped.

In addition, a plurality of objects 10 exist, and in order to perform a specific task, a sequence in which the plurality of objects 10 are combined is determined. An event for holding the object 10 that does not correspond to the order May occur.

In this case, as shown in (b) of FIG. 13, gripping on the object 10 should be performed in consideration of the changed direction and position of the changed object 10.

In addition, as shown in (b) of FIG. 13, a plurality of objects 10 exist, and in order to perform a specific task, the order in which the plurality of objects 10 are combined is determined. When an event for gripping the object 10 that does not correspond to occurs, the method of releasing the gripped object 10 and newly gripping the object 10 must be applied.

In addition, in re-grasping, a method of gripping a point of the object 10 optimized for the next process, or a method of gripping the number 10 of the object 10 by changing the position, direction, etc. of the object 10 to be suitable for the next process may also be used. Can be.

14 is a view for explaining the need for re-gripping according to the present invention.

In FIG. 14, a plurality of objects 10 exist, and an object is to perform a specific task of assembling the plurality of objects 10.

At this time, the control unit 180 may store information on a sequence of assembling the plurality of objects 10 in order to perform a specific task or receive instructions from the outside through the communication unit 110.

14 (a), the object 10 may be completed only when the first object 10a, the second object 10b, and the third object 10c are sequentially combined, first, After the object 10a and the second object 10b are combined, the corresponding task can be performed only when the third object 10c is combined with the second object 10b.

However, at this time, as illustrated in (b) of FIG. 14, an event is performed in which a plurality of robot hands 200 first grasp the second object 10b and the third object 10c to perform the operation of combining them. Can be.

Through the operation as shown in FIG. 14 (b), the second object 10b and the third object 10c are combined and positioned on the left, as shown in FIG. 14 (c), and the first object 10a ) May be located on the right side.

At this time, as illustrated in (d) of FIG. 14, an event in which the first object 10a is combined is generated at the bottom of the form in which the second object 10b and the third object 10c are combined, resulting in an object. (10) A problem that cannot be assembled occurs.

Therefore, in this case, in a predetermined order, first, the first object 10a and the second object 10b are combined, and then the third object 10c is combined with the second object 10b to perform the corresponding task. This can be done through re-grasping.

That is, in FIG. 14B, the first object 10a and the second object 10b must be gripped by the two robot hands 200, but the second object 10b and the third object 10c are gripped. When an event occurs, the robot hand 200 may release the gripped objects, and then grip the correct object again.

In addition, in the present invention, when an erroneous event occurs, it is also possible to change the object target and order to be gripped in response to the event to perform re-grasping.

That is, when an incorrect event occurs by gripping the second object 10b and the third object 10c in FIG. 14C, the second object 10b and the third object 10c in response to the event ) Releases the combined structure, rotates the combined structure to the left or right, or moves and rotates the first object 10a to the sinus side, and then performs a re-gripping operation to combine the combined structure and the first object 10a. Through it, you can perform specific tasks.

15 shows an example of a re- gripping learning model according to the present invention.

15, in ①, the robot hand 200 succeeds in gripping the object 10 vertically.

However, in performing a specific task, the controller 180 may determine that the form of ① is not desirable, and may release the object 10 in the gripping operation as in ② and ③.

In addition, since the controller 180 knows the following process to complete a specific task, in the liberation operations of ② and ③, the position and direction, etc., which are arranged when the object 10 is released can be changed.

Thereafter, under the control of the controller 180, the robot hand 200 performs an ④ operation of gripping and lifting the object 10 at a certain angle from the upper left.

④ In the execution of the operation, the operation ⑤ is performed based on the operation ① and grips the information obtained through the tactile sensor 144 and the information obtained through the camera 122 together.

Thereafter, the ⑥ operation of moving the object 10 gripped through the ⑤ operation according to a predetermined order and process may be performed.

In addition, FIG. 16 is a view for explaining a method of performing re-gripping according to the probability of successful gripping of the present invention.

Referring to (a) of FIG. 16, in the present invention, a kernel density estimation method can be utilized.

That is, based on P (grasped | s, a), which is the probability of successful gripping after re- gripping, uses whether or not a gripping operation has a high success rate, and P (s_new | s, a, grasped), which is the probability of an object state after successful re- gripping. Based on), it is possible to judge and use what kind of motion is good to center the object.

As shown in (b) of FIG. 16, in re-grasping the object 10, a Kernel density estimation method is utilized, and based on P (grasped | s, a), which is a probability of success in gripping after the re-grasping operation. Whether or not the gripping operation has a high success rate, and as shown in FIG. 16 (c), to re-center the object based on the object state probability P (s_new | s, a, grasped) It can be used to judge whether the operation is good.

17 is a diagram for explaining the performance of a task to which re-phage is applied in connection with the present invention.

17 (a) illustrates a process of moving a plurality of objects 10 according to a predetermined process to perform a specific task, and when an event that deviates from the predetermined process occurs, performing re-grasping to release and grip immediately. It shows the process.

On the contrary, in FIG. 17 (b), when a plurality of objects 10 are moved according to a predetermined process to perform a specific task, when an event that deviates from the predetermined process occurs, the specific task is performed in the corresponding state. The process most suitable for the following is newly determined by the controller 180, and accordingly, a process of performing re-grasping for higesting and gripping is illustrated.

In the present invention, all of the contents shown in (a) and (b) of FIG. 17 described above may be applied.

In addition, FIG. 18 is a flowchart illustrating a method capable of completing a specific task using regrasping when manipulation in accordance with a given purpose cannot be performed in the form of grasping in relation to the present invention.

Referring to FIG. 18, a step S40 in which the camera 122 acquires images of the plurality of objects 10 is performed.

Thereafter, based on the control of the controller 180, the robot hand 200 grasps and moves the first object among the plurality of objects in a predetermined order (S41).

Thereafter, the controller 180 determines whether the task cannot be performed (S41).

Here, in the event that the task cannot be performed, if at least one of the position and direction of the first object is changed, the first object cannot be gripped as desired, or an incorrect gripping and joining is already performed and the task cannot be performed according to the determined process. And the like.

If it is determined in step S41 that the task cannot be performed, the controller 180 controls the robot hand 200 to release the gripped first object (S42).

After the step S42, the controller 180 grips the first object according to the original plan or newly establishes a plan suitable for an incorrectly processed process and based on the image acquired through the camera so as to grasp the first object. At least one of the changed direction and position is determined (S43).

In addition, the robot hand 200 proceeds to step S44 of re-grasping the first object based on the changed direction and position.

The controller 180 may control the steps S41 to S44 to be repeatedly learned and performed until a specific task is performed.

The present invention, as regrasping when assembling, knows the information about the movement shape and the assembly sequence in advance, so that the next process is known, the object is released upon release from the phage so as to be in a good arrangement for the next operation, and a re-grasping operation is performed. You can.

In addition, in the present invention, when an event in which the object 10 is not gripped due to an unexpected shape occurs, the corresponding part of the object 10 is rotated through the robot hand 200, the visual information is acquired again, and then gripped again. You can also perform actions.

In addition, in the present invention, based on the obtained information, a geometric feature reflection file for assembling each part is generated (direction, size, etc. of holes), permutation, and a first assembly work sequence is secured through a combination of all parts. After that, it can be confirmed that all the holes of the parts to be assembled are used. Also, you can search the assembly work sequence and solve the problem with the Constraint satisfaction problem (make sure that interference between parts occurs in the assembly direction during assembly).

Therefore, the present invention can provide a technique for accomplishing a specific task by using regrasping when a robot hand performs a specific task, and it is not possible to perform manipulation for a given purpose in the form of grasping.

In addition, according to the present invention, a conventional tactile sensor capable of classifying an object by developing normal force measurement as well as measuring shear force and measuring the conductivity of an object and measuring the conductivity of the object is developed, and such tactile information can be utilized like a human. You can.

In addition, real-time position / posture / status recognition technology of parts using visual and tactile information, optimum gripping type reasoning technology for stable gripping of parts, and intelligent gripping technology based on recognition information and experience (single gripper / hand use, 30 types) Ideal object), position / direction manipulation (In-Hand) of parts using visual and tactile information, and learning strategy of assembling various parts based on experience. Can solve the current problem being performed by the.

Meanwhile, the above-described embodiments of the present invention may be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

For implementation by hardware, the method according to embodiments of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs) , Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. The software code can be stored in a memory unit and driven by a processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various known means.

The detailed description of preferred embodiments of the present invention disclosed as described above has been provided to enable those skilled in the art to implement and practice the present invention. Although described above with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and changes can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art can use each of the configurations described in the above-described embodiments in a manner of combining with each other. Accordingly, the present invention is not intended to be limited to the embodiments presented herein, but to give the broadest scope consistent with the principles and novel features disclosed herein.

The invention may be embodied in other specific forms without departing from the spirit and essential features of the invention. Accordingly, the above detailed description should not be construed as limiting in all respects, but should be considered illustrative. The scope of the invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the invention are included in the scope of the invention. The invention is not intended to be limited to the embodiments presented herein, but rather to give the broadest scope consistent with the principles and novel features disclosed herein. In addition, claims that do not have an explicit citation relationship in the claims can be combined to form an embodiment or included as a new claim by correction after filing.

Claims (11)

A camera that acquires an image of at least one first object;
A control unit for determining at least one of a position coordinate, a size, and a center of the first object using the acquired image; And
Including when the at least one factor (factor) of the position coordinates, size and center is determined, the hand for gripping the first object using the determined factor;

When the hand first fails to hold the first object,
The control unit,
The information about the first object stored in advance or received from the outside is compared with an image acquired by the camera to determine an error level, and among the position coordinates, size, and center of the first object based on the first failure event Decide at least one again,
The hand grips the first object again using the determined factor,

If the hand fails at least once after the first failure,
The control unit,
The obtained information is corrected by determining the error between the stored image and the input image by using the learned information, and a 2D model corresponding to the recognition region to be gripped among the regions of the first object is determined,
Converting the 2D model into a 3D model, the hand determines a gripping posture for gripping the 3D model,

The hand,
A robot hand, characterized in that the first object is gripped again based on the determined factor and the determined gripping posture.
According to claim 1,
The control unit,
In the absence of prior information on the first object, the robot hand, characterized in that for controlling the hand to grip the first object using only the determined factor. delete According to claim 1,
When the hand grips the first object again using the determined factor,
The control unit,
The hand grips the gripping stability of the gripped first object,
When the gripping stability is below a preset criterion, a robot hand, characterized in that the hand is controlled to release the first object. delete According to claim 1,
The hand includes a plurality of fingers,
Each of the plurality of fingers includes a distance sensor that senses a distance from the first object,
When holding the first object,
Based on the information obtained through the plurality of distance sensors, each of the plurality of fingers is a robot hand, characterized in that moving close to the first object at the same speed. A first step in which the camera acquires an image of at least one first object;
A second step in which a control unit determines at least one of a position coordinate, a size, and a center of the first object using the acquired image;
A third step in which a hand grips the first object using the determined factor when at least one factor of the location coordinate, size, and center is determined;
A fourth step in which the hand first fails to grip the first object;
The control unit compares information on the first object previously stored or received from the outside with the image acquired by the camera to determine an error level, and the position coordinates and size of the first object based on the first failure event And a fifth step of re-determining at least one of the centers;
A sixth step in which the hand grips the first object again using the determined factor;
A seventh step in which the hand fails at least once or more after the first failure;
The acquired information is corrected by determining the error between the stored image and the input image by using the learned information, and a two-dimensional model corresponding to the recognition region to be gripped among the regions of the first object is determined, and the two-dimensional model is An eighth step of converting to a three-dimensional model and determining a gripping posture for the hand to grip the three-dimensional model; And
And a ninth step in which the hand grips the first object again based on the determined factor and the determined gripping posture.
The method of claim 7,
In the third step,
The control method of a robot hand, characterized in that the hand grasps the first object using only the determined factor in a state in which there is no prior information on the first object. delete The method of claim 7,
After the sixth step,
Determining a gripping stability of the first object gripped by the hand; And
And when the gripping stability is below a preset criterion, the hand liberating the first object. The method of claim 7,
In the third step,
When the hand is in contact with the first object for the gripping, a tactile sensor detects at least one tactile factor of normal force, shear force and conductivity of the first object, ,
In the sixth step,
When the hand grips the first object again using the re-determined factor, the control method of the robot hand, characterized in that at least one tactile factor sensed by the tactile sensor is used together.

Images