KR20190102973A - Camera-based tactile sensor system - Google Patents

Abstract

The present invention provides a tactile sensor system which comprises: a tactile sensing device providing a tactile sensing member having a plurality of protrusion patterns formed therein and a finger including a single camera photographing an inner surface of the tactile sensing member; and a control device generating a parallax map including distance information on a 2D image inputted from the single camera by using a deep learning network and obtaining tactile sensing information based on the 2D image and the parallax map.

Description

Camera-based tactile sensor system

The present invention relates to a camera-based tactile sensor system.

With the development of robot technology, research on tactile sensing technology is being conducted in order for humans and robots to cooperate more closely.

However, the tactile sensing technology developed to date has various limitations such as the use of expensive tactile sensors and high accuracy.

Thus, there is a need for the development of new, more efficient tactile sensing techniques.

An object of the present invention is to provide a method for implementing a new method of more efficient tactile sensing technology.

In order to achieve the above object, the present invention provides a tactile sensing device including a tactile sensing member having a plurality of protrusion patterns formed on an inner surface thereof, and a finger including a single camera photographing the inner surface of the tactile sensing member; Tactile sensing system comprising a control device for generating a parallax map including distance information on the 2D image input from the single camera using a deep learning network, and obtains tactile sensing information based on the 2D image and parallax map To provide.

Here, the deep learning network learns a deep learning learning database constructed from a parallax map generated through a dual camera for stereo and a 2D image generated by one of the dual cameras corresponding to the stereo camera, and the deep learning network. A parallax map of the 2D image input by the single camera may be generated based on the learning information about the learning database.

The deep learning network may be configured as a GAN based network.

The control device generates a corrected 2D image by calibrating the 2D image, compares the corrected 2D image based on a reference 2D image in a non-contact state, and analyzes a 2D pattern regarding a position shift of the pattern protrusion. Information may be detected and 3D pattern information in which distance information of the parallax map is added to the 2D pattern analysis information may be analyzed.

The tactile sensing information may include a pressed position, a pressed direction, and an applied force.

The finger has a tube shape having a through hole formed therein, and a finger frame having the tactile sensing member coupled to one end thereof; It may include a lead coupled to the other end of the finger frame to cover the through hole, the single camera is mounted on the inner surface.

The present invention provides a new method of efficient tactile sensing technology, which performs high-performance tactile sensing by detecting distance information on a 2D image photographed by a single camera through a deep learning based learning technique using parallax maps. Technology.

Accordingly, it is possible to implement a robot in which the tactile sensing capability has evolved, and the robot industry can be greatly developed.

1 schematically illustrates a tactile sensing system according to an embodiment of the invention.
2 schematically illustrates a finger of a tactile sensing system according to an embodiment of the invention.
3 is a view schematically showing a pattern protrusion of a finger according to an embodiment of the present invention.
Figure 4 schematically illustrates the structure of the finger used in the process of building a deep learning database in the tactile sensing system according to an embodiment of the present invention
5 is a view for explaining a deep learning database for learning according to an embodiment of the present invention.
6 is a view for explaining a deep learning based distance extraction process according to an embodiment of the present invention.
7 is a view showing an image pattern analysis process in the pattern analyzer of the control device according to an embodiment of the present invention.
8 illustrates a feature point extraction algorithm according to an embodiment of the present invention.
9 is a diagram illustrating a result of extracting feature points by applying a feature point extraction algorithm to an input image according to an embodiment of the present invention.
10 illustrates a shape detection algorithm according to an embodiment of the present invention.
11 is a view for explaining a force detection process according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1 is a view schematically showing a tactile sensing system according to an embodiment of the present invention. 2 is a view schematically showing a finger of the tactile sensing system according to an embodiment of the present invention. On the other hand, Figure 2 shows the fingers viewed from the lower side and the upper side, respectively. 3 is a view schematically showing a pattern protrusion of a finger according to an embodiment of the present invention.

Referring to FIG. 1, a tactile sensing system according to an embodiment of the present invention corresponds to a tactile sensing system based on an image.

The tactile sensing system 10 may include the tactile sensing device 100 and the control device 200.

The tactile sensor 100 may be configured using a so-called robot arm that performs tactile sensing on the object OB and performs work on the object OB according to the sensing result.

The tactile sensor 100 configured as described above may include a robot arm body 110 and at least one finger 120 configured at an end (or end) of the robot arm body 110.

At least one joint member is configured in the robot arm main body 110 and may be configured to perform an operation similar to a human arm.

Finger 120 may be connected to the robot arm body 110 and the joint member. Meanwhile, a force sensor (eg, 6-axis force sensor) may be coupled to the robot arm body 110 side end of the finger 120.

The finger 120 includes a finger frame 121 defining a substantial shape of the finger 120 and a tactile sensing member (or tactile sensing tip) 130 mounted at one end of the finger frame 121. can do.

Referring to FIG. 2 with respect to such a finger 120, the finger frame 120 may include a through hole therein. That is, the finger frame 121 may be configured in a form in which both ends are opened along its longitudinal direction, that is, in a hollow form. For example, the finger frame 120 may be formed in a cylinder shape having a circular tube shape, but is not limited thereto and may be formed in a polygonal tube shape.

One end (or end) of the finger frame 121 may be coupled to the tactile sensing member 130.

The tactile sensing member 130 is a member for image-based tactile sensing, which may be formed of an elastic material such as rubber. That is, the tactile sensing member 130 may be formed of an elastic material that is deformed in contact with the object OB and is restored to its original shape when the contact is released.

Such a tactile sensing member 130 may be formed to protrude convexly in an outward direction.

On the inner surface of the tactile sensing member 130 (ie, the surface opposite to the outer surface of the contact surface), as shown in FIG. 3, a plurality of pattern protrusions 131 for substantially implementing image-based tactile sensing are provided. Can be formed.

The pattern protrusions 131 may be substantially spaced apart from each other by a predetermined distance along the entire inner surface of the tactile sensing member 130. Here, the spacing may be configured to be all the same or change depending on the position.

Meanwhile, the pattern protrusions 131 may be configured to be disposed on the inner surface of the tactile sensing member 130 according to a 2D-to-3D mapping method based on a Cartesian coordinate system or a polar coordinate system.

On the other hand, when capturing (or capturing) the inner surface of the tactile sensing member 130, the pattern protrusions 131 and the peripheral parts of the pattern protrusions 131 may be positioned in order to easily secure visibility of the pattern protrusions 131 in the image. Inner parts of the tactile sensor 130 may be formed to have different colors.

In this regard, for example, the surface of the pattern protrusions 131 may be formed to have reflection characteristics or fluorescence characteristics. For example, a white or silver material reflecting material may be applied or a fluorescent material may be applied. In addition, inner surface portions of the tactile sensing members 130 positioned around the pattern protrusions 131 may be formed in black so as to have light absorption characteristics.

As the pattern protrusions 131 are formed on the inner surface of the tactile sensing member 130 as described above, the distribution or position of the pattern protrusions 131 is deformed when the pattern protrusions 131 are in contact with the object OB, and based on the deformation. Tactile sensing can be implemented.

On the other hand, the finger 120 may be equipped with a camera 140 which is a photographing means for photographing the inner surface of the tactile sensing member 130. Furthermore, at least one light emitting diode 150 may be mounted on the finger 120 as a light source for providing light required for imaging.

Here, it is preferable that a single camera 140 is mounted on the finger 120. Such a camera 140 may be mounted at a position facing the inner surface of the tactile sensing member 130. In this regard, for example, the other end of the finger frame 121 may be provided with a lead 125 covering the through hole, the camera 140 on the inner surface of the lead 125 of the tactile sensing member 130 It may be arranged to face the inside. In this case, the single camera 140 is preferably located at the center of the inner surface of the lid 125.

In addition, the light emitting diodes 150 may be installed in the finger frame 121. For example, the light emitting diodes 150 may be installed at the bottom surface of one end of the finger frame 121. As another example, the light emitting diode 150 may be installed at the lower end of the inner circumferential surface of the finger frame 121.

The light generated by the light emitting diodes 150 installed as described above is provided to the inner surface of the tactile sensing member 130, and the camera 140 may photograph the tactile sensing member 130 while the light is provided.

An image for tactile sensing is transmitted to the control device 200 as a 2D (two-dimensional) image which is an image photographed by the camera 140.

The control device 200 is a configuration for overall control of the tactile sensor 100, and may control an operation of the tactile sensor 100 related to tactile detection.

In this regard, the control device 200 controls the photographing of the camera 140 and displays a 2D image of the inner surface of the tactile sensor 130 photographed by the camera 140, that is, a 2D image of the pattern protrusions 131. By analyzing, tactile sensing information may be obtained.

On the other hand, the pattern protrusions 131 in the input 2D image are analyzed as their feature patterns and their distribution patterns or arrangement patterns, thereby being applied to (or applied to or applied to the sensor sensing member 130 when the object OB is touched). Various tactile sensing information can be obtained, such as acting force, pressed (or contacted) position, pressed direction.

In addition, the control device 200 may control subsequent operations of the robot arm constituting the tactile sensing device 100 based on the obtained tactile sensing information.

Such a control device 200 may be configured using a so-called computer that performs various data operation processing, including image analysis.

Meanwhile, the 2D image photographed by the single camera 140 is a planar image, for example, does not include distance information (or depth information) based on the camera position. In order to obtain various and accurate tactile sensing information such as a pressed position, a pressed direction, and a three-axis force on the object OB, the distance information (or depth information) of the pattern protrusion 131 which is a feature point in the 2D image is obtained. It would be desirable to extract it.

To this end, in the present embodiment, the control device 200 may be configured to use the deep learning network 210, which is an artificial intelligence technology.

In this regard, for example, as shown in FIG. 4, by using two cameras 145, that is, dual cameras 145 for stereo, a disparity map corresponding to a 3D image in various contact environments ( Alternatively, a sight glass image or a stereo image may be generated and built as a database for deep learning learning. Here, in the deep learning database, 2D images generated by one of two stereo dual cameras 145 corresponding to each parallax map may be stored together.

On the other hand, Figure 4 schematically shows the structure of the finger 120 used in the process of building a database for learning deep learning, compared with the structure of the finger 120 in the tactile sensing process of Figure 2, it is constant with each other Two stereo dual cameras 145 spaced apart may be disposed to face the inner surface of the tactile sensing member 130 on the inner surface of the lid 125.

The control apparatus 200 may learn the deep learning learning database through the deep learning network, thereby more accurately inferring (or predicting) the distance information of the 2D image captured by the single camera 140.

In this regard, reference may be made to FIGS. 5 and 6.

5 is a diagram for describing a deep learning database according to an embodiment of the present invention. In such a learning database, each parallax map generated by a stereo camera and a 2D image corresponding thereto may be databased.

6 is a view for explaining a deep learning based distance extraction process according to an embodiment of the present invention. In the tactile sensing process, the deep learning network 210 installed in the control device 200 may generate a corresponding parallax map based on deep learning learning on the input 2D image. That is, based on the deep learning learning information, a parallax map may be generated by predicting distance information on the 2D image input by the single camera 140 and reflecting the distance information.

As described above, in the present exemplary embodiment, distance information with high accuracy can be extracted from a 2D image using a deep learning technique. Accordingly, there is an advantage that it is possible to effectively implement a tactile sensing substantially equivalent to the case of using a dual camera while using a single camera 140.

Meanwhile, the deep learning network 210 for generating the disparity map may use, for example, a GAN based network shown in the lower part of FIG. 6.

Meanwhile, referring to FIG. 7, a tactile sensing process through image pattern analysis in the control apparatus 200 may be referred to. 7 is a view illustrating an image pattern analysis process in the pattern analyzer of the control device according to an embodiment of the present invention.

Referring to FIG. 7, the control device 200 may be provided with a pattern analyzer, which receives a 2D image captured by a single camera 140 and corrects (ie, calibrates) the 2D image. Can be generated. In this regard, a calibrated corrected 2D image may be generated by converting the positions of the pattern protrusions 131, that is, the feature points of the input image, through a previously prepared transform matrix.

Such a corrected 2D image may be compared with a reference 2D image to perform planar, that is, 2D (two-dimensional) pattern analysis. Here, the reference 2D image corresponds to an image photographing the inner surface of the tactile sensing member 130 in a non-contact state, that is, a circular state, through a single camera 140.

The pattern analysis of the corrected 2D image based on the reference 2D image does not substantially reflect the distance information. In this 2D pattern analysis, each of them is based on the feature points (ie, reference feature points) of the reference 2D image. A 2D position shift (or change) of feature points in the corresponding corrected 2D image may be extracted. That is, 2D variation of the arrangement pattern of the feature points may be extracted.

Through the position shift of the feature points in the 2D pattern analysis, the position, shape, etc. pressed by the object OB may be detected.

Meanwhile, when inputting a 2D image captured by a single camera 140, the disparity map generated by the deep learning network 210 may be input to the pattern analyzer, and the input disparity map is linked with the generation of the corrected 2D image. Can be corrected.

Next, 3D pattern analysis information in which distance information z is added to 2D pattern analysis information may be generated. Such distance information z may be extracted from a parallax map input together with a 2D image.

Meanwhile, the distance information based on the 2D pattern analysis information is reflected in the distance information of the parallax map, so that the accuracy of the distance information z may be improved. On the other hand, since the position change in the 2D pattern analysis is related to the distance change, the distance information based on the 2D pattern analysis information is reflected in the distance information of the parallax map, so that the distance information z with improved accuracy can be extracted. have.

According to the above process, 3D pattern information including distance information z may be obtained in the 2D pattern analysis information.

The force and direction pressed by the object OB may be calculated by analyzing the 3D pattern information including the obtained distance information.

On the other hand, the control device 200 may be configured with a feature point extraction unit for feature point extraction, the feature point extraction unit may use an algorithm for robust feature point extraction.

In this regard, referring to FIG. 8, the feature extraction algorithm may include a binarization process, that is, a thresholding process and a pixel including converting an input image (or a source image) input to a feature extraction unit into a binarization image. It may include the grouping process at the level, the merging process at the blob level, the filtering process, and the ball detection, ie, the center and radius calculation processes.

9 shows a result of extracting feature points by applying a feature point extraction algorithm to an input image.

On the other hand, the control device 200 may be configured with a shape detector for detecting the shape (or shape) pressed by the tactile sensing object (OB), the shape detector may use an algorithm for accurate shape detection. The shape detector may be included in the pattern analyzer or may be separately configured, and may use 2D pattern analysis information.

Referring to FIG. 10, the shape detection algorithm may include an offline phase and an online phase.

In the off-line face before performing tactile sensing, an average background (mBG) is calculated by averaging contactless input images and stored for use in an online face.

In an online face where tactile sensing is performed, key points may be detected for the input image, a feature may be calculated for the feature points, and the shape may be estimated through an SVM classifier.

Here, the characteristic of the feature points is the number of feature points (nK), the mean of the feature points (mS), the standard deviation of the feature points (sS), the average difference (or mean difference image) (mD) between the background (mBG) and the input image It may include.

The SVM classifier is a shape classifier and may output a shape corresponding to a characteristic of feature points of an input image among a plurality of predefined and classified shapes. For example, a triangle, a quadrangle, or a contactless state may be defined, and a shape corresponding to the characteristic of the input image may be selected and output from among them.

Meanwhile, the control device 200 may calculate the area of the contact surface of the object OB, that is, the pressed area, by using the extracted shape and the distribution pattern of the feature points.

Meanwhile, the control device 200 may calculate the magnitude and / or direction of the triaxial force in more detail than the pressed force using the extracted shape and the distribution pattern of the feature points.

Referring to FIG. 11 regarding the calculation of the three-axis force, feature points may be extracted from the input image, and the position and direction of the force may be detected. After analyzing the feature points to confirm the direction in which the force is applied, the final force is calculated by analyzing the pattern shape of the pattern protrusions and the pattern shape of the surrounding protrusions in the corresponding direction.

As described above, an embodiment of the present invention provides a new method of efficient tactile sensing technology, which detects distance information on a 2D image captured by a single camera through a deep learning based learning technique using parallax maps. It provides a technology for performing high performance tactile sensing.

Accordingly, it is possible to implement a robot in which the tactile sensing capability has evolved, and the robot industry can be greatly developed.

Meanwhile, the tactile sensing method according to the embodiment of the present invention described above may be embodied in the form of program instructions that can be executed through a computer and recorded in a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the media may be those specially designed and constructed for the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical, and ROM, RAM, flash memory and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of one embodiment of the present invention, and vice versa.

Embodiment of the present invention described above is an example of the present invention, it is possible to change freely within the scope included in the spirit of the present invention. Accordingly, the invention includes modifications of the invention within the scope of the appended claims and their equivalents.

10: tactile sensing system 100: tactile sensing device
110: robot arm body 120: finger
121: finger frame 125: lead
130: tactile sensing member 131: pattern protrusion
140: camera 150: light emitting diode
200: controller 210: deep learning network

Claims (6)

A tactile sensing device comprising a tactile sensing member having a plurality of protrusion patterns formed therein and a finger including a single camera photographing the inner surface of the tactile sensing member;
Control device for generating a parallax map including distance information on the 2D image input from the single camera using a deep learning network, and obtains tactile sensing information based on the 2D image and parallax map
Tactile sensing system comprising a.
The method of claim 1,
The deep learning network,
Study a deep learning database constructed from a parallax map generated through a dual camera for stereo and a 2D image generated by one of the dual cameras corresponding to the stereo,
Generating a parallax map for the 2D image input from the single camera based on the learning information about the deep learning database
Tactile sensing system.
The method of claim 1,
The deep learning network is composed of a GAN-based network
Tactile sensing system.
The method of claim 1,
The control device,
Generate a corrected 2D image by calibrating the 2D image,
Comparing the corrected 2D image on the basis of the reference 2D image of the contactless state, to detect the 2D pattern analysis information about the position shift of the pattern projection,
Analyzing 3D pattern information to which the distance information of the parallax map is added to the 2D pattern analysis information
Tactile sensing system.
The method of claim 1,
The tactile sensing information includes a pressed position, a pressed direction, and an applied force
Tactile sensing system.
The method of claim 1,
The finger is,
A finger frame having a tube shape having a through hole formed therein and having the tactile sensing member coupled to one end thereof;
A lead coupled to the other end of the finger frame to cover the through hole and having the single camera mounted on an inner surface thereof;
Tactile sensing system.

Images