CN114964576A

CN114964576A - Vision-based sensor with haptic perception and near-field imaging

Info

Publication number: CN114964576A
Application number: CN202210151344.7A
Authority: CN
Inventors: 于宏宇; 陈侠; 王煜; 张冠澜
Original assignee: Hong Kong University of Science and Technology HKUST
Current assignee: Hong Kong University of Science and Technology HKUST
Priority date: 2021-02-25
Filing date: 2022-02-18
Publication date: 2022-08-30

Abstract

The present disclosure provides a sensor device and a sensor system. The sensor device includes: at least one visual element; a transparent elastomeric layer disposed on the at least one visual element; a marker disposed on a side of the elastomeric layer facing away from the at least one visual element and embedded in the elastomeric layer, the marker being formed into a specific pattern arranged in an array; an illumination source disposed at a side of the elastomeric layer and configured to illuminate the marker and an external environment; and a flexible transparent protective layer disposed over the tag and the elastomer. Under the condition that an object is in contact with the transparent protective layer, the specific pattern is deformed, and at least one visual unit captures the deformed pattern and obtains contact information through calculation; in the case where the foreign object is located outside the sensor device and is not in contact with the transparent protective layer, the at least one vision unit captures an image of the foreign object via an area where no marker is present or where the marker is obscured to being ignored.

Description

Vision-based sensor with haptic perception and near-field imaging

Technical Field

The present disclosure relates to optical sensors having both tactile and visual capabilities, and in particular to optical sensors for sensing three-dimensional surface deformation, triaxial force distribution, and near proximity activity prior to contact.

Background

Vision-based tactile sensors are a branch of artificial tactile sensors used in robotics and other applications. A typical application scenario is for fine robotic hand manipulation, touch information such as force distribution is important feedback for decision making execution.

Conventional tactile sensors, such as capacitive, piezoelectric and piezoresistive sensors, have been studied for a long time, and some commercialized tactile sensors have been developed based on these sensor methods. Even though the sensing portions of these types of tactile sensors may be as thin as sub-millimeters to several millimeters, most of them lack high spatial resolution. However, vision-based tactile sensors have high spatial resolution, such as the GelSight, GelSlim, or TacTip prototypes with CMOS image sensors. However, all of these employ a reflective layer behind the marker so that only the marker is visible and, therefore, the sensor cannot perceive the scene in front of the sensor. At the same time, the size is huge and no more information in the scene (such as non-contact movement of objects) can be obtained.

To overcome the above challenges, the present application proposes a vision-based tactile sensor for tactilely sensing and capturing an image outside a contact surface. The invention adopts the traditional single vision unit imaging method or adopts a pinhole array or a multi-vision unit of a lens array, thereby realizing a multifunctional sensor. It not only exhibits a high spatial density of force distribution mapping capability, but also visual information outside the contact surface.

Disclosure of Invention

In one aspect, the present disclosure provides a sensor device comprising: at least one visual element; a transparent elastomeric layer disposed on the at least one visual element; a marker disposed on a side of the elastomeric layer facing away from the at least one visual element and embedded in the elastomeric layer, the marker formed into a specific pattern arranged in an array; an illumination source disposed at a side of the elastomeric layer and configured to illuminate the marker and an external environment; and a flexible, transparent protective layer disposed over the marker and the elastomer, wherein the sensor device is configured to: in the case where an object is in contact with the transparent protective layer, the specific pattern formed by the marker is deformed, and the at least one visual unit captures the deformed pattern and calculates to obtain contact information; in the case where a foreign object is located outside the sensor device and is not in contact with the transparent protective layer, the at least one vision unit captures an image of the foreign object via an area where the marker is not present or is obscured to being ignored.

In an embodiment, the sensor device further comprises a transparent rigid layer located between the elastomeric layer and the at least one visual element.

In an embodiment, the at least one visual unit comprises a photosensor and at least one pinhole structure, the at least one pinhole structure being arranged between the photosensor and the elastomer layer and having a polygonal groove on a side of the at least one pinhole structure facing the photosensor and having an aperture on a side of the at least one pinhole structure facing away from the photosensor.

In an embodiment, the at least one vision unit comprises a camera module and a connection frame, and the connection frame is located between the rigid layer and the camera module for connecting the camera module and the rigid layer.

In an embodiment, the sensor device further comprises a plano-convex lens located between the at least one pinhole structure and the elastomer layer, and the protruding portion of the plano-convex lens is partially embedded in the hole of the pinhole structure.

In an embodiment, the sensor device further comprises a plano-convex lens located between the at least one pinhole structure and the elastomeric layer, and the protruding portion of the plano-convex lens is fully embedded in the hole of the pinhole structure.

In an embodiment, the sensor device further comprises a plano-convex lens located between the at least one pinhole structure and the elastomeric layer, and a convex portion of the plano-convex lens is arranged on a side of the plano-convex lens facing away from the at least one pinhole structure, and a central position of the convex portion is aligned with a central position of the aperture.

In an embodiment, the specific pattern of markers arranged in an array comprises: a plurality of arrays arranged in a 3 x 4 matrix, each array having disposed at it dots arranged in a 4 x 4 matrix, the dots being black or dark.

In an embodiment, the specific pattern of markers arranged in an array comprises: a plurality of arrays arranged irregularly, each array being provided with four other arrays in the diagonal direction, and dots arranged in a 4 × 4 matrix, which are black or dark, being provided at each array.

In an embodiment, the specific pattern of markers arranged in an array comprises: a plurality of dots arranged in a matrix, the dots being black or dark.

In an embodiment, the dots are circular dots or square dots.

In an embodiment, the material of the elastomeric layer comprises polydimethylsiloxane.

In an embodiment, the holes have a diameter of 32 μm and a depth of 25 μm, and the square trenches have a side length of 500 μm and a depth of 500 μm.

In an embodiment, the photosensor is a complementary metal oxide semiconductor image sensor.

In an embodiment, the material of the marker comprises a metal.

In an embodiment, the material of the flexible transparent protective layer comprises silicone or parylene.

In another aspect, the present disclosure provides a sensor system comprising a plurality of sensor devices as described above, wherein the at least one vision unit comprises a photosensor and at least one pinhole structure, and the plurality of sensor devices share one photosensor.

In another aspect, the present disclosure provides a sensor system comprising a plurality of sensor devices as described above and a flexible elastomer, wherein the plurality of sensor devices are disposed on the flexible elastomer.

Drawings

The following drawings and detailed description are provided to further understanding of the present disclosure. In the drawings:

FIG. 1 is a schematic diagram of a sensor showing one object contacting the sensor interface and another object not contacting the sensor interface;

fig. 2 is a schematic view illustrating a sensor using a conventional camera module having one vision unit.

FIG. 3 is a top view of a sensor interface showing a first type of pattern of markers;

FIG. 4 is a top view of a sensor interface showing a second type of pattern of markers;

FIG. 5 is a top view of a sensor interface showing a third type of pattern of markers;

FIG. 6 is a top view of a sensor interface showing a fourth type of pattern of markers;

FIG. 7 is a schematic diagram showing a sensor with another imaging system;

FIG. 8 is a schematic diagram of a sensor with a single imaging unit using a pinhole imaging method;

fig. 9 is a schematic diagram of a sensor having a single imaging unit using a lens imaging method;

FIG. 10 is a schematic diagram of a sensor system incorporating the structure of FIG. 1 and using one photosensor; and

FIG. 11 illustrates an embodiment of a flexible sensor system combining the plurality of structures of FIG. 1 with another elastomer.

Detailed Description

The present invention will be described below by way of detailed description of examples.

Such vision-based sensors have been developed for the purpose of performing functions of touch motion information, such as force distribution and visual information outside the touch surface. On top of the device, there is a flexible transparent protective layer on top of the elastomer layer in which the marker is embedded. The marker is specifically designed as a regular or irregular pattern with a circular or other shape or a random shape. The displacement and area of the marker is captured by an imaging subsystem comprising at least one vision unit using a camera with a single or an array of imaging units and a lens structure. An optional rigid transparent layer between the elastomer and the lens may be added to the structure to enhance robustness of the device. The photosensor is responsible for forming an image for further processing. Typically, illumination sources are added at the edges of the elastomeric layer or between the structural layers to provide stable imaging illumination. LEDs or optical fibers may be used for the illumination source.

In fig. 1, a schematic diagram of a sensor with two example objects is shown, one cubic object contacting the sensor interface and the other spherical object not contacting the sensor interface. It is understood that a sensor interface refers to a surface (also referred to as sensor surface) of a tactile sensor (also referred to as sensor, sensor device) to be in contact with an object. The tactile sensor comprises a photosensor 1, a pinhole structure 2, a rigid layer 3, an elastomeric layer 4, an illumination source 5, a marker 6, a transparent protective layer 7. The spherical object 8 is not in contact with the sensor surface and the cubic object 9 is in contact with the sensor surface, in particular the cubic object 9 is in contact with the transparent protective layer 7. The combination of the photosensor 1 and pinhole structure 2 provides a plurality of visual elements and the touch interface is composed of a rigid layer 3, an elastomeric layer 4, a marker 6 and a transparent protective layer 7. Illumination is provided by an illumination source 5.

Here, the photosensor 1 may be a Complementary Metal Oxide Semiconductor (CMOS) image sensor. The pinhole structure 2 serves here as a structure for imaging, while other types of lens structures can also be used (see for example fig. 7). The pinhole structure 2 is placed between the photosensor 1 and the rigid layer 3. In another embodiment, the rigid layer 3 may also be directly replaced by the elastomeric layer 4, i.e. the rigid layer 3 may be omitted. In this case the pinhole structure 2 is placed between the photosensor 1 and the elastomer layer 4. The pinhole structure 2 may be formed by double-sided etching, wherein the side facing the photosensor 1 is etched as an array of square or other polygonal trenches, and the other side is etched as an array of pinholes. For one particular design, the pinholes have a diameter of 32 μm and a depth of 25 μm, and the square trenches have side lengths of 500 μm and a depth of 500 μm. The dimensions of the pinhole structures 2 may vary according to specific requirements.

The elastomeric layer 4 is elastic and transparent. The working range depends on the thickness and material properties of the elastomeric layer 4. PDMS (polydimethylsiloxane) is a desirable choice for the elastomer layer 4, and typically the thickness of the elastomer layer 4 may be a few millimeters. A marker 6 having a specific pattern is embedded in the elastomeric layer 4 or in an additional layer as described above. To illustrate the contact information, the deformation can be demonstrated with given material properties. The transparent region where the marker 6 is not present can be used to obtain an image of the near field or the far field. In an embodiment, the transparent area may also be an area where the marker obscures to be negligible.

The illumination source 5 is arranged at the side of the elastomeric layer 4. Here, one Light Emitting Diode (LED) or four LEDs at the side of the elastomer layer 4 may be used to illuminate the environment including the embedded marker 6 and the outside.

The marker 6 is transferred or coated onto the elastomeric layer 4 and it will not cover the entire area of the elastomeric layer 4. Deformation of the pattern formed by the marker 6 (e.g., displacement of the center point of the marker and change in the area of the marker) is captured, and then contact information is calculated using an imaging process. Typically, the marker is a few microns thick. The shape of the pattern of the marker should be an enclosed area having a contrasting color to the transparent area. Circular, square or irregular shapes may be used. In the present embodiment, a dot shape is adopted. The distribution of the marker is another design parameter. In this embodiment, the markers in the form of a 4 x 4 dense array are distributed at each location in another 3 x 4 array, which means that the entire area of the elastomer layer is not completely covered by the markers 6, which may be ink-jet printed or by sputtered metallic material. For image processing, the markers 6 are considered to obtain contact information. In this process, the information of other areas will not be counted. In the pattern formed by this type of marker 6, each visual element may capture some dots, while for the entire interface deformation there is an edge portion overlap of the image between adjacent visual elements.

The transparent protection layer 7 is a flexible transparent layer. The transparent protective layer 7 may be a transparent material, such as silicone or parylene, which also protects the tag 6 and the elastomeric layer 4, and an additional protective layer may be added between the transparent protective layer 7 and the elastomeric layer 4, which additional protective layer should also be transparent and typically has a thickness of a few microns, such as parylene.

With such an embodiment, when the cubic object 9 contacts the surface of the sensor, the pattern formed by the markers 6 will be deformed, and the deformation will be captured by a plurality of visual units, which will be further processed. Meanwhile, a non-contact image of the spherical object 8 can be captured via the transparent region without the marker 6, and thus, a tactile sensor having tactile sensation and imaging capabilities is realized.

In fig. 2, the camera module 10 and the connection frame 11 serve as one visual unit for obtaining an image. The connection frame is used for connecting the camera module and the rigid layer. Instead of using multiple visual elements, tactile sensing and imaging capabilities may also be obtained by using only one visual element.

In fig. 3, a top view of the sensor interface shows the pattern formed by the markers (which may also be referred to as a marker pattern). The marker pattern consisted of a 3 × 4 array. Each array consists of 4 x 4 dots. In embodiments, the dots may be replaced by square dots (e.g., see fig. 6) or other shaped closed patterns. Except that the color of the marker is black or dark, the regions other than the marker should be transparent. The array may also be adjusted to a 2 x 2, 2 x 3 or denser array depending on the degree of need for contact information. Generally, the denser the array, the more abundant and accurate contact information, such as force distribution, can be guaranteed. At the same time, the density of the marker pattern will affect the clear area responsible for constructing the image beyond the marker without the marker. The denser the array, the fewer transparent areas. Maximum pattern density is required to construct a complete image of the exterior of the marker. Otherwise, a clear image without the marker is not available. Circular markers can also be square, triangular or non-square enclosed areas. The marker areas do not completely cover the contact surface on the elastomer layer 4. In this case, each visual unit may perceive one complete 4 × 4 dot and two incomplete 4 × 4 dots for the corresponding 3 × 4 visual unit. Thus, there is an overlap between two neighboring regions, which can be further stitched into the whole image. Meanwhile, the non-contact objects can be spliced or not spliced to obtain a complete image.

In fig. 4, a top view of the sensor interface shows a second type of pattern of markers. Each pattern consists of the same 4 x 4 dots as shown in fig. 2. However, the distribution of the arrays is different. In this case, only 6 arrays are formed in an irregular distribution, for example, each array is provided with four other arrays in a diagonal direction. In this case, it is ensured that information points that are less touched have a more transparent area. However, multiple visual units will be largely divided into two types. One to receive primarily the marker pattern to reflect touch information and the other to receive primarily the transparent areas to reflect images outside the touch surface. At a suitable distance, a visual element may have only transparent regions. It may not be necessary for all applications.

The top view of the sensor interface shows a third type of pattern for a given marker shown in fig. 5, where each pattern consists of one dot. And in this case the distribution is regular, with a sparse distribution on the surface. In this case, it is ensured that information points that are less touched have a more transparent area. Another common distribution is that the distribution is not regularly formed, and the appearance in fig. 3 can also be applied to a single dot pattern.

FIG. 6 is a top view of another embodiment of a sensor interface. In this case, the pattern shape is square rather than circular, as compared to the previous embodiment.

Fig. 7 is a sensor schematic of another imaging system consisting of an array of lens elements, differing in that the diameter of the pinhole structure differs from that shown in fig. 1, where the size of the pinhole is adapted to the size of the plano-convex lens 12, the plano-convex lens 12 being embedded in the pinhole structure 2, in particular the convex part of the plano-convex lens being completely or partially embedded in the pinhole of the pinhole structure 2. In another embodiment, the convex portion of the plano-convex lens is arranged on a side of the plano-convex lens facing away from the pinhole structure, and a central position of the convex portion is aligned with a central position of the pinhole. Thereby, the entire sensor is further provided with imaging capabilities. Other lens structures, such as lenticular or multi-lens structures based on the pinhole structure 2, may also be applied in this design.

Fig. 8 is a schematic diagram of a sensor having a single imaging unit using a pinhole imaging method. In contrast to the case of multiple visual elements, parts of the external object cannot be fully imaged by the sensor.

Fig. 9 is a schematic diagram of a sensor having a single imaging unit using a lens imaging method, here employing a lens imaging system. Generally, a plano-convex lens is used.

FIG. 10 is one embodiment of a sensor system incorporating the structure of FIG. 1 and using one photosensor. In an embodiment, the sensor system comprises a plurality of sensor devices, each sensor device comprising a photosensor 1, a pinhole structure 2, a rigid layer 3, an elastomeric layer 4, an illumination source 5, a marker 6, a transparent protective layer 7. A plurality of sensor devices share a single photosensor. This design ensures that a larger sensing area is available. Furthermore, if the photosensor is flexible, the multi-touch sensor combination may provide a larger field of view.

Fig. 11 shows an embodiment of a flexible tactile sensor system combining the plurality of structures of fig. 1 and another elastic body 13. The further elastomer body 13 is flexible. A plurality of sensors are provided on the flexible elastic body 13. Thereby providing a larger field of view.

It is to be understood that the above embodiments are merely exemplary embodiments that are employed to illustrate the principles of the present disclosure, and that the present disclosure is not limited thereto. It will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the disclosure, and these are to be considered as the scope of the disclosure.

Claims

1. A sensor device, comprising:

at least one visual element;

a transparent elastomeric layer disposed on the at least one visual element;

a marker disposed on a side of the elastomeric layer facing away from the at least one visual element and embedded in the elastomeric layer, the marker formed into a specific pattern arranged in an array;

an illumination source disposed at a side of the elastomeric layer and configured to illuminate the marker and an external environment; and

a flexible transparent protective layer disposed over the marker and the elastomer,

wherein the sensor device is configured to:

in the case where an object is in contact with the transparent protective layer, the specific pattern formed by the marker is deformed, and the at least one visual unit captures the deformed pattern and calculates to obtain contact information; in the case where a foreign object is located outside the sensor device and is not in contact with the transparent protective layer, the at least one vision unit captures an image of the foreign object via an area where the marker is not present or is obscured to being ignored.

2. The sensor device of claim 1, further comprising a transparent rigid layer positioned between the elastomeric layer and the at least one visual element.

3. The sensor device of claim 1 or 2,

the at least one vision unit comprises a photosensor and at least one pinhole structure,

the at least one pinhole structure is disposed between the photosensor and the elastomer layer, and

the side of the at least one pinhole structure facing the photosensor has a polygonal groove, and the side of the at least one pinhole structure facing away from the photosensor has a hole.

4. The sensor device of claim 2,

the at least one vision unit includes a camera module and a connection frame, and

the connection frame is located between the rigid layer and the camera module for connecting the camera module and the rigid layer.

5. The sensor device of claim 3, further comprising a plano-convex lens,

the plano-convex lens is located between the at least one pinhole structure and the elastomeric layer, and the protruding portion of the plano-convex lens is partially embedded in the aperture of the pinhole structure.

6. The sensor device of claim 3, further comprising a plano-convex lens,

the plano-convex lens is located between the at least one pinhole structure and the elastomeric layer, and the protruding portion of the plano-convex lens is fully embedded in the aperture of the pinhole structure.

7. The sensor device of claim 3, further comprising a plano-convex lens,

the plano-convex lens is located between the at least one pinhole structure and the elastomeric layer, and a protruding part of the plano-convex lens is arranged on a side of the plano-convex lens facing away from the at least one pinhole structure, and a central position of the protruding part is aligned with a central position of the aperture.

8. The sensor device of claim 1, wherein

The specific pattern of markers arranged in an array includes:

a plurality of arrays arranged in a 3 x 4 matrix, each array having disposed at it dots arranged in a 4 x 4 matrix, the dots being black or dark.

9. The sensor device of claim 1, wherein

The specific pattern of markers arranged in an array includes:

a plurality of arrays arranged irregularly, each array being provided with four other arrays in a diagonal direction, and dots arranged in a 4 × 4 matrix, which are black or dark, are provided at each array.

10. The sensor device of claim 1, wherein

The specific pattern of markers arranged in an array includes:

a plurality of dots arranged in a matrix, the dots being black or dark.

11. The sensor device of any one of claims 8 to 10, wherein

The dots are circular dots or square dots.

12. The sensor device of claim 1,

the material of the elastomeric layer includes polydimethylsiloxane.

13. The sensor device of claim 3, wherein the hole has a diameter of 32 μm and a depth of 25 μm, and the square trench has a side length of 500 μm and a depth of 500 μm.

14. The sensor device of claim 3, wherein the photosensor is a complementary metal oxide semiconductor image sensor.

15. The sensor device of claim 1, wherein the material of the marker comprises a metal.

16. The sensor device of claim 1, wherein the material of the transparent protective layer comprises silicone or parylene.

17. A sensor system comprising a plurality of sensor devices according to claim 1,

the at least one vision unit comprises a photosensor and at least one pinhole structure, and

the plurality of sensor devices share a single photosensor.

18. A sensor system comprising a plurality of sensor devices according to claim 1 and a flexible elastomer, wherein the plurality of sensor devices are disposed on the flexible elastomer.