CN114043506B - Multi-mode touch sensing device and application method thereof - Google Patents
Multi-mode touch sensing device and application method thereof Download PDFInfo
- Publication number
- CN114043506B CN114043506B CN202111256932.9A CN202111256932A CN114043506B CN 114043506 B CN114043506 B CN 114043506B CN 202111256932 A CN202111256932 A CN 202111256932A CN 114043506 B CN114043506 B CN 114043506B
- Authority
- CN
- China
- Prior art keywords
- unit
- square box
- touch
- information
- touch sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
- B25J13/081—Touching devices, e.g. pressure-sensitive
- B25J13/084—Tactile sensors
Abstract
The invention discloses a multi-mode touch sensing device and an application method thereof. Wherein the air pump passes through the pipe connection with the device, and flexible touch sensor unit installs in square box non-confined one side, and the light source unit includes that RGB (red, green and blue) three lamp pearls are installed respectively on the three side inner walls of square box, and the bottom surface at the square box is installed to the image acquisition unit, and the acceleration acquisition unit is installed on the bottom inner wall of square box. According to the invention, the flexible material canvas is used as a contact surface of the touch device, so that two types of heterogeneous touch information can be triggered simultaneously: and respectively representing a touch sequence and a touch image, and quantitatively describing information such as touch, slip sense and the like in the physical world through fitting. When the device contacts or slides with a measured object, the device can sense touch information of various modes such as force touch quantity, RGB image, depth image, acceleration and the like.
Description
Technical Field
The invention relates to the technical field of robot sensing, in particular to a multi-mode touch sensing device and an application method thereof.
Background
Haptic sensations are an important source of information for robots to perceive an external environment. With the wide application of haptic sensing technology and interactive technology, robot haptic sensing technology has received high attention from robot developers. The conventional touch sensors have piezoresistive, capacitive, photoelectric and the like, and most of them have the problems of difficult density improvement, complex circuit processing and the like. In order to overcome the defects, the visual-based touch sensing device has the advantages of simple structure, rich information and the like in recent years, but the conventional touch sensing device still has a great improvement space in the aspects of weak information pairing and single touch sensing mode.
Disclosure of Invention
In order to solve the problems existing in the prior art, the invention provides a multi-mode touch sensing device and an application method thereof, and the invention provides the multi-mode touch sensing device aiming at the two common problems of weak information pairing and single touch sensing mode existing in the conventional touch sensing device. The device main part is black square box, and one of them face is not closed, pastes canvas power touch sensor, and in the use, utilizes the air pump to pass through a little mouth of device side, inflates the device inside, makes the deformation of power touch sensor more obvious when being surveyed the thing and being contacted with canvas force sensor. The sensor is used as follows: the flexible touch sensor presses the tested object vertically downwards, the horizontal moving device enables the canvas surface to be in contact with the tested object and deform, the 16 contacts on the surface of the touch sensor return read AD signals to the MCU for processing, meanwhile, the device is provided with an acceleration sensor, acceleration signals in the measuring process can be obtained, RGB light beads are further placed in the device to provide red, green and blue light sources, the bottom binocular camera can read RGB images and depth images generated by sliding of an object on the canvas force touch sensor, and after fitting, touch sequences, touch images and acceleration information can be obtained at the same time, so that the device does not have weak pairing when sensing the object, and multi-mode touch information can be obtained.
To achieve the purpose, the invention adopts the following technical scheme:
a multi-mode touch sensing device comprises a square box unit, an air pump unit, a flexible touch sensor unit, a light source unit, an image acquisition unit and an acceleration acquisition unit; the air pump unit with square box unit passes through the pipe connection, flexible touch sensor unit sets up on the square box unit, the light source unit sets up on the square box unit lateral wall, the image acquisition unit sets up square box unit bottom, the acceleration acquisition unit sets up square box unit bottom.
Further, the square box unit is a cuboid hollow shell, the shell is made of black acrylic plate materials, and one surface of the cuboid is not closed.
Further, the air pump unit is an automatic air charging device.
Further, the flexible touch sensor unit comprises a canvas force touch sensor and a peripheral circuit, wherein the canvas force touch sensor is arranged on one non-closed side of the cuboid, and the peripheral circuit is arranged externally. The canvas force touch sensor is formed by combining nano force sensitive materials with soft canvas, and the canvas force touch sensor has five layers, including two canvas layers, two film layers and a nano force sensitive material layer.
Further, the light source unit comprises three light beads of RGB (red, green and blue), and the three light beads are respectively stuck on three side walls of the square box unit by hot melt adhesive.
Further, the image acquisition unit is a 100-ten-thousand-pixel high-definition binocular camera and is arranged at the bottom of the square box unit.
Further, the acceleration acquisition unit is a WT61C sensor based on an MPU6050 chip and is installed at the bottom of the square box unit.
Further, the force-sensitive material layer has 16 sensing point areas, which are symmetrically distributed in a 4×4 matrix, and each sensing point area is uniformly distributed with an equal amount of nano force-sensitive material.
An application method of a multi-mode touch sensing device comprises the following steps:
s1: firstly, 12V is electrified, the air pump starts to work and charges the inside of the device through the pipeline, and meanwhile, the flexible touch sensor unit starts to work after being electrified;
s2: when the flexible touch sensor unit works, the RGB light source unit continuously provides red, green and blue light sources, the binocular camera shoots a canvas force touch sensor which deforms when a measured object contacts or slides with the device, at the moment, the shape information of the measured object can be shot, then the shot RGB image and depth image are saved as a jpg file through the upper computer, the acceleration sensor also continuously outputs acceleration information to the upper computer through the serial port, and the upper computer saves all the acceleration information as a txt file;
s3: and (3) utilizing a long-and-short-term memory model to circulate a neural network algorithm, and carrying out centralized analysis on the haptic sequence, the txt information, the acceleration sequence, the txt information, the RGB image, the jpg information and the depth image, the jpg information, and then obtaining the shape information of the measured object through fitting.
The beneficial effects of the invention are as follows:
according to the invention, the flexible material canvas is used as a contact surface of the touch device, so that two types of heterogeneous touch information can be triggered simultaneously: and respectively representing a touch sequence and a touch image, and quantitatively describing information such as touch, slip sense and the like in the physical world through fitting. The device can obtain various modal information such as force touch quantity, RGB image, depth image, acceleration and the like when contacting or sliding with a measured object. The device breaks the current situation of single-mode information acquisition of the traditional touch sensing device, can integrate the measurement of information such as stress, acceleration, sliding shape and the like in the same sensing device, obtains high-precision contact texture information, realizes the measurement of multi-mode information, and can integrate the information into the identification and grabbing operation of an object. The device has the following advantages:
1. the gel sensing layer in the existing tactile sensor is replaced by a canvas force tactile sensor, so that the tactile sensor can trigger two heterogeneous tactile information simultaneously when sensing an object: the touch sequence and the touch image respectively solve the problem of insufficient sensing information caused by the single touch sensing mode of the traditional touch sensor.
2. The designed multi-mode touch sensing device is simultaneously provided with a plurality of sensors such as a pressure sensor, an acceleration sensor and a binocular camera, and the device collects multi-mode touch information of an object at the same time when sensing the object, so that the problem of weak pairing of multi-mode data existing when the traditional multi-mode sensor collects data is avoided, namely, the effect is poor when the data is fused due to the fact that the sensors are not synchronous when collecting the multi-mode data.
3. Compared with the existing tactile sensor, the flexible tactile sensor based on the nano force-sensitive material and the canvas has better flexibility, and the tactile image of the measured object can be captured through the shape change of the canvas force tactile sensor in the flexible tactile sensor when the object is sensed.
Drawings
FIG. 1 is a mechanical block diagram of a multi-modal haptic sensation apparatus in accordance with the present invention;
FIG. 2 is a block diagram of the overall structure of a multi-modal haptic sensation apparatus in accordance with the present invention;
figure 3 is a hierarchical view of the internal structure of the canvas force touch sensor according to the present invention (16 force touch sensing point areas on the force sensitive material layer).
In the figure, 1, PC end; 2. a power supply and a serial port; 3. a single chip microcomputer; 4. a flexible force tactile sensor; 5. black pellets of the object to be measured; 6. blue LED lamp beads; 7. an acceleration sensor; 8. a depth camera; 9. RGB camera; 10. red LED lamp beads; 11. green LED light beads; 12. an air pump.
Detailed Description
For the purpose of making apparent the objects, technical solutions and advantages of the present invention, the present invention will be further described in detail with reference to the following examples and the accompanying drawings, wherein the exemplary embodiments of the present invention and the descriptions thereof are for illustrating the present invention only and are not to be construed as limiting the present invention.
Referring to fig. 1-3, in the drawing, 1 is a PC end, the PC end receives data of all sensors, a user processes the data at the PC end, 2 is a power supply and a serial port, which provide power for the whole system and connect the PC and a single chip microcomputer, 3 is a single chip microcomputer minimum system drawn by us, 4 is a flexible force touch sensor designed by us independently, 5 is a black ball of a detected object, 6 is a blue LED lamp bead, 7 is an MPU6050 acceleration sensor, 8 is a depth camera, 9 is an RGB camera, 10 is a red LED lamp bead, 11 is a green LED lamp bead, and 12 is an air pump.
The square box unit is a cuboid hollow box made of black acrylic plate material, one surface of the cuboid is not closed, and the square box is used as a carrier of a circuit board of each module and is used for isolating the inside of the multi-mode touch sensing device from the external environment, so that the inside is absolutely isolated from the external environment; the air pump unit is an automatic air charging device;
the flexible touch sensor unit comprises a canvas force touch sensor, a 12V-to-5V voltage-stabilizing power supply circuit, a 5V-to-3.3V voltage-stabilizing power supply circuit, a stm32f103c8t6 minimum system circuit and a CD4066 switch selection circuit;
the light source unit comprises three light beads of red, green and blue, light sources of various colors are respectively provided, and each light bead is fixed on one side of the square box;
the image acquisition unit is a 100-ten-thousand-pixel high-definition binocular camera, is arranged at the bottom of the device, and shoots deformation information generated by the contact or sliding of a measured object with the canvas force touch sensor, wherein the deformation information comprises an RGB image and a depth image;
the acceleration acquisition unit is a WT61C sensor based on an MPU6050 chip and is arranged at the bottom and parallel to the horizontal plane;
the 12V power supply is mainly used for an air pump, the 3.3V voltage-stabilizing power supply circuit is mainly used for supplying power to a CD4066 chip and a stm32f103c8t6 singlechip chip, the stm32f103c8t6 minimum system circuit is mainly used for resetting the stm32f103c8t6 singlechip chip to prevent program running death and an external system clock, and the CD4066 switch selection circuit is mainly used for controlling the power supply of an internal circuit of a canvas force touch sensor and the selection function of an AD signal acquisition channel of an MCU.
The canvas force touch sensor combines nanometer force sensitive materials with soft canvas, the sensor has five layers altogether, including two canvas layers, two film layers and force sensitive material layers, wherein the canvas layer adopts thin canvas, make the sensor contact with measured object more silky, the film layer adopts flexible film, play the role of fixing the inner force sensitive material layer, the force sensitive material layer adopts nanometer force sensitive material, when the sensor receives the pressure, the resistance of the force sensitive material layer decreases with the pressure increase, 16 sense point areas altogether, become 4*4 matrix symmetrical distribution, each sense point area distributes the equal amount nanometer force sensitive materials evenly, when any sense point area produces deformation and can make the resistance of the area change, obtain this force change information through the sampling circuit, wherein the force acquisition resolution of the canvas force touch sensor can reach 0.5N, the sensor is mainly used for gathering the force touch information produced when measured object contacts with sensor;
the air pump unit is used for always filling air into the device in the working process of the sensor through the pipeline. The flexible touch sensor unit is mainly used for collecting force touch information generated by contact or sliding of a detected object and the device, wherein the canvas force touch sensor is fixed on one surface of the square box, which is not closed. Three RGB lamp beads in the light source unit are fixed on three sides of the inner wall of four sides of the square box, and mainly provide red, green and blue light sources. The image acquisition unit is fixed at the bottom of the square box and is mainly used for acquiring RGB information and depth information generated when a measured object contacts or slides with the device. The acceleration acquisition unit is fixed at the bottom of the square box and is parallel to the horizontal plane, and is mainly used for acquiring acceleration information generated when a measured object contacts or slides with the device. The shape information of the measured object is obtained by the fitting after the data are recorded and stored by the upper computer.
1. Firstly, 12V is electrified, an air pump starts to work and charges the inside of the device through a pipeline, meanwhile, a touch sensor unit is electrified, as the voltage value of a battery is reduced along with the longer service time, and the stm32f103c8t6 singlechip chip of the touch sensor unit needs 3.3V, a 3.3V and 5V voltage-stabilizing power supply circuit is designed for supplying power to the modules, and along with normal power supply, the CD4066 chip controls the on-off of 4 switches to enable 16 contacts of a canvas force touch sensor to supply power line by line, meanwhile, the stm32f103c8t6 singlechip returns an AD signal for collecting and supplying one line of contacts at a time to an upper computer, the information is 16 touch sequences at a time, and the upper computer stores all touch sequences as a txt file.
2. When the flexible touch sensor unit works, the RGB light source unit continuously provides red, green and blue light sources, the binocular camera shoots a canvas force touch sensor which deforms when a measured object contacts or slides with the device, at the moment, the shape information of the measured object can be shot, then the shot RGB image and depth image are saved as a jpg file through the upper computer, the acceleration sensor continuously outputs acceleration information to the upper computer through the serial port, and the upper computer saves all the acceleration information as a txt file.
3. The tactile sequence txt information, the acceleration sequence txt information, the RGB image jpg information, the depth image jpg information are intensively analyzed through a long-short-term memory model cyclic neural network algorithm, and then the shape information of the measured object with the accuracy rate of more than 90% can be obtained through fitting.
The above is an embodiment of the present invention. The foregoing embodiments and the specific parameters of the embodiments are only for clarity of description of the invention and are not intended to limit the scope of the invention, which is defined by the appended claims, and all equivalent structural changes made in the description and drawings of the invention are intended to be included in the scope of the invention.
Claims (4)
1. A multi-modal haptic sensation apparatus, characterized by: the device comprises a square box unit, an air pump unit, a flexible touch sensor unit, a light source unit, an image acquisition unit and an acceleration acquisition unit; the air pump unit is connected with the square box unit through a pipeline, the flexible touch sensor unit is arranged on the square box unit, the light source unit is arranged on the side wall of the square box unit, the image acquisition unit is arranged at the bottom of the square box unit, and the acceleration acquisition unit is arranged at the bottom of the square box unit; the square box unit is a cuboid hollow shell, the shell is made of black acrylic plate material, and one surface of the cuboid is not closed; the flexible touch sensor unit comprises a canvas force touch sensor and a peripheral circuit, wherein the canvas force touch sensor is arranged on one non-closed side of the cuboid, and the peripheral circuit is externally arranged; the canvas force touch sensor is formed by combining nano force sensitive materials with soft canvas, and comprises five layers, namely two canvas layers, two film layers and a nano force sensitive material layer; the light source unit comprises three RGB lamp beads which are respectively stuck to three side walls of the square box unit by hot melt adhesive; the image acquisition unit is a 100-ten-thousand-pixel high-definition binocular camera and is arranged at the bottom of the square box unit; the force-sensitive material layer is provided with 16 sensing point areas which are symmetrically distributed in a 4 multiplied by 4 matrix, and the sensing point areas are uniformly distributed with the same amount of nano force-sensitive material.
2. A multi-modal haptic device as claimed in claim 1 wherein: the air pump unit is an automatic air charging device.
3. A multi-modal haptic device as claimed in claim 1 wherein: the acceleration acquisition unit is a WT61C sensor based on an MPU6050 chip and is arranged at the bottom of the square box unit.
4. The method of claim 1, wherein the step of using the multi-modal tactile sensation device comprises: the method comprises the following steps:
s1: firstly, 12V is electrified, the air pump starts to work and charges the inside of the device through the pipeline, and meanwhile, the flexible touch sensor unit starts to work after being electrified;
s2: when the flexible touch sensor unit works, the RGB light source unit continuously provides red, green and blue light sources, the binocular camera shoots a canvas force touch sensor which deforms when a measured object contacts or slides with the device, at the moment, the shape information of the measured object can be shot, then the shot RGB image and depth image are saved as a jpg file through the upper computer, the acceleration acquisition unit also continuously outputs acceleration information to the upper computer through the serial port, and the upper computer saves all the acceleration information as a txt file;
s3: and (3) utilizing a long-and-short-term memory model to circulate a neural network algorithm, and carrying out centralized analysis on the haptic sequence, the txt information, the acceleration sequence, the txt information, the RGB image, the jpg information and the depth image, the jpg information, and then obtaining the shape information of the measured object through fitting.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111256932.9A CN114043506B (en) | 2021-10-27 | 2021-10-27 | Multi-mode touch sensing device and application method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111256932.9A CN114043506B (en) | 2021-10-27 | 2021-10-27 | Multi-mode touch sensing device and application method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114043506A CN114043506A (en) | 2022-02-15 |
| CN114043506B true CN114043506B (en) | 2023-05-30 |
Family
ID=80206428
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111256932.9A Active CN114043506B (en) | 2021-10-27 | 2021-10-27 | Multi-mode touch sensing device and application method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114043506B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115431289B (en) * | 2022-11-07 | 2023-03-07 | 之江实验室 | Robot-oriented four-in-one multi-modal touch sensor and method |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9164586B2 (en) * | 2012-11-21 | 2015-10-20 | Novasentis, Inc. | Haptic system with localized response |
| US9400557B2 (en) * | 2014-06-25 | 2016-07-26 | Intel Corporation | Multimodal haptic effect system |
| JP6703728B2 (en) * | 2016-07-11 | 2020-06-03 | 学校法人早稲田大学 | Near-contact sensor |
| CN108161994B (en) * | 2017-12-20 | 2020-07-10 | 清华大学 | Multi-modal touch sensing device |
| WO2021081084A1 (en) * | 2019-10-21 | 2021-04-29 | The Regents Of The University Of California | Multi-directional high-resolution optical tactile sensors |
| CN111241748A (en) * | 2020-01-13 | 2020-06-05 | 华北电力大学 | Wind driven generator fault diagnosis based on long-short term memory model recurrent neural network |
| CN111947814B (en) * | 2020-08-11 | 2021-12-17 | 上海海事大学 | Flexible three-dimensional touch sensor and manufacturing and detecting methods thereof |
| CN212567428U (en) * | 2020-08-20 | 2021-02-19 | 香港中文大学（深圳） | Touch sensor and robot |
| CN112284577B (en) * | 2020-09-27 | 2022-09-27 | 西安交通大学 | Piezoelectric piezoresistive composite touch sensor and preparation method thereof |
| CN112652060A (en) * | 2021-01-06 | 2021-04-13 | 上海交通大学 | Multi-modal visual and tactile sensing system and method based on particle image velocimetry |
| CN113146660A (en) * | 2021-04-08 | 2021-07-23 | 清华大学深圳国际研究生院 | Mechanical claw for tactile perception by depth vision |
| CN113295307A (en) * | 2021-05-20 | 2021-08-24 | 福州大学 | Flexible force touch sensing array capable of realizing body multi-position pressure bearing detection |
-
2021
- 2021-10-27 CN CN202111256932.9A patent/CN114043506B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114043506A (en) | 2022-02-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104679367B (en) | Sender unit and its control method | |
| CN103793099B (en) | The method that touch-sensing system and being used to reduces its delay | |
| CN106052914A (en) | Touch measurement sensor based on visual sense | |
| JP2017502535A (en) | Latency measurement and test system and method | |
| CN108161994A (en) | A kind of multi-modal tactile sensor | |
| CN114043506B (en) | Multi-mode touch sensing device and application method thereof | |
| TW201329815A (en) | Force sensitive interface device and methods of using same | |
| JP2007518966A (en) | Optical tactile sensor and force vector distribution reconstruction method using the sensor | |
| CN101206544B (en) | Man machine interactive touch sensing device and method thereof | |
| CN104007844A (en) | Electronic instrument and wearable type input device for same | |
| CN103495981A (en) | Manipulator based on touch sensor | |
| CN107145232B (en) | A kind of method and apparatus using capacitance detecting identification user's hand behavior | |
| CN101714046B (en) | Device and method for detecting object position and image display system using device | |
| CN104460951A (en) | Human-computer interaction method | |
| CN108151738A (en) | Codified active light marked ball with attitude algorithm | |
| CN110058711A (en) | Electronics blackboard eraser and its control method | |
| EP3691521A1 (en) | Electronic device and method for providing stress index corresponding to activity of user | |
| CN104881152B (en) | Sender unit and its method of signalling | |
| CN104142739A (en) | Laser point tracking system and method based on optical mouse sensing array | |
| CN111761579A (en) | Optical touch sensor, manufacturing method of elastic body module and information acquisition method | |
| Abad et al. | HaptiTemp: A next-generation thermosensitive GelSight-like visuotactile sensor | |
| Li et al. | L $^{3} $ F-TOUCH: A Wireless GelSight with Decoupled Tactile and Three-axis Force Sensing | |
| CN108068328A (en) | Recover the method for printing after 3D printing device, 3D printer and its interruption | |
| CN207301977U (en) | A kind of virtual reality glove | |
| CN114993542B (en) | Touch and slide sense integrated sensor based on single-contact soft vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |