CN111079333B - Deep learning sensing method of flexible touch sensor - Google Patents

Deep learning sensing method of flexible touch sensor Download PDF

Info

Publication number
CN111079333B
CN111079333B CN201911314290.6A CN201911314290A CN111079333B CN 111079333 B CN111079333 B CN 111079333B CN 201911314290 A CN201911314290 A CN 201911314290A CN 111079333 B CN111079333 B CN 111079333B
Authority
CN
China
Prior art keywords
sensor
data set
fusion
deep learning
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911314290.6A
Other languages
Chinese (zh)
Other versions
CN111079333A (en
Inventor
刘旺玉
郭正强
谢卫规
苟竞仁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to CN201911314290.6A priority Critical patent/CN111079333B/en
Publication of CN111079333A publication Critical patent/CN111079333A/en
Application granted granted Critical
Publication of CN111079333B publication Critical patent/CN111079333B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/048Activation functions

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Force Measurement Appropriate To Specific Purposes (AREA)

Abstract

The invention discloses a deep learning sensing method of a flexible touch sensor, which comprises the following steps in sequence: establishing a sensor unit structure mechanical model; establishing a sensor array structure mechanical model; obtaining an actual measurement data set; obtaining a finite element simulation dataset; the data set fusion improves the data resolution; and combining deep learning to establish a perception mechanism model. According to the invention, through fusing the data sets, the relation between the pressure signal and the three-dimensional multi-scale geometrical dimension, the surface morphology, the physical properties and other structures of the detection object is obtained by means of a deep learning model.

Description

Deep learning sensing method of flexible touch sensor
Technical Field
The invention relates to the field of sensing of flexible touch sensors, in particular to a deep learning sensing method of a flexible touch sensor.
Background
In the discrete numerical modeling of the perception mechanism, the numerical simulation precision based on the finite element depends on the size of the division of the finite element unit, and the contradiction between the calculation precision and the calculation efficiency can be brought. On the one hand, if a high-precision perception effect needs to be obtained, smaller and more finite element units are needed; on the other hand, more finite element units can bring the reduction of solving speed, and when the finite element units reach a certain scale, real-time numerical solving cannot be achieved by utilizing the finite element model of contact mechanics. At this point, a faster sensing mechanism approach is sought to achieve real-time prediction of sensor force signals and sensing models.
The perceptual mathematical model based on contact mechanics is a highly nonlinear model. In recent years, with the development of machine learning technology, deep learning attracts more and more attention with its excellent nonlinear fitting capability, and is widely used in semantic perception problems such as robot object recognition, object detection, and semantic segmentation. Furthermore, deep learning extends the perception based on two-dimensional images to three-dimensional space over some quantitative issues, such as object pose estimation, motion estimation, etc. The great success of these tasks demonstrates the ability to solve the quantitative estimation problem with deep learning. However, the current sensing methods such as CN106446948A only stay on the machine learning level, and the sensing mechanism research has not been performed by integrating the measured data, the finite element simulation data and the deep learning means, so as to improve the sensing precision and efficiency.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art and provides a deep learning sensing method of a flexible touch sensor, which is used for acquiring the relation between a pressure signal and structures such as three-dimensional multi-scale geometric dimensions, surface morphology, physical properties and the like of a detection object by means of a deep learning model through fusing the data sets.
The aim of the invention is achieved by the following technical scheme:
the deep learning sensing method of the flexible touch sensor comprises the following steps in sequence:
step 1), building a sensor unit structure mechanical model;
step 2), building a sensor array structure mechanical model;
step 3), obtaining an actual measurement data set;
step 4), obtaining a finite element simulation data set;
step 5), data set fusion improves data resolution;
and 6) combining deep learning to establish a perception mechanism model.
The sensor unit is a pyramid sensor unit formed by four sensitive units, and the four sensitive units are arranged at the inner part of the sensor unit in a 2x2 mode.
And the sensor unit integrates the elastic behavior model and the stress-capacitance conversion model into a sensor representative unit, performs discrete simplification and modularization on the sensor array, and constructs a response mechanism of the sensor array.
The actual measurement data set is finally established by touching an object with obvious boundary characteristics by utilizing a sensor array. The actual measurement database obtained at this time is rough and is used for training.
Further, a fine simulation data set is obtained using finite element simulation.
The data fusion is to establish a mapping relation to realize the fusion of the simulation data set and the actual measurement data set.
The establishment of the perception mechanism model comprises high-resolution pressure cloud image generation, fusion convolution operation, geometric relation reconstruction and microstructure and material attribute reconstruction.
The high-resolution pressure cloud image generation comprises the following steps:
(i) Firstly, obtaining a rough pressure cloud picture by using a linear interpolation mode from a low-resolution pressure cloud picture;
(ii) Performing downsampling self-coding operation on the low-resolution pressure cloud image by using convolution operation to obtain a series of characteristic layers;
(iii) And the characteristics of the layers are connected by adopting a U-Net mode, and finally, a high-resolution pressure cloud picture is obtained, so that higher precision is obtained.
The fusion convolution operation specifically comprises the following steps: providing a fusion convolution function for the properties of the boundary, the material and the like of the fusion object, and using the fusion convolution function in the convolution operation of each layer; let x be i Is the corresponding characteristic value, b is the offset value, N (x i ) Is x i Corresponding to the fusion convolution defined as:
x i =Ψ(M i )∑w(x j )·(x j e m j )+b
where w (·) can be regarded as a weight function, M i ={m j The corresponding adaptive matrix is the 'multiplication' operation, and the function of filtering the boundary value is achieved; in this, the function ψ (M i ) Is defined as a function related to object boundaries, microscopic appearance, material properties, etc.; thus, the function is combined with convolution operation to apply the features of the detected object to the high resolution pressure cloud reconstruction.
The geometric relationship reconstruction is specifically as follows: and carrying out feature extraction on the input three-dimensional pressure cloud image by using a graph convolution neural network in reconstruction, and then carrying out regression reconstruction on the three-dimensional model, so as to acquire the geometric shape of the detection object by using the sensor sensing signal.
The microstructure and material attribute reconstruction is specifically as follows: through actual measurement and finite element simulation, a corresponding database set of a sensor three-dimensional pressure cloud image and an object appearance microstructure, materials and the like is constructed, then the three-dimensional pressure cloud image is constructed into a 3×n matrix as an input characteristic, a multi-layer perceptron with DenseNet+ReLU as a block is adopted to conduct quantization regression training, and finally numerical vectors of the corresponding appearance microstructure and materials are output, so that the relation of material properties of the detected object is obtained through sensor perception signals.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. according to the invention, the established mechanical model of the touch sensor is utilized to carry out numerical simulation of the touch characteristic object of the sensor array, so as to obtain simulated touch data, the simulated touch data is used as an auxiliary database for collecting data of the sensor entity, and a mapping model is established to improve the accuracy of a training data set, so that a large-scale force signal and a corresponding high-resolution data set of a three-dimensional structure are constructed;
2. by utilizing the technology of convolutional neural network, constructing an end-to-end depth network to train on a data set and obtain a training model, quickly identifying the geometric and surface microstructure, material and other attributes of a three-dimensional object by utilizing force signals obtained by a sensor, and constructing a three-dimensional structural relation with the detected object.
Drawings
FIG. 1-1 is a schematic diagram of a sensor unit; fig. 1-2 are exploded views of a sensor unit.
FIG. 2 is a schematic diagram of the contact relationship between a sensor array and a surface feature of a test object.
Fig. 3 is a schematic diagram of a depth network for generating a high resolution pressure cloud.
Fig. 4 is a schematic diagram of a geometrical relationship reconstruction based on a three-dimensional pressure cloud.
Fig. 5 is a schematic diagram of a perception mechanism.
Wherein the meaning of the reference numerals is explained as follows:
1-adaptive convolution operation, 2-connection operation, 3-up acquisition operation, 4-sensor unit, 5-sensitive unit.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
The invention relates to the field of sensing mechanisms of flexible touch sensors, in particular to a deep learning sensing method of a flexible touch sensor. And establishing a three-dimensional mechanical model of the contact state of the surface microstructure of the flexible touch sensor and the three-dimensional microstructure of the surface of the touch object by adopting a numerical simulation and experiment method, fusing the actual measurement data set with the finite element contact simulation data set, and acquiring the relation between the pressure signal and the three-dimensional multi-scale geometric dimension, the surface morphology, the physical attribute and other structures of the detection object by means of a neural network deep learning model. The method comprises the following steps:
step 1), building a sensor unit structure mechanical model;
step 2), building a sensor array structure mechanical model;
step 3), obtaining an actual measurement data set;
step 4), obtaining a finite element simulation data set;
step 5), data set fusion improves data resolution;
and 6) combining deep learning to establish a perception mechanism model.
The sensor is designed to form a pyramid sensor unit by four sensitive units, wherein the four sensitive units are arranged at the inner part of the sensor unit by 2x2, as shown in figures 1-1 and 1-2. The upper and lower electrode layers of each sensing unit are connected in series with adjacent units so as to realize the array connection of the sensing units. In combination with the micro-capacitance detection circuit, high resolution tactile sensing can be achieved. Furthermore, the sensor unit elastic behavior model and the stress-capacitance conversion model are integrated into a sensor representative unit, matlab programming is adopted, modularization and discretization simulation are carried out on a sensor array mechanical response mechanism, and a cooperative strain mechanism of force loads in different directions is considered. And touching an object with obvious boundary characteristics by using a sensor array, and establishing a rough actual measurement database for training. By means of a finite element tool, simulating a stress response process when the sensor array faces touch features such as different micro geometric contact morphologies (as shown in fig. 2), and the like, touch feedback data under micro boundary disturbance is obtained and used as a fine data set for strengthening and supplementing the defects of an actual measurement data set, and fusion of the simulation data set and the actual measurement data set is realized by establishing a mapping relation, namely, the physical precision of the sensor is combined with the precision of a numerical algorithm, and the detection precision and sensitivity of the sensor are improved. And constructing a model of a perception mechanism by using a depth network, wherein the model comprises high-resolution pressure cloud image generation, fusion convolution operation, geometric relation reconstruction and microstructure and material property reconstruction.
The high-resolution pressure cloud image generating step is shown in fig. 3, (i) firstly, a rough pressure cloud image is obtained from a low-resolution pressure cloud image in a linear interpolation mode; (ii) Performing downsampling self-coding operation on the low-resolution pressure cloud image by using convolution operation to obtain a series of characteristic layers; (iii) And the characteristics of the layers are connected by adopting a U-Net mode, and finally, a high-resolution pressure cloud picture is obtained, so that higher precision is obtained. A fusion convolution function is provided for fusing object boundaries, materials and other attributes and is used in convolution operation of each layer. Let x be i Is the corresponding characteristic value, b is the offset value, N (x i ) Is x i Is to the neighborhood ofThe convolution is defined as:
x i =Ψ(M i )∑w(x j )·(x j e m j )+b
where w (·) can be regarded as a weight function, M i ={m j The corresponding adaptive matrix is the 'multiplication' operation, and the function of filtering the boundary value is achieved. In this, the function ψ (M i ) May be defined as a function related to object boundaries, microscopic appearance, material properties, etc. Thus, the feature of the detected object can be applied to the reconstruction of the high resolution pressure cloud image when the function is combined with the convolution operation.
And extracting features of the input three-dimensional pressure cloud image by using a graph convolution neural network, and then carrying out regression reconstruction on the three-dimensional model, so that the geometric shape of the detection object is obtained by using the sensor sensing signal, and the reconstruction flow is shown in fig. 4.
Through actual measurement and finite element simulation, a corresponding database set of a sensor three-dimensional pressure cloud image and an object appearance microstructure, materials and the like is constructed, then the three-dimensional pressure cloud image is constructed into a 3×n matrix as an input characteristic, a multi-layer perceptron with DenseNet+ReLU as a block is adopted to conduct quantization regression training, finally numerical vectors of the corresponding appearance microstructure, materials and the like are output, and therefore the relation of material properties of the detected object is obtained through sensor perception signals, and the perception mechanism is shown in figure 5.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.

Claims (4)

1. The deep learning sensing method of the flexible touch sensor is characterized by comprising the following steps in sequence:
step 1), building a sensor unit structure mechanical model;
step 2), building a sensor array structure mechanical model;
step 3), obtaining an actual measurement data set;
step 4), obtaining a finite element simulation data set;
step 5), data set fusion improves data resolution;
step 6), combining deep learning to establish a perception mechanism model;
the data set fusion is to establish a mapping relation to realize fusion of the simulation data set and the actual measurement data set;
the establishment of a perception mechanism model comprises high-resolution pressure cloud image generation, fusion convolution operation, geometric relation reconstruction and microstructure and material attribute reconstruction;
the high-resolution pressure cloud image generation comprises the following steps:
(i) Firstly, obtaining a rough pressure cloud picture by using a linear interpolation mode from a low-resolution pressure cloud picture;
(ii) Performing downsampling self-coding operation on the low-resolution pressure cloud image by using convolution operation to obtain a series of characteristic layers;
(iii) The characteristics of the layers are connected by adopting a U-Net mode, and finally, a high-resolution pressure cloud picture is obtained, so that higher precision is obtained;
the fusion convolution operation specifically comprises the following steps: providing a fusion convolution function for the boundary and material property of the fusion object, wherein the fusion convolution function is used in the convolution operation of each layer; let x be i Is the corresponding characteristic value, b is the offset value, N (x i ) Is x i Corresponding to the fusion convolution defined as:
x i =Ψ(M i )∑w(x j )·(x j ⊙m j )+b
wherein w (·) is a weight function, M i ={m j The corresponding adaptive matrix is the }, the # -is the multiplication operation, which plays a role in filtering the boundary value; in this, the function ψ (M i ) Defining as a function related to object boundaries, microscopic appearance, material properties; thus, combining the function to convolution operation applies the feature of the detected object to the high-resolution pressure cloudGraph reconstruction is carried out;
the geometric relationship reconstruction is specifically as follows: the reconstruction utilizes a graph convolution neural network to perform feature extraction on an input three-dimensional pressure cloud graph, and then performs regression reconstruction on a three-dimensional model, so that the geometric shape of a detection object is obtained by utilizing a sensor sensing signal;
the microstructure and material attribute reconstruction is specifically as follows: through actual measurement and finite element simulation, a corresponding database set of the sensor three-dimensional pressure cloud image and the microstructure and the material of the appearance of the object is constructed, then the three-dimensional pressure cloud image is constructed into a 3×n matrix as an input characteristic, a multi-layer perceptron with DenseNet+ReLU as a block is adopted to conduct quantization regression training, and finally a numerical vector of the corresponding microstructure and the material of the appearance is output, so that the relation of material properties of the detected object is obtained through sensor perception signals.
2. The method for deep learning sensing of a flexible touch sensor of claim 1, wherein the sensor unit is a pyramid sensor unit consisting of four sensing units, and the four sensing units are arranged 2x2 inside the sensor unit.
3. The method for deep learning sensing of a flexible touch sensor according to claim 1, wherein the sensor unit integrates an elastic behavior model and a stress-capacitance conversion model into a sensor representative unit, and performs discrete simplification and modularization on the sensor array to construct a response mechanism of the sensor array.
4. The method of claim 1, wherein the actual measurement data set is established by touching an object with distinct boundary features with a sensor array.
CN201911314290.6A 2019-12-19 2019-12-19 Deep learning sensing method of flexible touch sensor Active CN111079333B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911314290.6A CN111079333B (en) 2019-12-19 2019-12-19 Deep learning sensing method of flexible touch sensor

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911314290.6A CN111079333B (en) 2019-12-19 2019-12-19 Deep learning sensing method of flexible touch sensor

Publications (2)

Publication Number Publication Date
CN111079333A CN111079333A (en) 2020-04-28
CN111079333B true CN111079333B (en) 2024-03-12

Family

ID=70315541

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911314290.6A Active CN111079333B (en) 2019-12-19 2019-12-19 Deep learning sensing method of flexible touch sensor

Country Status (1)

Country Link
CN (1) CN111079333B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111796708B (en) * 2020-06-02 2023-05-26 南京信息工程大学 Method for reproducing three-dimensional shape features of image on touch screen
CN111964821A (en) * 2020-08-05 2020-11-20 清华大学深圳国际研究生院 Pressure touch prediction method and pressure touch prediction model for electronic skin
US20230306261A1 (en) * 2020-11-24 2023-09-28 Max-Planck-Gesellschaft zur Förderung der Wissenschaften e. V. Method for force inference of a sensor arrangement, methods for training networks, force inference module and sensor arrangement
CN112802182B (en) * 2021-01-20 2022-12-16 同济大学 Method and system for reconstructing anthropomorphic touch object based on touch sensor
CN114894354B (en) * 2022-04-11 2023-06-13 汕头大学 Pressure sensing feedback device based on surface structural color and deep learning identification method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20030027636A (en) * 2001-09-29 2003-04-07 홍동표 A Sensor capable of sensing objects
CN1539604A (en) * 2003-11-01 2004-10-27 中国科学院合肥智能机械研究所 Flexible touch sensor and touch information detection method
CN110135485A (en) * 2019-05-05 2019-08-16 浙江大学 The object identification and localization method and system that monocular camera is merged with millimetre-wave radar

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9092737B2 (en) * 2009-07-30 2015-07-28 Northwestern University Systems, methods, and apparatus for 3-D surface mapping, compliance mapping, and spatial registration with an array of cantilevered tactile hair or whisker sensors

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20030027636A (en) * 2001-09-29 2003-04-07 홍동표 A Sensor capable of sensing objects
CN1539604A (en) * 2003-11-01 2004-10-27 中国科学院合肥智能机械研究所 Flexible touch sensor and touch information detection method
CN110135485A (en) * 2019-05-05 2019-08-16 浙江大学 The object identification and localization method and system that monocular camera is merged with millimetre-wave radar

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Multimodal Material identification through recursive tactile sensing;A. Gómez Eguíluz 等;Robotics and Autonomous Systems;20180507;第106卷;130-139 *
基于卷积神经网络的软硬触觉感知方法研究;余乐 等;传感器与微系统;20171231;第36卷(第06期);35-37, 41 *
电容-电阻双模式材质识别传感器设计与实验;郭小辉 等;华中科技大学学报(自然科学版);20151031;第43卷(第S1期);220-223 *

Also Published As

Publication number Publication date
CN111079333A (en) 2020-04-28

Similar Documents

Publication Publication Date Title
CN111079333B (en) Deep learning sensing method of flexible touch sensor
Zhang et al. Fingervision tactile sensor design and slip detection using convolutional lstm network
CN110188598B (en) Real-time hand posture estimation method based on MobileNet-v2
Narang et al. Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections
Zhang et al. Effective estimation of contact force and torque for vision-based tactile sensors with helmholtz–hodge decomposition
Suresh et al. Shapemap 3-d: Efficient shape mapping through dense touch and vision
Wi et al. Virdo: Visio-tactile implicit representations of deformable objects
CN108594660B (en) Working modal parameter identification method and system of time invariant structure
Kuppuswamy et al. Fast model-based contact patch and pose estimation for highly deformable dense-geometry tactile sensors
CN111204476A (en) Vision-touch fusion fine operation method based on reinforcement learning
Lee et al. Predicting the force map of an ert-based tactile sensor using simulation and deep networks
Guo et al. Inverse simulation: Reconstructing dynamic geometry of clothed humans via optimal control
Soter et al. Shape reconstruction of CCD camera-based soft tactile sensors
Van der Merwe et al. Integrated object deformation and contact patch estimation from visuo-tactile feedback
CN116911079B (en) Self-evolution modeling method and system for incomplete model
CN113947119A (en) Method for detecting human gait by using plantar pressure signals
Saku et al. Spatio-temporal prediction of soil deformation in bucket excavation using machine learning
Du et al. 3D contact point cloud reconstruction from vision-based tactile flow
Wu et al. Example-based real-time clothing synthesis for virtual agents
Wang et al. Tactile sensory response prediction and design using virtual tests
CN116029205A (en) Flow field reconstruction method based on intrinsic orthogonal decomposition and deep learning fusion
CN100464151C (en) Method for inspecting article surface vein and its sensor
Rasoulzadeh et al. Linking early design stages with physical simulations using machine learning
Wang et al. Elastic interaction of particles for robotic tactile simulation
Azulay et al. Augmenting tactile simulators with real-like and zero-shot capabilities

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant