CN112971772A - Hand multi-movement mode recognition system based on palm multi-mode information - Google Patents
Hand multi-movement mode recognition system based on palm multi-mode information Download PDFInfo
- Publication number
- CN112971772A CN112971772A CN202110270965.2A CN202110270965A CN112971772A CN 112971772 A CN112971772 A CN 112971772A CN 202110270965 A CN202110270965 A CN 202110270965A CN 112971772 A CN112971772 A CN 112971772A
- Authority
- CN
- China
- Prior art keywords
- hand
- pressure sensor
- sensor
- palm
- glove
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000005452 bending Methods 0.000 claims abstract description 42
- 238000010801 machine learning Methods 0.000 claims abstract description 39
- 238000000034 method Methods 0.000 claims abstract description 19
- 238000003909 pattern recognition Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims abstract description 11
- 238000012549 training Methods 0.000 claims description 37
- 230000009471 action Effects 0.000 claims description 29
- 210000003811 finger Anatomy 0.000 claims description 22
- 241000282414 Homo sapiens Species 0.000 claims description 11
- 239000000758 substrate Substances 0.000 claims description 10
- 238000012360 testing method Methods 0.000 claims description 9
- 210000004247 hand Anatomy 0.000 claims description 7
- 210000000236 metacarpal bone Anatomy 0.000 claims description 5
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000009434 installation Methods 0.000 claims description 3
- 239000000463 material Substances 0.000 claims description 3
- 244000060701 Kaempferia pandurata Species 0.000 claims description 2
- 235000016390 Uvaria chamae Nutrition 0.000 claims description 2
- 238000009826 distribution Methods 0.000 claims description 2
- 210000004932 little finger Anatomy 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 abstract description 3
- 238000005070 sampling Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 210000003205 muscle Anatomy 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000010606 normalization Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 239000004744 fabric Substances 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 210000003813 thumb Anatomy 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 208000032170 Congenital Abnormalities Diseases 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000010146 3D printing Methods 0.000 description 1
- 208000004182 Acquired Hand Deformities Diseases 0.000 description 1
- 208000009004 Congenital Hand Deformities Diseases 0.000 description 1
- 208000025174 PANDAS Diseases 0.000 description 1
- 208000021155 Paediatric autoimmune neuropsychiatric disorders associated with streptococcal infection Diseases 0.000 description 1
- 240000000220 Panda oleosa Species 0.000 description 1
- 235000016496 Panda oleosa Nutrition 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 210000003710 cerebral cortex Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000013013 elastic material Substances 0.000 description 1
- 238000002567 electromyography Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 210000000245 forearm Anatomy 0.000 description 1
- 238000010030 laminating Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000002040 relaxant effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000009958 sewing Methods 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1118—Determining activity level
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1124—Determining motor skills
- A61B5/1125—Grasping motions of hands
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
- A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
- A61B5/6802—Sensor mounted on worn items
- A61B5/6804—Garments; Clothes
- A61B5/6806—Gloves
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Abstract
The invention discloses a hand multi-movement pattern recognition system based on palm multi-mode information, which comprises a sensing glove, a signal acquisition circuit, a computer and a display system, wherein: the glove comprises a sensing glove, wherein 5 bending sensors are arranged at the palm of the hand of the sensing glove, 3 pressure sensors are arranged at the back of the hand, and each bending sensor and each pressure sensor are connected with a signal acquisition circuit through wires; the signal acquisition circuit is connected with a computer, and the computer acquires and processes signals of the signal acquisition circuit and displays related information on a display. The curvature information of the hand and the pressure information of the back of the hand are simultaneously acquired, mutually supplemented and mutually backed up, the information is expanded into two modes, the number of the sensors is also increased relative to that of a single mode, the information quantity of machine learning is increased, and simple conversion between the single mode and the double mode can be realized to adapt to different working requirements.
Description
Technical Field
The invention belongs to the technical field of biomechanics integration (biomechanics), and relates to a hand multi-movement pattern recognition system based on palm multi-mode information.
Background
The artificial hand is a robot product combining various high and new technologies, and is also a representative technology in the field of bio-mechatronics. The main application scene of the artificial hand is to help the congenital or acquired hand deformity to complement the hand appearance and function. With the advance of human science and technology, the prosthetic hand is gradually changed from the original decorative prosthetic hand to the functional prosthetic hand.
The recognition of human intention is one of the supports of artificial hand functionality, and the accurate recognition of hand movements that a human desires to perform is the basis for completing artificial hand control, and has become a research hotspot in related fields in recent years. At present, the mainstream artificial hand adopts a non-invasive control mode which is mainly divided into three modes of electroencephalogram, electromyogram and muscle strength. The electroencephalogram recognition mode is that a special bioelectricity reaction formed by cerebral cortex when the brain controls a body to perform actions is detected to obtain an electroencephalogram, and an initial instruction sent by the brain is recognized by analyzing an electroencephalogram signal. Electromyography recognition is also a widely-used control mode, and is the most mature control mode, and the control intention of human beings on muscles is judged by monitoring the change of electric signals on the surface of skin. In addition, the muscle force signal recognition also has unique advantages, and the muscle force signal recognition mode acquires the muscle pressure signal through the pressure sensor attached to the skin and judges the intended gesture according to the muscle pressure signal.
The half palm deformity is a special condition of palm deformity, and the half palm artificial hand is also used for solving the problem. The remaining palm presents a challenge for the design of a prosthetic hand, but also presents a new possibility for hand motion recognition, some researchers hope to shift the signal acquisition position from the traditional position of the lower arm, the wrist and the like to the remaining palm, and the arrangement of muscle force sensors on the palm and the back of the hand is a good attempt, but the mode also has the defects of low recognition rate, uncomfortable wearing and the like. At present, no device or method for recognizing stable and technically mature human hand action aiming at the palm exists in the market.
Furthermore, the recognition patterns of human intent that apply more and more mature in the field of half-palm prosthetic hands tend to be monomodal. However, in practice, when a person performs a hand movement, a large number of variations of different physical quantities occur, and these signals can be used as a reference for recognition of the hand movement, so that the recognition rate and the recognition effect are improved, and it is significant to study a multi-modal recognition method.
Disclosure of Invention
The invention provides a hand multi-motion mode recognition system based on palm multi-mode information, which aims at solving the problems that the existing hand motion recognition method for a prosthetic hand at home and abroad has too single recognition reference physical quantity type, large recognition rate fluctuation, complex structure and poor wearing comfort, and most of schemes only use a single quantity as a recognition reference. The invention uses the bending degree and the pressure as multi-mode recognition, and the specific idea is that the judgment of the hand action mode is finished by machine learning after the bending degrees of different positions of the palm and the pressure conditions of different points of the back of the hand are collected, so that the collection and analysis of the bending information of the palm of the hand during working can be realized, and the classification recognition of the whole hand action is finished on the basis. The curvature information of the hand and the pressure information of the back of the hand are simultaneously acquired, mutually supplemented and mutually backed up, the information is expanded into two modes, the number of the sensors is also increased relative to that of a single mode, the information quantity of machine learning is increased, and simple conversion between the single mode and the double mode can be realized to adapt to different working requirements.
The purpose of the invention is realized by the following technical scheme:
a hand multi-movement mode recognition system based on palm multi-mode information comprises a sensing glove, a signal acquisition circuit, a computer and a display system, wherein:
the glove is characterized in that 5 bending sensors are arranged at the palm of the hand of the sensing glove, 3 pressure sensors (distributed as shown in figure 1) are arranged at the back of the hand, and each bending sensor and each pressure sensor are connected with a signal acquisition circuit through a lead;
the arrangement direction of the curvature sensors is along several key bending directions related to the motion of the hand part, the arrangement direction of the sensors corresponding to each finger is approximately the same as the finger direction (except the thumb part) in the finger root parts of the little finger, the ring finger, the middle finger and the ring finger, the connected palm part and the direction (reflecting the motion of the thumb) vertical to the first metacarpal bone and the second metacarpal bone, the motion of the corresponding finger can be reflected, and the curvature sensors can also independently provide curvature information for the system to serve as information for identifying the single mode;
the pressure sensor is attached to the back of the hand, and the three distribution points are respectively arranged on the metacarpal bones corresponding to the index finger, the middle finger and the ring finger and are close to the joint where the proximal phalanx is connected;
the signal acquisition circuit is composed of a power supply, a voltage division resistor, a potential limiting resistor, a multiplexer, a programmable gain amplifier, an A/D converter and a buffer, one pin of the bending sensor is connected with the negative electrode of the power supply, the other pin of the bending sensor is connected with one end of the voltage division resistor corresponding to the bending sensor, and the other end of the voltage division resistor corresponding to the bending sensor is connected with the positive electrode of the power supply; one pin of the pressure sensor is connected with the positive electrode of the power supply, the other pin of the pressure sensor is connected with one end of the divider resistor corresponding to the pressure sensor, and the other end of the divider resistor corresponding to the pressure sensor is connected with the negative electrode of the power supply; potential signals are led out between the bending sensor and the piezoelectric resistors corresponding to the bending sensor, between the pressure sensor and the voltage dividing resistors corresponding to the pressure sensor through leads, and the potential is limited through a potential limiting resistor; after entering, the multi-path signals are multiplexed into the programmable gain amplifier through the multiplexer, then are delivered to the A/D converter to be converted into digital signals, and the digital signals are buffered and then delivered to the computer;
the signal acquisition circuit is connected with a computer, and the computer acquires and processes signals of the signal acquisition circuit and displays related information on a display.
In the invention, the sensing glove is designed in a modularized way, and the sensing glove is divided into a glove substrate and a pressure sensor base on a mechanism, wherein: the curvature sensor is directly fixed on the glove substrate and is in place along with the glove substrate when worn; the pressure sensor is fixed on a pressure sensor base (the structure is shown in figure 3), the pressure sensor base is made of hard materials, the pressure sensor is attached to a movable salient point on the pressure sensor base, the movable salient point stretches through threads to adjust pre-pressure on the pressure sensor, the salient point and the pressure sensor are separated by elastic materials to isolate the influence of bolt rotation on the pressure sensor, and meanwhile, a holding torque is provided for the bolt through friction. The sensor base is secured to the hand with the glove base already in place by a strap which simultaneously provides a degree of pre-compression and securement for the curvature sensor. The sensing glove and the subsequent acquisition circuit adopt a plug-in design.
In the invention, the computer part in the hardware shares the tasks of data acquisition and writing, machine learning processing and processing result display. The tasks are carried out by means of software in a computer, and the software systems comprise a training sample acquisition system, a machine learning system and a real-time acquisition and display system, wherein: the training sample acquisition system provides training data for the machine learning system, the machine learning system provides classified calculation service for the real-time acquisition and display system, and the real-time acquisition system acquires data for the machine learning system and displays the final result to an operator through a display.
In the invention, a training sample acquisition system takes the task of data acquisition, an acquisition circuit sends a processed digital potential signal to a computer, the training sample acquisition system performs acquisition work, an operator sets an acquisition mode and the like in the training sample acquisition system, and the system displays an acquisition state according to corresponding settings and stores acquired data. The stored data is called by a machine learning system, the system processes the original data and constructs a training sample set, the training sample set is waited to be called by a real-time acquisition and display system, and the system carries out corresponding classification calculation after the training sample set is called. The real-time acquisition and display system is a system which works during actual identification, an operator can carry out identification work and system setting in the system, and the system can call partial functions in the machine learning system to obtain the classification result of the machine learning system and then display the classification result.
A method for recognizing a hand movement pattern based on palm bending information protection by using the system comprises the following steps:
step one, equipment connection: connecting each device through an interface, and checking to ensure the correctness and stability of the connection;
step two, wearing the glove substrate: the glove is worn on the palm, the position of the glove is adjusted to enable the curvature sensor to be in a proper position, the wearing is comfortable, trial actions are carried out after the wearing is finished, and the adjustment is carried out according to the actual situation;
step three, installing a pressure sensor: fixing a pressure sensor base on a palm with a glove base, so that the position is proper and the wearing is comfortable, performing trial action after the completion, adjusting according to actual conditions, wherein the installation step of the pressure sensor may have certain influence on the glove base, and adjusting according to the actual wearing conditions;
step four, testing and adjusting: opening a training sample acquisition system, monitoring the real-time waveform of the sensor by using a display control in the system so as to judge whether the state of the sensor is proper or not, and carrying out corresponding adjustment;
step five, sample collection: starting acquisition work on the basis of the fourth step, acquiring initial values of the sensors under the condition that hands are completely relaxed, and acquiring data of the sensors corresponding to a plurality of actions needing to be identified after the initial value acquisition is finished;
step six, action recognition: and C, the data acquired in the step five are delivered to a machine learning system, the machine learning system constructs a training set, a real-time acquisition display system is started, initial values are acquired, hands move freely after the acquisition is finished, actions in a sample set are made, the classification is carried out through machine learning, and the actions are fed back to a display screen.
Compared with the prior art, the invention has the following advantages:
1. the recognition mode for the hand motion of the person provided by the invention is novel and simple in form, high in working stability and reliability, good in cooperation between the equipment and the human body, high in affinity, simple and comfortable to wear, and capable of realizing integration of collection and recognition.
2. The invention realizes the recognition of various gestures through a machine learning method, has better flexibility, can be easily switched to various machine learning methods, and can easily add data and gestures to a training set so as to increase or decrease or update the gesture training set.
3. The circuit system adopted by the invention is simple and reliable, the signal processing method is quick and accurate, and the recognition rate is higher and stable after data processing such as filtering, normalization and the like in an algorithm. In a test, 9 common gestures in the database are recognized, the total recognition rate can exceed 93%, the recognition rate of some actions can even approach 100%, and the requirements of real-time recognition of the actions of the human hand and subsequent artificial hand control can be met.
4. The invention adopts a plurality of hand physical information, including the bending information of the hand and the pressure information of the back of the hand, the information is expanded into two modes, the number of the sensors is increased relative to the single mode, thus the number and the quality of the information input into the machine learning are greatly improved, and the identification accuracy and the identification stability of the system are greatly improved.
5. Multiple kinds of information are mutually supplemented and simultaneously mutually backed up, information with certain reliability can be provided for an operator only by identifying one mode, the number of the modes can be adjusted according to the actual use condition, and the flexibility is higher. The information of multiple modalities also provides more possibilities for optimization of the algorithm.
6. The flexibility of structure is good, and the design that the modularization can be inserted makes system architecture more reliable nimble, and tensioning and pressure sensor bolt are adjustable in addition, and is better to the adaptability of different people's palms on the whole.
Drawings
FIG. 1 is a schematic view of a sensor arrangement according to the present invention;
FIG. 2 is a diagram of the hardware configuration of the present invention;
FIG. 3 is a schematic view of a sensor base structure according to the present invention;
FIG. 4 is a schematic structural diagram of the working software system of the present invention.
Detailed Description
The technical solution of the present invention is further described below with reference to the accompanying drawings, but not limited thereto, and any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention shall be covered by the protection scope of the present invention.
The invention provides a hand multi-movement pattern recognition system based on palm multi-mode information, which consists of a sensing glove, a signal acquisition circuit, a computer and a display system on hardware and a training sample acquisition system, a machine learning system and a real-time acquisition and display system on software.
The base of the sensing glove is made of common fabric fibers, a plurality of bending sensors are arranged outside the fabric of the palm of the hand of the glove, the arrangement directions of the bending sensors are along a plurality of key bending directions related to the hand action, a plurality of pressure sensors are arranged on the back of the hand, the pressure sensors are fixed on a sensor base, and an adjusting device is arranged on the sensor base and can adjust the preset values of the pressure sensors. The sensor base of the back of the hand is connected with the glove through the fixing and tensioning device, and the tensioning device can also help the bending sensor at the palm to be closely attached to the glove and the hand, so that the deformation condition of the palm can be accurately reflected. And each bending sensor and each pressure sensor are connected with two wires to connect the sensors with a signal acquisition circuit. The main components of the signal acquisition circuit are composed of a power supply, a voltage dividing resistor, a potential limiting resistor, a multiplexer, a programmable gain amplifier, an A/D converter, a buffer and the like. One pin of the bending sensor is connected with the negative electrode of the power supply, the other pin of the bending sensor is connected with one end of a divider resistor of the bending sensor, and the other end of the divider resistor corresponding to the bending sensor is connected with the positive electrode of the power supply; one pin of the pressure sensor is connected with the positive electrode of the power supply, the other pin of the pressure sensor is connected with one end of a divider resistor of the pressure sensor, and the other end of the divider resistor corresponding to the pressure sensor is connected with the negative electrode of the power supply; potential signals are led out between the bending sensor and the bending sensor divider resistors and between the pressure sensor and the pressure sensor divider resistors through wires, and the potential is limited through the limiting resistors. Each sensor is connected with a signal acquisition circuit in such a way, after entering, a plurality of signals are multiplexed into a programmable gain amplifier through a multiplexer, then are delivered to an A/D converter to be converted into digital signals, then are cached and delivered to a computer, and the computer acquires and processes the signals and displays related information on a display.
The task of data acquisition in software is to train the sample acquisition system, and the acquisition circuit sends the processed digital potential signal to the computer. The computer is integrated into a dynamic array, initial values are stored when recording is started, then 8 channels of the dynamic array are stripped to form 8 dynamic waveforms which are displayed to an operator on a display screen, while averaging the dynamic data over a sampling frequency at the computer as part of the digital filtering, thereby obtaining a dynamic array with 8 elements, the number of the array is dynamically displayed to an operator after being subtracted from the initial array, and displayed in a waveform map and fed back to the operator, the array being dynamically refreshed, after acquisition is initiated, the dynamic array is retained to the computer if the system is in acquisition mode, different paths are switched to be respectively stored when different gesture actions are carried out, and only waveforms are generated without corresponding data being stored in the test mode. Before real-time identification, a machine learning system is used for importing and constructing a training set, then a real-time acquisition and display system is started, and the machine learning system needs to be continuously called in the identification process. After the test is started, the real-time acquisition and display system firstly acquires an initial value of a relaxation gesture, then acquires a dynamic array, 8 channel parts of the array are stripped, and an average value of dynamic data is acquired within a sampling frequency to serve as a part of digital filtering, so that a dynamic array with 8 elements can be obtained, the number of the array is subtracted from the initial array and serves as a test array to be sent to a machine learning system, the machine learning system receives and classifies the classification result and then returns the classification result, and the module displays the classification result in a mode of numbers and graphs. The machine learning system firstly calls data generated by a training sample acquisition system, establishes the data into a same array, performs normalization processing on all the data, uses the processed data as a training set, enters the training set into a classification algorithm, performs classification operation after receiving fact data from a real-time acquisition display system after waiting for the calling, and returns a return value after the operation is finished.
The training and identification process can be started after the gloves are worn correctly and connected correctly. Signals are collected under different gestures, collected through a training sample collection system and stored in a computer. During real-time identification, the operator makes several collected gestures, and the real-time signals are collected and classified by machine learning to generate an identification result and are fed back to the operator. The method specifically comprises the following steps:
step one, equipment connection: connecting the devices through interfaces, enabling the circuit elements to be connected to a computer after being correctly connected according to the diagram shown in FIG. 2, and checking to ensure the correctness and stability of the connection;
step two, wearing the glove substrate: the glove is worn on the palm, the position of the glove is adjusted to enable the curvature sensor to be in a proper position, the wearing is comfortable, trial actions are carried out after the wearing is finished, and the adjustment is carried out according to the actual situation;
step three, installing a pressure sensor: fixing a pressure sensor base on a palm with a glove base through a binding band, adjusting the position of the sensor and the tension of the binding band to enable the position to be proper and comfortable to wear, performing trial action after completion, adjusting according to actual conditions, wherein the installation step of the pressure sensor may have certain influence on the glove base, and adjusting according to actual wearing conditions;
step four, testing and adjusting: opening a training sample acquisition system, monitoring the real-time waveform of the sensor by using a display control in the system so as to judge whether the state of the sensor is proper or not, and carrying out corresponding adjustment;
step five, sample collection: starting acquisition work on the basis of the fourth step, acquiring initial values of the sensors under the condition that hands are completely relaxed, acquiring data of the sensors corresponding to a plurality of actions needing to be identified after the initial value acquisition is finished, sequentially acquiring data of each action, determining the quantity of the acquired data according to actual requirements, and adjusting the amplitude of force according to the actual conditions in the acquisition process of one action so as to enable the data acquisition to cover the conditions as much as possible;
step six, action recognition: and C, the data acquired in the step five are delivered to a machine learning system, the machine learning system constructs a training set, a real-time acquisition display system is started, the initial value is acquired firstly, the hand can be freely moved after the acquisition is finished, a plurality of actions in the sample set are made, the system is classified through machine learning, and the actions are fed back to a display screen.
The third to fifth steps are to collect and identify on the computer, and the specific process and logic are shown in fig. 4: after the training sample collection system starts to collect, firstly, whether a start key is a true value is judged, if yes, an initial value is collected and recorded, whether the training sample collection system is in a collection mode is judged, if not, only a channel is divided to display a waveform, if yes, the average value is further calculated, the difference between the average value and the initial value is calculated, the result is stored to be called by a standby device learning system such as a computer, then, whether the circulation is continued is judged, if yes, the collection is continued, and if not, the collection is stopped. The machine learning system receives data from the training sample acquisition system, performs mathematical processing such as normalization and the like, establishes a sample and a classifier, and establishes a subfunction capable of normalizing and classifying real-time data. And the real-time acquisition display system performs real-time sampling after judging that the initial value is acquired after starting, calculates the mean value and the difference value and gives the mean value and the difference value to a classification subfunction of the machine learning system, displays the obtained operation return value, judges whether to continue circulation or not, repeats the real-time identification process if the operation return value is judged to continue circulation, and stops the process if the operation return value is judged to continue circulation.
Example (b):
the hardware implementation of this embodiment can be seen in fig. 2. The traditional fabric glove is used as a substrate at a part contacted with the hand, the sensor is fixed on the glove by bonding, sewing and other modes on the substrate, 5 curved film sensors are distributed at the center of the hand, the fixing positions of the sensors are respectively along the deformation directions of 5 fingers, and the tail end position of the sensor is coincided with the position with larger deformation of the root of the finger, so that better deformation information can be obtained. The sensor is a film bending sensor produced by Sparkfun company, the resistance of the sensor in a straight state is 25K ohms, the resistance of the sensor in a bending state can reach 10-125K ohms, and the resistance value of the sensor is increased along with the increase of the bending degree. The back of the hand is attached with three film pressure sensors, the sensor is selected from FSR400 produced by Sparkfun company, the measuring range of the sensor can reach 10 kilograms, the resistance of the sensor is reduced along with the increase of pressure, when the pressure reaches about 10 kilograms, the resistance of the sensor can be reduced to about 0.25 kilo-ohm, the three pressure sensors are fixed on a sensor base, the manufacturing method of the sensor base is 3D printing, the design refers to ergonomics to realize better skin affinity, three threaded holes are formed in three corresponding positions on the base, three bolt knobs can penetrate through the threaded holes to apply pre-pressure to the pressure sensors, and the nominal diameter of each bolt is 8 millimeters and basically consistent with the effective area diameter of the pressure sensor. The pressure sensor base is through 4 firmly taking fixes on the gloves, and these 4 firmly take can play the effect of tensioning simultaneously, play the effect that compresses tightly and fix to the crookedness sensor of palm department to realize the laminating of crookedness sensor and people's palm of the hand in order to adopt more accurate data. Two pins of the curvature sensor are connected with a basic circuit through a lead, one pin is connected with the negative electrode of a power supply, the other pin is connected with a divider resistor to which the sensor belongs, and the other end of the divider resistor is connected with the positive electrode of the power supply; two pins of the pressure sensor are connected with the basic circuit through wires, one pin is connected with the anode of the power supply, the other pin is connected with a divider resistor to which the pressure sensor belongs, and the other end of the divider resistor is connected with the cathode of the power supply. The basic potential display circuit unit is thus constructed, and all the voltage dividing resistances are 100K ohms for the bending sensor and 10K ohms for the pressure sensor. The two ends of the sensor and the divider resistor are connected with a 5V power supply, 8 divider circuits are connected in parallel at the two ends of the power supply, a potential signal is led out to a signal processing circuit through a lead between each sensor and each divider resistor, firstly, the potential signal is limited by a limiting resistor (passing through 127K ohm, passing through 30.9 ohm to be connected with the 2.5V power supply, and passing through 39.2K ohm to be grounded), then, the signal is multiplexed by a multiplexer and amplified by a programmable gain amplifier, the amplified signal converts an analog quantity signal into a digital quantity signal through an analog-to-digital converter, and then, a related signal is transmitted to a computer and cached in a cache. The circuit for signal processing adopts a built-in circuit of an NI USB6008 signal acquisition card, and the acquisition card has 8 analog input channels.
In this embodiment, the training sample collection system uses LabVIEW to write a program. The front panel is composed of a picture of a hand part, a waveform oscilloscope corresponding to each sensor, an array display module for feeding back the difference between the value of the sensor and a zero point in real time and a waveform diagram thereof. The input control is provided with a file saving path selector, a start button, a collection button, a stop button and a mode selection button. The main program has a while loop envelope, thereby realizing uninterrupted display and file writing, and a small loop is arranged outside the loop of the main program to execute the power-off operation. Wrapping a condition structure in the main loop, entering the main loop when a start button is a true value, wherein the condition structure is internally a sequential structure, firstly returning each display control and a key to an initial value, and then collecting an initial value by the DAQ assistant after collecting and forming an initial value array. Nesting a while loop in the next part of the sequential structure, wherein the loop executes the core part of the program, continuously extracting information from the DAQ assistant to the program in the core part, the sampling frequency is 1000 Hz, the sampling mode is N sampling, the signal entering the program is an array containing 8 columns of information, and the array is split and is respectively displayed through a waveform diagram, so that the waveform of each sensor can be fed back to an operator. After the channel groups are separated, the channel groups enter a nested condition structure separately, and the condition structure can judge a test mode and an acquisition mode. The system executes the conditional structure in an acquisition mode, an array summation command is used in the process of averaging data in a sampling frequency in the conditional structure, after the operation is finished, each channel outputs a number in the sampling frequency, the 8 numbers in the 8 channels are combined into an array, the array is established by using the previous initial value number, and the difference value between each channel and the initial value can be obtained. Feeding back the difference value to a front panel through an array display control, simultaneously creating a waveform chart, taking the array as a Y coordinate, and creating an array constant of 1-8 as an X coordinate, thereby obtaining a complete graph and displaying the complete graph. And nesting a condition mechanism in the next sequential structure, entering the condition structure when the system is in an acquisition mode, and writing the average number array into the table file under the specified path through a write measurement file control in the condition structure. The form file will be used as the material for further training. The core content of a gesture recognition software part is a machine learning system, the module uses Python for compiling, modules needing to be called, such as nunpy, pandas, sklern and the like, are loaded, a table file is opened through an open command (9 files are available, each file represents a gesture, each file has 5 columns, and each column represents data of a sensor), after the table file is opened, the data in the table is exported through a list command, then the data is digitized through an array command of a numpy function library, then the first row of each sub-array is removed through a vstack of the numpy function library, a large array is pasted to form a training set, the size of the array before pasting is counted through shape, and then labels are added to the training array from 0 to 8 in sequence to form a label array. And (3) carrying out floating point processing on all data in the training array and the label array, solving the maximum value and the minimum value of each column and the difference between the maximum value and the minimum value in the training array, and dividing the difference after subtracting the minimum value from each element to finish the data after normalization processing. The final data is copied to the support vector machine (which may be replaced with other classifiers) and the support vector machine is designated as the linear classification kernel. And then, defining a special function to wait for calling from LabVIEW, taking the real-time data as the input of the function, carrying out the same normalization processing on the real-time data according to the maximum value and the minimum value of the training sample, calling the previous machine learning model to train, and obtaining a classification result as the return value of the function. The system is also written by LabVIEW, the front panel comprises an indicator light, a display control for starting a key return value and a display control for gestures made by a pull-down list, while a while loop is still adopted as a program main body in a program diagram of the system, wherein a sequence structure is nested, in part 1 of the sequence structure, the initial of the data is acquired by a DAQ assistant and stored as an initial value array, the latter part of the sequence structure is similar to the program of a sampling system, after the data is acquired by the DAQ assistant, the average value in a sampling frequency is directly calculated, the average value is output as a dynamic array with 8 elements, a Python module is called and input into the Python module as an input value, and the output of the Python module, namely the return value of a subfunction in a machine learning system, is led out to the display control for displaying the return value And the corresponding picture is displayed as an input of a drop-down list.
When the gesture information acquisition system is used, the sample data acquisition module is opened firstly, the sensing gloves are worn, the user clicks to start and start acquisition, the program acquires the initial value of data firstly, the initial acquisition of the data is completed after the green light is turned on, the user clicks to enable the system to be switched from a test state to an acquisition state at the moment, the user starts to acquire the gesture information at the moment, the gesture information under the more complex condition can be provided through force transformation, forearm position transformation and the like during the acquisition for about 30 seconds, and therefore the overall error rate is reduced. After the acquisition of one gesture is completed, the file storage directory is changed according to the same method, the acquisition of the rest 8 data is carried out, and finally 9 table files are formed, wherein the 9 gestures respectively represent 9 gestures, and the following 9 gestures are adopted in the experiment: relaxing, bending thumb, bending index finger, bending three fingers, stretching five fingers, stretching index finger, clenching fist, unfolding palm and pinching three fingers. These actions represent gestures that are often used in daily life. And after the sample collection is completed, opening the real-time identification and display module and entering an activity stage. And saving the file path in the activity stage, keeping the hands relaxed, clicking to start the system to start taking an initial value at the moment, and starting the free activity stage after the system is started to light up a green light after the system is adopted. After the free activity phase, the picture on the display screen can be seen as a result of the recognition of the current gesture. When the device is used, the sampling frequency and the sampling time can be adjusted according to the actual use environment to control the refreshing speed and the accuracy of the sampling result. On the basis, the accuracy of the whole system is tested, the test method is that 9 gestures contained in a sample set are made in a recognition and display module, each gesture is made 20 times, if the gesture is recognized as other gestures or cannot be recognized for a long time, the gesture is regarded as a recognition error, the recognition accuracy of the system is counted, and the comprehensive accuracy reaches 93%.
Claims (9)
1. A hand multi-movement pattern recognition system based on palm multi-mode information is characterized in that the hand multi-movement pattern recognition system comprises a sensing glove, a signal acquisition circuit, a computer and a display system, wherein:
the glove comprises a sensing glove, wherein 5 bending sensors are arranged at the palm of the hand of the sensing glove, 3 pressure sensors are arranged at the back of the hand, and each bending sensor and each pressure sensor are connected with a signal acquisition circuit through wires;
the signal acquisition circuit is connected with a computer, and the computer acquires and processes signals of the signal acquisition circuit and displays related information on a display.
2. The palm multi-modal information-based human hand multi-motion pattern recognition system of claim 1, wherein the bending sensor is arranged along a critical bending direction related to the hand motion.
3. The hand multi-modal information-based hand multi-movement pattern recognition system according to claim 2, wherein the bending sensors are respectively arranged at the finger roots of the little finger, the ring finger, the middle finger and the ring finger, the connected palm parts and the directions perpendicular to the first metacarpal bone and the second metacarpal bone.
4. The hand multi-movement pattern recognition system based on palm multi-mode information as claimed in claim 1, wherein the pressure sensor is attached to the back of the hand, and the three distribution points are respectively located on the metacarpals corresponding to the index finger, the middle finger and the ring finger and are located near the joint where the proximal phalanx meets.
5. The hand multi-movement pattern recognition system based on palm multi-mode information according to claim 1, characterized in that the signal acquisition circuit is composed of a power supply, a voltage dividing resistor, a potential limiting resistor, a multiplexer, a programmable gain amplifier, an A/D converter and a buffer, one pin of the curvature sensor is connected with the negative electrode of the power supply, the other pin of the curvature sensor is connected with one end of the voltage dividing resistor corresponding to the curvature sensor, and the other end of the voltage dividing resistor corresponding to the curvature sensor is connected with the positive electrode of the power supply; one pin of the pressure sensor is connected with the positive electrode of the power supply, the other pin of the pressure sensor is connected with one end of the divider resistor corresponding to the pressure sensor, and the other end of the divider resistor corresponding to the pressure sensor is connected with the negative electrode of the power supply; potential signals are led out between the bending sensor and the piezoelectric resistors corresponding to the bending sensor, between the pressure sensor and the voltage dividing resistors corresponding to the pressure sensor through leads, and the potential is limited through a potential limiting resistor; after entering, the multi-channel signals are multiplexed into the programmable gain amplifier through the multiplexer, then are delivered to the A/D converter to be converted into digital signals, and the digital signals are buffered and then delivered to the computer.
6. The palm multi-modal information-based human hand multi-motion pattern recognition system of claim 1, wherein the sensing glove is divided into two modules, a glove base and a pressure sensor base, wherein: the bending sensor is fixed on the glove substrate, the pressure sensor is fixed on the pressure sensor base, and the pressure sensor base is fixed on the glove substrate through the binding belt.
7. The palm multi-modal information-based human hand multi-movement pattern recognition system of claim 1, wherein the pressure sensor base is made of a hard material and the pressure sensor is attached to the movable salient points thereon.
8. The hand multi-movement pattern recognition system based on palm multi-modal information as claimed in claim 1, wherein the computer comprises a training sample collection system, a machine learning system, a real-time collection display system, wherein: the training sample acquisition system provides training data for the machine learning system, the machine learning system provides classified calculation service for the real-time acquisition and display system, and the real-time acquisition system acquires data for the machine learning system and displays the final result to an operator through a display.
9. A method for hand movement pattern recognition based on palm flexion information protection using the system of any one of claims 1 to 8, the method comprising the steps of:
step one, equipment connection: connecting each device through an interface, and checking to ensure the correctness and stability of the connection;
step two, wearing the glove substrate: the glove is worn on the palm, the position of the glove is adjusted to enable the curvature sensor to be in a proper position, the wearing is comfortable, trial actions are carried out after the wearing is finished, and the adjustment is carried out according to the actual situation;
step three, installing a pressure sensor: fixing a pressure sensor base on a palm with a glove base, so that the position is proper and the wearing is comfortable, performing trial action after the completion, adjusting according to actual conditions, wherein the installation step of the pressure sensor may have certain influence on the glove base, and adjusting according to the actual wearing conditions;
step four, testing and adjusting: opening a training sample acquisition system, monitoring the real-time waveform of the sensor by using a display control in the system so as to judge whether the state of the sensor is proper or not, and carrying out corresponding adjustment;
step five, sample collection: starting acquisition work on the basis of the fourth step, acquiring initial values of the sensors under the condition that hands are completely relaxed, and acquiring data of the sensors corresponding to a plurality of actions needing to be identified after the initial value acquisition is finished;
step six, action recognition: and C, the data acquired in the step five are delivered to a machine learning system, the machine learning system constructs a training set, a real-time acquisition display system is started, initial values are acquired, hands move freely after the acquisition is finished, actions in a sample set are made, the classification is carried out through machine learning, and the actions are fed back to a display screen.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110270965.2A CN112971772A (en) | 2021-03-12 | 2021-03-12 | Hand multi-movement mode recognition system based on palm multi-mode information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110270965.2A CN112971772A (en) | 2021-03-12 | 2021-03-12 | Hand multi-movement mode recognition system based on palm multi-mode information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112971772A true CN112971772A (en) | 2021-06-18 |
Family
ID=76334737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110270965.2A Pending CN112971772A (en) | 2021-03-12 | 2021-03-12 | Hand multi-movement mode recognition system based on palm multi-mode information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112971772A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103251419A (en) * | 2013-04-25 | 2013-08-21 | 西安交通大学苏州研究院 | Data gloves for function rehabilitation training and assessment of hands and monitoring method thereof |
| CN106383579A (en) * | 2016-09-14 | 2017-02-08 | 西安电子科技大学 | EMG and FSR-based refined gesture recognition system and method |
| CN108037821A (en) * | 2017-10-30 | 2018-05-15 | 上海念通智能科技有限公司 | A kind of wearable palm band for gesture identification |
| WO2019173678A1 (en) * | 2018-03-09 | 2019-09-12 | Siemens Aktiengesellschaft | Optimal hand pose tracking using a flexible electronics-based sensing glove and machine learning |
| CN110262664A (en) * | 2019-06-21 | 2019-09-20 | 济南大学 | A kind of intelligent interaction gloves with cognitive ability |
| CN111708433A (en) * | 2020-05-22 | 2020-09-25 | 重庆大学 | Gesture data acquisition glove and sign language and gesture recognition method based on gesture data acquisition glove |
-
2021
- 2021-03-12 CN CN202110270965.2A patent/CN112971772A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103251419A (en) * | 2013-04-25 | 2013-08-21 | 西安交通大学苏州研究院 | Data gloves for function rehabilitation training and assessment of hands and monitoring method thereof |
| CN106383579A (en) * | 2016-09-14 | 2017-02-08 | 西安电子科技大学 | EMG and FSR-based refined gesture recognition system and method |
| CN108037821A (en) * | 2017-10-30 | 2018-05-15 | 上海念通智能科技有限公司 | A kind of wearable palm band for gesture identification |
| WO2019173678A1 (en) * | 2018-03-09 | 2019-09-12 | Siemens Aktiengesellschaft | Optimal hand pose tracking using a flexible electronics-based sensing glove and machine learning |
| CN110262664A (en) * | 2019-06-21 | 2019-09-20 | 济南大学 | A kind of intelligent interaction gloves with cognitive ability |
| CN111708433A (en) * | 2020-05-22 | 2020-09-25 | 重庆大学 | Gesture data acquisition glove and sign language and gesture recognition method based on gesture data acquisition glove |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jiang et al. | Exploration of force myography and surface electromyography in hand gesture classification | |
| CN107378944B (en) | Multidimensional surface electromyographic signal artificial hand control method based on principal component analysis method | |
| Jiang et al. | Force exertion affects grasp classification using force myography | |
| Esposito et al. | A piezoresistive array armband with reduced number of sensors for hand gesture recognition | |
| Lu et al. | A hand gesture recognition framework and wearable gesture-based interaction prototype for mobile devices | |
| EP3203350A1 (en) | Wearable controller for wrist | |
| CN105575219A (en) | Intelligent glove | |
| CN103777752A (en) | Gesture recognition device based on arm muscle current detection and motion sensor | |
| CN104665820A (en) | Wearable Mobile Device And Method Of Measuring Biological Signal With The Same | |
| CN112057059A (en) | Psychological stress intelligent acquisition, test and analysis system based on multi-modal physiological data | |
| Yang et al. | Experimental study of an EMG-controlled 5-DOF anthropomorphic prosthetic hand for motion restoration | |
| CN205508194U (en) | Intelligence gloves | |
| Haggard et al. | Hand and brain: the neurophysiology and psychology of hand movements | |
| CN107943285B (en) | Man-machine interaction wrist ring, system and method based on biological myoelectricity | |
| CN104856707A (en) | Pressure sensing data glove based on machine vision and gripping process judgment method thereof | |
| CN111419237A (en) | Cerebral apoplexy hand motion function Carroll score prediction method | |
| Nagata et al. | A classification method of hand movements using multi channel electrode | |
| Saggio et al. | Sensory systems for human body gesture recognition and motion capture | |
| CN112971773B (en) | Hand motion mode recognition system based on palm bending information | |
| CN112971772A (en) | Hand multi-movement mode recognition system based on palm multi-mode information | |
| CN116304816A (en) | Wearable intelligent human motion state recognition system based on machine learning | |
| CN112380943B (en) | Multi-position limb motion capture method based on electrical impedance | |
| CN212067683U (en) | Hand ring for analyzing shooting gestures | |
| CN110232976B (en) | Behavior identification method based on waist and shoulder surface myoelectricity measurement | |
| CN213787446U (en) | Glove for measuring physiological signals of human body |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |