CN116301341A

CN116301341A - Wearable sign language recognition system and method integrating attachable flexible stretching sensor and inertial sensing unit

Info

Publication number: CN116301341A
Application number: CN202310113573.4A
Authority: CN
Inventors: 胡又凡; 刘宇轩; 蒋熙俊; 罗一鸣; 于兴阁
Original assignee: Xiangtan University
Current assignee: Xiangtan University
Priority date: 2023-02-14
Filing date: 2023-02-14
Publication date: 2023-06-23

Abstract

The invention provides a wearable sign language recognition system and a method integrating an attachable flexible stretching sensor and an inertial sensing unit, wherein the system comprises the following steps: the flexible stretching sensor is attached to the position of the joint of the right finger of the human body and is used for monitoring the bending movement of the finger; the inertial sensing unit is attached to the back of the hand and used for monitoring the movement of the hand in the space; the flexible acquisition circuit is in data communication with each sensor; the sign language recognition system receives the sensing data transmitted from the flexible acquisition circuit, judges whether the hand is in an active state according to the sensing data, and recognizes and outputs sign language actions based on a first recognition model based on CNN trained in advance in the active state. The invention adopts the attached flexible stretching sensor with good attachment and convenient wearing, combines the sensing of the inertial sensing unit to realize the sensing of the movement of the wrist and the arm in space, fuses the bending characteristic and the movement state characteristic of the hand, and rapidly and accurately responds and identifies the complex movement.

Description

Wearable sign language recognition system and method integrating attachable flexible stretching sensor and inertial sensing unit

Technical Field

The invention relates to the technical field of intelligent wearable equipment, in particular to a wearable sign language recognition system and method integrating a novel attachable flexible stretching sensor and an inertial sensing unit.

Background

Sign language is a form of language conveyed by hand, expression and body, and is mainly recognized by visual perception. However, without prior knowledge of sign language, it is difficult for non-sign language users to accept and understand such dialog mediums. Thus, communication barriers are caused between sign language users and non-sign language users.

Currently, the prior art attempts to use auxiliary means such as a wearable electronic device to help establish communication and perception between sign language users and non-sign language users, and the wearable electronic device has the advantages of light weight, low cost, high flexibility, strong adaptability and the like, so that the wearable electronic device can provide a technical solution for the communication obstacle through the form of the wearable sign language translation device.

In the field of gesture monitoring, commonly used sensors include flexible mechanical sensors (pressure, strain, etc.), electromyographic signal sensors, image sensors, and the like. For example, a standardized sign language simulation intelligent glove disclosed in chinese patent application publication No. CN110491251a monitors finger bending only by a flexible sensor during monitoring, cannot monitor hand movement in space, and loses important information. As another example, a self-adaptive correction type sign language inter-translation system disclosed in chinese patent application publication No. CN110189590a adopts a bending sensor and an accelerometer, and uses a glove to carry all sensors, but the commercial bending sensor is not light and thin, does not have good flexibility, and the glove is not comfortable to wear.

Therefore, in order to solve the communication obstacle between the sign language user and the non-sign language user, some more effective auxiliary tools are needed to help the deaf-mute using the sign language communicate with the outside in a light, efficient and flexible manner.

Prior art literature:

patent document 1: CN110491251A standardized sign language simulation intelligent glove

Patent document 2: CN110189590A self-adaptive correction type sign language inter-translation system and method

Patent document 3: CN109613976A intelligent flexible pressure sensing sign language recognition device

Disclosure of Invention

In view of the defects existing in the prior art, the invention aims to provide a wearable sign language recognition system and method integrating an attachable flexible tension sensor and an inertial sensing unit, which adopt the novel attachable flexible tension sensor, are good in attaching property and convenient to wear, do not cause uncomfortable feeling or action resistance influence on a wearer, do not influence the action of the wearer, have better dynamic performance and sensitivity, and can respond to complex actions rapidly and accurately; simultaneously, the sensing of the motions of the wrist and the arm in space can be realized by combining the sensing of the inertial sensing unit, the bending characteristics and the motion state characteristics of the hand are fused, on one hand, the judgment of the motions is realized, on the other hand, the training of the convolutional neural network sign language recognition model is performed based on the fused characteristics, the meaning of the sign language recognition motions is rapidly and accurately obtained during the actual use, and the sign language recognition results are output in real time.

According to a first aspect of the object of the present invention, there is provided a wearable sign language recognition system integrating an attachable flexible tension sensor and an inertial sensing unit, comprising:

a flexible stretch sensor attached to the joint position of each finger of the human body for monitoring bending movements of the fingers;

the inertial sensing unit is attached to the back of the hand and used for monitoring the movement of the hand in the space;

the flexible acquisition circuit is in data communication with each flexible tension sensor and each inertial sensing unit;

the sign language recognition system is arranged to receive the sensing data transmitted from the flexible acquisition circuit, judge whether the hand is in an active state according to the sensing data, and recognize and output sign language actions based on a first recognition model trained in advance in the active state.

As an alternative embodiment, the flexible stretching sensor is a resistive flexible stretching sensor, and is composed of a first substrate layer, a second substrate layer, a sensitive material layer and a stretchable wire layer, wherein the sensitive material layer is positioned between the first substrate layer and the second substrate layer, and the stretchable wire layer is electrically connected with the sensitive material layer. The sensitive material layer is a conductive carbon black layer. The stretchable conducting wire layer is a conductive polymer with a preset shape, wherein the conductive polymer is prepared by mixing an Ecoflex material and silver nano-sheets in a certain proportion. In some embodiments, the mass ratio of silver nanoplates in the conductive polymer is 68% ± 1%. The conductivity of the silver nano-sheets with less content is too low, and open circuits are easy to form under larger stretching; the silver nano-sheets with larger content are difficult to stir and mix in the preparation process, and the formed mud is too hard to be adhered to a substrate to form a pattern after the mask is stripped. Therefore, the mass ratio of the silver nano-sheet optimized by the invention can ensure the conductivity and simultaneously ensure that the mud-like substance is not too hard, and the patterning is easy.

Therefore, the attachable flexible tension sensor provided by the invention is light, thin and soft, has good attachable performance, can be well attached to the skin surface of the finger joint without other auxiliary means (such as glue, adhesive tape and the like), does not cause uncomfortable feeling to a wearer, and does not influence the action of the wearer.

The attachable flexible tensile sensor provided by the embodiment of the invention can be directly attached to the second joint of the finger of the right hand by means of the adhesive force of the material, and is attached to 5 fingers of the hand in total. When the finger is bent, the stretching of the skin at the joint drives the stretching action of the sensor, and the sensor is light, thin and soft, and has an elastic modulus close to that of human skin, so that the sensor can still be well attached when the finger is bent, and a wearer can not feel obvious resistance.

When the sensitive material layer in the sensor is stretched, the conductive network formed by the conductive carbon black is partially disconnected, so that the resistance of the conductive network is obviously increased; when the stretching is released, the disconnected portion of the conductive network is restored, so its resistance is also restored. Thus, the change in the degree of finger bending can be converted into a change in sensor resistance in real time.

As an alternative embodiment, the flexible stretch sensor is arranged to be made in the following way:

step 1, taking an acrylic plate with a certain thickness, and cutting the acrylic plate into a substrate with a certain size according to the shape of a sensor;

step 2, cutting a PI film with the thickness of 50 micrometers to obtain patterns of a sensitive material layer and a stretchable wire layer serving as a mask plate, and cutting the patterns into a certain size;

step 3, placing the substrate on a spin coater, uniformly pouring the prepared Ecoflex glue on the substrate, and carrying out spin coating;

step 4, placing the substrate subjected to spin coating on a hot plate, and heating and curing for 1 hour to prepare a first basal layer;

step 5, aligning the mask plate of the sensitive material layer with the substrate, and attaching the PI film of the sensitive material layer to the corresponding position of the solidified Ecoflex glue surface;

step 6, rolling and coating a certain amount of conductive carbon black on the surface of the Ecoflex glue exposed by the mask plate and corresponding to the position of the sensitive material layer, repeating for a plurality of times, and uniformly coating; then stripping the mask plate of the sensitive material layer, and based on the acting force between the Ecoflex glue and the conductive carbon black, leaving the conductive carbon black on the surface of the Ecoflex glue in a required pattern;

step 7, aligning the mask plate of the stretchable wire layer with the substrate, and attaching the PI film of the stretchable wire layer to the corresponding position on the surface of the solidified Ecoflex glue;

Step 8, uniformly coating the conductive polymer formed by mixing the Ecoflex material and the silver nano-sheet on the surface of the mask plate, wherein the pattern area corresponding to the stretchable wire layer is free from holes;

step 9, stripping the mask plate of the stretchable wire layer, wherein the conductive polymer is reserved in a required pattern;

step 10, taking conductive textile wires, and respectively adhering the conductive textile wires to the end parts of each stretchable wire layer by using a small amount of conductive polymers;

step 11, placing the prepared first basal layer, sensitive material layer and stretchable conducting layer substrate on a spin coater, uniformly distributing a second layer of Ecoflex glue again, and packaging the device;

and step 12, placing the substrate on a hot plate, heating and curing for 1 hour, preparing a second basal layer, and finishing the preparation of the sensor.

As an alternative example, the conductive polymer of Ecoflex material mixed with silver nanoplates is configured to be prepared as follows:

uniformly mixing a certain amount of silver nano-sheets, 50ml of ethanol and 2ml of deionized water by magnetic stirring to obtain a first mixed solution;

dripping a certain amount of potassium iodide solution with the concentration of 0.01mol/L into the first mixed solution, and continuously stirring to obtain a second mixed solution;

vacuum filtering the second mixed solution to complete solid-liquid separation and drying;

Exposing the dried silver nano-sheet powder to strong light to decompose silver iodide therein;

mixing the treated silver nano-sheet powder with an Ecoflex material according to a preset mass ratio to prepare the conductive polymer.

As an alternative embodiment, the sign language recognition system is configured to determine whether the hand is in an active state in the following manner:

receiving first sensing data acquired by 5 flexible stretching sensors according to a preset sampling period T, and second sensing data sampled by a triaxial accelerometer of an inertial sensing unit according to the preset sampling period T;

and judging that the root mean square after deriving any one of the first sensing data or the second sensing data exceeds a preset threshold value, and judging that the sensor is in an active state, otherwise, judging that the sensor is not in the active state.

As an alternative embodiment, the pre-trained first recognition model is configured to be obtained based on convolutional neural network training, the training process comprising:

under the action of a plurality of different sign languages, the flexible stretching sensor at each finger joint position and the inertial sensing unit at the back of the hand position are used for respectively monitoring the bending of the finger and the movement data of the hand; then, carrying out numerical value and length standardization on the monitoring data of each sign language action, and establishing a sample database of the sign language action; the method comprises the steps that the numerical standardization of monitoring data corresponding to each sign language action in a sample database means that output numerical values of three accelerometers of five flexible stretching sensors and an inertial sensing unit are respectively subjected to standardized conversion, and the length standardization means that the output numerical values are standardized according to a preset duration range to obtain 200 sampling point data;

Fusing eight data channels corresponding to each sample in the sample database to form an 8 x 200 matrix containing finger bending characteristics and hand movement characteristics;

forming a training set and a testing set according to the sample database at a ratio of 80:20;

and training a model for sign language action recognition through a convolutional neural network by taking the training set as training data, and performing test verification through a test set to obtain a model with recognition accuracy reaching a preset level as the first recognition model.

According to a second aspect of the object of the present invention, there is also provided a wearable sign language recognition method, comprising the following steps:

step A, acquiring monitoring data acquired by a flexible stretching sensor and an inertial sensing unit according to a preset sampling period T;

step B, filtering the monitoring data collected by the flexible stretching sensor and the inertia sensing unit;

step C, judging whether the flexible tension sensor is in an active state or not according to monitoring data acquired by the flexible tension sensor and the inertia sensing unit:

-in response to being in an active state, continuing to acquire monitoring data for a next sampling period T until a next inactive state, recording all monitoring data in an active state for that stage as identification object data;

-in response to the inactive state, discarding the sampling period data and returning to step a, continuing to acquire monitoring data for the next sampling period T;

step D, judging whether the duration time of the identification object data reaches a preset threshold time length or not:

-discarding the segment of identification object data in response to the preset threshold duration not being reached, and returning to step a, continuing to acquire monitoring data for the next sampling period T;

-in response to reaching a preset threshold duration, retaining the piece of identification object data;

e, respectively carrying out standardization processing on the numerical range and the length of the output values of the three accelerometers corresponding to the five flexible stretching sensors and the inertial sensing unit in the reserved identification object data to obtain eight standard data channel data; fusing the eight standardized data channels to form an 8 x 200 matrix containing finger bending characteristics and hand movement characteristics;

and F, inputting a matrix of 8 x 200 containing finger bending characteristics and hand movement characteristics into the first recognition model trained in advance to recognize the sign language actions, and outputting a corresponding sign language action recognition result.

It should be understood that all combinations of the foregoing concepts, as well as additional concepts described in more detail below, may be considered a part of the inventive subject matter of the present disclosure as long as such concepts are not mutually inconsistent. In addition, all combinations of claimed subject matter are considered part of the disclosed inventive subject matter.

The foregoing and other aspects, embodiments, and features of the present teachings will be more fully understood from the following description, taken together with the accompanying drawings. Other additional aspects of the invention, such as features and/or advantages of the exemplary embodiments, will be apparent from the description which follows, or may be learned by practice of the embodiments according to the teachings of the invention.

Drawings

The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. Embodiments of various aspects of the invention will now be described, by way of example, with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of a wearable sign language recognition system according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of wearing effect of the wearable sign language recognition system according to the embodiment of the present invention.

Fig. 3 is a schematic diagram of a flexible stretch sensor of a wearable sign language recognition system of an embodiment of the present invention.

FIGS. 4 a-4 c are test data for a flexible stretch sensor according to an embodiment of the present invention, wherein 4a is a stretch-release hysteresis curve, respectively; 4b is a repeated test at different degrees of stretching; 4c is a durability test for 2000 cycles of stretch release.

Fig. 5 is a graph of the resistivity of conductive polymers using different weight fractions of silver nanoplates in an embodiment of the present invention.

Fig. 6 is a system schematic diagram of a flexible acquisition circuit according to an embodiment of the invention.

Fig. 7 is a schematic circuit diagram of a resistor-voltage conversion circuit (i.e., an R-V converter in the figure) in a flexible acquisition circuit according to an embodiment of the present invention.

FIG. 8 is a schematic representation of acquired sensor data waveforms in an embodiment of the present invention.

Fig. 9 is a flowchart of a wearable sign language recognition method according to an embodiment of the present invention.

FIG. 10 is a data waveform diagram of 10 different sign language word samples according to an embodiment of the present invention.

The definition of the individual reference numerals in the figures is as follows:

the system comprises a 101-flexible stretching sensor, a 102-inertial sensing unit, a 103-flexible acquisition circuit and a 104-sign language recognition system;

201-substrate, 202-sensitive material layer, 203-stretchable wire layer, 204-conductive textile wire.

Detailed Description

For a better understanding of the technical content of the present invention, specific examples are set forth below, along with the accompanying drawings.

Aspects of the invention are described in this disclosure with reference to the drawings, in which are shown a number of illustrative embodiments. The embodiments of the present disclosure are not necessarily intended to include all aspects of the invention. It should be understood that the various concepts and embodiments described above, as well as those described in more detail below, may be implemented in any of a number of ways, as the disclosed concepts and embodiments are not limited to any implementation. Additionally, some aspects of the disclosure may be used alone or in any suitable combination with other aspects of the disclosure.

{ wearable sign language recognition System })

The wearable sign language recognition system of the embodiment shown in connection with fig. 1 and 2 comprises a flexible stretch sensor attached to the joint position of each finger of a human body for monitoring the bending movement of the finger, and an inertial sensing unit attached to the back of the hand for monitoring the movement of the hand in space. Therefore, the sensing assembly integrating the plurality of sensors has sensing capability of multiple dimensions, not only can sense bending of finger joints, but also can sense movement of wrists and arms in space, and the sign language sensing capability is improved.

The wearable sign language recognition system of the embodiment shown in fig. 1 and 2 further comprises a flexible acquisition circuit that acquires data of each flexible tension sensor and the inertial sensing unit.

In some embodiments, the sign language recognition system is configured to be computer system-based, including for example but not limited to embedded computer systems, desktop/laptop computer systems, cloud computer systems, etc., which may generally be configured according to different application scenarios and requirements, including for example but not limited to the following forms: integrated with a flexible acquisition circuit; physically separate from the flexible acquisition circuitry and located on the surface of the human body (e.g., hand, wrist, arm, chest, head, etc.); physically separate from the flexible acquisition circuit and from the human body; is arranged at a cloud server and the like.

In some embodiments, a computer system generally has a data communication interface module, a memory, and a processor. The data interface module is used for realizing data communication, for example, data interaction with the flexible tension sensor and the inertial sensing unit in a wireless (such as Bluetooth, wifi and the like) or wired communication mode, obtaining monitoring data and storing the monitoring data in the memory. The memory can be a nonvolatile memory, and the data can be read and written between the nonvolatile memory and the processor through a data bus. The processor is used for reading the monitoring data and executing a preset computer program to realize the recognition and output of the sign language actions.

In an alternative embodiment, the computer system is further configured with means for characterizing the recognition result, for example a sound and/or visual characterization means, in particular a loudspeaker which can be driven to sound, a display screen means which can be driven to display the recognition result.

Flexible stretching sensor

In an embodiment of the invention, FIG. 3 illustrates a schematic diagram of a flexible stretch sensor, which may be particularly useful with a resistive flexible stretch sensor, attached to a second joint location of a finger for monitoring bending movement of the finger.

The flexible stretch sensor may take the form of a "sandwich" structure, i.e. two substrates 201 with a layer of sensitive material 202 and a stretchable wire layer 203 between the two substrates. In the example of fig. 3, the flexible stretch sensor is composed of a first substrate layer (i.e., a lower substrate Ecoflex), a second substrate layer (i.e., an upper substrate Ecoflex), and a layer of sensitive material 202 and a stretchable wire layer 203 between the first and second substrate layers. The stretchable wire layer 203 is connected to the sensitive material layer 202.

The substrate 201 serves as a carrier and package for the sensor, carrying and protecting the sensor's internal components. In alternative embodiments, the substrate 201 may be selected from materials having a low modulus of elasticity and good biocompatibility, such as Ecoflex materials, which are selected in embodiments of the present invention

00-30 material with elastic modulus equivalent to human skin. Therefore, the attachable flexible tension sensor is light, thin and soft, has good adhesion, and can be well adhered to the skin surface of a human body without other auxiliary means (such as glue, adhesive tape and the like).

The sensitive material layer 202 is a conductive carbon black layer and is formed by using a conductive carbon black material, such as ECP600JD conductive carbon black. As a core for generating the sensing signal, the resistance of the conductive carbon black layer varies with stretching.

The stretchable wire 203 connecting the sensitive material layer and the external circuit is a conductive polymer formed by mixing Ecoflex material and silver nanoplates in a certain ratio, and is made into a conductive polymer of a predetermined shape, for example, a bar shape. The stretchable wire 203 adopted by the invention has the advantages of flexibility, stretchability and good conductivity, and still has good conductivity under larger stretching, thereby playing the roles of leading out the sensitive material layer and isolating the sensitive material layer from external physical contact.

As shown in connection with fig. 3, the conductive textile wire 204 of the flexible tensile sensor 101 serves as a wire that is led out of the end of the stretchable wire, which facilitates connection of the sensor to external circuitry outside the Ecoflex material package.

Referring to fig. 2 and 3, the flexible stretching sensor 101 with the sandwich structure provided by the invention can be directly attached to the second joint of the finger by means of the adhesion of the substrate material, and five flexible stretching sensors 101 are configured corresponding to five fingers. In the example of fig. 2, 5 fingers attached to the right hand are taken as an example.

When the finger is bent, the stretching of the skin at the joint drives the stretching of the flexible stretching sensor 101, and the flexible stretching sensor 101 is light, thin and soft, and has the elastic modulus equivalent to that of human skin, so that the flexible stretching sensor 101 can still be well attached when the finger is bent, and a wearer can not feel obvious resistance.

When a portion of the sensitive material layer (e.g., conductive carbon black layer) in the flexible tensile sensor 101 is stretched, the conductive network formed by the conductive carbon black therein is partially broken, causing its electrical resistance to increase significantly; when the stretching is released, the disconnected portion of the conductive network is restored, so its resistance is also restored. Thus, a change in the degree of finger bending can be converted into a change in sensor resistance.

Preparation of flexible tensile sensor

The embodiment of the invention discloses a preparation process for the flexible tension sensor, which comprises the following steps:

step 5, aligning the mask plate of the sensitive material layer with the substrate, and attaching the PI film of the sensitive material layer to the corresponding position on the surface of the solidified Ecoflex glue;

Accordingly, the process of preparing a flexible tensile sensor on a 2cm x 6cm substrate using the process described above in this example includes:

1) Taking an acrylic plate with the thickness of 3mm, and cutting the acrylic plate into a substrate with the thickness of 2cm by 6cm by a laser cutting machine according to the shape of a sensor;

2) Cutting out patterns of the sensitive material layer and the stretchable wire layer by using a laser cutting machine to serve as a mask plate, and also cutting into a size of 2cm x 6 cm;

3) Mixing a proper amount of Ecoflex A glue and Ecoflex B glue, removing bubbles in vacuum, and generally pumping for 5min to obtain the Ecoflex glue for later use;

4) Placing the substrate on a spin coater, and uniformly pouring a proper amount of prepared Ecoflex glue on the substrate; rotating for 20s at a rotating speed of about 1000rpm, and uniformly throwing the Ecoflex glue on the substrate, wherein the thickness of one layer of Ecoflex is in the level of tens of micrometers, so that the sensor is light and thin enough;

5) Placing the substrate on a hot plate, and heating and curing at 70 ℃ for 1 hour;

6) Aligning the mask plate of the sensitive material layer with the substrate, enabling PI to be attached to the surface of Ecoflex, and enabling the pattern to be in a correct position;

7) A small amount of carbon black is dipped by a cotton swab and is rolled on the surface of the Ecoflex exposed by the mask plate, and the coating is repeated for a plurality of times and is uniform. Stripping the PI mask, and reserving carbon black powder on the surface of the Ecoflex in a required pattern due to stronger acting force between the Ecoflex and the carbon black;

8) Aligning the reticle of the stretchable wire layer with the substrate similar to step 5);

9) Uniformly coating a conductive polymer formed by mixing Ecoflex and silver nano-sheets on the surface of the mask plate to ensure that no holes exist in a pattern area;

10 Peeling off the PI mask, it is also preserved in the desired pattern due to the tackiness of the conductive polymer itself;

11 Taking two sections of conductive textile wires with the length of about 5cm, respectively adhering a small amount of conductive polymer to two ends of the stretchable wire, and testing whether the sensor is conducted or not on the textile wires at the two ends by using a universal meter; if the material is not conducted, the error in the manufacturing process is indicated, and the material needs to be prepared again; if the current is conducted, the preparation is qualified;

12 Placing the substrate on a spin coater, uniformly coating a second layer of Ecoflex glue on the substrate, and packaging the device;

13 Placing the substrate on a hot plate, and heating and curing at 70 ℃ for 1 hour;

14 After curing, the sensor is peeled off from the acrylic substrate.

Thus, the preparation of the flexible tension sensor provided by the invention is completed.

As an alternative example, a conductive polymer in which Ecoflex material is used mixed with silver nanoplates may be prepared in the following manner:

mixing the treated silver nano-sheet powder with an Ecoflex material according to a preset mass ratio to complete the preparation of the conductive polymer.

Accordingly, the required conductive polymer is prepared by adopting the process, and the specific steps are as follows:

1) 2g commercial silver nanoplates, 50ml ethanol, 2ml deionized water were mixed and magnetically stirred for 1 hour;

2) Preparing a proper amount of potassium iodide solution with the concentration of 0.01 mol/L;

3) After the magnetic stirring of the step 1) is completed, dripping 5ml of potassium iodide solution into the mixed solution, and continuously stirring for 30min until the reaction is fully completed;

4) After stirring, vacuum filtering the mixed solution to complete solid-liquid separation, and drying;

5) Exposing the dried silver nano-sheet powder to strong light to decompose silver iodide therein;

6) And preparing an appropriate amount of Ecoflex glue, and mixing the treated silver nano-sheet powder with Ecoflex, wherein the mass ratio of the silver nano-sheet powder is 68% +/-1%, so as to complete the preparation of the conductive polymer.

Silver nanoplates are excellent conductive fillers. However, commercial silver nanoplates are typically coated with a lubricant to prevent cold welding during processing. The lubricant layer typically in the form of a long chain fatty acid silver salt impedes electrical conduction between adjacent silver flakes. Commercial silver nanoplates are first treated with potassium iodide to convert the surface lubricant and silver oxide to silver iodide, which then breaks down under light to form silver nanoparticles. The conductivity of these silver nanoplates is greatly enhanced due to the removal of the surface lubricant.

The mixing ratio of the silver nanoplate powder to Ecoflex is further described: because the Ecoflex is made conductive, the silver nano-sheet powder with relatively large content is required to be added, so that the Ecoflex is dispersed by the silver nano-sheet, and cannot be solidified and formed as pure Ecoflex, and the Ecoflex is in a mud shape. The silver nano-sheet with a small content has low conductivity and is easy to cause open circuit under larger stretching; while the silver nano-sheets with larger content are difficult to stir and mix, the formed mud is too hard and difficult to be adhered on a substrate to form a pattern after the mask is stripped. Fig. 5 shows that the resistivity of the conductive polymer prepared by different weight ratios of silver nanoplates, the silver nanoplates of the present invention selected to be suitable for 68% ± 1% mass ratio can ensure conductivity while making the paste not too hard and easy to pattern.

Fig. 4a-4c schematically show test results for the prepared flexible stretch sensor 201. In embodiments of the present invention, the performance of the prepared sensor, including sensitivity, linearity and repeatability durability, was tested using a linear displacement stage. The following is presented in connection with fig. 4a-4 c: 4a is a stretch-release hysteresis curve, and shows that the sensitivity GF value is about 4, the hysteresis is smaller, and the linearity is good; 4b is a repeated test at different degrees of stretching, showing good repeatability at all degrees of stretching; 4c is a durability test of 2000 cycle stretch release, and from the test results, the sensor still works stably after 2000 cycle bad tests.

Inertial sensing unit

In the embodiment of the invention, the inertial sensing unit 102 is attached to the back of the hand, and a commercial inertial sensor is selected and comprises a triaxial accelerometer, so that the change of three axial accelerations in the process of hand movement in space can be sensed, and monitoring data can be output.

As an alternative embodiment, inertial sensing unit 102 employs a commercially available MPU9250 sensor and uses the functionality of three of the accelerometers to communicate data via an I2C interface to a flexible acquisition circuit.

It should be appreciated that commercial MPU9250 sensors integrate 3-axis gyroscopes, 3-axis accelerometers, and 3-axis magnetometers internally, with high static measurement accuracy. In the embodiment of the present invention, only the sensing data of the 3-axis accelerometer of the MPU9250 is used, and the sensing data is connected and communicated with the flexible acquisition circuit through the I2C interface, for example, the MCU of the flexible acquisition circuit may access the register inside the MPU9250 through the I2C interface to acquire the data.

Flexible acquisition circuit

In the embodiment of the present invention, the flexible acquisition circuit 103 is configured to acquire signal data of the above-mentioned multiple sensors (i.e., signal data of 8 channels output by the three accelerometers of the 5 flexible tensile sensors and the inertial sensor unit 102), perform preprocessing, and then send the signal data to the computer system for subsequent identification processing.

As an alternative embodiment, the flexible acquisition circuit 103 is used to implement conversion, acquisition, filtering and transmission processes for the sensor signals. The flexible acquisition circuit 103 as shown in fig. 6 includes: 1) The interface of the sensor is convenient for connecting each sensor (the inertia sensing unit and the flexible stretching sensor) to realize data communication; 2) A resistance-voltage conversion circuit (i.e., the R-V converter in FIG. 6) converts the change in resistance of the flexible tension sensor into a voltage change for convenient acquisition by the digital-to-analog converter; with reference to fig. 7, the resistor-voltage conversion circuit includes a filtering function for the output signal of the flexible tension sensor, and a first-order RC low-pass filter circuit is used for filtering the signal; 3) The Microcontroller (MCU) module comprises functions of digital-to-analog conversion, data transmission and the like, realizes the control of data acquisition and transmission, and also comprises a digital filter to complete the operation required by the filtering of digital signals; MCU selects ESP32 of Lexin company; 4) The power supply module converts the voltage provided by the battery or external power supply into the voltage required by each module and provides the electric energy required by the normal operation of each module.

It should be appreciated that the digital filter circuit of the flexible acquisition circuit 103 is used to filter the accelerometer signal and is implemented using a kalman filter algorithm. Since the acceleration raw data obtained from the inertial sensing unit is easily affected by vibration, and contains large noise, the subsequent analysis of the data is not facilitated, and therefore, the data of the accelerometer is filtered by adopting a Kalman filtering algorithm in the embodiment of the invention. Kalman filtering is an algorithm for optimally estimating the state of a system by using a linear system state equation and through system input and output observation data. Because the filtering algorithm is a time domain algorithm, and the current filtering result can only depend on the last result and the current measured value, the method has the advantages of small calculated amount and no need of historical data buffering, and therefore the filtering effect is good in the embodiment of the invention.

With reference to fig. 6, each flexible tensile sensor of each channel is correspondingly configured with a resistor-voltage conversion circuit (i.e., the R-V converter in fig. 6), so that the change of the resistor of the attachable flexible tensile sensor is converted into a voltage change, and the voltage change is convenient for the digital-analog converter to collect. Meanwhile, as shown in fig. 7, the resistance-voltage conversion circuit further includes an RC filter process for the output signal of the attachable flexible tension sensor.

The resistor-voltage conversion circuit shown in fig. 7 is combined, and the resistor-voltage conversion circuit is realized by using a homodromous proportional amplifying circuit, so that the linear conversion of the resistor to the voltage can be realized. The attachable flexible tension sensor is used as a feedback resistor to be connected into a circuit, and the output voltage of the circuit is as follows:

next, the signal is passed through a first order RC low pass filter to filter out high frequency noise, the cut-off frequency of the filter being:

since the signal bandwidth of the finger movement to be detected is low, the cut-off frequency is set to 100Hz in this example in order to filter out the vast majority of electromagnetic interference.

Since there are 5 tensile sensors, 5 sets of resistor-to-voltage conversion circuits as shown in fig. 7 are required, and can be implemented using a combination of LMV324 four op-amp and LMV321 single op-amp.

In the resistor-voltage conversion circuit shown in fig. 7, the resistors R1 and R2 generate Vref by dividing VCC, and the value thereof is set to about 0.5V. In order to ensure that the signal is in a reasonable dynamic range, the value of Rref can be 2-3 times of the initial value of Rsensor. The initial resistance of the flexible tensile sensor of the present invention is around 20k, so Rref is preferably 40k-60k. For the low pass filter portion, the resistor R and the capacitor C can be 16k and 0.1 mu F respectively, so that the cut-off frequency is close to 100Hz. The operational amplifier can be an LMV324 or an LMV321 general operational amplifier, and the low-voltage rail-to-rail characteristics of the operational amplifier are suitable for the embodiment of the invention.

In an embodiment of the present invention, the MCU of the flexible acquisition circuit 103 is in operation, and its typical processing flow includes:

1) Initializing peripheral equipment, a timer and filter parameters;

2) Waiting for timer interrupt triggering;

3) After the timer is triggered in an interruption way, 5 stretching sensor data are collected by using an ADC (analog to digital converter), and the data of triaxial acceleration of the MPU9250 are collected by using an I2C interface;

4) Filtering data output by the accelerometer by using a Kalman filter;

5) Transmitting the data to an upper computer system;

6) Returning to step 2)

In the embodiment of the invention, the timer adopts a hardware timer provided inside the MCU, which can periodically trigger the corresponding interrupt service routine according to the set time, and the sampling rate of the whole system is determined by the timer. The sampling rate is set to be 100Hz for comprehensive consideration of data volume and data reduction degree, so that the data processing capacity of the upper computer is not too high, and the gesture action can be well reduced.

The data may be transmitted to a computer system as a host computer by a wired or wireless method. The wired connection with the PC is realized by a mode of converting a serial port into a USB, and the wireless connection can be realized by a mode of Bluetooth, wiFi and other transmission protocols.

The MCU should also be equipped with program download circuitry to facilitate the adjustment and testing of the program. In the embodiment of the invention, a cp2102 USB-to-serial port chip is selected to support program downloading of the MCU.

Sign language recognition system

Fig. 8 shows waveforms acquired by the flexible tension sensor 201 and the inertial sensing unit 202 employing the configuration of the present invention. In an embodiment of the present invention, the sign language recognition system 204 is configured to determine whether a hand is in an active state in the following manner:

Activity status determination

Typically for a signal with a mean value of 0, a root mean square value may be used to determine whether the signal is active. However, in this system, the mean value of the signal is not 0, that is, the signal does not fluctuate around 0, so it is not appropriate to directly calculate the root mean square value to determine whether the signal is active.

Thus, in an embodiment of the invention, the originally monitored signal f (t) is derived such that f '(t) fluctuates around 0, so the root mean square value can be calculated using f' (t) to determine if the signal is active, i.e.:

where T represents the sampling time, here taken to be 0.5s. As discrete signals, i.e. 50 sampling points.

Since the sum of the accelerations of the 3 axes and the 5 stretch sensors are configured in this example, a total of 8 channels of signals are considered to be active as long as the root mean square value of the derivative of any one of the signals exceeds a certain threshold. Namely:

(rms′ _x ＞thr1)∨(rms′ _y ＞thr1)∨(rms′ _z ＞thr1)∨(rms′ _R1 ＞thr2)∨(rms′ _R2 ＞thr2)

∨(rms′ _R3 ＞thr2)∨(rms′ _R4 ＞thr2)∨(rms′ _R5 ＞thr2)＝1

in the above formula, thr1 is a threshold value configured corresponding to the accelerometer, thr2 is a threshold value configured corresponding to the attachable flexible tension sensor, and if any one of the above formulas is true, the active state is judged.

Then, based on the judgment that the hand is in the active state, sign language action recognition is further performed by a first recognition model trained in advance (for example, trained based on the convolutional neural network CNN) based on the sensing data of the 8-channel sensors as input, and a recognition result is output.

First recognition model trained in advance

Alternatively, the pre-trained first recognition model is configured to be obtained based on convolutional neural network training, the training process comprising:

Under the action of a plurality of different sign languages, the flexible stretching sensor at each finger joint position and the inertial sensing unit at the back of the hand position are used for respectively monitoring the bending of the finger and the movement data of the hand; then, carrying out numerical value and length standardization on the monitoring data of each sign language action, and establishing a sample database of the sign language action; the numerical standardization of the monitoring data corresponding to each sign language action in the sample database means that output numerical values of three accelerometers of the five flexible stretching sensors and the inertial sensing unit are respectively subjected to standardized conversion, and the length standardization means that the output numerical values are standardized according to a preset duration (for example, 2 s) range to obtain 200 sampling point data;

forming a training set and a testing set according to the ratio of 80:20 by a sample database;

the training set is used as training data, a model for sign language action recognition is trained through a convolutional neural network, test verification is carried out through a test set, and a model with recognition accuracy reaching a preset level (for example, verification accuracy is higher than 95%) is obtained and used as a first recognition model for recognizing the sign language action.

As an alternative embodiment, the numerical normalization of the detection data includes normalization of the accelerometer output values and normalization of the flexible stretch sensor output values.

The normalization of the accelerometer output values can be converted into the unit gravitational acceleration g by means of a suitable numerical scaling in a manner known in the art.

Normalization of the output value of the flexible stretch sensor requires calibration of the maximum and minimum values of the sensor resistance, thereby normalizing the variation of the resistance R of the detected value to between 0 and 1:

wherein R is _max Represents the maximum value of resistance of the flexible stretching sensor, R _min Representing the minimum resistance of the flexible stretch sensor. R is R _normalized Representing the flexible stretch sensor output value after normalization.

It should be appreciated that in embodiments of the present invention, where a data sample of a sign language action is desired to be sampled, the length of each sample may be different because the time may not be exactly uniform and stumbling when a person completes a particular sign language action. The same sign language action may result in a sample of 1.5s in length, a sample of 2.0s in length, or a sample of 2.5s in length. Although of different lengths, they all represent the same sign language action, which makes processing of the recognition algorithm difficult. In addition to the normalization of the data values, the embodiment of the invention aims to reduce the influence of different time samples of the same action and normalize the data length.

According to the observation and analysis of the inventor of the invention, most sign language actions can be completed within 1-3s, so that the sign language action samples are standardized to 2s, namely 200 sampling points in the example, so that the data length is not excessively long, and enough information can be carried for analysis.

As an alternative example, the effect of "scaling" the acquired data waveform may be achieved by means of extraction, interpolation, averaging, etc., so that the waveform after "scaling" should be substantially identical in morphology to the original waveform, and the information of the original waveform should be preserved as much as possible.

As an alternative example, we sort the eight data channels into an 8 x 200 matrix incorporating the characteristic information of the attachable flexible stretch sensor and the inertial sensing unit, including all the information of the sign language action, namely:

wherein a normalized matrix of the sensing data of the 8 channels is represented, respectively. Wherein,,

representing the 1 st to 200 th features of an X-axis accelerometer,>

representing the 1 st to 200 th features of a Y-axis accelerometer,

features 1-200 of the Z-axis accelerometer are shown.

In the same way, the processing method comprises the steps of,

representing the 1 st to 200 th features of the first flexible stretch sensor, +. >

Representing the 1 st to 200 th features of the second flexible stretch sensor, +.>

Representing the 1 st to 200 th features of the third flexible stretch sensor, +.>

Representing the 1 st to 200 th features of the fourth flexible stretch sensor, +.>

Features 1 to 200 of the fifth flexible tension sensor are shown.

In order to identify the meaning of the sign language actions, data of different sign language actions are acquired for multiple times, a data set and a test set are established, a convolutional neural network model is trained and tested, parameters of the model are adjusted according to test results, and therefore an identification model of the convolutional neural network capable of accurately identifying the sign language words is obtained.

In connection with the flow of the wearable sign language recognition method of the wearable sign language recognition system shown in fig. 9, the implementation includes the following procedures:

step A, acquiring monitoring data acquired by a flexible tension sensor and an inertial sensing unit according to a preset sampling period T (for example, taking 0.5 s);

step B, filtering the monitoring data acquired by the flexible tension sensor and the inertia sensing unit, performing first-order RC low-pass filtering on the output data of the flexible tension sensor, and performing Kalman filtering on the output data of the accelerometer of the inertia sensing unit;

step D, judging whether the duration of the identification object data reaches a preset threshold duration (generally may be set to 1 s):

Thus, the real-time recognition and output of the sign language actions are realized.

Alternatively, the recognition result may be characterized by sound and/or visualization (e.g., display via a display screen), or the like.

An example of data waveforms for 10 different sign language word samples is shown in fig. 10, it being seen that the different sign language word samples show good differentiation. In the embodiment of the invention, 50 commonly used sign language words are trained, 2 samples are collected for each word to serve as a training set, 8 samples are also collected for each word to serve as a test set, and the recognition accuracy rate can reach 95% after training.

In a further alternative, preferably, after training the convolutional neural network model for recognizing the independent sign language words, the recognition of the sentences composed of the continuous sign language can be further realized. For example, in a sentence in which a sign language is recognized, a sign language action sequence is accurately and automatically split, and effective words are extracted by splitting in a continuous sign language action sequence, thereby recognizing the sentence and outputting the result.

While the invention has been described with reference to preferred embodiments, it is not intended to be limiting. Those skilled in the art will appreciate that various modifications and adaptations can be made without departing from the spirit and scope of the present invention. Accordingly, the scope of the invention is defined by the appended claims.

Claims

1. A wearable sign language recognition system integrating an attachable flexible tension sensor and an inertial sensing unit, comprising:

the flexible stretching sensor is attached to the joint position of each finger of the human body and is used for monitoring the bending movement of the finger;

the flexible acquisition circuit acquires data of each flexible stretching sensor and the inertial sensing unit;

2. The wearable sign language recognition system of claim 1, wherein the flexible stretch sensor is a resistive flexible stretch sensor, and comprises a first substrate layer, a second substrate layer, a sensitive material layer and a stretchable wire layer, wherein the sensitive material layer and the stretchable wire layer are positioned between the first substrate layer and the second substrate layer, and the stretchable wire layer is electrically connected with the sensitive material layer.

3. The wearable sign language recognition system of claim 2, wherein the sensitive material layer is a conductive carbon black layer.

4. The wearable sign language recognition system of claim 2, wherein the stretchable wire layer is a conductive polymer of a predetermined shape made by mixing Ecoflex material with silver nanoplates in a certain ratio.

5. The wearable sign language recognition system of claim 4, wherein the mass ratio of silver nanoplates in the conductive polymer is 68% ± 1%.

6. The wearable sign language recognition system of integrated attachable flexible tension sensor and inertial sensing unit of any one of claims 1-5, wherein the flexible tension sensor is configured to be made in the following manner:

step 8, uniformly coating a conductive polymer formed by mixing the Ecoflex material and the silver nano-sheet on the surface of the mask plate to ensure that the pattern area corresponding to the stretchable wire layer is free from holes;

7. The wearable sign language recognition system of claim 6, wherein the conductive polymer of Ecoflex material mixed with silver nanoplates is configured to be prepared as follows:

8. The wearable sign language recognition system of claim 1, wherein the sign language recognition system is configured to determine whether a hand is active by:

9. The wearable sign language recognition system of claim 1, wherein the pre-trained first recognition model is configured to be obtained based on convolutional neural network training, the training process comprising:

10. The wearable sign language recognition method of the wearable sign language recognition system of the integrated attachable flexible tension sensor and inertial sensing unit according to any one of claims 1 to 9, comprising the following process: