CN116301341A - Wearable Sign Language Recognition System and Method Integrating Attachable Flexible Stretch Sensor and Inertial Sensing Unit - Google Patents

Abstract

The invention provides a wearable sign language recognition system and method that integrates a flexible stretch sensor and an inertial sensing unit. The system includes: a flexible stretch sensor that is attached to the joints of the fingers of the right hand of the human body and used to monitor the fingers The bending movement of the hand; the inertial sensing unit, attached to the back of the hand, is used to monitor the movement of the hand in space; the flexible acquisition circuit communicates with each sensor data; the sign language recognition system receives the sensory data transmitted from the flexible acquisition circuit , and accordingly judge whether the hand is in an active state, and in the active state, based on the pre-trained CNN-based first recognition model, recognize and output sign language actions. The present invention adopts a flexible stretch sensor with good adhesion and is easy to wear, combined with the perception of the inertial sensing unit, it can realize the sensing of the movement of the wrist and arm in space, and integrates the bending characteristics and the movement state characteristics of the hand, and the Complex movements for fast and accurate response and recognition.

Description

Wearable Sign Language Integrated with Attachable Flexible Stretch Sensor and Inertial Sensing Unit Identification system and method

technical field

The invention relates to the technical field of smart wearable devices, in particular to a wearable sign language recognition system and method integrating a novel attachable flexible stretch sensor and an inertial sensing unit.

Background technique

Sign language is a form of language communicated through hands, facial expressions, and body, with recognition primarily through visual perception. However, without prior knowledge of sign language, it is difficult for non-sign language users to pick up and understand this medium of conversation. Thus, a communication barrier is created between sign language users and non-sign language users.

At present, existing technologies try to use auxiliary means such as wearable electronic devices to help establish communication and perception between sign language users and non-sign language users. Wearable electronic devices have the advantages of light weight, low cost, high flexibility and strong adaptability, etc. Making it possible to provide a technological solution to this communication barrier in the form of wearable sign language interpreting devices.

In the field of gesture monitoring, commonly used sensors include flexible mechanical sensors (pressure, strain, etc.), myoelectric signal sensors, image sensors, and the like. For example, the Chinese patent application publication number CN110491251A discloses a standardized sign language simulating smart glove, which only monitors the bending of the finger through a flexible sensor, but cannot monitor the movement of the hand in space and loses important information. Another example is the self-adaptive corrective sign language inter-interpretation system disclosed in the Chinese patent application with the publication number CN110189590A, which uses a bending sensor and an accelerometer, and carries all the sensors in the form of a glove, but the commercial bending sensor is not light and thin, and does not have Good softness, gloves are not comfortable enough to wear.

Therefore, in order to solve the communication barrier between sign language users and non-sign language users, some more effective auxiliary tools are needed to help deaf-mute people using sign language communicate with the outside world in a light, efficient and flexible way.

Prior art literature:

Patent Document 1: CN110491251A A Standardized Sign Language Simulated Smart Glove

Patent Document 2: CN110189590A An Adaptive Correction Sign Language Mutual Interpretation System and Method

Patent Document 3: CN109613976A An Intelligent Flexible Pressure Sensing Sign Language Recognition Device

Contents of the invention

In view of the defects existing in the prior art, the present invention aims to propose a wearable sign language recognition system and method integrating an attachable flexible stretch sensor and an inertial sensing unit. Good adhesion, easy to wear, will not cause discomfort or movement resistance to the wearer, does not affect the wearer's own movement, and has better dynamic performance and sensitivity, and can quickly and accurately respond to complex movements; At the same time, combined with the perception of inertial sensing units, the movement of the wrist and arm in space can be sensed, and the bending characteristics and the movement state characteristics of the hand can be fused. On the one hand, the judgment of the action can be realized, and on the other hand, convolution can be performed based on the fused features. The training of the neural network sign language recognition model can quickly and accurately obtain the meaning of sign language recognition actions in actual use, and output sign language recognition results in real time.

According to the first aspect of the purpose of the present invention, a wearable sign language recognition system integrating an attachable flexible stretch sensor and an inertial sensing unit is provided, including:

A flexible stretch sensor attached to the joint position of each finger of the human body to monitor the bending movement of the finger;

Inertial sensing unit, attached to the back of the hand, is used to monitor the movement of the hand in space;

Flexible acquisition circuit, data communication with each flexible stretch sensor and inertial sensing unit;

The sign language recognition system is configured to receive the sensory data transmitted from the flexible acquisition circuit, and judge whether the hand is in an active state accordingly, and recognize and output sign language actions based on the pre-trained first recognition model in the active state.

As an optional embodiment, the flexible stretch sensor is a resistive flexible stretch sensor, which consists of a first base layer, a second base layer, and a sensitive material layer between the first base layer and the second base layer. The stretchable wire layer is formed, and the stretchable wire layer is electrically connected with the sensitive material layer. The sensitive material layer is a conductive carbon black layer. The stretchable wire layer is a predetermined shape conductive polymer made by mixing Ecoflex material and silver nanosheets in a certain ratio. In some embodiments, the mass ratio of the silver nanosheets in the conductive polymer is 68%±1%. The conductivity of silver nanosheets with a small content is too low, and it is easy to form an open circuit under a large stretch; while the silver nanosheets with a large content are difficult to stir and mix during the preparation process, and the formed mud is too hard to be exposed to the mask. After the template is peeled off, it is attached to the substrate to form a pattern. Therefore, the mass ratio of silver nanosheets optimized in the present invention can ensure electrical conductivity while making the slime not too hard and easy to pattern.

Therefore, the attachable flexible tensile sensor proposed by the present invention is light, thin and soft, and has good attachability, and can be well attached to the skin surface of the finger joints without other auxiliary means (such as glue, tape, etc.), It will not cause discomfort to the wearer, and will not affect the wearer's own movements.

The attachable flexible stretch sensor of the embodiment of the present invention can be directly attached to the second joint of the right hand finger by virtue of the adhesive force of the material itself. There are 5 pieces in total, which are attached to the 5 fingers of the hand. When the finger is bent, the stretching of the skin at the joint drives the stretching action of the sensor. Since the sensor is light and soft, and its elastic modulus is close to that of human skin, the sensor can still maintain a good attachment when the finger is bent, and the wearer cannot feel it. Obvious resistance.

When the sensitive material layer in the sensor is stretched, the conductive network formed by the conductive carbon black will be partially disconnected, causing its resistance to increase significantly; when the stretch is released, the disconnected part of the conductive network will recover, So its resistance is also restored. Thus, changes in the degree of finger bending can be converted into changes in sensor resistance in real time.

As an optional embodiment, the flexible stretch sensor is configured to be made in the following manner:

Step 1. Take an acrylic sheet of a certain thickness and cut it into a substrate of a certain size according to the shape of the sensor;

Step 2. Take the PI film with a thickness of 50 microns, cut out the pattern of the sensitive material layer and the stretchable wire layer as a mask, and cut it into a certain size;

Step 3. Place the substrate on the glue homogenizer, pour the configured Ecoflex glue evenly on the substrate, and shake it evenly;

Step 4, placing the uniformly glued substrate on a hot plate, heating and curing for 1 hour, and preparing the first base layer;

Step 5. Align the mask plate of the sensitive material layer with the substrate, so that the PI film of the sensitive material layer is attached to the corresponding position on the surface of the cured Ecoflex glue;

Step 6. Roll-coat a certain amount of conductive carbon black on the surface of the Ecoflex adhesive corresponding to the position of the sensitive material layer exposed by the mask, repeat it several times, and apply it evenly; then peel off the mask of the sensitive material layer, based on Ecoflex adhesive and The force between the conductive carbon blacks, and the conductive carbon blacks stay on the surface of the Ecoflex glue in the desired pattern;

Step 7. Align the mask plate of the stretchable wire layer with the substrate, so that the PI film of the stretchable wire layer is attached to the corresponding position on the surface of the cured Ecoflex glue;

Step 8. Apply the conductive polymer formed by mixing Ecoflex material and silver nanosheets evenly on the surface of the mask, and there is no hole in the pattern area corresponding to the stretchable wire layer;

Step 9, peeling off the mask plate of the stretchable wire layer, and the conductive polymer remains in the required pattern;

Step 10, take the conductive textile thread, and use a small amount of conductive polymer to glue the conductive textile thread to the end of each stretchable wire layer;

Step 11, put the prepared first base layer, sensitive material layer and stretchable conductive layer substrate on the glue spreader, spread the second layer of Ecoflex glue evenly again, and package the device;

Step 12, placing the substrate on a hot plate, heating and curing for 1 hour, preparing a second base layer, and completing the sensor preparation.

As an optional embodiment, the conductive polymer formed by mixing the Ecoflex material with silver nanosheets is set to be prepared in the following manner:

Mix a certain amount of silver nanosheets, 50ml ethanol, and 2ml deionized water with magnetic force and stir evenly to obtain the first mixed solution;

Drop a certain amount of 0.01mol/L potassium iodide solution into the first mixed solution, and continue stirring to obtain the second mixed solution;

The second mixed solution is vacuum filtered to complete solid-liquid separation and dried;

Expose the dried silver nanosheet powder to strong light to decompose the silver iodide therein;

The treated silver nanosheet powder is mixed with the Ecoflex material according to a preset mass ratio to complete the preparation of the conductive polymer.

As an optional embodiment, the sign language recognition system is configured to determine whether the hand is active in the following manner:

receiving the first sensing data collected by the five flexible tensile sensors according to the preset sampling period T, and the second sensing data sampled by the three-axis accelerometer of the inertial sensing unit according to the preset sampling period T;

If the root-mean-square derivative of any one of the first sensing data or the second sensing data exceeds a preset threshold, it is judged to be in an active state; otherwise, it is deemed not to be in an active state.

As an optional embodiment, the pre-trained first recognition model is set to be obtained based on convolutional neural network training, and the training process includes:

Under a variety of different sign language actions, the flexible stretch sensor at the position of each finger joint and the inertial sensing unit at the back of the hand are used to monitor the finger bending and hand activity data respectively; The standardization of the sign language action sample database is established; the numerical standardization of the monitoring data corresponding to each sign language action in the sample database refers to the standardized conversion of the output values of the five flexible stretch sensors and the three accelerometers of the inertial sensing unit. Length standardization refers to standardization according to the predetermined time length range to obtain 200 sampling point data;

Fuse the eight data channel data corresponding to each sample in the sample database to form an 8*200 matrix including finger bending features and hand activity features;

The sample database is formed into a training set and a test set according to 80:20;

Using the above-mentioned training set as training data, train a model for sign language action recognition through a convolutional neural network, and conduct test verification through a test set to obtain a model with a recognition accuracy rate reaching a predetermined level as the first recognition model.

According to the second aspect of the purpose of the present invention, a wearable sign language recognition method is also proposed, including the following process:

Step A, obtaining the monitoring data collected by the flexible tensile sensor and the inertial sensing unit according to the preset sampling period T;

Step B, filtering the monitoring data collected by the flexible tensile sensor and the inertial sensing unit;

Step C, judging whether it is in an active state according to the monitoring data collected by the flexible stretch sensor and the inertial sensing unit:

- in response to being in an active state, continue to obtain the monitoring data of the next sampling period T until the next inactive state, and record all the monitoring data in the active state at this stage as the identification object data;

- in response to the inactive state, abandon the sampling period data, and return to step A, and continue to obtain the monitoring data of the next sampling period T;

Step D. Judging whether the duration of the identified object data reaches a preset threshold duration:

- In response to not reaching the preset threshold duration, abandon the segment of identification object data, and return to step A, and continue to obtain the monitoring data of the next sampling period T;

- in response to reaching the preset threshold length of time, then retain the segment of identified object data;

Step E, standardize the value range and length of the output values corresponding to the five flexible stretch sensors and the three accelerometers of the inertial sensing unit in the retained identification object data, and obtain eight data channel data after standardization; The data of the eight data channels after standardized processing are fused to form an 8*200 matrix including finger bending features and hand activity features;

Step F: Input the 8*200 matrix including finger bending features and hand movement features into the pre-trained first recognition model for sign language action recognition, and output the corresponding sign language action recognition results.

It should be understood that all combinations of the foregoing concepts, as well as additional concepts described in more detail below, may be considered part of the inventive subject matter of the present disclosure, provided such concepts are not mutually inconsistent. Additionally, all combinations of claimed subject matter are considered a part of the inventive subject matter of this disclosure.

The foregoing and other aspects, embodiments and features of the present teachings can be more fully understood from the following description when taken in conjunction with the accompanying drawings. Other additional aspects of the invention, such as the features and/or advantages of the exemplary embodiments, will be apparent from the description below, or learned by practice of specific embodiments in accordance with the teachings of the invention.

Description of drawings

The figures are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like reference numeral. For purposes of clarity, not every component may be labeled in every drawing. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings.

FIG. 1 is a schematic diagram of the principle of a wearable sign language recognition system according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of the wearing effect of the wearable sign language recognition system according to the embodiment of the present invention.

Fig. 3 is a schematic diagram of a flexible stretch sensor of a wearable sign language recognition system according to an embodiment of the present invention.

Figures 4a to 4c are the test data of the flexible tensile sensor of the embodiment of the present invention, in which 4a is the stretch-release hysteresis curve; 4b is repeated tests under different stretching degrees; 4c is 2000 cycles of stretch release durability test.

FIG. 5 is a graph of resistivity of conductive polymers with different weight ratios of silver nanosheets according to an embodiment of the present invention.

Fig. 6 is a system schematic diagram of the flexible acquisition circuit of the embodiment of the present invention.

Fig. 7 is a schematic circuit diagram of the resistance-voltage conversion circuit (ie, the R-V converter in the figure) in the flexible acquisition circuit of the embodiment of the present invention.

Fig. 8 is a schematic diagram of sensor data waveforms collected in an embodiment of the present invention.

Fig. 9 is a schematic flowchart of a wearable sign language recognition method according to an embodiment of the present invention.

Fig. 10 is a schematic diagram of data waveforms of 10 different sign language word samples according to the embodiment of the present invention.

The definition of each reference sign in the figure is as follows:

101-flexible tensile sensor, 102-inertial sensing unit, 103-flexible acquisition circuit, 104-sign language recognition system;

201-substrate, 202-sensitive material layer, 203-stretchable wire layer, 204-conductive textile thread.

Detailed ways

In order to better understand the technical content of the present invention, specific embodiments are given together with the attached drawings for description as follows.

Aspects of the invention are described in this disclosure with reference to the accompanying drawings, which show a number of illustrated embodiments. Embodiments of the present disclosure are not necessarily intended to include all aspects of the invention. It should be appreciated that the various concepts and embodiments described above, as well as those described in more detail below, can be implemented in any of numerous ways, since the concepts and embodiments disclosed herein are not limited to any implementation. In addition, some aspects of the present disclosure may be used alone or in any suitable combination with other aspects of the present disclosure.

{Wearable Sign Language Recognition System}

The wearable sign language recognition system combined with the embodiment shown in Figures 1 and 2 includes a flexible stretch sensor attached to the joint position of each finger of the human body for monitoring the bending movement of the finger, and attached to the back of the hand, Inertial sensing unit for monitoring the movement of the hand in space. As a result, the sensing component fused with multiple sensors has multi-dimensional sensing capabilities, which can not only sense the bending of finger joints, but also sense the movement of wrists and arms in space, improving sign language perception .

The wearable sign language recognition system of the embodiment shown in Figures 1 and 2 also includes a flexible acquisition circuit, which acquires data from each flexible stretch sensor and inertial sensing unit.

The sign language recognition system is configured to receive the sensory data transmitted from the flexible acquisition circuit, and judge whether the hand is in an active state accordingly, and recognize and output sign language actions based on the pre-trained first recognition model in the active state.

In some embodiments, the sign language recognition system is configured to be implemented based on a computer system, such as including but not limited to an embedded computer system, a desktop/laptop computer system, a cloud computer system, etc. And requirements are configured, for example, including but not limited to the following forms: integrated with flexible acquisition circuits; physically separated from flexible acquisition circuits and located on the surface of the human body (such as hands, wrists, arms, chest, head, etc.); The acquisition circuit is physically separated and separated from the human body; it is arranged on a cloud server, etc.

In some embodiments, a computer system generally has a data communication interface module, a memory, and a processor. The data interface module is used to realize data communication, such as performing data interaction with the flexible tensile sensor and the inertial sensing unit in a wireless (such as bluetooth, wifi, etc.) or wired communication mode to obtain monitoring data and store it in the memory. In particular, the memory can be a non-volatile memory, and data can be read and written between the processor and the processor through a data bus. The processor is used to read the monitoring data and execute preset computer programs to realize the recognition and output of sign language actions.

In an optional embodiment, the computer system is further configured with a module for characterizing the recognition result, such as an audio and/or visual characterizing module, especially a speaker that can be driven to produce sound, a display that can be driven to display the recognition result screen module.

flexible stretch sensor

In an embodiment of the present invention, FIG. 3 exemplarily shows a schematic diagram of a flexible stretch sensor, which can especially adopt a resistive flexible stretch sensor attached to the second joint position of the finger to monitor the bending of the finger. sports.

The flexible stretch sensor can adopt a "sandwich" structure, that is, two substrates 201 and a sensitive material layer 202 and a stretchable wire layer 203 between the two substrates. In the example of Figure 3, the flexible tensile sensor consists of a first base layer (i.e. the lower base Ecoflex), a second base layer (i.e. the upper base Ecoflex) and a sensitive material layer between the first base layer and the second base layer 202 and stretchable wire layer 203 constitute. The stretchable wire layer 203 is connected to the sensitive material layer 202 .

00-30材料，具有与人体皮肤相当的弹性模量。由此，可贴附的柔性拉伸传感器轻薄柔软，具有良好的可贴附性，不需要其他辅助手段(如胶水、胶带等)即可良好地贴附在人体皮肤表面。The substrate 201 serves as the carrier and package of the sensor, carrying and protecting the internal components of the sensor. In an optional embodiment, the base 201 can be selected from a material with low elastic modulus and good biocompatibility, such as Ecoflex material, which is selected in the embodiment of the present invention

00-30 material, with a modulus of elasticity comparable to that of human skin. Therefore, the attachable flexible stretch sensor is light, thin and soft, has good attachability, and can be well attached to the surface of human skin without other auxiliary means (such as glue, tape, etc.).

The sensitive material layer 202 is a conductive carbon black layer, made of conductive carbon black material, such as ECP600JD conductive carbon black. As the core for generating the sensing signal, the electrical resistance of the conductive carbon black layer changes as it is stretched.

The stretchable wire 203 connecting the sensitive material layer and the external circuit is a conductive polymer mixed with Ecoflex material and silver nanosheets in a certain proportion, and is made into a conductive polymer in a predetermined shape, such as a strip. The stretchable wire 203 used in the present invention has the advantages of flexibility, stretchability, and good conductivity, and it still has good conductivity under relatively large stretching, which can lead out the sensitive material layer and isolate the sensitive material layer from the outside. The role of physical contact.

As shown in FIG. 3 , the conductive textile wire 204 of the flexible tensile sensor 101 is used as a wire leading out from the end of the stretchable wire, which is outside the package of the Ecoflex material to facilitate the connection of the sensor to an external circuit.

As shown in Figures 2 and 3, the flexible tensile sensor 101 of the "sandwich" structure proposed by the present invention can be directly attached to the second joint of the finger by virtue of the adhesive force of the base material itself, and five fingers are configured correspondingly. Flexible stretch sensor 101. In the example of Fig. 2, take the 5 fingers attached to the right hand as an example.

When the finger is bent, the stretching of the skin at the joint drives the stretching of the flexible stretch sensor 101. Since the flexible stretch sensor 101 is light and soft, and its elastic modulus is equivalent to that of human skin, the flexible stretch sensor 101 can be stretched when the finger is bent. It still maintains a good attachment, and the wearer does not feel obvious resistance.

When the sensitive material layer (such as the conductive carbon black layer) in the flexible tensile sensor 101 is partially stretched, the conductive network formed by the conductive carbon black will be partially disconnected, so that its resistance will increase significantly; When , the disconnected part of the conductive network will recover, so its resistance will also recover. Thus, changes in the degree of bending of the finger can be converted into changes in the resistance of the sensor.

Fabrication of Flexible Stretch Sensors

The embodiment disclosed in the present invention proposes a preparation process for the above-mentioned flexible stretch sensor, which includes the following process:

Step 1. Take an acrylic sheet of a certain thickness and cut it into a substrate of a certain size according to the shape of the sensor;

Step 2. Take the PI film with a thickness of 50 microns, cut out the pattern of the sensitive material layer and the stretchable wire layer as a mask, and cut it into a certain size;

Step 3. Place the substrate on the glue homogenizer, pour the configured Ecoflex glue evenly on the substrate, and shake it evenly;

Step 4, placing the uniformly glued substrate on a hot plate, heating and curing for 1 hour, and preparing the first base layer;

Step 5. Align the mask plate of the sensitive material layer with the substrate, so that the PI film of the sensitive material layer is attached to the corresponding position on the surface of the cured Ecoflex adhesive;

Step 6. Roll-coat a certain amount of conductive carbon black on the surface of the Ecoflex adhesive corresponding to the position of the sensitive material layer exposed by the mask, repeat it several times, and apply it evenly; then peel off the mask of the sensitive material layer, based on Ecoflex adhesive and The force between the conductive carbon blacks, and the conductive carbon blacks stay on the surface of the Ecoflex glue in the desired pattern;

Step 7. Align the mask plate of the stretchable wire layer with the substrate, so that the PI film of the stretchable wire layer is attached to the corresponding position on the surface of the cured Ecoflex glue;

Step 8. Spread the conductive polymer formed by mixing Ecoflex material and silver nanosheets evenly on the surface of the mask, and there is no hole in the pattern area corresponding to the stretchable wire layer;

Step 9, peeling off the mask plate of the stretchable wire layer, and the conductive polymer remains in the required pattern;

Step 10, take the conductive textile thread, and use a small amount of conductive polymer to glue the conductive textile thread to the end of each stretchable wire layer;

Step 11, put the prepared first base layer, sensitive material layer and stretchable conductive layer substrate on the glue spreader, spread the second layer of Ecoflex glue evenly again, and package the device;

Step 12, placing the substrate on a hot plate, heating and curing for 1 hour, preparing a second base layer, and completing the sensor preparation.

Accordingly, the process of preparing a flexible stretch sensor on a 2cm*6cm substrate using the above-mentioned process in this example includes:

1) Take a 3mm thick acrylic sheet and cut it into a 2cm*6cm substrate with a laser cutting machine according to the shape of the sensor;

2) Take a PI film of about 50 microns, use a laser cutting machine to cut out the pattern of the sensitive material layer and the stretchable wire layer as a mask, and also cut it into a size of 2cm*6cm;

3) Take an appropriate amount of EcoflexA glue + B glue, and after mixing, remove the air bubbles in a vacuum, usually for 5 minutes, to obtain the Ecoflex glue to be used;

4) Place the substrate on the glue leveler, and evenly pour an appropriate amount of Ecoflex glue on the substrate; rotate at a speed of about 1000rpm for 20s, and shake the Ecoflex glue on the substrate evenly, so that the thickness of a layer of Ecoflex is several The level of ten microns makes the sensor thin enough;

5) Place the substrate on a hot plate, heat and cure at 70°C for 1 hour;

6) Align the mask plate of the sensitive material layer with the substrate, so that PI is attached to the surface of Ecoflex, and the pattern is in the correct position;

7) Dip a small amount of carbon black with a cotton swab, and roll it on the surface of Ecoflex exposed by the mask, repeat it several times, and apply it evenly. Peel off the PI mask, and due to the strong force between Ecoflex and carbon black, the carbon black powder stays on the surface of Ecoflex in the desired pattern;

8) Similar to step 5), align the mask plate of the stretchable wire layer with the substrate;

9) Evenly spread the conductive polymer made of Ecoflex and silver nanosheets on the surface of the mask to ensure that there are no holes in the pattern area;

10) Peel off the PI mask, it is also retained in the required pattern due to the viscosity of the conductive polymer itself;

11) Take two sections of conductive textile thread of about 5cm, and use a small amount of conductive polymer to adhere to the two ends of the stretchable wire respectively, and use a multimeter to test whether the sensor has been conducted on the textile thread at both ends; Mistakes have occurred during the production process and need to be re-prepared; if it is turned on, it indicates that the preparation is qualified;

12) Put the substrate on the glue spreader, spread the second layer of Ecoflex glue here, and package the device;

13) Place the substrate on a hot plate, heat and cure at 70°C for 1 hour;

14) After the curing is completed, the sensor is peeled off from the acrylic substrate.

So far, the preparation of the flexible stretch sensor proposed by the present invention is completed.

As an optional example, the conductive polymer in which Ecoflex material is mixed with silver nanosheets can be prepared in the following way:

Mix a certain amount of silver nanosheets, 50ml ethanol, and 2ml deionized water with magnetic force and stir evenly to obtain the first mixed solution;

Drop a certain amount of 0.01mol/L potassium iodide solution into the first mixed solution, and continue stirring to obtain the second mixed solution;

The second mixed solution is vacuum filtered to complete solid-liquid separation and dried;

Expose the dried silver nanosheet powder to strong light to decompose the silver iodide therein;

The processed silver nanosheet powder is mixed with the Ecoflex material according to a preset mass ratio to complete the preparation of the conductive polymer.

Accordingly, we use the above process to prepare the required conductive polymer, the specific steps are as follows:

1) Mix 2g of commercial silver nanosheets, 50ml of ethanol, and 2ml of deionized water with magnetic stirring for 1 hour;

2) Prepare an appropriate amount of potassium iodide solution of 0.01mol/L;

3) After the magnetic stirring in step 1) is completed, drop 5ml of potassium iodide solution into the above-mentioned mixed solution, and continue to stir for 30min until the reaction is fully completed;

4) After the stirring is completed, vacuum filter the mixed solution to complete solid-liquid separation and dry;

5) exposing the dried silver nanosheet powder to strong light to decompose the silver iodide therein;

6) Prepare an appropriate amount of Ecoflex glue, mix the treated silver nanosheet powder with Ecoflex, the mass ratio of the silver nanosheet powder is 68%±1%, and thus complete the preparation of the conductive polymer.

Silver nanosheets are excellent conductive fillers. However, the surface of commercial silver nanosheets is usually covered with a layer of lubricant to prevent cold welding during processing. This lubricant layer typically hinders electrical conduction between adjacent silver flakes in the form of long-chain fatty acid silver salts. Commercially available silver nanosheets are first treated with potassium iodide, which converts the surface lubricant and silver oxide into silver iodide, which is then decomposed under light to form silver nanoparticles. Due to the removal of the surface lubricant, the electrical conductivity of these silver nanosheets was greatly enhanced.

The above-mentioned mixing ratio of silver nanosheet powder and Ecoflex is further explained: due to the need to add a large proportion of silver nanosheet powder to make Ecoflex conductive, Ecoflex is dispersed by silver nanosheets and cannot be solidified and formed like pure Ecoflex. shape. A small amount of silver nanosheets has too low electrical conductivity, and it is easy to cause an open circuit under a large stretch; while a large amount of silver nanosheets is difficult to stir and mix, and the formed mud is too hard, and it is difficult to attach after the mask is peeled off. A pattern is formed on the substrate. Fig. 5 has shown the electrical resistivity of the prepared conducting polymer of different silver nanosheet weight proportions, and the present invention selects the silver nanosheet of appropriate 68% ± 1% mass ratio, can make mud-like thing Not too rigid, easy to pattern.

4a-4c exemplarily show the test results of the fabricated flexible stretch sensor 201. In the embodiment of the present invention, the performance of the prepared sensor is tested using a linear displacement stage, including sensitivity, linearity and repeatability durability. As shown in Figures 4a-4c: 4a is the stretch-release hysteresis curve, showing that the sensitivity GF value is about 4, the hysteresis is small, and the linearity is good; 4b is repeated tests under different stretching degrees, which shows that in Good repeatability is shown under different degrees of stretching tests; 4c is the durability test of 2000 cycles of stretching and release. It can be seen from the test results that the sensor is still working stably after 2000 cycles of damage testing.

Inertial Sensing Unit

In the embodiment of the present invention, the inertial sensing unit 102 is attached to the back of the hand, and a commercial inertial sensor is selected, which includes a three-axis accelerometer, which can sense the three axial accelerations during the movement of the hand in space. changes, and output monitoring data.

As an optional embodiment, the inertial sensing unit 102 adopts a commercial MPU9250 sensor, uses the functions of three accelerometers, and connects with the flexible acquisition circuit through an I2C interface to realize data communication.

It should be understood that the commercial MPU9250 sensor integrates a 3-axis gyroscope, a 3-axis accelerometer, and a 3-axis magnetometer, and has high static measurement accuracy. In the embodiment of the present invention, only the sensing data of the 3-axis accelerometer of the MPU9250 is used, and the connection and data communication with the flexible acquisition circuit are carried out through the I2C interface. For example, the MCU of the flexible acquisition circuit can access the internal registers of the MPU9250 through the I2C interface to get the data.

Flexible Acquisition Circuit

In an embodiment of the present invention, the flexible acquisition circuit 103 is used to acquire the signal data of the above-mentioned multiple sensors (that is, the signal data of 8 channels output by the three accelerometers of the five flexible stretch sensors and the inertial sensing unit 102 ), and after preprocessing, it is sent to the computer system for subsequent recognition processing.

As an optional embodiment, the flexible acquisition circuit 103 is used to realize the conversion, acquisition, filtering and transmission processing of the sensing signal. The flexible acquisition circuit 103 as shown in Figure 6 comprises: 1) the interface of sensor, is convenient to each sensor (inertial sensing unit and flexible tensile sensor) connect, realizes data communication; 2) resistance-voltage conversion circuit (being that in Figure 6 R-V converter), the change of the resistance of the flexible stretch sensor is converted into a voltage change, which is convenient for the acquisition of the digital-to-analog converter; combined with Figure 7, the resistance-voltage conversion circuit includes a filter function for the output signal of the flexible stretch sensor, using The first-order RC low-pass filter circuit filters the signal; 3) Microcontroller (MCU) module, including digital-to-analog conversion, data transmission and other functions, realizes the control of data acquisition and transmission, which also includes digital filters, Complete the calculations required for digital signal filtering; MCU chooses ESP32 from Espressif; 4) The power supply module converts the voltage provided by the battery or external power supply into the voltage required by each module, and provides the power required for the normal operation of each module .

It should be understood that the digital filter circuit of the flexible acquisition circuit 103 is used to filter the accelerometer signal, which is realized by using a Kalman filter algorithm. Since the raw acceleration data obtained from the inertial sensing unit is easily affected by vibrations and contains relatively large noises, it is not conducive to the subsequent analysis of the data, so in the embodiments of the present invention, the Kalman filter algorithm is used to analyze the acceleration of the accelerometer. The data is filtered. Kalman filtering is an algorithm that uses the linear system state equation to optimally estimate the system state through the input and output observation data of the system. Since the filtering algorithm is a time-domain algorithm, and the result of the current filtering can only depend on the last result and the current measurement value, it has the advantages of less calculation and no need for historical data cache, so in the embodiment of the present invention, the filtering effect good.

As shown in Figure 6, each channel of the flexible stretch sensor is equipped with a resistance-voltage conversion circuit (that is, the R-V converter in Figure 6), which converts the change in the resistance of the attachable flexible stretch sensor into a voltage change , which is convenient for digital-to-analog converter acquisition. Meanwhile, as shown in FIG. 7 , the resistance-voltage conversion circuit also includes RC filter processing for the output signal of the attachable flexible tensile sensor.

Combined with the resistance-to-voltage conversion circuit shown in FIG. 7 , it is realized by using a non-inverting proportional amplification circuit, which can realize the linear conversion of resistance to voltage. The attachable flexible stretch sensor is connected to the circuit as a feedback resistor, and the output voltage of the circuit is as follows:

Next, the signal passes through a first-order RC low-pass filter to filter out high-frequency noise. The cut-off frequency of the filter is:

Since the signal bandwidth of the finger movement to be detected is very low, in order to filter out most of the electromagnetic interference, the cutoff frequency is set to 100Hz in this example.

Since there are 5 stretch sensors, 5 sets of resistance-voltage conversion circuits as shown in Figure 7 are required, which can be realized by using the combination of LMV324 four-op amp and LMV321 single-op amp.

In the resistance-to-voltage conversion circuit shown in FIG. 7 , resistors R1 and R2 generate Vref by dividing the voltage of VCC, and the value of Vref is only about 0.5V. In order to ensure that the signal is in a reasonable dynamic range, the value of Rref can be 2-3 times the initial value of Rsensor. The initial resistance of the flexible tensile sensor of the present invention is about 20k, so Rref may be 40k-60k. For the low-pass filter part, the resistor R and the capacitor C can be 16k and 0.1μF respectively, so that the cut-off frequency is close to 100Hz. The op-amp can be LMV324 or LMV321 general-purpose op-amp, and its low-voltage, rail-to-rail characteristics are suitable for the examples of the present invention.

In an embodiment of the present invention, when the MCU of the flexible acquisition circuit 103 is working, its typical processing flow includes:

1) Initialize peripherals, timers, and filter parameters;

2) Wait for the timer interrupt trigger;

3) After the timer interrupt is triggered, use the ADC to collect the data of 5 tensile sensors, and the I2C interface to collect the data of the three-axis acceleration of the MPU9250;

4) Use a Kalman filter to filter the data output by the accelerometer;

5) Send the data to the upper computer system;

6) Go back to step 2)

In the embodiment of the present invention, the timer adopts the hardware timer provided inside the MCU, which can periodically trigger the corresponding interrupt service routine according to the set time, and the sampling rate of the whole system is determined by it. In consideration of the amount of data and the degree of data restoration, the sampling rate is set at 100Hz, so that the data processing capacity of the host computer is not too large, and the sign language movements can be restored better.

For the way of sending data to the computer system as the host computer, wired or wireless way can be selected. The wired connection is connected to the PC through serial port to USB, and the wireless connection can be realized through transmission protocols such as Bluetooth and WiFi.

The MCU should also be equipped with a program download circuit, which is convenient for adjusting and testing the program. In the example of the present invention, the cp2102USB-to-serial port chip is selected to support the program download of the MCU.

Sign Language Recognition System

FIG. 8 is a schematic diagram of waveforms collected by the flexible tensile sensor 201 and the inertial sensing unit 202 configured in the present invention. In an embodiment of the present invention, the sign language recognition system 204 is set to judge whether the hand is active in the following manner:

receiving the first sensing data collected by the five flexible tensile sensors according to the preset sampling period T, and the second sensing data sampled by the three-axis accelerometer of the inertial sensing unit according to the preset sampling period T;

If the root-mean-square derivative of any one of the first sensing data or the second sensing data exceeds a preset threshold, it is judged to be in an active state; otherwise, it is deemed not to be in an active state.

Activity status judgment

Usually for a signal with a mean value of 0, the RMS value can be used to determine whether the receiver is active. But in this system, the mean value of the signal is not 0, that is, the signal does not fluctuate around 0, so it is not suitable to directly calculate the root mean square value to judge whether the signal is active or not.

Therefore, in the embodiment of the present invention, the original monitored signal f(t) is derived so that f'(t) fluctuates around 0, so f'(t) can be used to calculate the root mean square value to judge whether the signal is active, that is:

In the above formula, T represents the sampling time, which is 0.5s here. Since it is a discrete signal, that is 50 sampling points.

Since the acceleration of 3 axes and 5 stretch sensors are configured in this example, there are 8 channels of signals in total. As long as the root mean square value of the derivative of any one of the signals exceeds a certain threshold, the signal is considered to be active. state. Right now:

(rms′ _x ＞thr1)∨(rms′ _y ＞thr1)∨(rms′ _z ＞thr1)∨(rms′ _R1 ＞thr2)∨(rms′ _R2 ＞thr2)

∨(rms′ _R3 > thr2) ∨ (rms′ _R4 > thr2) ∨ (rms′ _R5 > thr2) = 1

In the above formula, thr1 is the threshold corresponding to the configuration of the accelerometer, and thr2 is the threshold corresponding to the configuration of the attachable flexible stretch sensor. As long as any item in the above formula is true, it is judged to be in an active state.

Then, based on the judgment that the hand is in an active state, the sensory data of the sensors of the 8 channels is further used as input, and the sign language action recognition is performed through the pre-trained first recognition model (for example, based on convolutional neural network CNN training), and the output recognition result.

Pre-trained first recognition model

As an optional manner, the pre-trained first recognition model is set to be obtained based on convolutional neural network training, and its training process includes:

Under a variety of different sign language actions, the flexible stretch sensor at the position of each finger joint and the inertial sensing unit at the back of the hand are used to monitor the finger bending and hand activity data respectively; The standardization of the sign language action sample database is established; the numerical standardization of the monitoring data corresponding to each sign language action in the sample database refers to the standardized conversion of the output values of the five flexible stretch sensors and the three accelerometers of the inertial sensing unit. Length standardization refers to standardization according to a predetermined time length (for example, 2s) to obtain 200 sampling point data;

Fuse the eight data channel data corresponding to each sample in the sample database to form an 8*200 matrix including finger bending features and hand activity features;

The sample database is formed into a training set and a test set according to 80:20;

Using the above-mentioned training set as the training data, train a model for sign language action recognition through a convolutional neural network, and perform test verification through a test set to obtain a model with a recognition accuracy rate reaching a predetermined level (for example, the verification accuracy rate is higher than 95%), As the first recognition model, it is used to recognize sign language actions.

As an optional embodiment, the numerical standardization processing of the detection data includes normalization processing of the output value of the accelerometer and normalization processing of the output value of the flexible stretch sensor.

The standardized processing of the output value of the accelerometer can be converted into the unit of gravitational acceleration g by means of the prior art through appropriate numerical scaling.

To standardize the output value of the flexible stretch sensor, it is necessary to calibrate the maximum and minimum values of the sensor resistance, thereby normalizing the change of the resistance R of the detected value to between 0-1:

Wherein, R _max represents the maximum resistance value of the flexible stretch sensor, and R _min represents the minimum resistance value of the flexible stretch sensor. R _normalized represents the output value of the flexible stretch sensor after normalization.

It should be understood that, in the embodiment of the present invention, when data samples of sign language movements need to be sampled, the lengths obtained by each sampling may be different, because when a person completes a specific sign language movement, the time cannot be completely consistent. Fast and slow. The same sign language action may get a sample with a length of 1.5s, or a sample with a length of 2.0s, or a sample with a length of 2.5s. Although they are different in length, they all represent the same sign language action, which makes it difficult for the recognition algorithm to process. Therefore, in the embodiment of the invention, in addition to standardizing the value of the data, it also aims to reduce the influence of samples of different durations of the same action, and standardize the length of the data.

According to the observation and analysis of the inventors of the present invention, most of the sign language movements can be completed within 1-3s, so this example normalizes the sign language movement samples to 2s, that is, 200 sampling points, so that the data length will not be reduced. Too long, but can carry enough information for analysis.

As an optional example, the effect of "zooming" the collected data waveform can be achieved by means of extraction, interpolation, averaging, etc., so that the waveform after "zooming" should be basically the same as the original waveform in shape, and the waveform should be preserved as much as possible. information about the original waveform.

As an optional example, we organize the data of the eight data channels into an 8*200 matrix that combines the characteristic information of the attachable flexible stretch sensor and the inertial sensing unit, which contains all the information of the sign language action, namely :

表示X轴加速度计的第1～第200个特征，/>

表示Y轴加速度计的第1～第200个特征，

表示Z轴加速度计的第1～第200个特征。Wherein, respectively represent the normalized matrices of the sensing data of 8 channels. in,

Indicates the 1st to 200th features of the X-axis accelerometer, />

Indicates the 1st to 200th features of the Y-axis accelerometer,

Indicates the 1st to 200th features of the Z-axis accelerometer.

表示第一个柔性拉伸传感器的第1～第200个特征，/>

表示第二个柔性拉伸传感器的第1～第200个特征，/>

表示第三个柔性拉伸传感器的第1～第200个特征，/>

表示第四个柔性拉伸传感器的第1～第200个特征，/>

表示第五个柔性拉伸传感器的第1～第200个特征。In the same way,

Indicates the 1st to 200th features of the first flexible stretch sensor, />

Indicates the 1st to 200th features of the second flexible stretch sensor, />

Indicates the 1st to 200th features of the third flexible stretch sensor, />

Indicates the 1st to 200th features of the fourth flexible stretch sensor, />

Indicates the 1st to 200th features of the fifth flexible stretch sensor.

In order to recognize the meaning of sign language actions, it is necessary to collect data of different sign language actions multiple times, establish a data set and a test set, train and test the convolutional neural network model, and adjust the parameters of the model according to the test results, so as to obtain the ability to accurately recognize sign language A recognition model for convolutional neural networks of words.

Combining the flow of the wearable sign language recognition method of the wearable sign language recognition system shown in Figure 9, its implementation includes the following processes:

Step A. Obtain the monitoring data collected by the flexible tensile sensor and the inertial sensing unit according to the preset sampling period T (for example, 0.5s);

Step B, filtering the monitoring data collected by the flexible stretch sensor and the inertial sensing unit, such as performing first-order RC low-pass filtering on the output data of the flexible stretch sensor, and performing the first-order RC low-pass filtering on the accelerometer output data of the inertial sensing unit Kalman filter;

Step C, judging whether it is in an active state according to the monitoring data collected by the flexible stretch sensor and the inertial sensing unit:

- in response to being in an active state, continue to obtain the monitoring data of the next sampling period T until the next inactive state, and record all the monitoring data in the active state at this stage as the identification object data;

- in response to the inactive state, abandon the sampling period data, and return to step A, and continue to obtain the monitoring data of the next sampling period T;

Step D, judging whether the duration of the identified object data reaches a preset threshold duration (generally, it can be set to 1s):

- In response to not reaching the preset threshold duration, abandon the segment of identification object data, and return to step A, and continue to obtain the monitoring data of the next sampling period T;

- in response to reaching the preset threshold length of time, then retain the segment of identified object data;

Step E, standardize the value range and length of the output values corresponding to the five flexible stretch sensors and the three accelerometers of the inertial sensing unit in the retained identification object data, and obtain eight data channel data after standardization; The data of the eight data channels after standardized processing are fused to form an 8*200 matrix including finger bending features and hand activity features;

Step F: Input the 8*200 matrix including finger bending features and hand movement features into the pre-trained first recognition model for sign language action recognition, and output the corresponding sign language action recognition results.

Thus, the real-time recognition and output of sign language movements is realized.

As an optional manner, the recognition result is also characterized by means of sound and/or visualization (for example, displayed on a display screen).

Figure 10 shows examples of data waveforms of 10 different sign language word samples, and it can be seen that the different sign language word samples show good discrimination. In the example of the present invention, 50 commonly used sign language words are trained, and 2 samples are collected for each word as a training set, and 8 samples are also collected for each word as a test set. After training, the test can reach 95% recognition Accuracy.

In a further optional solution, preferably, after the convolutional neural network model for recognizing independent sign language words is trained, it can further realize the recognition of sentences composed of continuous sign language. For example, for the recognition of sign language sentences, accurate and automatic sign language action sequences are used to segment and extract effective words from continuous sign language action sequences, so as to recognize and output results.

Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Those skilled in the art of the present invention may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be defined by the claims.

Claims (10)

1. A wearable sign language recognition system that can be attached to a flexible stretch sensor and an inertial sensing unit, characterized in that it includes: A flexible stretch sensor, attached to the joint position of each finger of the human body, is used to monitor the bending movement of the finger; Inertial sensing unit, attached to the back of the hand, is used to monitor the movement of the hand in space; The flexible acquisition circuit collects the data of each flexible stretch sensor and inertial sensing unit; The sign language recognition system is configured to receive the sensory data transmitted from the flexible acquisition circuit, and judge whether the hand is in an active state accordingly, and recognize and output sign language actions based on the pre-trained first recognition model in the active state. 2. The wearable sign language recognition system that can be attached to an integrated flexible stretch sensor and an inertial sensing unit according to claim 1, wherein the flexible stretch sensor is a resistive flexible stretch sensor, and is formed by The first base layer, the second base layer, and the sensitive material layer between the first base layer and the second base layer are composed of a stretchable wire layer, and the stretchable wire layer is electrically connected with the sensitive material layer. 3. The wearable sign language recognition system integrating an attachable flexible stretch sensor and an inertial sensing unit according to claim 2, wherein the sensitive material layer is a conductive carbon black layer. 4. The wearable sign language recognition system that can be attached to an integrated flexible stretch sensor and an inertial sensing unit according to claim 2, is characterized in that, the stretchable wire layer is Ecoflex material and silver nanosheet with a certain Predetermined shapes of conductive polymers produced by mixing in proportions. 5. The wearable sign language recognition system that can be attached to an integrated flexible stretch sensor and an inertial sensing unit according to claim 4, wherein the mass ratio of silver nanosheets in the conductive polymer is 68% ±1%. 6. The wearable sign language recognition system integrating an attachable flexible stretch sensor and an inertial sensing unit according to any one of claims 1 to 5, wherein the flexible stretch sensor is set to Made in the following way: Step 1. Take an acrylic sheet of a certain thickness and cut it into a substrate of a certain size according to the shape of the sensor; Step 2. Take the PI film with a thickness of 50 microns, cut out the pattern of the sensitive material layer and the stretchable wire layer as a mask, and cut it into a certain size; Step 3. Place the substrate on the glue homogenizer, pour the configured Ecoflex glue evenly on the substrate, and shake it evenly; Step 4, placing the uniformly glued substrate on a hot plate, heating and curing for 1 hour, and preparing the first base layer; Step 5. Align the mask plate of the sensitive material layer with the substrate, so that the PI film of the sensitive material layer is attached to the corresponding position on the surface of the cured Ecoflex glue; Step 6. Roll-coat a certain amount of conductive carbon black on the surface of the Ecoflex adhesive corresponding to the position of the sensitive material layer exposed by the mask, repeat it several times, and apply it evenly; then peel off the mask of the sensitive material layer, based on Ecoflex adhesive and The force between the conductive carbon blacks, and the conductive carbon blacks stay on the surface of the Ecoflex glue in the desired pattern; Step 7. Align the mask plate of the stretchable wire layer with the substrate, so that the PI film of the stretchable wire layer is attached to the corresponding position on the surface of the cured Ecoflex glue; Step 8. Apply the conductive polymer formed by mixing Ecoflex material and silver nanosheets evenly on the surface of the mask to ensure that there are no holes in the pattern area corresponding to the stretchable wire layer; Step 9, peeling off the mask plate of the stretchable wire layer, and the conductive polymer remains in the required pattern; Step 10, take the conductive textile thread, and use a small amount of conductive polymer to glue the conductive textile thread to the end of each stretchable wire layer; Step 11, put the prepared first base layer, sensitive material layer and stretchable conductive layer substrate on the glue spreader, spread the second layer of Ecoflex glue evenly again, and package the device; Step 12, placing the substrate on a hot plate, heating and curing for 1 hour, preparing a second base layer, and completing the sensor preparation. 7. The wearable sign language recognition system that can be attached to an integrated flexible stretch sensor and an inertial sensing unit according to claim 6, wherein the conductive polymer formed by mixing the Ecoflex material with silver nanosheets, is set to prepare as follows: Mix a certain amount of silver nanosheets, 50ml ethanol, and 2ml deionized water with magnetic force and stir evenly to obtain the first mixed solution; Drop a certain amount of 0.01mol/L potassium iodide solution into the first mixed solution, and continue stirring to obtain the second mixed solution; The second mixed solution is vacuum filtered to complete solid-liquid separation and dried; Expose the dried silver nanosheet powder to strong light to decompose the silver iodide therein; The processed silver nanosheet powder is mixed with the Ecoflex material according to a preset mass ratio to complete the preparation of the conductive polymer. 8. The wearable sign language recognition system integrating a flexible stretch sensor and an inertial sensing unit according to claim 1, wherein the sign language recognition system is set to judge whether the hand is active in the following manner state: receiving the first sensing data collected by the five flexible tensile sensors according to the preset sampling period T, and the second sensing data sampled by the three-axis accelerometer of the inertial sensing unit according to the preset sampling period T; If the root-mean-square derivative of any one of the first sensing data or the second sensing data exceeds a preset threshold, it is judged to be in an active state; otherwise, it is deemed not to be in an active state. 9. The wearable sign language recognition system integrating an attachable flexible stretch sensor and an inertial sensing unit according to claim 1, wherein the pre-trained first recognition model is set to be based on a convolutional neural network The network training is obtained, and its training process includes: Under a variety of different sign language actions, the flexible stretch sensor at the position of each finger joint and the inertial sensing unit at the back of the hand are used to monitor the finger bending and hand activity data respectively; The standardization of the sign language action sample database is established; the numerical standardization of the monitoring data corresponding to each sign language action in the sample database refers to the standardized conversion of the output values of the five flexible stretch sensors and the three accelerometers of the inertial sensing unit. Length standardization refers to standardization according to the predetermined time length range to obtain 200 sampling point data; Fuse the eight data channel data corresponding to each sample in the sample database to form an 8*200 matrix including finger bending features and hand activity features; The sample database is formed into a training set and a test set according to 80:20; Using the above-mentioned training set as training data, train a model for sign language action recognition through a convolutional neural network, and conduct test verification through a test set to obtain a model with a recognition accuracy rate reaching a predetermined level as the first recognition model. 10. The wearable sign language recognition method of a wearable sign language recognition system integrating a flexible stretch sensor and an inertial sensing unit according to any one of claims 1 to 9, characterized in that it includes the following process : Step A, obtaining the monitoring data collected by the flexible tensile sensor and the inertial sensing unit according to the preset sampling period T; Step B, filtering the monitoring data collected by the flexible tensile sensor and the inertial sensing unit; Step C, judging whether it is in an active state according to the monitoring data collected by the flexible stretch sensor and the inertial sensing unit: - in response to being in an active state, continue to obtain the monitoring data of the next sampling period T until the next inactive state, and record all the monitoring data in the active state at this stage as the identification object data; - in response to the inactive state, abandon the sampling period data, and return to step A, and continue to obtain the monitoring data of the next sampling period T; Step D. Judging whether the duration of the identified object data reaches a preset threshold duration: - In response to not reaching the preset threshold duration, abandon the segment of identification object data, and return to step A, and continue to obtain the monitoring data of the next sampling period T; - in response to reaching the preset threshold length of time, then retain the segment of identified object data; Step E, standardize the value range and length of the output values corresponding to the five flexible stretch sensors and the three accelerometers of the inertial sensing unit in the retained identification object data, and obtain eight data channel data after standardization; The data of the eight data channels after standardized processing are fused to form an 8*200 matrix including finger bending features and hand activity features; Step F: Input the 8*200 matrix including finger bending features and hand movement features into the pre-trained first recognition model for sign language action recognition, and output the corresponding sign language action recognition results.

Images