CN117873327A - Portable intelligent sign language translator - Google Patents
Portable intelligent sign language translator Download PDFInfo
- Publication number
- CN117873327A CN117873327A CN202410063144.5A CN202410063144A CN117873327A CN 117873327 A CN117873327 A CN 117873327A CN 202410063144 A CN202410063144 A CN 202410063144A CN 117873327 A CN117873327 A CN 117873327A
- Authority
- CN
- China
- Prior art keywords
- sign language
- data
- model
- acquisition module
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 claims abstract description 32
- 238000003062 neural network model Methods 0.000 claims abstract description 16
- 230000001133 acceleration Effects 0.000 claims abstract description 15
- 230000009471 action Effects 0.000 claims abstract description 10
- 238000005452 bending Methods 0.000 claims abstract description 9
- 238000000034 method Methods 0.000 claims abstract description 9
- 238000012544 monitoring process Methods 0.000 claims abstract description 9
- 239000011159 matrix material Substances 0.000 claims abstract description 8
- 230000006870 function Effects 0.000 claims description 3
- 238000001514 detection method Methods 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 4
- 238000012360 testing method Methods 0.000 description 7
- 206010011878 Deafness Diseases 0.000 description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Abstract
The invention provides an intelligent sign language translator convenient to carry, and relates to the technical field of sign language identification. The portable intelligent sign language translator comprises the following specific steps: 1) The sign language acquisition module is used for collecting data of a bending sensor and a triaxial acceleration sensor of human sign language actions; 2) The sample acquisition module is used for continuously monitoring and judging the acquired data of the bending sensor and the triaxial acceleration sensor, storing the time sequence data judged in a period of time as a data matrix, further storing the data into an image, and completing acquisition of a sample; 3) The neural network model module is divided into two processes: in the training stage, based on a training sample set, a MMEDU MMClassification algorithm library and a pre-training model are utilized to train a MobileNet deep neural network model. In the recognition stage, the sign language is recognized by using the trained model, and the sign language is broadcasted by voice.
Description
Technical Field
The invention relates to the technical field of sign language identification, in particular to an intelligent sign language translator convenient to carry.
Background
The deaf-mute can see on the subway and roadside, and the communication between them is by sign language, but because few normal people learn the sign language, the communication with the normal people is difficult. The deaf-mute passengers often experience difficulty in buying tickets. To help the deaf-mute communicate with normal people, some methods for sign language recognition translation using images or videos have emerged. However, the method depends on the camera, and the use place is only suitable for indoor fixed positions, such as before a computer, is inconvenient in daily life, and is inconvenient to carry and use when going out. Accordingly, a person skilled in the art provides a portable intelligent sign language translator to solve the above-mentioned problems.
Disclosure of Invention
(one) solving the technical problems
Aiming at the defects of the prior art, the invention provides an intelligent sign language translator convenient to carry, which collects time sequence change data of both hands and sign language in a period of time by using ten bending sensors and two three-dimensional acceleration sensors, trains a model by using an artificial intelligent deep neural network, carries out sign language identification by using the model and can broadcast by using voice. Tests show that the device can primarily recognize daily sign language, is independent of a camera, and is convenient for the deaf-mute to use in daily life.
(II) technical scheme
In order to achieve the above purpose, the invention is realized by the following technical scheme: an intelligent sign language translator convenient to carry comprises 2 gloves, a control main board, 10 bending sensors and 2 triaxial acceleration sensors, wherein the control main board comprises a line-space board with a server and an identification function and a control board for collecting and sending data;
the method comprises the following specific steps:
1) The sign language acquisition module is used for collecting data of a bending sensor and a triaxial acceleration sensor of human sign language actions;
2) The sample acquisition module is used for acquiring data of the bending sensor and the triaxial acceleration sensor, monitoring and judging the data, storing the judged data in a continuous period of time as a data matrix, further storing the data as an image, and completing sample acquisition;
3) The neural network model module is divided into two stages: in the training model stage, based on a training sample set, a MMClassification algorithm library of MMedu and a pre-training model are utilized to train a MobileNet deep neural network model, and in the recognition stage, the trained model is utilized to recognize sign language and broadcast the sign language through voice.
Through the technical scheme, the intelligent sign language translator is composed of 2 gloves, a control main board, 10 curvature sensors and 2 triaxial acceleration sensors, a blank plate and a control board are used as main server ends and data acquisition and transmission ends, when the intelligent sign language translator trains a model, an MMClassification library of MMEdu is imported, then a MobileNet and a pre-training model are set, and optimal model parameters are obtained through training.
Preferably, the sign language acquisition module acquires data of 16 sensors once every 0.1 second, normalizes the data and sends the data to a server in a range of 0-255;
preferably, the sample acquisition module adopts a kind label of 'hello, thank you, bang me' to facilitate later acquisition of sample data and form a training data set.
Preferably, the sample acquisition module is accessed to the internet of things server through the MQTT to acquire sign language time sequence data;
according to the technical scheme, when the device is started, the MQTT is accessed to the Internet of things server to collect sign language time sequence data, each data change condition in the sign language process within 2 seconds is collected, the data is collected once in 0.1 seconds, 20 times of data are collected in total, and 20 times of data are formed and used as data sampling once.
Preferably, the sample acquisition module uses a monitoring server to obtain a data matrix [20, 16] with data and messages;
through the technical scheme, 100 kinds of samples with kinds of labels are respectively sampled in the three categories, wherein 70 samples are used for training the model, and 30 samples are used for testing the model.
Preferably, the neural network model module includes a training model stage and a sign language recognition stage. In the training model stage, the model is trained by using a training data set with class labels.
Preferably, the training neural network model module adopts an MMClassification library imported with MMEdu, and sets a MobileNet and a pre-training model to set training parameters;
through the technical scheme, training parameters are set:
Epoch＝20 lr＝0.005
validate＝True,
checkpoint＝checkpoint,
device＝'cuda',
the optimal = 'Adam', and the optimal model checkpoint is obtained when the accuracy requirement is met.
Through the technical scheme, the input layer is 20×16 nodes, the output layer is 3 nodes, and the three sign language labels are respectively corresponding to the input layer and the output layer. In the sign language recognition stage, a sign language action is directly made, the acquired time sequence sample is input into an artificial intelligent model, the category is recognized by the model, and the category is broadcasted through voice.
Preferably, the neural network model module further comprises voice broadcasting;
through the technical scheme, in the test stage, sign language actions are directly made, the categories are identified by the model, and voice broadcasting is performed.
Working principle: before the portable intelligent sign language translator is put into use, firstly, 16 sensors are fixed on gloves, then, as a preliminary attempt, three sign language contents are selected as identified categories: you are, thank you, help me, marked with 1, 2, 3 respectively, fix the tortuosity sensor and 3-dimensional acceleration sensor (16 data total) on the glove. The data is collected once in 0.1 second and is used as data collection once, normalization processing is carried out during the data collection, the data is kept in the range of (0-255), the data is sent to a server, the server judges whether each data is in the range of (0-255), the data is collected in the range and is not to be collected again in the range, the monitoring is carried out by monitoring whether the server has data or not during the starting of a sample acquisition module, the data in 2 seconds are collected together when the server has a message, 20 times of data are collected in total, 16 times of 20 data are formed, a data matrix is stored as an image, one sample acquisition is completed, and the monitoring is carried out again when the server has no message. In the training model stage, 100 samples are sampled for three categories, 70 are used for training models, 30 are used for testing models, then in the training model stage, an MMClassification library of MMEdu is imported, the category of the MobileNet model and the pre-training model are set, training parameters are set, the input layer is 20 x 16 nodes, the output layer is 3 nodes, and three sign language labels are respectively corresponding to the three sign language labels. Wherein the training parameters are set as follows:
Epoch＝20 lr＝0.005
validate＝True,
checkpoint＝checkpoint,
device＝'cuda',
and (3) obtaining an optimal model checkpoint by using an optimal= 'Adam' to meet the accuracy requirement.
In the sign language recognition stage, a sign language action is directly made, a time sequence sample acquired through collection is input into a trained artificial intelligent model, the category is recognized by the model, and the category is broadcasted through voice.
(III) beneficial effects
The invention provides an intelligent sign language translator convenient to carry. The beneficial effects are as follows:
compared with the existing sign language identification equipment, the intelligent sign language translator convenient to carry provided by the invention has the advantages that the equipment collects sign language time sequence data of both hands by using 10 curvature sensors and 2 three-dimensional acceleration sensors, trains and identifies a model of sign language by using an artificial intelligent deep neural network, and can identify a category and broadcast by voice when doing the action by using the model. Through test display, the device can primarily recognize daily sign language, is independent of a camera, is convenient for the deaf-mute to use in daily life, and has strong practicability.
Drawings
FIG. 1 is a portable schematic diagram of a portable intelligent sign language translator of the present invention;
FIG. 2 is a system workflow diagram of a portable intelligent sign language translator of the present invention;
FIG. 3 is a flow chart of acquiring sign language time sequence data of the portable intelligent sign language translator of the present invention;
FIG. 4 is a flow chart of a sample acquisition module of the portable intelligent sign language translator of the present invention;
fig. 5 is a flowchart of a training neural network model stage of a portable intelligent sign language translator of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1:
as shown in fig. 1-5, the embodiment of the invention provides a portable intelligent sign language translator, which comprises 2 gloves, a control main board, 10 curvature sensors and 2 triaxial acceleration sensors, wherein the control main board comprises a line-space board with a server and sign language recognition functions and a palm control board for collecting and sending data;
the method comprises the following specific steps:
1) The sign language acquisition module is used for collecting data of a bending sensor and a triaxial acceleration sensor of human sign language actions, acquiring data of 16 sensors every 0.1 second, normalizing the data, keeping the range between 0 and 225, and sending the data to the server;
2) The sample acquisition module is accessed into the Internet of things server by adopting the MQTT to acquire data, and the monitoring server is utilized to acquire data or not and the information is utilized to acquire the data, so that the time sequence data in a period of time after judgment is stored into a data matrix and an image, and one sample acquisition is completed, for example, each item of data change condition in the sign language process in 2 seconds is acquired, the data is acquired once in 0.1 seconds, 20 times of data are acquired in total, and a data matrix of [20, 16] is formed and is used as one data sampling;
3) And the neural network model module adopts an MMClassification library of MMEdu, then sets a MobileNet model, trains a checkpoint parameter and obtains a neural network model for identifying sign language.
The intelligent sign language translator is composed of 2 gloves, a control main board, 10 bending sensors and 2 triaxial acceleration sensors, and a traveling blank board and a control board are used as main server ends and data acquisition and transmission ends.
Three sign language categories are set, namely 'hello, thank you, help you me'.
In the process of generating the training sample set, corresponding sign language actions are respectively made, and the sign language acquisition module and the sample acquisition module acquire 100 samples with labels for each category, wherein 70 samples are used for training the model, and 30 samples are used for testing the model.
In the training model stage, an MMClassification library of MMEdu is adopted, then a MobileNet model is set, the input layer is 20×16 nodes, the output layer is 3 nodes, the three sign language labels are respectively corresponding to the input layer and the output layer, and then a checkpoint parameter is trained to obtain a neural network model capable of identifying sign language. Wherein training parameters are set:
Epoch＝20 lr＝0.005
validate＝True,
checkpoint＝checkpoint,
device＝'cuda',
the optimal = 'Adam', and the optimal model checkpoint is obtained when the accuracy requirement is met.
In the sign language recognition stage, when a certain sign language action is made, the sign language acquisition module and the sample acquisition module adopt the MQTT to access the Internet of things server to acquire the sign language time sequence data, and the category of the sign language time sequence data can be recognized by inputting the sign language time sequence data into the neural network model and is broadcasted by voice.
The test was performed using the remaining 30 samples, and the accuracy is shown in the following table:
| sign language content | Correct times (30 times in total) | Accuracy rate of |
| You like | 28 | 93％ |
| Thank you | 29 | 97％ |
| I need help | 29 | 97％ |
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (7)
1. The utility model provides an intelligence sign language translator convenient to carry, includes 2 gloves, control mainboard, 10 crookedness sensors and 2 triaxial acceleration sensor, its characterized in that: the control main board comprises a row blank board with a server and an identification function and a control board for collecting and sending data;
the method comprises the following specific steps:
1) The sign language acquisition module is used for acquiring data of the actions of the hands and the sign language in real time by using the curvature sensor and the triaxial acceleration sensor;
2) The sample acquisition module is used for continuously monitoring and judging the acquired data of the bending sensor and the triaxial acceleration sensor, storing the judged data in a period of time as a data matrix, further storing the data as an image, and completing acquisition of a sample;
3) The neural network model module is divided into two stages, namely, in a training model stage, based on a training sample set, a MMClassification algorithm library of MMedu and a pre-training model are utilized to train the Mobi LENet deep neural network model, and in a recognition stage, the trained model is utilized to recognize sign language and broadcast voice.
2. The portable intelligent sign language translator of claim 1, wherein: the sample acquisition module adopts a classification label of 'hello, thank you, help me'.
3. The portable intelligent sign language translator of claim 1, wherein: and the sign language acquisition module adopts an MQTT access internet of things server to receive and transmit sign language time sequence data.
4. The portable intelligent sign language translator of claim 1, wherein: the neural network model module comprises two stages of training models and detection and identification.
5. The portable intelligent sign language translator of claim 1, wherein: the neural network model module also comprises voice broadcasting.
6. The portable intelligent sign language translator of claim 1, wherein: the sign language acquisition module acquires data of 16 sensors once every 0.1 second, normalizes the data, and sends the data to the server in a range of 0-225.
7. The portable intelligent sign language translator of claim 1, wherein: the sample acquisition module obtains a data matrix [20, 16] using monitoring servers for data and messages for a period of time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410063144.5A CN117873327A (en) | 2024-01-16 | 2024-01-16 | Portable intelligent sign language translator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410063144.5A CN117873327A (en) | 2024-01-16 | 2024-01-16 | Portable intelligent sign language translator |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117873327A true CN117873327A (en) | 2024-04-12 |
Family
ID=90594556
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410063144.5A Pending CN117873327A (en) | 2024-01-16 | 2024-01-16 | Portable intelligent sign language translator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117873327A (en) |
-
2024
- 2024-01-16 CN CN202410063144.5A patent/CN117873327A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109359215A (en) | Video intelligent method for pushing and system | |
| CN111915148B (en) | Classroom teaching evaluation method and system based on information technology | |
| CN104094287A (en) | A method, an apparatus and a computer software for context recognition | |
| CN108090857A (en) | A kind of multi-modal student classroom behavior analysis system and method | |
| CN108234610A (en) | A kind of classroom interaction and reponse system | |
| CN110232346A (en) | A kind of video intelligent analysis system based on deep learning | |
| CN106407993A (en) | Intelligent voice robot system based on image recognition technology and method thereof | |
| CN112541529A (en) | Expression and posture fusion bimodal teaching evaluation method, device and storage medium | |
| CN111948487A (en) | High-voltage power equipment fault diagnosis method and system based on artificial intelligence | |
| CN111524578A (en) | Psychological assessment device, method and system based on electronic psychological sand table | |
| CN111895991B (en) | Indoor positioning navigation method combined with voice recognition | |
| CN112768070A (en) | Mental health evaluation method and system based on dialogue communication | |
| CN113744872A (en) | Intelligent interactive Alzheimer's disease auxiliary screening system | |
| CN110414581A (en) | Picture detection method and device, storage medium and electronic device | |
| CN108764640A (en) | A kind of personalized dynamic teaching diagnosis and the system of improvement based on big data | |
| CN111523445A (en) | Examination behavior detection method based on improved Openpos model and facial micro-expression | |
| CN113076999B (en) | Information data acquisition method based on artificial intelligence | |
| CN111898528B (en) | Data processing method, device, computer readable medium and electronic equipment | |
| CN117873327A (en) | Portable intelligent sign language translator | |
| CN116825288A (en) | Autism rehabilitation course recording method and device, electronic equipment and storage medium | |
| CN112016387A (en) | Contraband identification method and device suitable for millimeter wave security check instrument | |
| CN109272262B (en) | Method for analyzing natural language features | |
| CN115905187A (en) | Intelligent proposition system for cloud computing engineering technician authentication | |
| CN115659056A (en) | Accurate matching system of user service based on big data | |
| CN115527404A (en) | Artificial intelligence self-adaptation interactive teaching system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |