CN111428802B - Sign language translation method based on support vector machine - Google Patents
Sign language translation method based on support vector machine Download PDFInfo
- Publication number
- CN111428802B CN111428802B CN202010242973.1A CN202010242973A CN111428802B CN 111428802 B CN111428802 B CN 111428802B CN 202010242973 A CN202010242973 A CN 202010242973A CN 111428802 B CN111428802 B CN 111428802B
- Authority
- CN
- China
- Prior art keywords
- sign language
- support vector
- vector machine
- voltage signals
- words
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
Abstract
The invention relates to a sign language translation method based on a support vector machine, which is characterized by comprising the following steps: comprises the following steps: 1. collecting gesture voltage signals through the wearable data glove by using an STM32 development board; 2. making sign language words and common sign language sentences corresponding to each group of gesture voltage signals into a sign language sentence library by using a signal screening program; 3. compiling a support vector machine program comprising a support vector machine classification module, a data transmission module and a storage module, wherein the support vector machine classification module is a multi-classification model formed by a directed acyclic scheme based on a support vector machine two-classification algorithm; 4. converting the gesture voltage signals received each time into sign language words through a support vector machine classification module; 5. converting the sign language words obtained in the step 4 within a period of time into sign language word groups, matching the sign language word groups with the sign language word library, associating and filling the sign language word groups into sentence output results. The invention realizes automatic real-time translation and recognition of sign language by combining a support vector machine and a sensing technology.
Description
Technical Field
The invention relates to a sign language translation method, in particular to a sign language translation method based on a support vector machine, which combines the support vector machine and a sensing technology to realize automatic translation and identification of sign language.
Background
At present, in the process of communication between deaf-mutes and normal people in society, because normal people cannot understand sign language, a gap exists between the deaf-mutes and the normal people, the communication circle of the deaf-mutes is limited, and great limitation is brought to the living and developing space of the deaf-mutes. The deaf-mute auxiliary equipment in the market at present has two types, one type is an electronic throat from the 50 s of the last century, the electronic throat is arranged at the throat section, senses the vibration of a vocal cord and expands the vibration to help sound production, but the price of materials for sound production is high, and the disabled who does not have social work guarantee generally cannot afford the electronic throat. The other is sign language translation equipment based on computer vision, which appears in recent years, the price of the equipment is not high, but the limb movement recognition technology is still in a starting stage, meanwhile, the requirements of image processing on the acquisition environment are strict, and the recognition rate is not high under the condition that the acquisition environment is complex.
Support Vector Machines (Support Vector Machines) are a supervised learning algorithm. Support vectors (Support vectors) are those points closest to the separating hyperplane and machines (machines) represent an algorithm rather than a Machine. SVMs were originally designed to solve the binary problem, and their simple, high-efficiency computational speed provides a solution for many binary decision models. In order to solve the problem of supervised multi-classification of converting a voltage value group into a sign language word, a DAG (directed acyclic graph) technology and a support vector machine are combined, so that the method has the advantages of small calculated amount and high accuracy. Therefore, constructing a nonlinear data classification model by combining DAG and SVM is a good method for improving the multi-environment recognition rate.
Disclosure of Invention
The invention aims to solve the defects of the prior art, designs a sign language translation method based on a Support Vector Machine, utilizes a sensing technology to collect a gesture voltage signal as the input of the Support Vector Machine, utilizes the simple and efficient classification of the Support Vector Machine and the classification diversity of a directed acyclic graph technology to convert the voltage signal into corresponding semantic output required normally, provides feasibility for manufacturing a sign language translation system based on gesture voltage and an SVM (Support Vector Machine) Support Vector Machine, facilitates the communication between disabled people and normal people, reduces the distance between the disabled people and the normal people, and enables deaf-mutes to be better integrated into the normal society.
The invention is realized by the following steps: a sign language translation method based on a support vector machine is characterized by comprising the following steps:
step 1: the method comprises the following steps that an STM32 development board collects gesture voltage signals through a flexible sensor and an acceleration sensor which are arranged on a wearable data glove, and transmits the gesture voltage signals to a storage device for storage through a Bluetooth module integrated with the gesture voltage signals after filtering and amplifying;
the wearable data glove in the step 1 is provided with flexible sensors which are strain gauges fixed on 10 finger positions, the acceleration sensors are two six-axis sensors fixed on the back of the left hand and the back of the right hand respectively, gesture voltage signals are represented by the bending degree of the strain gauges following the fingers and the mutual position change of the two six-axis sensors, and the acquisition of the gesture voltage signals is 22 signals including 10 finger bending signals and 12 gesture direction signals.
And 2, step: sign language words and common sign language sentences corresponding to each group of signals are coded into a sign language library by a signal screening program to prepare a sign language sentence library, and gesture voltage signals and corresponding sign language words collected for many times are divided into a training set and a test set by a ratio of 9;
the sign language sentence library in the step 2 comprises a sign language sentence library and a sign language sub-library, and the sign language sentence library is manufactured by recording the current 22 gesture voltage signals in Excel, performing normalization and then storing the signals in the Access database.
And step 3: writing a program for establishing a support vector machine classification module, wherein the program mainly comprises a support vector machine classification module, a data transmission module and a storage module, training the support vector machine classification module through the training set in the step 2, guiding the trained support vector machine classification module into a test set for testing, and storing the support vector machine classification module into the storage module after a test result is in accordance with a preset period, wherein the support vector machine classification module adopts a directed acyclic scheme to form a multi-classification model based on a support vector machine two-classification algorithm;
the support vector machine classification module in the step 3 has 22 input vectors, which correspond to 22 voltage signals and 153 sub-classification models, adopts a directed acyclic classification scheme, and obtains 18 classification results with the serial numbers of 0 to 17 through 17 secondary classifications, which correspond to 18 commonly used phrases in sequence, and the commonly used phrases are randomly combined into 53 commonly used phrases.
And 4, step 4: converting the gesture voltage signals collected each time into sign language words through a support vector machine classification module, and the method comprises the following steps:
step 4.1: receiving gesture voltage signals of the wearable data gloves, and screening complete signals by using a signal screening program;
step 4.2: and converting the gesture voltage signals into words through a trained support vector machine classification module.
And 5: converting sign language words obtained by converting the gesture voltage signals in the step 4 within a period of time into sign language word groups, matching the sign language word groups with the sign language word library, associating and filling the matched sign language word groups with the sign language word library to form a sentence output result, and the method comprises the following steps of:
step 5.1: dividing the sentences in the sign language sentence library or the collected vocabulary groups into words and counting, defining the vocabulary with high frequency and symbolic meaning in the sentences as element 1, and defining the rest vocabularies as element 0;
and step 5.2: all common sign language sentences in the sign language sentence library are specified into a word frequency vector format according to the step 5.1, and corresponding word frequency vectors are generated;
step 5.3: converting the words obtained after the gesture voltage signals collected in the step 4 are converted within a period of time into sign language word groups, wherein the obtained sign language word groups are in a word frequency vector format according to the step 5.1 and are converted into corresponding word frequency vectors;
step 5.4: calculating cosine similarity between the word frequency vector converted in the step 5.3 and the word frequency vector in the sign language sentence library in the step 5.2, and selecting the sign language word in the sign language sentence library with the largest cosine similarity as an output word;
and step 5.5: and matching the output words obtained in the step 5.4 with corresponding written language sentences according to the indexes of all the commonly used sign language sentences in the sign language sentence library, and taking the matched written language sentences as final output results.
The invention has the beneficial effects that: according to statistics, more than 2000 tens of thousands of people with physiological and language disabilities can be listened to in the whole country, the people have disabilities in communication with normal people, and the difficulty in communication among deaf-mute people is an important reason why the deaf-mute people cannot work normally. The method of the invention can convert the gesture into a normal statement audio or video form for output by connecting the STM32 development board with external sound or video playing equipment, has high translation accuracy and quick response, can provide great convenience for the disabled and normal people to communicate, and meets the urgent requirements of the deaf-mute and disabled people to communicate with the normal people. The hardware equipment adopted by the method is low in cost, can quickly and effectively solve the problem of communication barrier encountered by the deaf-mute, opens a larger space for the employment of the deaf-mute, helps the deaf-mute barrier to better integrate into the society, and improves the living level of the deaf-mute.
Drawings
FIG. 1 is a block diagram showing the working steps of the method of the present invention.
FIG. 2 is a simplified schematic diagram of a single wearable data glove for acquiring gesture voltage signals in accordance with the method of the present invention.
FIG. 3 is a schematic diagram of the signal screening process of the method of the present invention.
FIG. 4 is a schematic hyperplane representation of the method of the present invention that enables the partitioning of two training copies.
FIG. 5 is a schematic diagram of the support vectors and intervals of the method of the present invention.
FIG. 6 is a schematic diagram of a directed acyclic algorithm framework for the method of the present invention.
Fig. 7 is a schematic diagram of the working process flow of matching, associating, filling, and sentence outputting of the words generated after conversion by the support vector machine classification module according to the method of the present invention.
Detailed Description
According to the attached figure 1, the invention relates to a sign language translation method based on a support vector machine, which comprises the following steps:
step 1: gather gesture voltage signal by STM32 development board through flexible sensor and the acceleration sensor who locates on the wearable data gloves to transmit to the accumulator storage through its integrated bluetooth module after with gesture voltage signal filtering, enlargiing.
And 2, step: sign language words and common sign language sentences corresponding to each group of signals are coded into a sign language library by a signal screening program to prepare a sign language sentence library, and gesture voltage signals and corresponding sign language words collected for many times are divided into a training set and a test set by a ratio of 9;
step 2.1: writing hand database recording software based on C # language, wherein the software can record a hand gesture voltage signal received each time and semantics represented by the signal in Excel;
step 2.2: checking whether the received signals are complete or not by using a signal screening program, recording the signals in an Excel table if the signals are complete, and otherwise, rejecting the signals;
step 2.3: the gesture voltage signals and corresponding sign language words collected for multiple times are numbered from number 0 and divided into training sets and data sets in a proportion of 9.
Step 2.4: and normalizing the sign language words and the common sign language sentences corresponding to the gesture voltage signals recorded in the Excel table, and then importing the sign language words and the common sign language sentences into an Access database to manufacture a sign language sentence library.
And step 3: writing a program for establishing a support vector machine classification module, wherein the program mainly comprises a support vector machine classification module, a data transmission module and a storage module, training the support vector machine classification module through the training set in the step 2, guiding the trained support vector machine classification module into a test set for testing, storing the support vector machine classification module in the storage module after a test result is in accordance with a preset result, and adopting a directed acyclic scheme to form a multi-classification model based on a support vector machine two-classification algorithm by the support vector machine classification module;
step 3.1: generating a sub model of a learning algorithm of a support vector machine;
step 3.2: a DAG (directed acyclic graph) algorithm framework is established.
And 4, step 4: converting the gesture voltage signals acquired each time into sign language words through a support vector machine classification module;
step 4.1: receiving gesture voltage signals of the wearable data gloves, and screening complete signals by using a signal screening program;
step 4.2: and converting the gesture voltage signals into sign language words by using a trained support vector machine classification module.
And 5: converting sign language words obtained by the gesture voltage signal conversion in the step 4 into sign language word groups within a period of time, matching the sign language word groups with a sign language word library, and filling the matched sign language word groups into sentence output results;
step 5.1: dividing the sentences in the sign language sentence library or the collected word groups into words and counting the words, wherein the words with high frequency and symbolic significance in the sentences are defined as element 1, and the other words are element 0;
step 5.2: all the frequently used sign language sentences in the sign language sentence library are in a word frequency vector format according to the specification of the step 5.1, and corresponding word frequency vectors are generated;
step 5.3: converting words obtained after the gesture voltage signals collected in the step 4 are converted within a period of time into sign language word groups, and converting the obtained sign language word groups into corresponding word frequency vectors in a word frequency vector format according to the specification of the step 5.1;
step 5.4: calculating the cosine similarity between the word frequency vector converted in the step 5.3 and the word frequency vector in the sign language sentence library in the step 5.2, and selecting the sign language words in the sign language sentence library with the largest cosine similarity as output words;
step 5.5: and matching the output words obtained in the step 5.4 with corresponding written language sentences according to the indexes of all the commonly used sign language sentences in the sign language sentence library, and taking the matched written language sentences as final output results.
The invention is described in further detail below with reference to the figures and specific examples.
The specific working steps of the sign language translation method based on the support vector machine are as follows:
step 1: flexible sensor, acceleration sensor and the constant resistance series connection on STM32 development board and the wearable data gloves, flexible sensor and acceleration sensor can change output voltage according to the change of finger bending and hand position, transmit to the storage after gesture voltage signal filtering, the amplification that will gather by the bluetooth module of STM32 single board computer.
According to the attached figure 2, the flexible sensors arranged on the wearable data gloves in the step 1 are strain gauges fixed at the positions of 10 fingers, the acceleration sensors are two six-axis sensors respectively fixed at the positions of the backs of the left and right hands, each pair of wearable data gloves is provided with two gloves for the left and right hands, the flexible sensors composed of ten strain gauges output ten voltage values, each six-axis sensor outputs twelve voltage values in the front and back directions of the x, y and z axes, and each output comprises a start signalNumiAnd an end signalNumoA total of 24 signals were in order:Numi，X 1 ，X 2 ，……X 22 ，Numo. WhereinX 1 ，X 2 ，……X 22 Representing the voltage value of the gesture.
Step 2: and (4) compiling sign language words and common sign language sentences corresponding to each group of the collected gesture voltage signals into a sign language library to manufacture the sign language sentence library. The sign language sentence library is divided into a training set and a test set by 9.
Step 2.1: the sign language library recording software is written based on C # language, and can record gesture voltage signals received each time and semantics represented by the signals in an Excel table.
Step 2.2: fig. 3 is a schematic diagram of the signal screening program, which is used to check whether the signals received each time are complete, and if complete, the signals are recorded in an Excel table, otherwise, the signals are rejected. The signal screening program receives the gesture voltage signal and meets the starting signalNumiWhen a stop signal is met, counting is startedNumoThe counting stops. When the counting K value is 22, the transmission data is completed and recorded in the Excel table, and if the counting K value is not completed, the counting is restarted.
Step 2.3: the gesture voltage signals and corresponding sign language words collected for multiple times are numbered from number 0 and divided into training sets and data sets in a proportion of 9.
Step 2.4: and (4) normalizing the gesture voltage signals, the corresponding sign language words and the common sign language sentences in the Excel table, and importing the normalized sign language words and the common sign language sentences into an Access database to prepare a database.
And step 3: the method comprises the steps of writing a program for establishing a support vector machine classification module, wherein the program mainly comprises a support vector machine classification module, a data transmission module and a storage module, and the support vector machine classification module is based on a support vector machine two-classification algorithm and adopts a directed acyclic scheme to form a multi-classification model. And (3) respectively importing the data sets with the numbers of 0 to 17 into corresponding support vector machine sub-classifiers through the training set in the step 2 to train the support vector machine classification modules, importing the trained support vector machine classification modules into a test set for testing, and storing the weight data of the support vector machine classification modules into a storage module after the test result meets the expectation.
For each gesture, the wearable data glove with the flexible sensor will transmit 22 sensor values to build a model. It is a method to build an index library by associating different sets of voltage values with gestures. However, chinese grammar has many gestures, and it takes much time to establish a relevant grammar library. And the voltage value groups formed by different people making gestures are not completely the same, and as the number of users increases, the voltage group corresponding to each gesture becomes larger and larger, which results in the increase of the index duration. Moreover, due to the limitation of the sensitivity of the device, different gesture voltage value sets are not very different, which brings many limitations to accurate recognition. In order to achieve quick and accurate identification, the invention adopts a support vector machine in machine learning to build a classification model of the voltage group.
The support vector machine classification module is composed of a support vector machine classification part and a model prediction part. The classification part of the support vector machine is also divided into two classes of a sub-model of the learning algorithm of the support vector machine and a multi-classification program. The support vector machine classification part is written by Python language, and the support vector machine is constructed in the Python language, so that the number of the support vector submodels is conveniently modified, and a large amount of data is quickly calculated. The programming environment is PyCharm, and the used program libraries are Numpy, pandas and SciPy. Numpy is used for compiling the sub-model of the learning algorithm of the support vector machine, pandas is used for importing data, and SciPy is used for outputting and storing.
The classification part of the support vector machine is divided into a sub-model of the learning algorithm of the support vector machine and a multi-classification program. The sub-model of the learning algorithm of the support vector machine adopts a two-classification idea, and mainly aims to generate a two-classification decision program. The multi-classification program adopts a DAG (directed acyclic graph) method and combines a two-classification decision program to divide the gesture voltage into a plurality of classes, the invention divides the gesture voltage into 18 common phrases to be output, each output serial number is 0 to 17, the 18 common phrases are sequentially corresponded, and the common phrases can be randomly combined to form 53 common phrases. The correspondence of the commonly used phrases and numbers are given in the following table.
Table 1 table for common word group number
Step 3.1: generating a sub model of a learning algorithm of a support vector machine;
taking the example of distinguishing the number 1, the word being "good" and the number 17, and the word being "favorite", the concept of generating the sub-model of the learning algorithm of the support vector machine is as follows:
given a training sample set
Wherein
The basic idea of the learning algorithm of the support vector machine is to find a partition hyperplane in a sample space based on a training set D and separate different class samples.
There are particularly many separate hyperplanes that can be used to divide the training samples, but one should go to find the dividing hyperplane that is "in the middle" of the two classes of training samples, as shown by the solid line in fig. 4, because this dividing hyperplane is best "tolerant" to local perturbations of the training samples, e.g., due to the limitations of the training set or noise, samples outside the training set may be closer to the separation boundary of the two classes than the training samples in fig. 4, which will make many dividing hyperplanes erroneous. As can be seen from fig. 4, the line division at the position of the solid line has the least influence on the hyperplane, and in short, the result of this division is robust.
As shown in fig. 4, a linear function can separate the samples, which is usually called that the data samples are linearly separable, i.e. the dashed lines in fig. 4. It is clear that not only one linear function line can separate the samples, but there are countless lines, and the linear branched support vector machine described in the present invention corresponds to the line that can correctly divide the data and has the largest interval. The linear equation can be described by the following equation:
whereinWThe normal vector of the hyperplane determines the direction of the hyperplane,bfor the displacement amount, the distance from the hyperplane to the origin is determined. The hyperplane is assumed to correctly classify the training samples. I.e. for training samples
And satisfies the following formula:
To obtain the hyperplane with the largest spacing, the spacing function needs to be set, as shown in fig. 5, and the sample point closest to the hyperplane satisfies
Or
These two points are called "support vectors". The dashed lines are called boundaries and the separation is the distance between the two dashed lines.
The interval is equal to the difference between two heterogeneous support vectorsWThe projection in the direction is that of the direction,Wthe direction is the normal direction of the solid line shown in fig. 5. Representing circular samples is calledx + The representative rectangular sample is calledx - It is possible to obtain:
and further can deduce:
therefore, the method comprises the following steps:
to maximize the spacing, namely:
Apparently maximum
I.e. minimize
The above formula is written as follows:
And (3) solving the dual problem by adopting a Lagrange multiplier method:
for is tow，bThe partial derivatives are obtained:
making the partial derivative 0, we can get:
a substitution into the lagrange function can be derived:
so that
，
，
Finally, the following formula is substituted:
solve outw，bAnd finally obtaining models of the support vector machine classification numbers 1 and 17:
working out in the same wayf 0-17 (x)，f 2-17 (x)，f 3-17 (x)…f 16-17 (x)，f 0-16 (x)，f 1-16 (x)…f 15-16 (x)…f 0-1 (x) Equal to 153 sub-classifier models. The training sample of the invention is imported from a read _ Excel function in a Pandas library into a sign language library stored as an Excel file. The phrase library is an Excel file, columns from 1 st to 22 th are voltage values, and column 23 is a corresponding gesture phrase. The invention adopts a support vector machine modelThe type prediction can reach 86% accuracy.
And after model training, the model needs to be stored, and 153 sub-classification model coefficient matrixes after training are stored into a Mat format. The savemat function in the Scipy library in Python can store the trained parameter matrix as a data file in a Mat format, and the loadmat function can import data in the Mat file into a program for prediction calling.
Step 3.2: a DAG (directed acyclic graph) technical algorithm framework is built as shown in fig. 6.
After the sub-classifier models are obtained, all sub-classifiers are constructed into a two-way directed acyclic graph comprising 153 nodes and 18 leaves. Each node is a classifier and is connected to two nodes (or leaves) in the next layer. When an unknown sample is classified, the classification is continued by using the left node or the right node of the next layer from the top and the classification result until a certain leaf of the bottom layer is reached, and the class represented by the leaf is the class of the unknown sample.
And 4, step 4: and converting the gesture voltage signals acquired each time into sign language words through a support vector machine classification module.
Step 4.1: and receiving gesture voltage signals of the wearable data gloves, and screening complete signals by using a signal screening program.
Step 4.2: and converting the gesture voltage signals into sign language words by using a trained support vector machine classification module.
The specific operation is as follows: and acquiring a current 2-second gesture voltage signal, and converting the acquired gesture voltage signal into sign language words through a support vector machine classification module.
And 5: converting the sign language words obtained by converting the gesture voltage signals in the step 4 into sign language word groups in a period of time, and matching and filling the sign language word groups into sentences.
The specific working process flow from the sign language words generated after conversion by the support vector machine classification module to the matching association filling sentence output is shown in fig. 7.
Step 5.1: dividing the words of the sentences in the sign language sentence library or the collected word groups into words and counting the words, defining the words with high frequency and symbolic meaning in the sentences as element 1, and defining the other words as element 0, if: according to the sentences in the sign language sentence library, i.e., ' I receives a sentence ', ' she is beautiful and ' you are poor ' ], word segmentation statistics is carried out to obtain the words appearing in the sentence library and the frequency of the appearance thereof [ I: 1, you: 1, she: 1, clothes: 1, poor: 1, beautiful: 1, receiving: 1, comprising: 1, very: 1], specifying the word-frequency vector format as [ I, you, her, clothing, poor, beautiful, received ];
and step 5.2: all the commonly used sign language sentences in the sign language sentence library are defined as a word frequency vector format according to the step 5.1, and corresponding word frequency vectors are generated, for example: the sentence "I received" the corresponding word frequency vector is [1,0, 1];
step 5.3: converting sign language words obtained by converting the gesture voltage signals acquired in the step 4 within a period of time into sign language word groups, converting the obtained sign language word groups into corresponding word frequency vectors according to the word frequency vector format specified in the step 5.1, and performing the following steps: the word frequency vector corresponding to the word group "she is beautiful" is [0, 1,0, 1,0];
step 5.4: and (4) calculating the cosine similarity between the word frequency vector converted in the step 5.3 and the word frequency vector in the sign language sentence library in the step 5.2, and selecting the sign language word in the sign language sentence library with the largest cosine similarity as an output word.
And step 5.5: and (5) matching the output words obtained in the step (5.4) with corresponding written language sentences according to the indexes of all the common sign language sentences in the sign language sentence library, and taking the matched written language sentences as final output results.
The output result can be output by a hardware device or a device of external sound and/or video playing equipment of a raspberry pi 3B single board computer, and the translation result can be output in a sentence audio or video form.
Claims (6)
1. A sign language translation method based on a support vector machine is characterized by comprising the following steps:
step (1): the method comprises the following steps that an STM32 development board collects gesture voltage signals through a flexible sensor and an acceleration sensor which are arranged on a wearable data glove, and the gesture voltage signals are transmitted to a storage device for storage through a Bluetooth module integrated with the gesture voltage signals after being filtered and amplified;
step (2): sign language words and common sign language sentences corresponding to each group of signals are coded into a sign language library by a signal screening program to prepare a sign language sentence library, and gesture voltage signals and corresponding sign language words collected for many times are divided into a training set and a test set by a ratio of 9;
and (3): writing a program for establishing a support vector machine classification module, wherein the program mainly comprises a support vector machine classification module, a data transmission module and a storage module, training the support vector machine classification module through the training set in the step (2), introducing the trained support vector machine classification module into a test set for testing, storing the support vector machine classification module in the storage module after a test result is in accordance with a preset period, and adopting a directed acyclic scheme to form a multi-classification model based on a support vector machine two-classification algorithm;
and (4): converting the gesture voltage signals acquired each time into sign language words through a support vector machine classification module;
and (5): converting sign language words obtained by the gesture voltage signal conversion in the step (4) within a period of time into sign language word groups, matching the sign language word groups with the sign language word library, and associatively filling the sign language word groups into sentence output results.
2. The sign language translation method based on the support vector machine according to claim 1, characterized in that: the wearable data glove in the step (1) is provided with flexible sensors which are strain gauges fixed at positions of 10 fingers, the acceleration sensors are six-axis sensors fixed at positions of the backs of the left hand and the right hand respectively, gesture voltage signals are represented by the bending degree of the fingers followed by the strain gauges and the mutual position change of the two six-axis sensors, and the acquisition of the gesture voltage signals is 22 signals which are 10 finger bending signals and 12 gesture direction signals.
3. The sign language translation method based on the support vector machine according to claim 1, characterized in that: the sign language sentence library in the step (2) comprises a sign language sentence library and a sign language sub-library, wherein the sign language sentence library is manufactured by firstly recording the received current 22 gesture voltage signals in Excel, and then storing the gesture voltage signals in the Access database after normalization.
4. The sign language translation method based on the support vector machine according to claim 1, characterized in that: the support vector machine classification module in the step (3) has 22 input vectors, corresponds to 22 voltage signals, has 153 sub-classification models, adopts a directed acyclic classification scheme, obtains 18 classification results with the serial numbers of 0 to 17 through 17 secondary classification, sequentially corresponds to 18 common phrases, and randomly combines the common phrases into 53 common phrases.
5. The sign language translation method based on the support vector machine according to claim 1, wherein the step (4) of converting the gesture voltage signal into the sign language words by the support vector machine classification module comprises the following steps:
step (4.1): receiving gesture voltage signals of the wearable data glove, and screening complete signals by using a signal screening program;
step (4.2): and converting the gesture voltage signals into words through a trained support vector machine classification module.
6. The sign language translation method based on the support vector machine as claimed in claim 1, wherein the step (5) of matching and associating with the sign language sentence library after being converted by the support vector machine classification module and filling into sentence output results comprises the following steps:
step (5.1): dividing the sentences in the sign language sentence library or collected vocabulary groups into words and counting, defining the vocabulary with high frequency and symbolic significance in the sentences as element 1, and defining the rest vocabularies as element 0;
step (5.2): defining all the frequently-used sign language sentences in the sign language sentence library into a word frequency vector format according to the step (5.1) to generate corresponding word frequency vectors;
step (5.3): converting words obtained after the gesture voltage signals collected in the step (4) are converted within a period of time into sign language word groups, and converting the obtained sign language word groups into corresponding word frequency vectors in a word frequency vector format according to the rule of the step (5.1);
step (5.4): calculating the cosine similarity between the word frequency vector converted in the step (5.3) and the word frequency vector in the sign language sentence library in the step (5.2), and selecting the sign language words in the sign language sentence library with the largest cosine similarity as output words;
step (5.5): and (5.4) matching the output words obtained in the step (5.4) with corresponding written language sentences according to the indexes of all the common sign language sentences in the sign language sentence library, and taking the matched written language sentences as final output results.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242973.1A CN111428802B (en) | 2020-03-31 | 2020-03-31 | Sign language translation method based on support vector machine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242973.1A CN111428802B (en) | 2020-03-31 | 2020-03-31 | Sign language translation method based on support vector machine |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111428802A CN111428802A (en) | 2020-07-17 |
| CN111428802B true CN111428802B (en) | 2023-02-07 |
Family
ID=71549260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010242973.1A Active CN111428802B (en) | 2020-03-31 | 2020-03-31 | Sign language translation method based on support vector machine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111428802B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001059741A1 (en) * | 2000-02-10 | 2001-08-16 | Koninklijke Philips Electronics N.V. | Sign language to speech converting method and apparatus |
| CN101794528A (en) * | 2010-04-02 | 2010-08-04 | 北京大学软件与微电子学院无锡产学研合作教育基地 | Gesture language-voice bidirectional translation system |
| CN103116576A (en) * | 2013-01-29 | 2013-05-22 | 安徽安泰新型包装材料有限公司 | Voice and gesture interactive translation device and control method thereof |
| CN203149569U (en) * | 2013-01-29 | 2013-08-21 | 安徽安泰新型包装材料有限公司 | Voice and gesture interactive translation device |
| JP2016051171A (en) * | 2014-08-28 | 2016-04-11 | 日本放送協会 | Proper name translation device and proper name translation method |
| CN109696963A (en) * | 2019-01-23 | 2019-04-30 | 深圳大学 | Sign language interpretation system, glove for sign language translation and sign language interpretation method |
| CN109885166A (en) * | 2019-02-25 | 2019-06-14 | 浙江工商大学 | Intelligent sign language translation gloves and its gesture identification method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10446059B2 (en) * | 2016-12-07 | 2019-10-15 | Thomas William Pryor | Hand motion interpretation and communication apparatus |
-
2020
- 2020-03-31 CN CN202010242973.1A patent/CN111428802B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001059741A1 (en) * | 2000-02-10 | 2001-08-16 | Koninklijke Philips Electronics N.V. | Sign language to speech converting method and apparatus |
| CN101794528A (en) * | 2010-04-02 | 2010-08-04 | 北京大学软件与微电子学院无锡产学研合作教育基地 | Gesture language-voice bidirectional translation system |
| CN103116576A (en) * | 2013-01-29 | 2013-05-22 | 安徽安泰新型包装材料有限公司 | Voice and gesture interactive translation device and control method thereof |
| CN203149569U (en) * | 2013-01-29 | 2013-08-21 | 安徽安泰新型包装材料有限公司 | Voice and gesture interactive translation device |
| JP2016051171A (en) * | 2014-08-28 | 2016-04-11 | 日本放送協会 | Proper name translation device and proper name translation method |
| CN109696963A (en) * | 2019-01-23 | 2019-04-30 | 深圳大学 | Sign language interpretation system, glove for sign language translation and sign language interpretation method |
| CN110096153A (en) * | 2019-01-23 | 2019-08-06 | 深圳大学 | A kind of sign language interpretation system, glove for sign language translation and sign language interpretation method |
| CN109885166A (en) * | 2019-02-25 | 2019-06-14 | 浙江工商大学 | Intelligent sign language translation gloves and its gesture identification method |
Non-Patent Citations (1)
| Title |
|---|
| 面向机器翻译的中国手语的理解与合成;徐琳等;《计算机学报》;20000112(第01期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111428802A (en) | 2020-07-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209806B (en) | Text classification method, text classification device and computer readable storage medium | |
| Kadous | Temporal classification: Extending the classification paradigm to multivariate time series | |
| CN110634491B (en) | Series connection feature extraction system and method for general voice task in voice signal | |
| CN105229725B (en) | Multilingual depth neural network | |
| Ahmed et al. | An ensemble 1D-CNN-LSTM-GRU model with data augmentation for speech emotion recognition | |
| CN106228977A (en) | The song emotion identification method of multi-modal fusion based on degree of depth study | |
| CN105575388A (en) | Emotional speech processing | |
| CN111428871B (en) | Sign language translation method based on BP neural network | |
| CN112700794B (en) | Audio scene classification method and device, electronic equipment and storage medium | |
| CN109461441B (en) | Self-adaptive unsupervised intelligent sensing method for classroom teaching activities | |
| CN110110059A (en) | A kind of medical conversational system intention assessment classification method based on deep learning | |
| CN111931616A (en) | Emotion recognition method and system based on mobile intelligent terminal sensor equipment | |
| Song et al. | Contrastive embeddind learning method for respiratory sound classification | |
| CN115862684A (en) | Audio-based depression state auxiliary detection method for dual-mode fusion type neural network | |
| Janbakhshi et al. | Automatic dysarthric speech detection exploiting pairwise distance-based convolutional neural networks | |
| CN116110565A (en) | Method for auxiliary detection of crowd depression state based on multi-modal deep neural network | |
| CN114188022A (en) | Clinical children cough intelligent pre-diagnosis system based on textCNN model | |
| CN111428802B (en) | Sign language translation method based on support vector machine | |
| CN113380418A (en) | System for analyzing and identifying depression through dialog text | |
| CN102141812A (en) | Robot | |
| Anindya et al. | Development of Indonesian speech recognition with deep neural network for robotic command | |
| CN115455142A (en) | Text retrieval method, computer device and storage medium | |
| Mouaz et al. | A new framework based on KNN and DT for speech identification through emphatic letters in Moroccan dialect | |
| Glüge et al. | SEGMENTED–MEMORY RECURRENT NEURAL NETWORKS VERSUS HIDDEN MARKOV MODELS IN EMOTION RECOGNITION FROM SPEECH | |
| KR20220005232A (en) | Method, apparatur, computer program and computer readable recording medium for providing telemedicine service based on speech recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |