CN111240489A - Gesture recognition method and system based on gesture recognition glove and gesture recognition glove - Google Patents
Gesture recognition method and system based on gesture recognition glove and gesture recognition glove Download PDFInfo
- Publication number
- CN111240489A CN111240489A CN202010126325.XA CN202010126325A CN111240489A CN 111240489 A CN111240489 A CN 111240489A CN 202010126325 A CN202010126325 A CN 202010126325A CN 111240489 A CN111240489 A CN 111240489A
- Authority
- CN
- China
- Prior art keywords
- gesture
- user terminal
- gesture recognition
- glove
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a gesture recognition method and system based on a gesture recognition glove and the gesture recognition glove. The gesture recognition method comprises the following steps: receiving gesture data uploaded by a user terminal; the gesture data are obtained by the user terminal according to the finger bending angle and the palm motion posture collected by the gesture recognition glove; based on a deep learning algorithm, recognizing and processing the gesture data according to pre-stored standard gesture data to obtain gesture information; and returning the gesture information to the user terminal, and transmitting the gesture information to the gesture recognition glove by the user terminal. The invention can quickly and accurately identify the gesture information of the wearing person and help the deaf-mute to perform barrier-free communication with the normal person.
Description
Technical Field
The invention relates to the technical field of gesture recognition, in particular to a gesture recognition method and system based on a gesture recognition glove and the gesture recognition glove.
Background
Sign language is a language used for communication of deaf-mutes, is a relatively stable expression system formed by hand movements supplemented with facial expressions, and is a special language for communication depending on movements and vision. However, sign language is a huge and complex language system, and most of normal people except some professionals have difficulty in interpreting sign language according to gestures of deaf-mutes, so that the deaf-mutes have barriers to communication with the normal people.
Currently, MACHINE-VISION based gesture recognition technology is mainly applied to recognize gestures of deaf-mutes. Due to the influence of factors such as camera imaging, environment, light and the like, and the problems of shielding, direction and the like during finger action recognition, the gesture of the deaf-mute is difficult to be recognized quickly and accurately.
Disclosure of Invention
The invention provides a gesture recognition method and system based on gesture recognition gloves and the gesture recognition gloves, which are used for overcoming the defects of the prior art.
In order to solve the above technical problem, in a first aspect, an embodiment of the present invention provides a gesture recognition method based on a gesture recognition glove, including:
receiving gesture data uploaded by a user terminal; the gesture data are obtained by the user terminal according to the finger bending angle and the palm motion posture collected by the gesture recognition glove;
based on a deep learning algorithm, recognizing and processing the gesture data according to pre-stored standard gesture data to obtain gesture information;
and returning the gesture information to the user terminal, and transmitting the gesture information to the gesture recognition glove by the user terminal.
Further, before the receiving gesture data uploaded by the user terminal, the method further includes:
and performing identity authentication on the user terminal, and receiving the gesture data uploaded by the user terminal when the user terminal passes the identity authentication.
Further, the gesture recognition method based on the gesture recognition glove further comprises the following steps:
responding to a gesture query request initiated by the user terminal, and returning a gesture query result to the user terminal;
and responding to the voice translation request initiated by the user terminal, and returning a voice translation result to the user terminal.
Further, the gesture information comprises one or more of voice information, text information or picture information.
In a second aspect, an embodiment of the present invention provides a gesture recognition system based on gesture recognition gloves, including a first gesture recognition glove, a second gesture recognition glove, a user terminal, and a server;
the first gesture recognition glove and the second gesture recognition glove are used for collecting a finger bending angle and a palm movement posture of a user wearing a person and transmitting the finger bending angle and the palm movement posture to the user terminal;
the user terminal is used for obtaining gesture data according to the finger bending angle and the palm movement posture and uploading the gesture data to the server;
the server is used for recognizing and processing the gesture data according to pre-stored standard gesture data based on a deep learning algorithm to obtain gesture information, returning the gesture information to the user terminal, and transmitting the gesture information to the second gesture recognition glove by the user terminal.
Further, the first gesture recognition glove comprises a first bending acquisition module, a first gesture acquisition module, a first main control module and a first communication module;
the second gesture recognition glove comprises a second bending acquisition module, a second gesture acquisition module, a second main control module, a second communication module and a second display module.
Further, the server is further configured to respond to a gesture query request initiated by the user terminal, and return a gesture query result to the user terminal;
the server is also used for responding to the voice translation request initiated by the user terminal and returning the voice translation result to the user terminal.
Further, the gesture information comprises one or more of voice information, text information or picture information.
In a third aspect, an embodiment of the present invention provides a gesture recognition glove, including a glove body; the glove comprises a glove body and is characterized in that a bending acquisition module is arranged at the finger position of the glove body, a posture acquisition module is arranged at the back of the hand of the glove body, and a main control module, a communication module and a display module are arranged at the wrist position of the glove body; the main control module is respectively connected with the bending acquisition module, the posture acquisition module, the communication module and the display module; wherein the content of the first and second substances,
the bending acquisition module is used for transmitting the acquired finger bending angle to the main control module;
the gesture acquisition module is used for transmitting the acquired palm motion gesture to the main control module;
the main control module is used for transmitting the finger bending angle and the palm movement posture to the communication module;
the communication module is used for transmitting the palm bending angle and the palm movement posture to external equipment and transmitting gesture information returned by the external equipment to the main control module;
the main control module is also used for transmitting the gesture information to the display module;
the display module is used for displaying the gesture information.
Further, the finger position of gloves body is provided with crooked collection module, specifically is:
and each finger position of the glove body is provided with the bending acquisition module.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
the gesture recognition glove comprises a gesture recognition glove, a gesture information acquisition module, a gesture information processing module and a gesture information processing module, wherein the gesture data are acquired by the user terminal according to finger bending angles and palm motion postures acquired by the gesture recognition glove, the gesture data are recognized and processed according to pre-stored standard gesture data based on a deep learning algorithm to obtain gesture information, the gesture information is returned to the user terminal, and the gesture information is transmitted to the gesture recognition glove by the user terminal. The invention utilizes the gesture recognition glove to collect the finger bending angle and the palm movement gesture of the wearing person, so that various collected data can be provided for the user terminal in time when the wearing person strokes the gesture, the user terminal obtains the gesture data according to the collected data, and the gesture data is recognized and processed based on the deep learning algorithm, so that the processing efficiency of the gesture data can be improved, the gesture information of the wearing person can be rapidly and accurately recognized, and the deaf-mute and the normal person can be helped to perform barrier-free communication.
Drawings
FIG. 1 is a flowchart illustrating a gesture recognition method based on a glove for gesture recognition according to a first embodiment of the present invention;
FIG. 2 is a diagram of a DeepConvLSTM network architecture in a first embodiment of the present invention;
fig. 3 is a network structure diagram of an RNN network in the first embodiment of the present invention;
FIG. 4 is a schematic flow chart of a preferred embodiment of the first embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a gesture recognition system based on a glove for gesture recognition according to a second embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a glove for gesture recognition according to a third embodiment of the present invention;
FIG. 7 is a circuit diagram of a glove for gesture recognition according to a third embodiment of the present invention.
Detailed Description
The technical solutions in the present invention will be described clearly and completely with reference to the accompanying drawings, and it is obvious that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that, the step numbers in the text are only for convenience of explanation of the specific embodiments, and do not serve to limit the execution sequence of the steps. The method provided by the embodiment can be executed by the relevant server, and the server is taken as an example for explanation below.
Please refer to fig. 1-4.
As shown in fig. 1, the first embodiment provides a gesture recognition method based on a gesture recognition glove, which includes steps S11 to S13:
s11, receiving gesture data uploaded by a user terminal; the gesture data are obtained by the user terminal according to the finger bending angle and the palm movement posture collected by the gesture recognition glove.
And S12, recognizing the gesture data according to the pre-stored standard gesture data based on a deep learning algorithm to obtain gesture information.
And S13, returning the gesture information to the user terminal, and transmitting the gesture information to the gesture recognition glove by the user terminal.
The user terminal includes a mobile phone, a computer, a tablet, and other communication devices that are held by the user and can be connected to the gesture recognition glove and the server. When the gesture recognition glove acquires the finger bending angle and the palm movement posture of a wearer, the gesture recognition glove transmits the acquired data such as the finger bending angle and the palm movement posture to a user terminal; when the user terminal receives the collected data, analyzing and packaging the collected data to obtain gesture data, and uploading the gesture data to the server; when the server receives the gesture data, based on a deep learning algorithm, the gesture data are recognized according to pre-stored standard gesture data to obtain gesture information, the gesture information is returned to the user terminal, and the gesture information is transmitted to the gesture recognition glove by the user terminal.
Take the mobile phone of the user terminal as the deaf-mute as an example.
After the deaf-mute wears the gesture recognition glove, the gesture recognition glove can acquire the finger bending angle and the palm motion gesture of the deaf-mute when the deaf-mute gestures, and transmits the acquired data to the mobile phone of the deaf-mute, the gesture data obtained by analyzing and packaging the acquired data is uploaded to the server through an application program installed on the mobile phone, and the gesture information returned by the server is received, so that the gesture information is transmitted to the gesture recognition glove through the application program installed on the mobile phone, and the deaf-mute can display the gesture information received by the mobile phone or the gesture recognition glove to a normal person and perform barrier-free communication with the normal person.
Take the user terminal as the mobile phone of the normal person as an example.
After the deaf-mute wears the gesture recognition glove, the gesture recognition glove can collect finger bending angles and palm motion postures of the deaf-mute when the deaf-mute gestures, and transmits collected data to a mobile phone of a normal person, the gesture data obtained by analyzing and packaging the collected data is uploaded to a server through an application program installed on the mobile phone, and gesture information returned by the server is received, so that the gesture information is transmitted to the gesture recognition glove through the application program installed on the mobile phone, and the normal person can directly check the gesture information received by the mobile phone or indirectly check the gesture information received by the gesture recognition glove and perform barrier-free communication with the deaf-mute.
The gesture recognition glove mainly collects the finger bending angle and the palm movement posture of a wearer through two groups of sensors arranged on the gesture recognition glove. Time series data X for a set of sensor inputsac＝{xac1，xac2，xac3...xacn}，Yac＝{yac1，yac2，yac3....yacn}，Zac＝{zac1，zac2，zac3....zacnAnd calculating the variance of the signals respectively. Taking x-axis data as an example, the variance calculation formula is: sigma2＝(∑(xacMu))/n, where mu represents the mean of the x-axis data and n represents the total number of x-axis data. And when the data of any axis is larger than a threshold value, judging that the wearing person is in a gesture of stroke, wherein the threshold value is 0.5.
The user terminal analyzes and packages the finger bending angle and the palm motion posture collected by the gesture recognition glove to obtain gesture data, and uploads the gesture data to the server.
The server selects an RNN recurrent neural network more suitable for analyzing time series data to identify and process the gesture data based on a deep learning algorithm. As shown in fig. 2, taking the deep convlstm network as an example, the network has eight layers, including one input layer, four convolutional layers, two LSTM layers and one softmax output layer. Model training sets batch-size to 128, the loss function is a cross entropy loss function, the random gradient decreases, and the learning rate is set to 0.001. The identification processing process of the gesture data is as follows:
(1) the collected data of the two groups of sensors are combined into a (128,8) -dimensional data matrix to serve as input data, and a DeepConvLSTM model is created for each group of sensors to obtain two models. After two convolution operations and two LSTM operations, the data is abstracted into 128-dimensional LSTM output vectors.
(2) In the convolution unit of CNN, the data of 128 dimensions is finally converted into the high-level features of 16 dimensions by successively performing the combined operations of two sets of convolution (1 × 1 convolution kernel), BatchNormalization, two-dimensional MaxPooling, and Dropout, wherein Dropout is performed only in the second set of operations. The convolution is calculated by
Wherein the content of the first and second substances,
representing the eigenvector j in layer I, sigma being a non-linear function, FlIs the number of feature vectors in layer i,
the kernel that is convolved on the feature vector f in layer l to create the feature map j is in layer (l +1), PlIs the length of the kernel in layer l, blIs a deviation vector. When processing sensor data, this convolution calculation will be applied independently to each sensor in the inputAnd (4) a channel.
(3) In the time sequence unit of RNN, the number of hidden layer neurons is set to 128, the 16-dimensional convolution characteristic is converted into the 128-dimensional time sequence characteristic, and then Dropout operation is executed. RNN structure as shown in fig. 3, the calculation procedure for this network is as follows: x is the number oftSensor input, s, representing the t-th sequencetThe state of the t-th step representing the hidden layer is the memory unit of the network. stThe method is to calculate according to the output of the current input layer and the state of the previous hidden layer, and the calculation formula is st＝f(U·xt+W·st-1) Setting the activation function f to ReLU, otRepresenting the output of the t step, classifying by using a softmax function, and calculating by using a formula of ot＝softmax(V·st)。
(4) Combining two model outputs of two groups of sensors into one input, namely 128 × 2 — 256, then performing operations such as Dropout, full connection, Batch Normalization and the like, and finally using a Softmax activation function to output probabilities of 16 categories so as to determine the maximum probability category as gesture information.
In the embodiment, gesture data obtained by the user terminal according to the finger bending angle and the palm movement posture collected by the gesture recognition glove is received, the gesture data is recognized and processed according to the pre-stored standard gesture data based on the deep learning algorithm to obtain gesture information, so that the gesture information is returned to the user terminal, and the gesture information is transmitted to the gesture recognition glove by the user terminal.
According to the embodiment, the finger bending angle and the palm movement posture of the wearer are collected by the gesture recognition gloves, so that various collection data can be timely provided for the user terminal when the wearer gestures, the user terminal obtains gesture data according to the collection data, the gesture data are recognized and processed based on a deep learning algorithm, the processing efficiency of the gesture data can be improved, the gesture information of the wearer can be rapidly and accurately recognized, and the deaf-mute and the normal person are helped to perform barrier-free communication.
In a preferred embodiment, before receiving the gesture data uploaded by the user terminal, step S1 further includes: and performing identity authentication on the user terminal, and receiving gesture data uploaded by the user terminal when the user terminal passes the identity authentication.
For example, the deaf-mute or the normal person needs to perform user registration when using an application installed in a mobile phone for the first time. After that, when the deaf-mute or the normal person uses the application program, the deaf-mute or the normal person needs to log in the application program and finish the identity verification to upload the gesture data.
In a preferred embodiment, as shown in fig. 4, the gesture recognition method based on the glove for gesture recognition further includes steps S14 to S15:
and S14, responding to the gesture inquiry request initiated by the user terminal, and returning the gesture inquiry result to the user terminal.
And S15, responding to the voice translation request initiated by the user terminal, and returning the voice translation result to the user terminal.
For example, when a deaf-mute or a normal person needs to inquire a gesture corresponding to certain information, the inquiry information can be input through an application program on a mobile phone, and gesture pictures and the like returned by a server are received, so that the deaf-mute or the normal person can learn the gesture to communicate. When the deaf-mute or the normal person needs to translate the gesture corresponding to a certain voice, the voice information can be input through the application program on the mobile phone, and the gesture pictures and the like returned by the server are received, so that the deaf-mute or the normal person can learn the gesture to communicate.
The function of the voice translation can be realized by directly calling the open SDK of the Baidu voice translation platform, and the calling process specifically comprises the following steps: initializing, starting to recognize/wake up, calling back an event, controlling to recognize/wake up, and exiting the event manager. The SDK calling initialization comprises the steps of initializing an eventManager class, customizing and registering an output event class; setting input parameters in the start recognition/wake-up and sending a start event; the callback event is acquired from EventListener; a stop event or cancel event may be sent to the SDK in control recognition/wake-up; and finally releasing the resources when the event manager exits.
In a preferred embodiment, the gesture information includes one or more of voice information, text information, or picture information in combination.
In the embodiment, the gesture information is sent in various forms such as voice, text and pictures, so that the user terminal and the gesture recognition glove display the gesture information by adopting shortcuts such as voice playing and text or picture displaying.
Please refer to fig. 5.
As shown in fig. 5, the second embodiment provides a gesture recognition system based on gesture recognition gloves, which includes a first gesture recognition glove 21, a second gesture recognition glove 22, a user terminal 23 and a server 24; the first gesture recognition glove 21 and the second gesture recognition glove 22 are used for acquiring a finger bending angle and a palm movement posture of a wearer and transmitting the finger bending angle and the palm movement posture to the user terminal 23; the user terminal 23 is configured to obtain gesture data according to the finger bending angle and the palm movement posture, and upload the gesture data to the server 24; and the server 24 is configured to perform recognition processing on the gesture data according to the pre-stored standard gesture data based on a deep learning algorithm to obtain gesture information, return the gesture information to the user terminal 23, and transmit the gesture information to the second gesture recognition glove 22 by the user terminal 23.
The first gesture recognition glove 21 and the second gesture recognition glove 22 are used to capture the finger bending angle and the palm movement posture of both hands of the wearer. The user terminal 23 includes a mobile phone, a computer, a tablet held by the user, and the like, which can be connected to the gesture recognition gloves (21, 22) and the server 24.
When the first gesture recognition glove 21 and the second gesture recognition glove 22 acquire the finger bending angle and the palm movement posture of the wearer, the first gesture recognition glove 21 and the second gesture recognition glove 22 transmit the acquired data such as the finger bending angle and the palm movement posture to the user terminal 23; when the user terminal 23 receives the collected data, analyzing and packaging the collected data to obtain gesture data, and uploading the gesture data to the server 24; when the server 24 receives the gesture data, based on the deep learning algorithm, the gesture data is recognized according to the pre-stored standard gesture data to obtain gesture information, the gesture information is returned to the user terminal 23, and the user terminal 23 transmits the gesture information to the second gesture recognition glove 22.
Take the user terminal 23 as a mobile phone of the deaf-mute as an example.
After the deaf-mute wears the first gesture recognition glove 21 and the second gesture recognition glove 22, the first gesture recognition glove 21 and the second gesture recognition glove 22 can collect finger bending angles and palm motion gestures of the deaf-mute when the deaf-mute gestures, and transmit collected data to a mobile phone of the deaf-mute, and through an application program installed on the mobile phone, the gesture data obtained by analyzing and packaging the collected data is uploaded to the server 24, and gesture information returned by the server 24 is received, so that the gesture information is transmitted to the second gesture recognition glove 22 through the application program installed on the mobile phone, and the deaf-mute can display the gesture information received by the mobile phone or the second gesture recognition glove 22 to normal persons and perform barrier-free communication with the normal persons.
Take the user terminal 23 as a normal mobile phone as an example.
After the deaf-mute wears the first gesture recognition glove 21 and the second gesture recognition glove 22, the first gesture recognition glove 21 and the second gesture recognition glove 22 can collect finger bending angles and palm motion gestures of the deaf-mute when the deaf-mute gestures, and transmit collected data to a mobile phone of a normal person, the gesture data obtained by analyzing and packaging the collected data is uploaded to the server 24 through an application program installed on the mobile phone, and gesture information returned by the server 24 is received, so that the gesture information is transmitted to the second gesture recognition glove 22 through the application program installed on the mobile phone, and the normal person can directly check the gesture information received by the mobile phone or indirectly check the gesture information received by the second gesture recognition glove 22 to perform barrier-free communication with the deaf-mute.
The first gesture recognition glove 21 and the second gesture recognition glove 22 mainly collect the finger bending angle and the palm movement posture of the wearer through two sets of sensors arranged on the corresponding gesture recognition gloves respectively. Time series data X for a set of sensor inputsac＝{xac1，xac2，xac3...xacn}，Yac＝{yac1，yac2，yac3....yacn}，Zac＝{zac1，zac2，zac3....zacnAnd calculating the variance of the signals respectively. Taking x-axis data as an example, the variance calculation formula is: sigma2＝(∑(xacMu))/n, where mu represents the mean of the x-axis data and n represents the total number of x-axis data. And when the data of any axis is larger than a threshold value, judging that the wearing person is in a gesture of stroke, wherein the threshold value is 0.5.
The user terminal 23 analyzes and packages the finger bending angles and the palm movement postures collected by the first gesture recognition glove 21 and the second gesture recognition glove 22 to obtain gesture data, and uploads the gesture data to the server 24.
The server 24 selects an RNN recurrent neural network more suitable for analyzing the time series data to perform recognition processing on the gesture data based on a deep learning algorithm. Taking the deep ConvLSTM network as an example, the network has eight layers, including an input layer, four convolutional layers, two LSTM layers and a softmax output layer. Model training sets batch-size to 128, the loss function is a cross entropy loss function, the random gradient decreases, and the learning rate is set to 0.001. The identification processing process of the gesture data is as follows:
(1) the collected data of the two groups of sensors are combined into a (128,8) -dimensional data matrix to serve as input data, and a DeepConvLSTM model is created for each group of sensors to obtain two models. After two convolution operations and two LSTM operations, the data is abstracted into 128-dimensional LSTM output vectors.
(2) In the convolution unit of CNN, the data of 128 dimensions is finally converted into the high-level features of 16 dimensions by successively performing the combined operations of two sets of convolution (1 × 1 convolution kernel), BatchNormalization, two-dimensional MaxPooling, and Dropout, wherein Dropout is performed only in the second set of operations. The convolution is calculated by
Wherein the content of the first and second substances,
expressing the direction of features in the l-th layerThe quantity j, σ being a non-linear function, FlIs the number of feature vectors in layer i,
the kernel that is convolved on the feature vector f in layer l to create the feature map j is in layer (l +1), PlIs the length of the kernel in layer l, blIs a deviation vector. This convolution calculation will be applied independently to each sensor channel in the input when processing the sensor data.
(3) In the time sequence unit of RNN, the number of hidden layer neurons is set to 128, the 16-dimensional convolution characteristic is converted into the 128-dimensional time sequence characteristic, and then Dropout operation is executed. The calculation procedure for the RNN network is as follows: x is the number oftSensor input, s, representing the t-th sequencetThe state of the t-th step representing the hidden layer is the memory unit of the network. stThe method is to calculate according to the output of the current input layer and the state of the previous hidden layer, and the calculation formula is st＝f(U·xt+W·st-1) Setting the activation function f to ReLU, otRepresenting the output of the t step, classifying by using a softmax function, and calculating by using a formula of ot＝softmax(V·st)。
(4) Combining two model outputs of two groups of sensors into one input, namely 128 × 2 — 256, then performing operations such as Dropout, full connection, Batch Normalization and the like, and finally using a Softmax activation function to output probabilities of 16 categories so as to determine the maximum probability category as gesture information.
In this embodiment, the server 24 receives gesture data obtained by the user terminal 23 according to the finger bending angle and the palm movement gesture collected by the first gesture recognition glove 21 and the second gesture recognition glove 22, and based on a deep learning algorithm, performs recognition processing on the gesture data according to pre-stored standard gesture data to obtain gesture information, so as to return the gesture information to the user terminal 23, and the user terminal 23 transmits the gesture information to the second gesture recognition glove 22.
The embodiment utilizes the first gesture recognition glove 21 and the second gesture recognition glove 22 to collect the finger bending angle and the palm motion gesture of the wearing person, so that various collection data can be provided for the user terminal 23 in time when the wearing person scratches the gesture, the gesture data is obtained by the user terminal 23 according to the collection data, the gesture data is recognized and processed based on a deep learning algorithm through the server 24, the processing efficiency of the gesture data can be improved, the gesture information of the wearing person can be recognized quickly and accurately, and the deaf-mute and the normal person are helped to perform barrier-free communication.
In this embodiment, the first gesture recognition glove 21 includes a first bending collection module 211, a first gesture collection module 212, a first main control module 213, and a first communication module 214; the second gesture recognition glove 22 includes a second bending collection module 221, a second gesture collection module 222, a second main control module 223, a second communication module 224, and a second display module 225.
The first bending acquisition module 211 is configured to transmit the acquired finger bending angle to the first main control module 213; a first gesture collection module 212, configured to transmit the collected palm movement gesture to a first main control module 213; a first main control module 213, configured to transmit the finger bending angle and the palm movement gesture to a first communication module 214; a first communication module 214 for transmitting the palm flexion angle and the palm movement posture to the user terminal 23. Similarly, the second bending collection module 221 is configured to transmit the collected finger bending angle to the second main control module 223; the second posture acquisition module 222 is configured to transmit the acquired palm movement posture to the second main control module 223; the second main control module 223 is used for transmitting the finger bending angle and the palm movement posture to the second communication module 224; the second communication module 224 is configured to transmit the palm bending angle and the palm movement posture to the user terminal 23, and transmit gesture information returned by the user terminal 23 to the second main control module 223; the second main control module 223 is further configured to transmit the gesture information to the second display module 225; and a second display module 225 for displaying the gesture information.
In the embodiment, the second display module 225 is additionally arranged on the second gesture recognition glove 22, so that the second gesture recognition glove 22 can conveniently display gesture information by displaying shortcuts such as texts or pictures.
In this embodiment, the server 24 is further configured to respond to a gesture query request initiated by the user terminal 23, and return a gesture query result to the user terminal 23; the server 24 is further configured to respond to the voice translation request initiated by the user terminal 23 and return the voice translation result to the user terminal 23.
For example, when the deaf-mute or the normal person needs to inquire a gesture corresponding to certain information, the inquiry information can be input through an application program on the mobile phone, and a gesture picture and the like returned by the server 24 are received, so that the deaf-mute or the normal person can learn the gesture to communicate. When the deaf-mute or the normal person needs to translate the gesture corresponding to a certain voice, the voice information can be input through the application program on the mobile phone, and the gesture pictures and the like returned by the server 24 are received, so that the deaf-mute or the normal person can learn the gesture to communicate.
The function of the voice translation can be realized by directly calling the open SDK of the Baidu voice translation platform, and the calling process specifically comprises the following steps: initializing, starting to recognize/wake up, calling back an event, controlling to recognize/wake up, and exiting the event manager. The SDK calling initialization comprises the steps of initializing an eventManager class, customizing and registering an output event class; setting input parameters in the start recognition/wake-up and sending a start event; the callback event is acquired from EventListener; a stop event or cancel event may be sent to the SDK in control recognition/wake-up; and finally releasing the resources when the event manager exits.
In a preferred embodiment, the gesture information includes one or more of voice information, text information, or picture information.
In the embodiment, the gesture information is sent in various forms such as voice, text and pictures, so that the user terminal and the gesture recognition glove display the gesture information by adopting shortcuts such as voice playing and text or picture displaying.
Please refer to fig. 6-7.
As shown in fig. 6, the third embodiment provides a gesture recognition glove, which includes a glove body 1; a bending acquisition module 101 is arranged at the finger position of the glove body 1, a gesture acquisition module 102 is arranged at the back of the hand of the glove body 1, and a main control module 103, a communication module 104 and a display module 105 are arranged at the wrist position of the glove body 1; the main control module 103 is respectively connected with the bending acquisition module 101, the posture acquisition module 102, the communication module 104 and the display module 105; the bending acquisition module 101 is used for transmitting the acquired finger bending angle to the main control module 103; the gesture acquisition module 102 is used for transmitting the acquired palm movement gesture to the main control module 103; the main control module 103 is used for transmitting the finger bending angle and the palm movement posture to the communication module 104; the communication module 104 is used for transmitting the palm bending angle and the palm movement posture to the external device and transmitting the gesture information returned by the external device to the main control module 103; the main control module 103 is further configured to transmit the gesture information to the display module 105; and the display module 105 is used for displaying the gesture information.
The external device includes a mobile phone, a computer, a tablet and other intelligent devices, and is configured to recognize gesture information according to a finger bending angle and a palm movement posture.
This embodiment sets up crooked collection module 101 through the finger position at gloves body 1, set up gesture collection module 102 in the back of the hand position, set up host system 103 in the wrist position, communication module 104 and display module 105, and with host system 103 respectively with crooked collection module 101, gesture collection module 102, communication module 104 and display module 105 are connected, make host system 103 can transmit the finger bending angle that crooked collection module 101 gathered and the palm motion gesture that gesture collection module 102 gathered to external equipment through communication module 104, and control display module 105 to show gesture information when external equipment returns gesture information through communication module 104.
The embodiment sets up crooked collection module 101 and gesture collection module 102 respectively in the finger position and the back of the hand position of gloves body 1 to can in time provide multiple gesture data to external equipment when the user strokes the gesture, thereby realize discerning user's gesture fast accurately, help the user to carry out accessible and communicate with other people.
In this embodiment, the finger position of the glove body 1 is provided with a bending acquisition module 101, specifically: each finger position of the glove body 1 is provided with a bending acquisition module 101.
According to the glove, the bending acquisition module 101 is arranged at each finger position of the glove body 1, the bending angle of each finger is acquired by the aid of the plurality of bending acquisition modules 101, so that complete gesture data are provided for external equipment when a user gestures, and the external equipment can accurately recognize gesture information.
In the present embodiment, the bending collection module 101 includes a bending sensor.
Because when the user gestures the gesture, the bending of the user's finger changes the resistance value of the bending sensor, and the bending angle of the finger can be converted into a voltage signal to be output through a connection circuit between the bending sensor and the main control module 103, such as an amplifying circuit and an ADC circuit, so as to realize the acquisition of the bending angle of the finger. Wherein the model of the bending sensor comprises Flex 2.2.
In the present embodiment, the gesture collection module 102 includes an acceleration sensor.
The core chip of the acceleration sensor adopts MPU 6050 which comprises an accelerometer and a gyroscope. The accelerometer can measure components in X, Y, Z at a certain moment, the inclination angle is obtained by calculating the ratio of the components in the three directions to the gravity acceleration, the gyroscope can measure the angular velocity, the angular velocity is multiplied by a certain period of time to obtain the rotated angle within a specified time, and the obtained angle is accumulated to obtain the target angle, so that the palm movement posture is acquired.
In the present embodiment, the master control module 103 includes an STM 32.
Among them, the model number of STM32 includes STM32F103RCT 6.
In the present embodiment, the communication module 104 includes an HC-05 Bluetooth module.
The HC-05 Bluetooth module is a master-slave integrated Bluetooth serial port module, and in brief, after the HC-05 Bluetooth module is successfully matched and connected with Bluetooth equipment such as a mobile phone, a communication protocol in the HC-05 Bluetooth module can be ignored, and the HC-05 Bluetooth module can be directly used as a serial port.
The HC-05 Bluetooth module is connected with the main control module 103 and the external device, so that the main control module 103 and the external device share one channel, namely the same serial port, when the main control module 103 sends a finger bending angle and a palm movement posture to the channel, the external device can receive the finger bending angle and the palm movement posture in the channel, and when the external device sends gesture information to the channel, the main control module 103 can receive the gesture information in the channel, so that the gesture information returned by the external device can be obtained in time, and the gesture of a user can be recognized quickly and accurately.
In the present embodiment, the display module 105 includes an OLED display screen.
The OLED display is a display made using organic electroluminescent diodes. The display panel has the excellent characteristics of self-luminescence, no need of backlight source, high contrast, thin thickness, wide viewing angle, high reaction speed, wide application temperature range, simple structure and process, and the like, and is considered as a new application technology of the next generation of flat panel displays.
Since the OLED (organic electroluminescent display technology) is an active light emitting display, it has the advantages of high contrast, wide viewing angle, fast response, high light emitting efficiency, low operating voltage, ultra lightness and thinness, etc. Compared with a liquid crystal display screen and an OLED display screen, the LED display screen has a lighter, thinner and attractive appearance, more excellent color display image quality, a wider viewing range and greater design flexibility, more importantly, the environment adaptability of the OLED display screen is far superior to that of the liquid crystal display screen, the tolerable temperature range reaches the temperature range of-40-85 ℃, and the service life of the gesture recognition glove is prolonged.
In a preferred implementation manner of this embodiment, the curved acquisition module 101 is flex2.2, the gesture acquisition module 102 is MPU 6050, the main control module 103 is STM32F103RCT6, the communication module 104 is HC-05 bluetooth module, and the display module 105 is an OLED display. The circuit structure is shown in fig. 7.
In summary, the embodiment of the present invention has the following advantages:
the gesture recognition glove comprises a gesture recognition glove, a gesture information acquisition module, a gesture information processing module and a gesture information processing module, wherein the gesture data are acquired by the user terminal according to finger bending angles and palm motion postures acquired by the gesture recognition glove, the gesture data are recognized and processed according to pre-stored standard gesture data based on a deep learning algorithm to obtain gesture information, the gesture information is returned to the user terminal, and the gesture information is transmitted to the gesture recognition glove by the user terminal. According to the embodiment, the finger bending angle and the palm movement posture of the wearer are collected by the gesture recognition gloves, so that various collection data can be timely provided for the user terminal when the wearer gestures, the user terminal obtains gesture data according to the collection data, the gesture data are recognized and processed based on a deep learning algorithm, the processing efficiency of the gesture data can be improved, the gesture information of the wearer can be rapidly and accurately recognized, and the deaf-mute and the normal person are helped to perform barrier-free communication.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
It will be understood by those skilled in the art that all or part of the processes of the above embodiments may be implemented by hardware related to instructions of a computer program, and the computer program may be stored in a computer readable storage medium, and when executed, may include the processes of the above embodiments. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Claims (10)
1. A gesture recognition method based on a gesture recognition glove is characterized by comprising the following steps:
receiving gesture data uploaded by a user terminal; the gesture data are obtained by the user terminal according to the finger bending angle and the palm motion posture collected by the gesture recognition glove;
based on a deep learning algorithm, recognizing and processing the gesture data according to pre-stored standard gesture data to obtain gesture information;
and returning the gesture information to the user terminal, and transmitting the gesture information to the gesture recognition glove by the user terminal.
2. The gesture recognition method based on the gesture recognition glove of claim 1, wherein before the receiving gesture data uploaded by the user terminal, further comprising:
and performing identity authentication on the user terminal, and receiving the gesture data uploaded by the user terminal when the user terminal passes the identity authentication.
3. The gesture recognition method based on the gesture recognition glove of claim 1, further comprising:
responding to a gesture query request initiated by the user terminal, and returning a gesture query result to the user terminal;
and responding to the voice translation request initiated by the user terminal, and returning a voice translation result to the user terminal.
4. The gesture recognition method of claim 1, wherein the gesture information comprises one or more combinations of voice information, text information, or picture information.
5. A gesture recognition system based on gesture recognition gloves is characterized by comprising first gesture recognition gloves, second gesture recognition gloves, a user terminal and a server;
the first gesture recognition glove and the second gesture recognition glove are used for collecting a finger bending angle and a palm movement posture of a wearer and transmitting the finger bending angle and the palm movement posture to the user terminal;
the user terminal is used for obtaining gesture data according to the finger bending angle and the palm movement posture and uploading the gesture data to the server;
the server is used for recognizing and processing the gesture data according to pre-stored standard gesture data based on a deep learning algorithm to obtain gesture information, returning the gesture information to the user terminal, and transmitting the gesture information to the second gesture recognition glove by the user terminal.
6. The gesture recognition system based on the gesture recognition glove of claim 5, wherein the first gesture recognition glove comprises a first bend acquisition module, a first pose acquisition module, a first master control module, a first communication module;
the second gesture recognition glove comprises a second bending acquisition module, a second gesture acquisition module, a second main control module, a second communication module and a second display module.
7. The gesture recognition system based on the gesture recognition glove of claim 5, wherein the server is further configured to return a gesture inquiry result to the user terminal in response to a gesture inquiry request initiated by the user terminal;
the server is also used for responding to the voice translation request initiated by the user terminal and returning the voice translation result to the user terminal.
8. The gesture recognition system of claim 5, wherein the gesture information comprises one or more combinations of voice information, text information, or picture information.
9. A gesture recognition glove is characterized by comprising a glove body; the glove comprises a glove body and is characterized in that a bending acquisition module is arranged at the finger position of the glove body, a posture acquisition module is arranged at the back of the hand of the glove body, and a main control module, a communication module and a display module are arranged at the wrist position of the glove body; the main control module is respectively connected with the bending acquisition module, the posture acquisition module, the communication module and the display module; wherein the content of the first and second substances,
the bending acquisition module is used for transmitting the acquired finger bending angle to the main control module;
the gesture acquisition module is used for transmitting the acquired palm motion gesture to the main control module;
the main control module is used for transmitting the finger bending angle and the palm movement posture to the communication module;
the communication module is used for transmitting the palm bending angle and the palm movement posture to external equipment and transmitting gesture information returned by the external equipment to the main control module;
the main control module is also used for transmitting the gesture information to the display module;
the display module is used for displaying the gesture information.
10. The gesture recognition glove of claim 9, wherein the finger position of the glove body is provided with a bending acquisition module, specifically:
and each finger position of the glove body is provided with the bending acquisition module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010126325.XA CN111240489A (en) | 2020-02-27 | 2020-02-27 | Gesture recognition method and system based on gesture recognition glove and gesture recognition glove |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010126325.XA CN111240489A (en) | 2020-02-27 | 2020-02-27 | Gesture recognition method and system based on gesture recognition glove and gesture recognition glove |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111240489A true CN111240489A (en) | 2020-06-05 |
Family
ID=70864518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010126325.XA Pending CN111240489A (en) | 2020-02-27 | 2020-02-27 | Gesture recognition method and system based on gesture recognition glove and gesture recognition glove |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111240489A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112068700A (en) * | 2020-09-04 | 2020-12-11 | 北京服装学院 | Character information input method, device and storage medium |
| CN113084848A (en) * | 2021-04-08 | 2021-07-09 | 河北省科学院应用数学研究所 | Master-slave type dexterous hand following method and terminal equipment |
| CN114120770A (en) * | 2021-03-24 | 2022-03-01 | 张银合 | Barrier-free communication method for hearing-impaired people |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7565295B1 (en) * | 2003-08-28 | 2009-07-21 | The George Washington University | Method and apparatus for translating hand gestures |
| CN206003392U (en) * | 2016-05-18 | 2017-03-08 | 福州大学 | A kind of deaf-mute's social activity gloves |
| CN110189590A (en) * | 2019-06-18 | 2019-08-30 | 合肥工业大学 | A kind of adaptively correcting formula sign language mutual translation system and method |
| CN211578174U (en) * | 2020-02-27 | 2020-09-25 | 广州大学华软软件学院 | Gesture recognition glove |
-
2020
- 2020-02-27 CN CN202010126325.XA patent/CN111240489A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7565295B1 (en) * | 2003-08-28 | 2009-07-21 | The George Washington University | Method and apparatus for translating hand gestures |
| CN206003392U (en) * | 2016-05-18 | 2017-03-08 | 福州大学 | A kind of deaf-mute's social activity gloves |
| CN110189590A (en) * | 2019-06-18 | 2019-08-30 | 合肥工业大学 | A kind of adaptively correcting formula sign language mutual translation system and method |
| CN211578174U (en) * | 2020-02-27 | 2020-09-25 | 广州大学华软软件学院 | Gesture recognition glove |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112068700A (en) * | 2020-09-04 | 2020-12-11 | 北京服装学院 | Character information input method, device and storage medium |
| CN114120770A (en) * | 2021-03-24 | 2022-03-01 | 张银合 | Barrier-free communication method for hearing-impaired people |
| CN113084848A (en) * | 2021-04-08 | 2021-07-09 | 河北省科学院应用数学研究所 | Master-slave type dexterous hand following method and terminal equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11080520B2 (en) | Automatic machine recognition of sign language gestures | |
| US11763599B2 (en) | Model training method and apparatus, face recognition method and apparatus, device, and storage medium | |
| WO2020199926A1 (en) | Image recognition network model training method, image recognition method and device | |
| US20220076000A1 (en) | Image Processing Method And Apparatus | |
| Li et al. | Deep Fisher discriminant learning for mobile hand gesture recognition | |
| CN111240489A (en) | Gesture recognition method and system based on gesture recognition glove and gesture recognition glove | |
| CN110008839B (en) | Intelligent sign language interaction system and method for self-adaptive gesture recognition | |
| Kim et al. | Watch & Do: A smart iot interaction system with object detection and gaze estimation | |
| CN107766403B (en) | Photo album processing method, mobile terminal and computer readable storage medium | |
| WO2021047587A1 (en) | Gesture recognition method, electronic device, computer-readable storage medium, and chip | |
| CN112148128B (en) | Real-time gesture recognition method and device and man-machine interaction system | |
| CN111722713A (en) | Multi-mode fused gesture keyboard input method, device, system and storage medium | |
| CN111149104A (en) | Apparatus, method and computer program product for biometric identification | |
| CN111444488A (en) | Identity authentication method based on dynamic gesture | |
| CN112820299A (en) | Voiceprint recognition model training method and device and related equipment | |
| CN111523559A (en) | Abnormal behavior detection method based on multi-feature fusion | |
| Saraswat et al. | An incremental learning based gesture recognition system for consumer devices using edge-fog computing | |
| CN115187824A (en) | Model training method, scene recognition method and related equipment | |
| Zhang et al. | Fine-grained and real-time gesture recognition by using IMU sensors | |
| WO2021258862A1 (en) | Typing method and apparatus, and device and storage medium | |
| WO2024055748A1 (en) | Head posture estimation method and apparatus, and device and storage medium | |
| WO2018171196A1 (en) | Control method, terminal and system | |
| Tao et al. | Research on communication APP for deaf and mute people based on face emotion recognition technology | |
| Memmesheimer et al. | Gesture recognition on human pose features of single images | |
| CN211578174U (en) | Gesture recognition glove |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |