CN112507822B - Method for responding to system instruction based on gesture action - Google Patents
Method for responding to system instruction based on gesture action Download PDFInfo
- Publication number
- CN112507822B CN112507822B CN202011353426.7A CN202011353426A CN112507822B CN 112507822 B CN112507822 B CN 112507822B CN 202011353426 A CN202011353426 A CN 202011353426A CN 112507822 B CN112507822 B CN 112507822B
- Authority
- CN
- China
- Prior art keywords
- gesture
- palm
- model
- face
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
- G06F18/295—Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method for responding to system instructions based on gesture actions, which comprises the following steps of B1, shooting by a camera module to obtain a hand three-dimensional model; B2. extracting characteristic quantity of the three-dimensional modeling to form two-dimensional plane data; B3. judging the front side and the back side of the hand according to the formed two-dimensional plane data, and identifying static gestures; B4. extracting palm position data (x) c ,y c ) Judging the moving position of the palm of the hand according to the time function of the palm position data, and identifying a dynamic gesture; B5. and selecting a corresponding executed command according to the identified static gesture or dynamic gesture. The beneficial effects of the invention are: 1) A contact type sensor is not needed for gesture recognition; 2) The positive and negative direction recognition of the gesture is increased, and the number of gesture instructions is expanded; 3) And the HMM model algorithm is used for gesture recognition training, so that the accuracy of recognizing the response speed block is high.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method for responding to system instructions based on gesture actions.
Background
With the popularity of computers in society, the development of technologies that facilitate Human-Computer Interaction (HCI) will have a positive impact on the use of computers. Therefore, there is an increasing emphasis on developing new technologies for cross-domain man-machine barriers. The ultimate goal of research is to make human-computer interaction as natural as human-to-human interaction. Gestures have long been recognized as an interactive technique that can provide more natural, creative, and intuitive communication with our computer. For this reason, adding gestures in human-computer interaction is an important research area.
Gesture recognition this term refers to the entire process of tracking human gestures, recognizing their representations and translating into semantically meaningful commands. Research in gesture recognition is directed to designing and developing systems that can recognize gestures for device control as inputs and by mapping commands to outputs. Generally, the approach of collecting information from gesture interaction is contact or non-contact, and gesture interaction systems can be divided into two types, namely contact-based sensors and non-contact-based sensors.
The gesture recognition technology based on the non-contact sensor has the technical problems of inconvenient use and high manufacturing cost, the main principle of the gesture recognition technology based on the non-contact sensor is image processing, gesture image information is collected through a camera, and collected data are preprocessed, wherein the preprocessing comprises denoising and information enhancement. Then, a target gesture in the image is acquired by using a segmentation algorithm. And finally, recognizing the target gesture through a gesture recognition algorithm. The image acquired by the camera may be three-dimensional or two-dimensional, and although more gesture information can be recognized by analyzing and recognizing the three-dimensional image, the two-dimensional analysis is mostly adopted at present because a large amount of data needs to be processed, the time consumption is long, and the response is slow, but the information which can be acquired by the two-dimensional analysis is limited, and generally, only four basic information, namely, the upper information, the lower information, the left information and the right information, can be read. The gesture recognition technology based on the non-contact sensor has the technical problems of high recognition difficulty and limited number of recognized gestures.
In order to solve the above technical problem, publication No. CN 111722717A discloses a gesture recognition method. The gesture recognition method comprises the following steps: acquiring a gesture image to be recognized; inputting the gesture image to be recognized into a pre-trained gesture recognition model to obtain a gesture recognition result; the gesture recognition model is obtained by training based on a training sample combination obtained by combination and a preset loss function. The gesture recognition method and the gesture recognition system can realize accurate recognition of the gesture with extremely small image difference degree, and improve the robustness of the gesture recognition model and the accuracy of the gesture recognition result. However, the invention still does not solve the technical problem of limited number of gestures that can be recognized.
Disclosure of Invention
The invention mainly solves the technical problems that the existing gesture recognition technology based on the contact sensor is inconvenient to use and high in manufacturing cost, and the number of gestures which can be recognized by the existing gesture recognition technology based on the non-contact sensor is limited.
The invention provides a method for responding to system instructions based on gesture actions, which comprises the following steps
B1. Shooting by adopting a camera module to obtain a hand three-dimensional model;
B2. extracting characteristic quantity of the three-dimensional modeling to form two-dimensional plane data;
B3. judging the front side and the back side of the hand according to the density of the formed two-dimensional plane data, and identifying static gestures;
B4. recognizing the gesture according to the change of the data density and the change of the data shape twice before and after the change of the data density in the time function, and extracting the palm position data (x) c ,y c ) Judging the moving position of the palm of the hand according to the time function of the palm position data, and identifying a dynamic gesture;
B5. and selecting a corresponding executed command according to the recognized static gesture or dynamic gesture, and restarting recognition when an unknown gesture is recognized.
Preferably, the static gestures include front-up, front-down, front-left, front-right and back-up, back-down, back-left, back-right. The front and back of the gesture are judged according to different gray values of the palm print and the nail relative to the skin of the human body.
Preferably, the dynamic gesture includes face-up, face-down, face-left, face-right, back-up, back-down, back-left, back-right, flip, sign, approach, distance, and tap. The preferred scheme expands the number of gesture recognition instructions, so that the functions which can be executed by the invention are richer.
Preferably, the step B3 includes the following steps
B301. Completing gesture recognition training according to HMM model algorithm
B301-1, setting the integral state number and the observation symbol number of the HMM model as N and M respectively, so as to obtain the original form of the set parameter value lambda = (pi, A, B):
the probability distribution of the initial state is defined as:
π＝(π 1 ，1－π 1 ，0，…，0)
defining the state transition probability matrix A as:
wherein the sum of the elements of each row of matrix A is 1,
the probability output matrix B of the observation symbols is:
wherein the sum of the elements of each row of matrix B is 1,
b301-2, inputting a training sample;
b301-3 according to the formula
p(O|λ)＝∑Sp(O,S|λ)＝∑Sp(O|S,λ)p(S|λ)
Calculating the conditional probability of the observation sequence O appearing under the model lambda, also called forward probability P (O | lambda),
b301-4 iterative re-estimation of parameters
Calculating front and rear probabilities
B301-5 calculation
Comparing the difference value with P (O | lambda) and a set value epsilon, and recording the result if the difference value is less than epsilon
Inputting the next training sample, repeating the steps B301-3 to B301-5 until the gesture recognition training of the last training sample is finished, and if the difference value between the two is larger than epsilon, ordering
Then repeating the steps B301-3 to B301-5;
b301-6, normalizing to obtain a final result lambda', and detecting the gesture recognition accuracy after training;
B302. comparing the two-dimensional plane data of the input gesture with each gesture model obtained by B301 training,
and sorting according to the similarity degree of the input data and each gesture model, and selecting the gesture model with the highest similarity degree.
Preferably, the step b4. Segmenting the human hand model according to the infrared principle, extracting the palm position data (xc, yc) and calculating the shortest distance z between the palm and the plane where the camera is located. The preferred scheme increases the types of gesture instructions capable of being recognized.
Preferably, when detecting that the variation of the palm position data (xc, yc) is smaller than the set value S1 within the time T1 and the peak-to-valley difference zmax-zmin of the shortest distance z within the time T1 is larger than the set value S2, the gesture command is recognized as tapping.
Preferably, the recognition training of the dynamic finger instruction in the step b4 is also performed by means of the HMM model algorithm.
Preferably, the present invention is implemented by computer programming.
The beneficial effects of the invention are: 1) A contact type sensor is not needed for gesture recognition; 2) The positive and negative direction recognition of the gesture is increased, and the number of gesture instructions is expanded; 3) And the HMM model algorithm is used for gesture recognition training, so that the accuracy of recognizing the response speed block is high. The technical problems that the existing gesture recognition technology based on the contact sensor is inconvenient to use and high in manufacturing cost, and the number of gestures which can be recognized by the existing gesture recognition technology based on the non-contact sensor is limited are solved.
Drawings
FIG. 1 is a block flow diagram of a method of an embodiment of the invention.
FIG. 2 is a flow chart of an HMM model algorithm according to an embodiment of the present invention.
Detailed Description
The technical scheme of the invention is further specifically described by the following embodiments and the accompanying drawings.
As shown in fig. 1, comprises the following steps
B1. Shooting by adopting a camera module to obtain a hand three-dimensional model;
B2. extracting characteristic quantity of the three-dimensional modeling to form two-dimensional plane data;
B3. judging the front side and the back side of the hand according to the formed two-dimensional plane data, and identifying static gestures; as shown in fig. 2, B3 specifically includes:
B301. completing gesture recognition training according to HMM model algorithm
B301-1, setting the integral state number and the observation symbol number of the HMM model as N and M respectively, so as to obtain the original form of the set parameter value lambda = (pi, A, B):
the probability distribution of the initial state is defined as:
π＝(π 1 ，1－π 1 ，0，…，0)
the state transition probability matrix a is defined as:
wherein the sum of the elements of each row of matrix A is 1,
the probability output matrix B of the observation symbols is:
wherein the sum of the elements of each column of matrix B is 1,
b301-2, inputting a training sample;
b301-3 is based on the formula
p(O|λ)＝∑Sp(O,S|λ)＝∑Sp(O|S,λ)p(S|λ)
Calculating the conditional probability of the observation sequence O appearing under the model lambda, also called forward probability P (O | lambda),
b301-4 iterative reestimation parameters
Calculating front and rear probabilities
B301-5 calculation
Comparing the difference value with the P (O | lambda) and the set value epsilon, and recording the result if the difference value is less than epsilon
Inputting the next training sample, repeating the steps B301-3 to B301-5 until the gesture recognition training of the last training sample is finished, and if the difference value between the two is larger than epsilon, ordering
Then repeating the steps B301-3 to B301-5;
and B301-6, normalizing to obtain a final result lambda', and detecting the gesture recognition accuracy after training.
B302. Comparing the two-dimensional plane data of the input gesture with each gesture model obtained by B301 training, sorting according to the similarity degree of the input data and each gesture model, and selecting the gesture model with the highest similarity degree.
B4. Extracting palm position data (x) c ,y c ) Judging the moving position of the palm of the hand according to the time function of the palm position data, and identifying dynamic gestures; and B4. The recognition training of the dynamic finger instruction is also completed by means of the HMM model algorithm.
B5. And selecting a corresponding executed command according to the recognized static gesture or dynamic gesture, inquiring whether the user executes the command, if the user selects 'yes', executing, and if the user selects 'no', restarting recognition.
The static gestures include face up, face down, face left, face right, and back up, back down, back left, back right. The front and back of the gesture are judged according to different gray values of the palm print and the nail relative to the skin of the human body.
The dynamic gesture includes a face up, face down, face left, face right, back up, back down, back left, back right, flip, sign, approach, away, and pat. And B4, segmenting the hand model according to the infrared principle, extracting palm position data (xc, yc) and calculating the shortest distance z between the palm and the plane where the camera is located. Taking tapping as an example, when detecting that the variation of the palm position data (xc, yc) is smaller than the set value S1 within the time T1 and the peak-to-valley difference zmax-zmin of the shortest distance z within the time T1 is larger than the set value S2, the gesture command is identified as tapping.
The above-described embodiments are only preferred embodiments of the present invention, and are not intended to limit the present invention in any way, and other variations and modifications may be made without departing from the spirit of the invention as set forth in the claims.
Claims (7)
1. A method for responding to system instructions based on gesture actions, comprising: comprises the following steps
B1. Shooting by adopting a camera module to obtain a hand three-dimensional model;
B2. extracting characteristic quantity of the three-dimensional modeling to form two-dimensional plane data;
B3. judging the front side and the back side of the hand according to the formed two-dimensional plane data, and identifying static gestures;
B4. extracting the position data (xc, yc) of the palm, judging the moving position of the palm according to the time function of the position data of the palm, and identifying dynamic gestures;
B5. selecting a corresponding executed command according to the recognized static gesture or dynamic gesture, inquiring whether the user executes the command, if the user selects 'yes', executing, and if the user selects 'no', restarting recognition;
b3, judging the front and the back of the gesture according to different gray values of the palm print and the fingernail relative to the skin of the human body;
the step B3 comprises the following steps
B301. Completing gesture recognition training according to HMM model algorithm
B301-1, setting the integral state number and the observation symbol number of the HMM model as N and M respectively, so as to obtain the original form of the set parameter value lambda = (pi, A, B):
the probability distribution of the initial state is defined as:
π＝(π1，1－π1，0，…，0)
defining the state transition probability matrix A as:
wherein the sum of the elements of each row of matrix A is 1,
the probability output matrix B of the observation symbol is:
wherein the sum of the elements of each row of matrix B is 1,
b301-2, inputting a training sample;
b301-3 according to the formula
p(O|λ)＝∑Sp(O,S|λ)＝∑Sp(O|S,λ)p(S|λ)
Calculating the conditional probability of the observation sequence O appearing under the model lambda, also called forward probability P (O | lambda), B301-4 iterates to estimate the parameter
Calculating front and rear probabilities
B301-5 calculation
Comparing the difference value with P (O | lambda) and a set value epsilon, and recording the result if the difference value is less than epsilon
Inputting the next training sample, repeating the steps B301-3 to B301-5 until the gesture recognition training of the last training sample is finished, and if the difference value between the two is larger than epsilon, ordering
Then repeating the steps B301-3 to B301-5; b301-6 is normalized to obtain a final result lambda', and the gesture recognition accuracy after training is detected;
and B4, segmenting the hand model according to the infrared principle, extracting palm position data (xc, yc) and calculating the shortest distance z between the palm and the plane where the camera is located.
2. The method of claim 1, wherein the method comprises: the static gestures include face up, face down, face left, face right, and back up, back down, back left, back right.
3. The method of claim 1, wherein the method comprises: the dynamic gesture includes a face up, face down, face left, face right, back up, back down, back left, back right, flip, sign, approach, distance, and tap.
4. The method of claim 1, wherein the method comprises: the step B3 comprises the following steps
B302. Comparing the two-dimensional plane data of the input gesture with each gesture model obtained by B301 training,
and sorting according to the similarity degree of the input data and each gesture model, and selecting the gesture model with the highest similarity degree.
5. The method of claim 1, wherein the method comprises: and when detecting that the variation of the palm position data (xc, yc) is smaller than a set value S1 within the time T1 and the peak-to-valley difference zmax-zmin of the shortest distance z within the time T1 is larger than a set value S2, identifying the gesture command as beating.
6. The method of claim 4, wherein the system command is responded to based on a gesture, and the method comprises the following steps: and the step B4. The recognition training of the dynamic finger instruction is also completed by means of the HMM model algorithm.
7. The method for responding to system instructions based on gesture actions according to any of claims 1-6, characterized by: the method is realized through computer programming.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011353426.7A CN112507822B (en) | 2020-11-26 | 2020-11-26 | Method for responding to system instruction based on gesture action |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011353426.7A CN112507822B (en) | 2020-11-26 | 2020-11-26 | Method for responding to system instruction based on gesture action |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112507822A CN112507822A (en) | 2021-03-16 |
| CN112507822B true CN112507822B (en) | 2022-12-13 |
Family
ID=74966567
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011353426.7A Active CN112507822B (en) | 2020-11-26 | 2020-11-26 | Method for responding to system instruction based on gesture action |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112507822B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331158A (en) * | 2014-10-29 | 2015-02-04 | 山东大学 | Gesture-controlled human-computer interaction method and device |
| CN109948592A (en) * | 2019-04-04 | 2019-06-28 | 北京理工大学 | A kind of design idea method of discrimination and system based on hand signal identification |
| CN110084192A (en) * | 2019-04-26 | 2019-08-02 | 南京大学 | Quick dynamic hand gesture recognition system and method based on target detection |
| CN110837792A (en) * | 2019-11-04 | 2020-02-25 | 东南大学 | Three-dimensional gesture recognition method and device |
-
2020
- 2020-11-26 CN CN202011353426.7A patent/CN112507822B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331158A (en) * | 2014-10-29 | 2015-02-04 | 山东大学 | Gesture-controlled human-computer interaction method and device |
| CN109948592A (en) * | 2019-04-04 | 2019-06-28 | 北京理工大学 | A kind of design idea method of discrimination and system based on hand signal identification |
| CN110084192A (en) * | 2019-04-26 | 2019-08-02 | 南京大学 | Quick dynamic hand gesture recognition system and method based on target detection |
| CN110837792A (en) * | 2019-11-04 | 2020-02-25 | 东南大学 | Three-dimensional gesture recognition method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112507822A (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Aloysius et al. | Understanding vision-based continuous sign language recognition | |
| CN103415825A (en) | System and method for gesture recognition | |
| CN113033398B (en) | Gesture recognition method and device, computer equipment and storage medium | |
| Yang et al. | Simultaneous spotting of signs and fingerspellings based on hierarchical conditional random fields and boostmap embeddings | |
| Subburaj et al. | Survey on sign language recognition in context of vision-based and deep learning | |
| Kolivand et al. | A new framework for sign language alphabet hand posture recognition using geometrical features through artificial neural network (part 1) | |
| CN112749646A (en) | Interactive point-reading system based on gesture recognition | |
| Kawulok | Energy-based blob analysis for improving precision of skin segmentation | |
| CN109190443A (en) | It is a kind of accidentally to know gestures detection and error correction method | |
| Wu et al. | Combining hidden Markov model and fuzzy neural network for continuous recognition of complex dynamic gestures | |
| Agha et al. | A comprehensive study on sign languages recognition systems using (SVM, KNN, CNN and ANN) | |
| KR20200080419A (en) | Hand gesture recognition method using artificial neural network and device thereof | |
| CN111368762A (en) | Robot gesture recognition method based on improved K-means clustering algorithm | |
| Choudhury et al. | A CNN-LSTM based ensemble framework for in-air handwritten Assamese character recognition | |
| Patil et al. | Performance analysis of static hand gesture recognition approaches using artificial neural network, support vector machine and two stream based transfer learning approach | |
| CN110032948B (en) | Sketch gesture recognition method based on interaction time sequence information | |
| CN113269089B (en) | Real-time gesture recognition method and system based on deep learning | |
| Altun et al. | SKETRACK: stroke-based recognition of online hand-drawn sketches of arrow-connected diagrams and digital logic circuit diagrams | |
| Thangakrishnan et al. | RETRACTED ARTICLE: Automated Hand-drawn sketches retrieval and recognition using regularized Particle Swarm Optimization based deep convolutional neural network | |
| CN112507822B (en) | Method for responding to system instruction based on gesture action | |
| Yadav et al. | Removal of self co-articulation and recognition of dynamic hand gestures using deep architectures | |
| Axyonov et al. | Method of multi-modal video analysis of hand movements for automatic recognition of isolated signs of Russian sign language | |
| Dandashy et al. | Enhanced face detection based on Haar-Like and MB-LBP features | |
| Ying et al. | Dynamic random regression forests for real-time head pose estimation | |
| Li et al. | Cross-people mobile-phone based airwriting character recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |