CN112486317B - Digital reading method and system based on gestures - Google Patents
Digital reading method and system based on gestures Download PDFInfo
- Publication number
- CN112486317B CN112486317B CN202011343542.0A CN202011343542A CN112486317B CN 112486317 B CN112486317 B CN 112486317B CN 202011343542 A CN202011343542 A CN 202011343542A CN 112486317 B CN112486317 B CN 112486317B
- Authority
- CN
- China
- Prior art keywords
- gesture
- gesture recognition
- result
- image
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2135—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on approximation criteria, e.g. principal component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Abstract
The invention provides a digital reading method and a digital reading system based on gestures, wherein the method can finally determine the gestures with mapping relations and the gestures without mapping relations in the user-defined gestures through a gesture recognition experiment process formed by any two user-defined gestures, namely the transition gestures are recognized; the current gesture recognition real result is a transition gesture or two consecutive gesture recognition real results are the same, and the gesture corresponding to the current gesture recognition real result is abandoned to be executed before the duration time of the corresponding gesture reaches a set value, so that although delay on customer requirements is possibly caused, the continuous execution of the same gesture in a short time and the execution of transition gestures which are not required to be executed can be avoided, and misoperation of reading behaviors is avoided; the current gesture recognition real result is a non-transitional gesture, and when the gesture recognition real results of two consecutive times are different, the current gesture recognition real result is immediately executed, so that the reading requirement of a user can be immediately responded, and the influence on the user experience caused by time delay is avoided.
Description
Technical Field
The invention relates to the technical field of gesture recognition, in particular to a digital reading method and system based on gestures.
Background
In the field of digital reading, reading behaviors are usually realized by a gesture control terminal, the terminal firstly defines operable behaviors, such as a limited number of gesture behaviors of page turning, exiting, entering and the like, then obtains a result from a gesture recognition engine, binds the result to a corresponding intention, and finally executes the behavior corresponding to the gesture.
Under normal conditions, for the same gesture, the interval time between two times of execution of the same gesture behavior is greater than 1S, if the current gesture of the user corresponds to page turning and the gesture is not changed in 1S, the page turning behavior is not executed in 1S after the first page turning behavior is executed, that is, the gesture in 1S maintained by the same gesture is an invalid gesture. However, the recognition result is continuously thrown within millisecond time after the gesture recognition engine recognizes the gesture, and the terminal cannot judge the invalid gesture within 1S before the user changes the gesture, and the page turning behavior is continuously executed, so that the misoperation of the reading behavior is caused. In addition, when a user sends a gesture for establishing, changing or retracting, the user respectively experiences the processes from no gesture to a target gesture, from a first target gesture to a second target gesture, and from the target gesture to no gesture, and transition gestures may occur in all the three processes, that is, gestures in which the user does not want to execute gesture behaviors, but the terminal cannot judge the transition gestures, and all the gestures corresponding to the transition gestures are executed, which also causes misoperation of reading behaviors.
Disclosure of Invention
In view of the above, in one aspect, the present invention provides a gesture-based digital reading method to solve the problem of incorrect reading behavior caused by the execution of invalid gestures and transitional gestures in the conventional gesture-based digital reading method.
The technical scheme of the invention is realized as follows: a gesture-based digital reading method, comprising:
acquiring all custom gestures, taking one of the custom gestures as an initial gesture, taking each of the rest gestures as a target gesture in sequence, and performing a gesture recognition experiment process of converting the initial gesture into each target gesture in sequence;
acquiring a gesture recognition experiment result in a gesture recognition experiment process, setting gestures except an initial gesture and a target gesture in the gesture recognition experiment result as transition gestures, and establishing a mapping relation between the initial gesture and all the transition gestures;
continuously acquiring a real gesture recognition result when digital reading is carried out;
if the current gesture recognition real result is the same as the last gesture recognition real result or a mapping relation exists, acquiring the duration of the gesture corresponding to the current gesture recognition real result; before the duration time of the gesture corresponding to the current gesture recognition real result reaches a set value, giving up executing the gesture corresponding to the current gesture recognition real result; when the duration of the gesture corresponding to the current gesture recognition real result reaches a set value, immediately executing the gesture corresponding to the current gesture recognition real result and resetting the duration;
and if the current gesture recognition real result is different from the last gesture recognition real result and no mapping relation exists, immediately executing the gesture corresponding to the current gesture recognition real result.
Optionally, the set value is 1S.
Optionally, the step of obtaining the gesture recognition experiment result or the gesture recognition real result includes:
acquiring a video image and segmenting a gesture image from a background of the video image;
binarizing the segmented gesture image, and converting the segmented gesture image into an image with a specified size;
extracting HOG features and Hu moments of the image, and combining the HOG features and the Hu moments into a fusion feature vector;
using a PCA algorithm to perform dimensionality reduction on the fusion feature vector;
and finishing the classification of the fusion feature vectors after the dimensionality reduction treatment by using the trained SVM classifier to obtain a gesture recognition result.
Optionally, the step of obtaining the video image and segmenting the gesture image from the background of the video image includes:
acquiring a video image, tracking gesture movement, and acquiring the area of a moving hand;
and carrying out skin color segmentation on the area where the hand is located, and obtaining a gesture image by adopting boundary scanning on the binary segmented image.
Optionally, the step of performing the dimensionality reduction processing on the fusion feature vector by using the PCA algorithm includes:
performing sign normalization processing on the fusion feature vector;
calculating a covariance matrix of the sample characteristics;
calculating the eigenvalue and the eigenvector of the covariance matrix by adopting a singular value decomposition algorithm to obtain a dimension reduction matrix;
the samples are mapped onto a low-dimensional space by a dimension reduction matrix.
Optionally, the step of obtaining the gesture recognition experiment result or the gesture recognition real result further includes:
and if the results of SVM classification continuous multi-frame gesture images are consistent, adding the multi-frame binarized gesture images with consistent results into an image queue which is classified correctly, acquiring the binarized gesture image next time, carrying out phase comparison on the gesture image and each gesture image in the image queue, and if the repeated area accounts for more than 95% of the area of all the gesture images and the image area of the gesture image per se in the image queue, not classifying the gesture image, and representing the recognition result of the gesture image by using the classification of the image queue.
Optionally, the step of obtaining the gesture recognition experiment result or the gesture recognition real result further includes:
and adding the gesture images which are not subjected to SVM classification into the corresponding image queue.
Compared with the prior art, the digital reading method based on the gestures has the following beneficial effects:
(1) through a gesture recognition experiment process formed by any two custom gestures, the gestures with mapping relations and the gestures without mapping relations in the custom gestures can be finally determined, namely transition gestures are recognized;
(2) the current gesture recognition real result is a transition gesture or two consecutive gesture recognition real results are the same, and the gesture corresponding to the current gesture recognition real result is abandoned to be executed before the duration time of the corresponding gesture reaches a set value, so that although delay on customer requirements is possibly caused, the continuous execution of the same gesture in a short time and the execution of transition gestures which are not required to be executed can be avoided, and misoperation of reading behaviors is avoided;
(3) the current gesture recognition real result is a non-transitional gesture, and when the gesture recognition real results of two consecutive times are different, the current gesture recognition real result is immediately executed, so that the reading requirement of a user can be immediately responded, and the influence on the user experience caused by time delay is avoided;
(4) for the same gesture, the type of the gesture can be recognized in a gesture image contrast mode, SVM classification does not need to be repeatedly carried out, and workload of SVM classification is reduced.
On the other hand, the invention also provides a digital reading system based on the gesture, so as to solve the problem of reading behavior misoperation caused by the fact that the traditional digital reading system based on the gesture executes invalid gestures and transitional gestures.
The technical scheme of the invention is realized as follows: a gesture-based digital reading system, comprising:
the gesture recognition experiment module is used for acquiring all the custom gestures, taking one of the custom gestures as an initial gesture, taking each of the rest gestures as a target gesture in sequence, and performing a gesture recognition experiment process for converting the initial gesture into each target gesture in sequence;
the mapping relation establishing module is used for acquiring a gesture recognition experiment result in the gesture recognition experiment process, setting gestures except an initial gesture and a target gesture in the gesture recognition experiment result as transition gestures, and establishing a mapping relation between the initial gesture and all the transition gestures;
the recognition result acquisition module is used for continuously acquiring a gesture recognition real result when digital reading is carried out;
the recognition result execution module is used for acquiring the duration of the gesture corresponding to the current gesture recognition real result if the current gesture recognition real result is the same as the previous gesture recognition real result or a mapping relation exists between the current gesture recognition real result and the previous gesture recognition real result; before the duration time of the gesture corresponding to the current gesture recognition real result reaches a set value, giving up executing the gesture corresponding to the current gesture recognition real result; when the duration of the gesture corresponding to the current gesture recognition real result reaches a set value, immediately executing the gesture corresponding to the current gesture recognition real result and clearing the duration;
the recognition result executing module is further used for immediately executing the gesture corresponding to the current gesture recognition real result if the current gesture recognition real result is different from the previous gesture recognition real result and no mapping relation exists.
The advantages of the digital reading system based on gestures and the digital reading method based on gestures are the same as the advantages of the digital reading system based on gestures and the digital reading method based on gestures in comparison with the prior art, and are not repeated herein.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a gesture-based digital reading method of the present invention;
fig. 2 is a block diagram of the digital reading system based on gestures according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
As shown in fig. 1, the digital reading method based on gestures of the present embodiment includes:
the method comprises the following steps that firstly, all user-defined gestures are obtained, one of the user-defined gestures is used as an initial gesture, each of the rest gestures is sequentially used as a target gesture, and a gesture recognition experiment process for converting the initial gesture into each target gesture is sequentially carried out;
secondly, acquiring a gesture recognition experiment result in the gesture recognition experiment process, setting gestures except for an initial gesture and a target gesture in the gesture recognition experiment result as transition gestures, and establishing a mapping relation between the initial gesture and all the transition gestures;
thirdly, continuously acquiring a real gesture recognition result when the digital reading is carried out;
fourthly, if the current gesture recognition real result is the same as the last gesture recognition real result or a mapping relation exists, acquiring the duration time of the gesture corresponding to the current gesture recognition real result; before the duration time of the gesture corresponding to the current gesture recognition real result reaches a set value, giving up executing the gesture corresponding to the current gesture recognition real result; when the duration of the gesture corresponding to the current gesture recognition real result reaches a set value, immediately executing the gesture corresponding to the current gesture recognition real result and resetting the duration;
and fifthly, if the current gesture recognition real result is different from the last gesture recognition real result and no mapping relation exists, immediately executing the gesture corresponding to the current gesture recognition real result.
In the first step, the user-defined gesture comprises a no-gesture and an effective gesture, the effective gesture comprises an entry gesture, a page turning gesture, an exit gesture and the like, and the gesture of the user without the effective gesture range is the no-gesture. Assuming that a gesture of extending one finger represents a gesture A, a gesture of extending two fingers represents a gesture B, a gesture of extending three fingers represents a gesture C, a gesture of extending four fingers represents a gesture D, and a gesture of extending five fingers represents a gesture E, a valid gesture is defined to include ABCDE.
Step two, if the starting gesture is no gesture: the target gesture is A, and it can be thought that excessive gestures do not exist in the gesture recognition experiment results of the gesture recognition experiment process from no gesture to A; the target gesture is B, and it can be thought that in the gesture recognition experiment process from the no-gesture to B, because the situation that one finger is stretched out first may be experienced in the process of stretching out two fingers, the gesture A may appear, and if the gesture A appears in the gesture recognition experiment result, the gesture A is taken as a transition gesture, and the mapping relation between the no-gesture and the gesture A is established; the target gesture is E, and it is conceivable that, in the gesture recognition experiment process from the non-gesture to E, since the situation that one finger, two fingers, three fingers or four fingers are firstly stretched may be experienced in the process of stretching out five fingers, any one of the gestures ABCD may occur, and if the gesture ABCD occurs in the gesture recognition experiment result, the ABCD is used as a transition gesture, and the mapping relationship between the non-gesture and the ABCD is established. By analogy, when the starting gesture is a no-gesture, the mapping relations between the starting gesture and all transition gestures, including the mapping relations between the no-gesture and a, between the no-gesture and B, between the no-gesture and C, and between the no-gesture and D, can be finally obtained.
If the initial gesture is A: the target gesture is gesture-free, and it is conceivable that excessive gestures do not exist in gesture recognition experiment results in the gesture-free gesture recognition experiment process from A to A; the target gesture is B, and it can be thought that excessive gestures do not exist in the gesture recognition experiment results from A to B; the target gesture is E, and it is conceivable that, in the gesture recognition experiment process from a to E, since the situation that two fingers, three fingers or four fingers are firstly stretched may be experienced in the process from one finger to stretching five fingers, any one of the gestures BCD may occur, and if the gesture BCD occurs in the gesture recognition experiment result, the BCD is used as a transition gesture to establish the mapping relationship between a and BCD. By analogy, when the starting gesture is a, the mapping relationship between the starting gesture and all transition gestures can be finally obtained, including the mapping relationship between a and B, A and C, A and D.
Finally, through repeated gesture recognition experiment processes, big data statistics can be carried out, and only gesture E does not have a mapping relation for the user-defined gesture consisting of the gesture-free gesture and the ABCDE.
In the traditional digital reading method based on gestures, because the transition gesture cannot be judged, the execution of the transition gesture cannot be abandoned. In this embodiment, through the gesture recognition experiment process formed by any two custom gestures, the gesture with the mapping relationship and the gesture without the mapping relationship in the custom gestures can be finally determined, that is, the transition gesture is recognized.
And thirdly, the real gesture recognition result is a gesture recognition result in the actual reading process, the time is recorded in each gesture recognition real result, and the duration of each gesture can be calculated.
The fourth step and the fifth step are parallel steps. In the fifth step, if the current gesture recognition real result is different from the previous gesture recognition real result and no mapping relation exists, taking the current gesture recognition real result as E as an example, because the gesture E does not have a mapping relation, the gesture E cannot be a transition gesture, that is, the gesture E must be a gesture which needs to be immediately executed by the user, the gesture corresponding to the current gesture recognition real result is immediately executed at this time, the reading requirement of the user can be immediately responded, and the influence of time delay on the user experience is avoided.
In the fourth step, if the current gesture recognition real result is the same as the previous gesture recognition real result, because the embodiment immediately executes the current gesture recognition real result and clears the duration when the duration of the gesture corresponding to the current gesture recognition real result reaches the set value, the current gesture recognition real result is not executed for the second time within the set value (such as 1S) before the duration of the gesture corresponding to the current gesture recognition real result reaches the set value, so that the execution of the gesture corresponding to the current gesture recognition real result is abandoned before the duration of the gesture corresponding to the current gesture recognition real result reaches the set value, and the continuous execution of the same gesture within a short time can be avoided. If the mapping relationship exists between the current gesture recognition real result and the last gesture recognition real result, two situations exist: the real result of the current gesture recognition is a gesture which needs to be executed immediately by the user, and the real result of the current gesture recognition is a transition gesture which does not need to be executed. Since it is not possible to determine which situation is at this time, in order to avoid the influence of the transition gesture that does not need to be performed, it is necessary to confirm the transition gesture that does not need to be performed. In general, the duration of the transition gesture which does not need to be executed is less than 1S, for example, when the user changes from the gesture a to the gesture C, the transition gesture B which does not need to be executed may appear in the middle, so that the transition gesture which does not need to be executed can be recognized by obtaining the duration of the gesture corresponding to the current gesture recognition real result. If the duration time of the gesture corresponding to the current gesture recognition real result does not reach a set value, the situation is the second situation, the current gesture recognition real result is a transition gesture which does not need to be executed, and therefore the execution of the gesture corresponding to the current gesture recognition real result is abandoned, the transition gesture which does not need to be executed can be avoided, and misoperation is avoided; if the duration of the gesture corresponding to the current gesture recognition real result reaches the set value, the first situation is shown at this time, the current gesture recognition real result is the gesture which needs to be executed immediately by the user, and since it cannot be determined which situation is in this time when timing starts, the user requirement is not executed immediately in this embodiment, but the user requirement is executed immediately when the duration of the gesture corresponding to the current gesture recognition real result reaches the set value, although a delay with the same duration as the set value is caused, the execution of the transition gesture which does not need to be executed can be avoided.
As can be seen from the above, the current gesture recognition real result is a non-transitional gesture, and the current gesture recognition real result is immediately executed when the gesture recognition real results of two consecutive times are different, so that the reading requirement of the user can be immediately responded, and the influence on the user experience due to time delay is avoided; the current gesture recognition real result is a transition gesture or the gesture recognition real results of two consecutive times are the same, and before the duration time of the corresponding gesture reaches a set value, the gesture corresponding to the current gesture recognition real result is abandoned to be executed, although delay to customer requirements may be caused, the continuous execution of the same gesture in a short time and the execution of transition gestures which are not required to be executed can be avoided, and misoperation of reading behaviors is avoided.
In this embodiment, gesture recognition is required during the gesture recognition experiment process or formal reading, and accurate and efficient gesture recognition has great significance for smooth reading and fast skip of reading behaviors. In this embodiment, preferably, the step of obtaining the gesture recognition experiment result or the gesture recognition real result includes:
acquiring a video image and segmenting a gesture image from a background of the video image; binarizing the segmented gesture image, and converting the segmented gesture image into an image with a specified size; extracting HOG features and Hu moments of the image, and combining the HOG features and the Hu moments into a fusion feature vector; using a PCA algorithm to perform dimensionality reduction on the fusion feature vector; and finishing the classification of the fusion feature vectors after the dimensionality reduction treatment by using the trained SVM classifier to obtain a gesture recognition result.
The method for acquiring the video image and segmenting the gesture image from the background of the video image specifically comprises the following steps: acquiring a video image, tracking gesture movement, and acquiring the area of a moving hand; and carrying out skin color segmentation on the region, and obtaining a gesture image by adopting boundary scanning on the binary segmentation image. Therefore, the gesture image can be effectively segmented from the background by utilizing the motion and skin color information, the area where the hand is located is obtained through gesture motion tracking, and the skin color segmentation is only carried out on the area where the hand is located, so that the cost of program calculation is reduced. And binarizing the segmented gesture image, and converting the binary gesture image into an image with a specified size, wherein the purpose is to keep the feature dimension of the calculated image consistent. Assuming that the fused feature vector is a high-dimensional feature vector of M × N, where N is the dimension of the feature and M is the total number of samples, performing a dimensionality reduction process on the fused feature vector by using a PCA algorithm, which specifically includes: performing sign normalization processing on the fusion feature vector; calculating a covariance matrix of the sample characteristics; calculating eigenvalues and eigenvectors of a covariance matrix by using a singular value decomposition algorithm, assuming that U is all eigenvectors of the covariance matrix obtained by calculation, each column is an eigenvector, the eigenvectors are sorted according to corresponding eigenvalues, the dimension of the U is N x N, the U is also called a dimension reduction matrix, dimension reduction can be performed on a sample by using the U, the default U contains all eigenvectors of the covariance matrix, if the sample is to be reduced to K dimension, the front K column of the U can be selected, and the Uk can be used for reducing the dimension of the sample to K dimension; and after the dimension reduction matrix is obtained, mapping the sample to a low-dimensional space through the dimension reduction matrix. Thus, the dimensionality of the fused feature vectors can be reduced using the PCA algorithm, resulting in reduced computational complexity.
Further, in this embodiment, preferably, the step of obtaining the gesture recognition experiment result or the gesture recognition real result further includes: and if the results of SVM classification continuous multi-frame gesture images are consistent, adding the multi-frame binarized gesture images with consistent results into an image queue which is classified correctly, acquiring the binarized gesture image next time, carrying out phase comparison on the gesture image and each gesture image in the image queue, if the repeated area accounts for more than 95% of the area of all the gesture images and the self image in the image queue, not classifying the gesture images, representing the recognition result of the gesture image by using the classification of the image queue, and adding the gesture image into the image queue. The classified correct gesture images in the image queue all correspond to the same gesture, such as gesture a, and if the next input gesture image is more than 95% of the area of the multiple images in the image queue and the area of the image per se, the next input gesture image and the gesture image in the image queue all correspond to the same gesture. And taking the gesture image and each gesture image in the image queue, actually taking the gesture image and all image queues, if the gesture A corresponding to the gesture image is not known in advance, taking the image queue corresponding to the gesture image and the image queue corresponding to the gesture B, and taking the image queues corresponding to the gesture E and the image queues corresponding to the gesture A. Therefore, for the same gesture, the type of the gesture can be recognized in a gesture image contrast mode, SVM classification does not need to be repeatedly carried out, and workload of SVM classification is reduced. In addition, every time a frame of gesture image with the same gesture corresponding to the image queue is added, the gesture image is added to the image queue, so that more and more gesture images are in the image queue, the probability that the repeated area of the next acquired gesture image and all the gesture images in the image queue accounts for more than 95% is more difficult, and the error rate of recognizing the gesture type in a gesture image comparison mode is lower and lower. In addition, in this embodiment, all image queues corresponding to gestures such as ABCDE and the like can be gradually established, SVM classification is required in the process of establishing the image queues, SVM classification is no longer required in most cases after the image queues corresponding to all gestures are established, and the gesture types are identified in a gesture-image comparison manner, so that the workload of SVM classification is finally greatly reduced.
The present embodiment further provides a digital reading system based on gestures, including:
the gesture recognition experiment module is used for acquiring all the custom gestures, taking one of the custom gestures as an initial gesture, taking each of the rest gestures as a target gesture in sequence, and performing a gesture recognition experiment process for converting the initial gesture into each target gesture in sequence;
the mapping relation establishing module is used for acquiring a gesture recognition experiment result in the gesture recognition experiment process, setting gestures except an initial gesture and a target gesture in the gesture recognition experiment result as transition gestures, and establishing a mapping relation between the initial gesture and all the transition gestures;
the recognition result acquisition module is used for continuously acquiring a gesture recognition real result when digital reading is carried out;
the recognition result execution module is used for acquiring the duration of the gesture corresponding to the current gesture recognition real result if the current gesture recognition real result is the same as the previous gesture recognition real result or a mapping relation exists between the current gesture recognition real result and the previous gesture recognition real result; before the duration time of the gesture corresponding to the current gesture recognition real result reaches a set value, giving up executing the gesture corresponding to the current gesture recognition real result; when the duration of the gesture corresponding to the current gesture recognition real result reaches a set value, immediately executing the gesture corresponding to the current gesture recognition real result and resetting the duration;
the recognition result executing module is further used for immediately executing the gesture corresponding to the current gesture recognition real result if the current gesture recognition real result is different from the previous gesture recognition real result and no mapping relation exists.
The digital reading system of the embodiment can finally determine the gesture with the mapping relation and the gesture without the mapping relation in the user-defined gestures through the gesture recognition experiment process formed by any two user-defined gestures, namely, the transition gesture is recognized; the current gesture recognition real result is a transition gesture or two consecutive gesture recognition real results are the same, and the gesture corresponding to the current gesture recognition real result is abandoned to be executed before the duration time of the corresponding gesture reaches a set value, so that although delay on customer requirements is possibly caused, the continuous execution of the same gesture in a short time and the execution of transition gestures which are not required to be executed can be avoided, and misoperation of reading behaviors is avoided; the current gesture recognition real result is a non-transitional gesture, and when the gesture recognition real results of two consecutive times are different, the current gesture recognition real result is immediately executed, so that the reading requirement of a user can be immediately responded, and the influence on the user experience caused by time delay is avoided.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (8)
1. A method of gesture-based digital reading, comprising:
acquiring all custom gestures, taking one of the custom gestures as an initial gesture, taking each of the rest gestures as a target gesture in sequence, and performing a gesture recognition experiment process of converting the initial gesture into each target gesture in sequence; the gesture representing gesture A stretching out one finger, the gesture representing gesture B stretching out two fingers, the gesture representing gesture C stretching out three fingers, the gesture representing gesture D stretching out four fingers and the gesture representing gesture E stretching out five fingers are defined to comprise ABCDE;
acquiring a gesture recognition experiment result in a gesture recognition experiment process, setting gestures except an initial gesture and a target gesture in the gesture recognition experiment result as transition gestures, and establishing a mapping relation between the initial gesture and all the transition gestures; if the gesture ABCD appears in the gesture recognition experiment result, the ABCD is used as a transition gesture, and a mapping relation between the initial gesture and the ABCD is established;
continuously acquiring a real gesture recognition result when digital reading is carried out;
if the current gesture recognition real result is the same as the last gesture recognition real result or a mapping relation exists, acquiring the duration of the gesture corresponding to the current gesture recognition real result; before the duration time of the gesture corresponding to the current gesture recognition real result reaches a set value, giving up executing the gesture corresponding to the current gesture recognition real result; when the duration of the gesture corresponding to the current gesture recognition real result reaches a set value, immediately executing the gesture corresponding to the current gesture recognition real result and resetting the duration;
and if the current gesture recognition real result is different from the last gesture recognition real result and no mapping relation exists, immediately executing the gesture corresponding to the current gesture recognition real result.
2. The gesture-based digital reading method of claim 1, wherein the set value is 1S.
3. The gesture-based digital reading method according to claim 1, wherein the step of obtaining the experimental result of gesture recognition or the real result of gesture recognition comprises:
acquiring a video image and segmenting a gesture image from the background of the video image;
binarizing the segmented gesture image, and converting the segmented gesture image into an image with a specified size;
extracting HOG features and Hu moments of the image, and combining the HOG features and the Hu moments into a fusion feature vector;
using a PCA algorithm to perform dimensionality reduction on the fusion feature vector;
and finishing the classification of the fusion feature vectors after the dimensionality reduction treatment by using the trained SVM classifier to obtain a gesture recognition result.
4. The method of gesture-based digital reading according to claim 3, wherein the step of capturing a video image and segmenting the gesture image from the background of the video image comprises:
acquiring a video image, tracking gesture movement, and acquiring the area of a moving hand;
and carrying out skin color segmentation on the area where the hand is located, and obtaining a gesture image by adopting boundary scanning on the binary segmented image.
5. The gesture-based digital reading method according to claim 3, wherein the step of performing dimension reduction processing on the fused feature vectors using the PCA algorithm comprises:
performing sign normalization processing on the fusion feature vector;
calculating a covariance matrix of the sample characteristics;
calculating the eigenvalue and the eigenvector of the covariance matrix by adopting a singular value decomposition algorithm to obtain a dimension reduction matrix;
the samples are mapped onto a low-dimensional space by a dimension reduction matrix.
6. The gesture-based digital reading method according to claim 3, wherein the step of obtaining the experimental result of gesture recognition or the real result of gesture recognition further comprises:
and if the results of SVM classification continuous multi-frame gesture images are consistent, adding the multi-frame binarized gesture images with consistent results into an image queue which is classified correctly, acquiring the binarized gesture image next time, carrying out phase comparison on the gesture image and each gesture image in the image queue, and if the repeated area accounts for more than 95% of the area of all the gesture images and the image area of the gesture image per se in the image queue, not classifying the gesture image, and representing the recognition result of the gesture image by using the classification of the image queue.
7. The gesture-based digital reading method according to claim 6, wherein the step of obtaining the experimental result of gesture recognition or the real result of gesture recognition further comprises:
and adding the gesture images which are not subjected to SVM classification into the corresponding image queue.
8. A gesture-based digital reading system, comprising:
the gesture recognition experiment module is used for acquiring all the custom gestures, taking one of the custom gestures as an initial gesture, taking each of the rest gestures as a target gesture in sequence, and performing a gesture recognition experiment process for converting the initial gesture into each target gesture in sequence; the gesture representing gesture A stretching out one finger, the gesture representing gesture B stretching out two fingers, the gesture representing gesture C stretching out three fingers, the gesture representing gesture D stretching out four fingers and the gesture representing gesture E stretching out five fingers are defined to comprise ABCDE;
the mapping relation establishing module is used for acquiring a gesture recognition experiment result in the gesture recognition experiment process, setting gestures except an initial gesture and a target gesture in the gesture recognition experiment result as transition gestures, and establishing a mapping relation between the initial gesture and all the transition gestures; if the gesture ABCD appears in the gesture recognition experiment result, the ABCD is used as a transition gesture, and a mapping relation between the initial gesture and the ABCD is established;
the recognition result acquisition module is used for continuously acquiring a gesture recognition real result when digital reading is carried out;
the recognition result execution module is used for acquiring the duration of the gesture corresponding to the current gesture recognition real result if the current gesture recognition real result is the same as the previous gesture recognition real result or a mapping relation exists between the current gesture recognition real result and the previous gesture recognition real result; before the duration time of the gesture corresponding to the current gesture recognition real result reaches a set value, giving up executing the gesture corresponding to the current gesture recognition real result; when the duration of the gesture corresponding to the current gesture recognition real result reaches a set value, immediately executing the gesture corresponding to the current gesture recognition real result and resetting the duration;
the recognition result executing module is further used for immediately executing the gesture corresponding to the current gesture recognition real result if the current gesture recognition real result is different from the previous gesture recognition real result and no mapping relation exists.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011343542.0A CN112486317B (en) | 2020-11-26 | 2020-11-26 | Digital reading method and system based on gestures |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011343542.0A CN112486317B (en) | 2020-11-26 | 2020-11-26 | Digital reading method and system based on gestures |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112486317A CN112486317A (en) | 2021-03-12 |
| CN112486317B true CN112486317B (en) | 2022-08-09 |
Family
ID=74934788
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011343542.0A Active CN112486317B (en) | 2020-11-26 | 2020-11-26 | Digital reading method and system based on gestures |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486317B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113033398B (en) * | 2021-03-25 | 2022-02-11 | 深圳市康冠商用科技有限公司 | Gesture recognition method and device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105278763A (en) * | 2015-05-28 | 2016-01-27 | 维沃移动通信有限公司 | Gesture recognition method and apparatus capable of preventing mistaken touch |
| CN106484108A (en) * | 2016-09-30 | 2017-03-08 | 天津大学 | Chinese characters recognition method based on double vision point gesture identification |
| CN109725723A (en) * | 2018-12-29 | 2019-05-07 | 百度在线网络技术（北京）有限公司 | Gestural control method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8418085B2 (en) * | 2009-05-29 | 2013-04-09 | Microsoft Corporation | Gesture coach |
| US10156909B2 (en) * | 2015-04-17 | 2018-12-18 | Mitsubishi Electric Corporation | Gesture recognition device, gesture recognition method, and information processing device |
-
2020
- 2020-11-26 CN CN202011343542.0A patent/CN112486317B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105278763A (en) * | 2015-05-28 | 2016-01-27 | 维沃移动通信有限公司 | Gesture recognition method and apparatus capable of preventing mistaken touch |
| CN106484108A (en) * | 2016-09-30 | 2017-03-08 | 天津大学 | Chinese characters recognition method based on double vision point gesture identification |
| CN109725723A (en) * | 2018-12-29 | 2019-05-07 | 百度在线网络技术（北京）有限公司 | Gestural control method and device |
Non-Patent Citations (1)
| Title |
|---|
| 李东东.实时动态手势动作分割与研究.《电光与控制》.2020,第27卷(第7期), * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112486317A (en) | 2021-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Abu Farha et al. | When will you do what?-anticipating temporal occurrences of activities | |
| Makhmudkhujaev et al. | Facial expression recognition with local prominent directional pattern | |
| US10002290B2 (en) | Learning device and learning method for object detection | |
| Mita et al. | Discriminative feature co-occurrence selection for object detection | |
| US8842883B2 (en) | Global classifier with local adaption for objection detection | |
| JP4858612B2 (en) | Object recognition system, object recognition method, and object recognition program | |
| CN109033955B (en) | Face tracking method and system | |
| Ribeiro et al. | Hand Image Segmentation in Video Sequence by GMM: a comparative analysis | |
| Zakaria et al. | Hierarchical skin-adaboost-neural network (h-skann) for multi-face detection | |
| Samad et al. | Extraction of the minimum number of Gabor wavelet parameters for the recognition of natural facial expressions | |
| Zhao et al. | Applying contrast-limited adaptive histogram equalization and integral projection for facial feature enhancement and detection | |
| Agha et al. | A comprehensive study on sign languages recognition systems using (SVM, KNN, CNN and ANN) | |
| CN112486317B (en) | Digital reading method and system based on gestures | |
| CN114519877A (en) | Face recognition method, face recognition device, computer equipment and storage medium | |
| JP4749884B2 (en) | Learning method of face discriminating apparatus, face discriminating method and apparatus, and program | |
| Kiran et al. | Facial expression classification using Support Vector Machine based on bidirectional Local Binary Pattern Histogram feature descriptor | |
| WO2009096208A1 (en) | Object recognition system, object recognition method, and object recognition program | |
| Li et al. | Ming-HsuanYang,“ | |
| JP2006244385A (en) | Face-discriminating apparatus, program and learning method for the apparatus | |
| Ganakwar et al. | Face detection using logit boost algorithm with YCbCr color space | |
| Pujol et al. | Face recognition using a hybrid SVM–LBP approach and the Indian movie face database | |
| CN112232217B (en) | Gesture recognition system | |
| JP7302719B2 (en) | Information processing device, information processing method, and information processing program | |
| CN109583262B (en) | Adaptive system and method for object detection | |
| Wilkowski12 | An efficient system for continuous hand-posture recognition in video sequences |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 04, Floor 2, Building 1, HongT Fashion Creative Street, No. 8, Jiangwang Road, Jianghan Economic and Technological Development Zone, Jianghan District, Wuhan, Hubei Province, 430000 Patentee after: Hubei Dingsen Intelligent Technology Co.,Ltd. Address before: 430000 room 08, 3 / F, building 5, fashion creative block, No.8, JiangWang Road, Jianghan Economic Development Zone, Jianghan District, Wuhan City, Hubei Province Patentee before: Hubei Dingsen Intelligent Technology Co.,Ltd. |