CN105160323A - Gesture identification method - Google Patents
Gesture identification method Download PDFInfo
- Publication number
- CN105160323A CN105160323A CN201510563293.9A CN201510563293A CN105160323A CN 105160323 A CN105160323 A CN 105160323A CN 201510563293 A CN201510563293 A CN 201510563293A CN 105160323 A CN105160323 A CN 105160323A
- Authority
- CN
- China
- Prior art keywords
- hand
- sequence
- point
- user
- gesture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000012360 testing method Methods 0.000 claims description 42
- 210000004247 hand Anatomy 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 34
- 238000013507 mapping Methods 0.000 claims description 26
- 230000008878 coupling Effects 0.000 claims description 14
- 238000010168 coupling process Methods 0.000 claims description 14
- 238000005859 coupling reaction Methods 0.000 claims description 14
- 210000000707 wrist Anatomy 0.000 claims description 10
- 244000060701 Kaempferia pandurata Species 0.000 claims description 9
- 235000016390 Uvaria chamae Nutrition 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 9
- 241000905957 Channa melasoma Species 0.000 claims description 6
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 238000003384 imaging method Methods 0.000 claims description 4
- 238000003708 edge detection Methods 0.000 claims description 3
- 230000009471 action Effects 0.000 description 39
- 230000008859 change Effects 0.000 description 9
- 230000013011 mating Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
Landscapes
- Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a gesture identification method including that a detected image sequence including depth information of the hands of a user is acquired; based on the image depth information and image color information, the hand contour of the user can be detected from every frame of the detected image sequence; for each hand of the user, the detected characteristic point sequence of the hand is determined in the every frame of the detected image sequence by means of a preset hand structure template; for each hand of the user, the matching sequence of the detected characteristic point sequence of the hand is determined in a plurality of preset characteristic point sequences, and the motion names and positions of the hand are determined based on the matching sequence; and gestures matched with the motion names and positions of the hands of the user are selected from a preset gesture table to be the gesture identification result of the detected image sequence. The gesture identification method can accurately identify the gestures of users, and is high in identification precision and fast in identification speed.
Description
Technical field
The present invention relates to signal processing technology, particularly relate to a kind of gesture identification method.
Background technology
Along with the development of science and technology, such as the electronic equipment of notebook computer, mobile phone, panel computer etc. has increasing function gradually, and gesture identification one of this various functions just.
At present, the algorithm that the identification processing procedure of existing Gesture Recognition utilizes is comparatively complicated, not only expends time in, and identifies that the precision of gesture is lower, accuracy is poor thus.In addition, the real-time based on the existing Gesture Recognition of pattern-recognition is poor, is not suitable for the occasion of real-time interaction demand.
Summary of the invention
Give hereinafter about brief overview of the present invention, to provide about the basic comprehension in some of the present invention.Should be appreciated that this general introduction is not summarize about exhaustive of the present invention.It is not that intention determines key of the present invention or pith, and nor is it intended to limit the scope of the present invention.Its object is only provide some concept in simplified form, in this, as the preorder in greater detail discussed after a while.
Given this, the invention provides a kind of gesture identification method, with at least solve existing for existing Gesture Recognition gesture identification precision ground, poor accuracy problem.
According to an aspect of the present invention, provide a kind of gesture identification method, described gesture identification method comprises: the testing image sequence of step one, the user hand of acquisition containing depth information; Step 2, according to image depth information and image color information, in every two field picture of described testing image sequence, detect the hand profile of described user; Step 3, every hand for described user, utilize the hand structure template preset, determine the characteristic point sequence to be measured of this hand in every two field picture of described testing image sequence; Step 4, every hand for described user, determine the matching sequence of the characteristic point sequence to be measured of this hand, to determine denomination of dive and the position of this hand according to described matching sequence in multiple default characteristic point sequence; Step 5, the gesture that selection matches with denomination of dive and the position of described user's both hands in default gesture table, as the gesture identification result of described testing image sequence.
Further, step 3 can comprise: step 3 one, every hand for described user, utilizes the hand structure template preset, determines a predetermined number unique point of this hand in the hand profile of every two field picture of described testing image sequence; Step 3 two, every hand for described user, utilize the predetermined number unique point that this hand is corresponding in each two field picture of described testing image sequence, obtain the characteristic point sequence to be measured of this hand.
Further, step one can comprise: by catching the image of the described user's hand in predetermined imaging region, obtains Detection Method in Optical Image Sequences
and infrared image sequence
for the pixel value at described Detection Method in Optical Image Sequences i-th two field picture coordinate (x, y) place, and
for the pixel value at described infrared image sequence i-th two field picture coordinate (x, y) place, obtain the image sequence extracting user's both hands information according to following formula:
Wherein, α, β, λ are parameter preset threshold value,
for the image sequence containing user's both hands of depth information obtained, as described testing image sequence, i=1,2 ..., the number of image frames that M, M comprise for described testing image sequence.
Further, step 2 can comprise: for described testing image sequence
in every two field picture
this two field picture of color combining information deletion
in noise spot and non-area of skin color, utilize edge detection operator E () to the image obtained after deleting described noise spot and described non-area of skin color
carry out rim detection, obtain edge image
described edge image
be the image only comprising described user's hand profile.
Further, step 3 one can comprise: the every two field picture for described testing image sequence performs following process: find the finger tip point in this outline line according to the curvature in the outline line of this image and refer to root articulation point; Using described finger tip point as setting base, mate each finger root articulation point singly referred to, obtain the benchmark of each length singly referred to as scaling; Based on the position of described finger tip point and described finger root articulation point and each length singly referred to, convergent-divergent and deformation are carried out to corresponding described hand structure template, obtained each articulations digitorum manus unique point and the wrist mid point unique point of every hand by coupling; Wherein, described hand structure template comprises left-handed configuration template and right hand configurations template, and described left-handed configuration template and right hand configurations template comprise separately: the fingertip characteristic point of each finger, each articulations digitorum manus unique point, topological relation respectively between finger root joint characteristic point, wrist mid point unique point and each unique point.
Further, step 4 can comprise: step 4 one, characteristic point sequence to be measured for every hand, be divided into multiple subsequence according to schedule time window by this characteristic point sequence to be measured, and obtain mean place corresponding to each subsequence; Step 4 two, for each subsequence corresponding to every hand, this subsequence is mated respectively with each in described multiple default characteristic point sequence, to select in described multiple default characteristic point sequence with the matching degree of this subsequence higher than the matching threshold preset and maximum default characteristic point sequence, as the matching sequence of this subsequence; Step 4 three, the denomination of dive that mean place corresponding for each subsequence is corresponding with the matching sequence of this subsequence to be associated; Step 4 four, for every hand, using the matching sequence of each subsequence corresponding for this hand as multiple matching sequences corresponding to this hand, and using the multiple denominations of dive of each for the plurality of matching sequence self-corresponding denomination of dive as this hand.
Further, step 5 can comprise: step May Day, pre-set following map listing and be used as described default gesture table: the left end of each mapping in this map listing be set title to and the right position of each denomination of dive; The right-hand member of each mapping in this map listing is a gesture HandSignal; Step 5 two, the left end of each mapping in described default gesture table to be mated with the denomination of dive of described user's both hands and position, wherein, the coupling of denomination of dive performs strict coupling, position is then calculate relative position information by user's both hands mean place separately, and then the similarity calculated between this relative position information and the position mapping left end realizes.
The above-mentioned gesture identification method according to the embodiment of the present invention, adopt and first identify that single-handed exercise and then the mode by double-handed exercise identification gesture realize, accurately can identify the gesture of user, accuracy of identification is higher, and recognition speed is very fast.
Above-mentioned gesture identification method of the present invention adopts Hierarchical Design algorithm, and algorithm complex is low, is convenient to realize.
In addition, apply above-mentioned gesture identification method of the present invention, (such as revise when needs change, increase or minimizing etc.) definition to action and/or gesture time, can by means of only adjustment template (namely, the definition of action is changed by revising denomination of dive corresponding to default characteristic point sequence, preset characteristic point sequence and respective action title thereof increase by increasing or reducing, subtract action) and default gesture table is (namely, the definition of gesture is changed by revising multiple actions that in default gesture table, gesture is corresponding, gesture in gesture table is preset and respective action increases by increasing or reducing, subtract gesture), and do not need to change algorithm or re-training sorter, substantially increase the adaptability of algorithm.
In addition, above-mentioned gesture identification method of the present invention real-time, can be applicable to the occasion of real-time interaction demand.
By below in conjunction with the detailed description of accompanying drawing to most preferred embodiment of the present invention, these and other advantage of the present invention will be more obvious.
Accompanying drawing explanation
The present invention can be better understood by reference to hereinafter given by reference to the accompanying drawings description, wherein employs same or analogous Reference numeral in all of the figs to represent identical or similar parts.Described accompanying drawing comprises in this manual together with detailed description below and forms the part of this instructions, and is used for illustrating the preferred embodiments of the present invention further and explaining principle and advantage of the present invention.In the accompanying drawings:
Fig. 1 is the process flow diagram of the example process that gesture identification method of the present invention is shown;
Fig. 2 is the process flow diagram of the example process that the step 3 shown in Fig. 1 is shown;
Fig. 3 is the process flow diagram of the example process that the step 4 shown in Fig. 1 is shown;
Fig. 4 is the process flow diagram of the example process that the step 5 shown in Fig. 1 is shown.
The element that it will be appreciated by those skilled in the art that in accompanying drawing be only used to simple and clear for the purpose of illustrate, and not necessarily to draw in proportion.Such as, in accompanying drawing, the size of some element may be exaggerated relative to other elements, to contribute to improving the understanding to the embodiment of the present invention.
Embodiment
To be described one exemplary embodiment of the present invention by reference to the accompanying drawings hereinafter.For clarity and conciseness, all features of actual embodiment are not described in the description.But, should understand, must make a lot specific to the decision of embodiment in the process of any this practical embodiments of exploitation, to realize the objectives of developer, such as, meet those restrictive conditions relevant to system and business, and these restrictive conditions may change to some extent along with the difference of embodiment.In addition, although will also be appreciated that development is likely very complicated and time-consuming, concerning the those skilled in the art having benefited from present disclosure, this development is only routine task.
At this, also it should be noted is that, in order to avoid the present invention fuzzy because of unnecessary details, illustrate only in the accompanying drawings with according to the closely-related apparatus structure of the solution of the present invention and/or treatment step, and eliminate other details little with relation of the present invention.
The embodiment provides a kind of gesture identification method, this gesture identification method comprises: the testing image sequence of step one, the user hand of acquisition containing depth information; Step 2, according to image depth information and image color information, in every two field picture of described testing image sequence, detect the hand profile of described user; Step 3, every hand for described user, utilize the hand structure template preset, determine the characteristic point sequence to be measured of this hand in every two field picture of described testing image sequence; Step 4, every hand for described user, determine the matching sequence of the characteristic point sequence to be measured of this hand, to determine denomination of dive and the position of this hand according to described matching sequence in multiple default characteristic point sequence; Step 5, the gesture that selection matches with denomination of dive and the position of described user's both hands in default gesture table, as the gesture identification result of described testing image sequence.
Fig. 1 shows the process flow diagram of an example process of gesture identification method of the present invention.The exemplary process of above-mentioned gesture identification method starts from step one.
As shown in Figure 1, in step one, obtain the testing image sequence of the user's hand containing depth information.
According to a kind of implementation, the process of step one can be realized as follows: by catching the image of the user's hand in predetermined imaging region, (such as can utilize the visible light image sensor in depth camera and infrared image sensor) obtains Detection Method in Optical Image Sequences
and infrared image sequence
for the pixel value at Detection Method in Optical Image Sequences i-th two field picture coordinate (x, y) place, and
for the pixel value at infrared image sequence i-th two field picture coordinate (x, y) place, can obtain according to following formula the image sequence extracting user's both hands information:
Wherein, α, β, λ are parameter preset threshold value, these parameter preset threshold values can set based on experience value, also can be determined by the method for test (such as being obtained by the actual sample image training using the depth camera of specific model to collect), repeat no more here.
for the image sequence containing user's both hands of depth information obtained, as above-mentioned testing image sequence.In addition, i=1,2 ..., M, M are number of image frames included in testing image sequence.
It should be noted that, according to the difference (single or two) of the hand quantity that user's gesture uses, the image of catching in predetermined imaging region may be the image comprising user's both hands, also may be the image only comprising user's single hand.In addition, the testing image sequence of acquisition can obtain in a period of time, and this time period can be arranged in advance based on experience value, such as, can be 10 seconds.
Then, in step 2, according to image depth information and image color information, in every two field picture of testing image sequence, detect the hand profile of user.Wherein, the hand profile detected may be both hands profile, also may be singlehanded profile.
According to a kind of implementation, the process of step two can be realized as follows: for testing image sequence
in every two field picture
this two field picture of color combining information deletion
in noise spot and non-area of skin color, utilize edge detection operator E () to the image obtained after erased noise point and non-area of skin color
carry out rim detection, thus obtain edge image
Edge image
be the image only comprising user's hand profile.
Wherein, in the processing procedure of " noise spot in this two field picture of color combining information deletion and non-area of skin color ", the noise spot that existing denoising method is come in deleted image can be utilized, and can computed image be passed through
average obtain area of skin color, then the region outside area of skin color is non-area of skin color, can realize the deletion to non-area of skin color.Such as, image is obtained
average after, to fluctuate a scope in this average, obtain the color gamut comprising this average, when the color value of certain point drops within this color gamut in image, then this point is determined it is colour of skin point, otherwise not think it is colour of skin point; All colour of skin points form area of skin color, and all the other are non-area of skin color.
Thus, by the process of step 2, the hand profile of user can be detected fast, improve speed and the efficiency of whole process.
Then, in step 3, for every hand of user, utilize the hand structure template preset, in every two field picture of testing image sequence, determine the characteristic point sequence to be measured of this hand.
Wherein, hand structure template comprises left-handed configuration template and right hand configurations template, and left-handed configuration template and right hand configurations template comprise the topological relation between a predetermined number unique point and each unique point separately.
In one example in which, left-handed configuration template and right hand configurations template can comprise following 20 separately (as the example of predetermined number, but predetermined number is not limited to 20, also can be the numerical value such as 19,21) individual unique point: the fingertip characteristic point (5) of each finger, each articulations digitorum manus unique point (9), respectively finger root joint characteristic point (5), wrist mid point unique point (1).
According to a kind of implementation, the process in step 3 can be realized by step 3 one as shown in Figure 2 and step 3 two.
As shown in Figure 2, in step 3 one, for every hand of user, above-mentioned default hand structure template can be utilized, respectively the hand profile in every two field picture of testing image sequence is carried out mating, aliging with hand structure template (tiled configuration template and right hand configurations template), obtain predetermined number (the such as 20) unique point in this two field picture hand profile.
Then, in step 3 two, for every hand of user, the predetermined number unique point (i.e. feature point set) that this hand is corresponding in each two field picture of testing image sequence can be utilized, obtain the characteristic point sequence to be measured of this hand.
Like this, carry out by hand structure template and each hand profile obtained before (i.e. hand profile in every two field picture of testing image sequence) process such as mating, the predetermined number unique point in each hand profile can be obtained quickly and accurately.Thereby, it is possible to make subsequent treatment utilize the described predetermined number unique point in these profiles to realize gesture identification further, compared to prior art, improve speed and the accuracy of whole gesture recognition process.
In the prior art, when needing to change (such as revise, increase or minimizing etc.) definition to action according to different application scene, amendment algorithm and re-training sorter is needed; In the present invention, the change that can realize action definition by means of only adjustment action template (namely preset characteristic point sequence), substantially increases the adaptability of algorithm.
In one example in which, the process of step three one can be realized in the following way.
According to the physiological structure feature of mankind's both hands, 20 (example as predetermined number) individual unique point can be got to often only portable.For every two field picture of testing image sequence
perform following process: first, according to this image
in profile curvature of a curve find finger tip point in this outline line and refer to root articulation point; Then using finger tip point as setting base, mate each finger root articulation point singly referred to, obtain the benchmark of each length singly referred to as scaling; Finally based on the finger tip point found and the position referring to root articulation point and the parameter of each length two aspect singly referred to that obtains, convergent-divergent and deformation are carried out to corresponding hand structure template, remaining 10 unique point of every hand are obtained, i.e. each articulations digitorum manus unique point of every hand and wrist mid point unique point by coupling.
Such as, outline line is being looked for
in finger tip point and refer in the process of root articulation point, can using salient point maximum for its mean curvature as finger tip point, using concave point maximum for curvature as webs minimum point, and be the unit length that this finger tip point is corresponding by the distance definition between each finger tip point to the adjacent webs minimum point of this finger tip point.To every two adjacent webs minimum points, this mid point of 2 is extended again the point of 1/3rd unit lengths (unit length that the finger tip point of unit length now for this reason between 2 is corresponding) toward volar direction, be defined as the finger root articulation point that this finger tip point is corresponding, 3, the centre that can obtain every hand thus refers to root articulation point.In addition, for every hand, root articulation point can be referred to by the head and the tail two obtaining this hand in the process of follow-up convergent-divergent and deformation; Or, also can using the distance between two of this hand webs minimum point that (such as selecting arbitrarily two) is adjacent as finger reference width, then by each for head and the tail two webs minimum points of this hand tangentially, stretch out half finger reference width, the point obtained refers to root articulation point respectively as the head and the tail two of this hand.
It should be noted that, if the salient point found for single hand is more than 5, unnecessary salient point can be removed itself and hand structure template being carried out mate in the process of aliging.
Thus, 20 unique point Pl={pl of left hand corresponding to each two field picture are obtained by such method coupling
1, pl
2..., pl
20and 20 unique point Pr={pr of the right hand
1, pr
2..., pr
20.It should be noted that, if user's gesture only comprises single hand, then what obtained by above coupling is 20 unique points (be called feature point set) of this single hand in every two field picture, i.e. Pl={pl
1, pl
2..., pl
20or Pr={pr
1, pr
2..., pr
20.Wherein, pl
1, pl
2..., pl
20be respectively the position of left hand 20 unique points, and pr
1, pr
2..., pr
20be respectively the position of the right hand 20 unique points.
If user's gesture comprises both hands, then can be obtained the characteristic point sequence { Pl to be measured of left hand by the process of above step 3 one and step 3 two
i, i=1,2 ..., the characteristic point sequence { Pr to be measured of M} and the right hand
i, i=1,2 ..., M}.Wherein, Pl
ifor 20 (example as predetermined number) individual unique point that user's left hand is corresponding in the i-th two field picture of testing image sequence, and Pr
ifor 20 (example as predetermined number) individual unique point that user's right hand is corresponding in the i-th two field picture of testing image sequence.
If user's gesture only comprises single hand, then the every two field picture in the testing image sequence of catching is all the images only comprising this single hand, thus can obtain the characteristic point sequence to be measured of this single hand after process by step 3 one and step 3 two, i.e. { Pl
i, i=1,2 ..., M} or { Pr
i, i=1,2 ..., M}.
Like this, step 4 is as shown in Figure 1 performed after execution of step three.
In step 4, for every hand of user, in multiple default characteristic point sequence, determine the matching sequence of the characteristic point sequence to be measured of this hand, to determine denomination of dive and the position of this hand according to this matching sequence.
Thus, by step 4, utilize the characteristic point sequence to be measured of every hand to mate with each default characteristic point sequence, obtain matching sequence, and then the denomination of dive corresponding according to matching sequence determines the action of this hand fast.
According to a kind of implementation, the process of step four can be realized by step 4 one to step 4 four as shown in Figure 3.
First, preset a hand motion list of file names, comprise basic hand motion, such as: wave, push away, draw, opening and closing, to turn, the template that each action has unique name identification and represents with normalized hand-characteristic point sequence (namely default characteristic point sequence).It should be noted that, for the both hands of user, every hand all has an above-mentioned hand motion list of file names.That is, for left hand, each action that the hand motion list of file names of left hand (being called for short left hand action list of file names) comprises, except having respective title respectively, also has a left hand template (i.e. a default characteristic point sequence of left hand); For the right hand, each action that the hand motion list of file names of the right hand (being called for short right hand action list of file names) comprises, except having respective title respectively, also has a right hand template (i.e. a default characteristic point sequence of the right hand).
Such as, the multiple default characteristic point sequence of single hand is designated as sequence A respectively
1, sequence A
2..., sequence A
h, wherein, the sequence number that above-mentioned multiple default characteristic point sequence that H is this single hand comprise, then in the hand motion list of file names of this single hand: the name identification of action 1 is " waving " and the template of correspondence (namely presetting characteristic point sequence) is sequence A
1; The name identification of action 2 is " pushing away " and the template of correspondence is sequence A
1; The name identification of action H is " turning " and the template of correspondence is sequence A
1.
In step 4 one, for the characteristic point sequence to be measured of every hand, according to schedule time window, this characteristic point sequence to be measured is divided into multiple subsequence, and obtains mean place corresponding to each subsequence.Wherein, the mean place that each subsequence is corresponding can choose specific characteristic point (as wrist mid point, or also can be other unique points) mean place in this subsequence.Wherein, schedule time window is about a singlehanded elemental motion (namely singlehanded hold, the grab) time from start to end, and can set based on experience value, maybe can be determined by the method for test, such as, can be 2.5 seconds.
In one example in which, suppose that characteristic point sequence to be measured gathered in 10 seconds, utilize the time window of 2.5 seconds the characteristic point sequence to be measured of the characteristic point sequence to be measured of left hand and the right hand can be divided into 4 subsequences respectively.With the characteristic point sequence { Pl to be measured of left hand
i, i=1,2 ..., M} is the example (characteristic point sequence { Pr to be measured of the right hand
i, i=1,2 ..., M} is similar with it, no longer describes in detail here), suppose collection 10 two field picture per second, then that characteristic point sequence to be measured is corresponding is 100 two field pictures, i.e. M=100, that is, { Pl
i, i=1,2 ..., M} comprises 100 stack features point set Pl
1, Pl
2..., Pl
100.Like this, by the time window of above-mentioned 2.5 seconds, can by { Pl
i, i=1,2 ..., M} is divided into { Pl
i, i=1,2 ..., 25}, { Pl
i, i=25,26 ..., 50}, { Pl
i, i=51,52 ..., 75} and { Pl
i, i=76,77 ..., 100}4 subsequence, and each corresponding 25 two field pictures of each subsequence, also, each subsequence respectively comprises 25 stack features point sets.Specific characteristic point chooses wrist mid point, with subsequence { Pl
i, i=1,2 ..., 25} is example (its excess-three sub-sequence is similar to its process, no longer describes in detail here), and wrist mid point is at { Pl
i, i=1,2 ..., the position that the 25 stack features points that 25} is corresponding are concentrated is respectively position p
1, p
2..., p
25so wrist mid point is at subsequence { Pl
i, i=1,2 ..., the mean place in 25} is (p
1+ p
2+ ... + p
25)/25, as subsequence { Pl
i, i=1,2 ..., the mean place that 25} is corresponding.
Then, in step 4 two, for each subsequence that every hand is corresponding, this subsequence is mated respectively with each in multiple default characteristic point sequence, in multiple default characteristic point sequence, select with the matching degree of this subsequence that (this matching threshold can set based on experience value higher than the matching threshold preset, or also can be determined by the method for test) and maximum that default characteristic point sequence, as the matching sequence of this subsequence.Wherein, the similarity between subsequence and default characteristic point sequence can be calculated, be used as matching degree therebetween.
It should be noted that, for each subsequence, and not necessarily can find the matching sequence that this subsequence is corresponding in multiple default characteristic point sequence.When certain subsequence for single hand does not find its matching sequence, then the matching sequence of this subsequence is designated as " sky ", but the mean place of this subsequence can not be " sky ".According to a kind of implementation, if the matching sequence of subsequence is " sky ", then the mean place of this subsequence is set to " sky "; According to another kind of implementation, if the matching sequence of subsequence is " sky ", the mean place of this subsequence is the actual average position of specifying unique point in this subsequence; According to other a kind of implementations, if the matching sequence of subsequence is " sky ", the mean place of this subsequence is set to "+∞ ".
In addition, according to a kind of implementation, if there is not specific characteristic point (also namely there is not the actual average position of this specific characteristic point) in subsequence, the mean place of this subsequence can be set to "+∞ ".
Then, in step 4 three, the denomination of dive that mean place corresponding for each subsequence is corresponding with the matching sequence of this subsequence is associated.
In step 4 four, for every hand, can using the matching sequence of each subsequence corresponding for this hand as multiple matching sequences corresponding to this hand, and using the multiple denominations of dive of each for the plurality of matching sequence self-corresponding denomination of dive (in chronological order after sequence) as this hand.
Such as, suppose that for multiple subsequences of the characteristic point sequence to be measured of left hand be { Pl
i, i=1,2 ..., 25}, { Pl
i, i=25,26 ..., 50}, { Pl
i, i=51,52 ..., 75} and { Pl
i, i=76,77 ..., 100}, finds { Pl in multiple default characteristic point sequence leftward respectively
i, i=1,2 ..., 25}, { Pl
i, i=25,26 ..., 50}, { Pl
i, i=51,52 ..., the matching sequence of 75} is followed successively by Pl
1', Pl
2', Pl
3', and do not find { Pl
i, i=76,77 ..., the matching sequence of 100}.Suppose Pl
1', Pl
2', Pl
3' denomination of dive corresponding in action list of file names respectively leftward is " waving ", " pushing away ", " drawing ", { Pl
i, i=1,2 ..., 25}, { Pl
i, i=25,26 ..., 50}, { Pl
i, i=51,52 ..., 75} and { Pl
i, i=76,77 ..., 100} mean place is separately respectively pm
1, pm
2, pm
3and pm
4, then denomination of dive and the position of the left hand obtained thus comprise: " waving " (position pm
1); " push away " (position pm
2); " draw " (position pm
3); " sky " (position " pm
4").Should be noted that and be, in different embodiments, pm
4may be actual position value, also may be " sky " or "+∞ " etc.
Thus, by the process of step 4 one to step 4 four, multiple denominations of dive corresponding to user every hand can be obtained (as the denomination of dive of this hand, that is, the denomination of dive of this hand), and each denomination of dive is associated with a mean place (as the position of this hand, " position of this hand " comprises one or more mean place, and quantity is identical with the quantity of denomination of dive) respectively.Compared to only identifying the recognition technology of individual part as gesture, the respective multiple action of the both hands that above-mentioned process identifies and position, provide array mode more flexibly, make the accuracy of identification of gesture higher on the one hand, the gesture making it possible on the other hand identify is more various, abundant.
Finally, in step 5, in default gesture table, select the gesture matched with the denomination of dive of user's both hands and position, as the gesture identification result of testing image sequence.
According to a kind of implementation, can by realizing the process of step five on step May Day as shown in Figure 4 and step 5 two.
As shown in Figure 4, step May Day, predefine one from two manually do and two, position key element to the map listing of gesture as default gesture table: the left end of each mapping be set title to and the right position of each denomination of dive; The right-hand member of each mapping is a gesture HandSignal.Then, step 5 two is performed.
Wherein, " set title to " comprises multiple denomination of dive pair, and each denomination of dive is to comprising left hand denomination of dive ActName
leftwith right hand denomination of dive ActName
right, the right position of each denomination of dive comprises the relative position of two hands.
Such as, in default gesture table, map one for { (" drawing ", " sky "), (" drawing ", " draw "), (" sky ", " conjunction "), (" sky ", " sky ") (as key element one), { (x
1, y
1), (x
2, y
2), (x
3, y
3), (x
4, y
4) (relative position, as key element two) to the mapping of gesture " switch "; Map two for { (" drawing ", " drawing "), (" opening ", " opening "), (" sky ", " sky "), (" sky ", " sky ") }, { (x
5, y
5), (x
6, y
6), (x
7, y
7), (x
8, y
8) to the mapping of gesture " blast "; Etc..Wherein, each action corresponds to left hand action to the denomination of dive on the left side in (as (" drawing ", " sky ")), and the denomination of dive on the right corresponds to right hand action.
To map one, (x
1, y
1) what represent is that left hand first element " draws " relative position between right hand first element " sky " (namely action is to the relative position of left hand action in (" drawing ", " sky ") and two hands corresponding to right hand action); (x
2, y
2) represent be left hand second action " draw " and the right hand second action " draw " between relative position; (x
3, y
3) what represent is relative position between left hand the 3rd action " sky " and the right hand the 3rd action " conjunction "; And (x
4, y
4) what represent is relative position between left hand the 4th action " sky " and the right hand the 4th action " sky ".Elocutionary meaning in other mappings is similar, repeats no more.
In step 5 two, the left end of each mapping in default gesture table is mated with the denomination of dive of user's both hands and position.
Wherein, the coupling of denomination of dive performs strict coupling, also, judges that these two denominations of dive are couplings of verbatim account between two denominations of dive; Position is then calculate relative position information by user's both hands mean place separately, and then (as a similarity threshold can be set, judging that when the similarity calculated is more than or equal to this similarity threshold position is coupling) that the similarity calculated between this relative position information and the position mapping left end realizes.
Such as, suppose to obtain user's both hands denomination of dive separately for { (" drawing ", " drawing "), (" opening " by step 4, " open "), (" sky ", " sky "), (" sky ", " sky "), position is { (x
11, y
12), (x
21, y
22), (x
31, y
32), (x
41, y
42) (corresponding left hand); (x '
11, y '
12), (x '
21, y '
22), (x '
31, y '
32), (x '
41, y '
42) (corresponding left hand).
Then, the left end of the denomination of dive of user's both hands with each mapping in default gesture table is mated.
When mating with mapping one, can draw, the denomination of dive of user's both hands does not mate with the denomination of dive of the left end mapping, therefore ignores mapping one, continues coupling mapping two.
When mating with mapping two, can draw, the denomination of dive of user's both hands mates completely with the denomination of dive of the left end mapping two, and then is mated by the relative position of the position of user's both hands with the left end mapping two.
Carrying out in the process of mating by the position of user's both hands with the relative position of the left end mapping two, the relative position first calculating user's both hands is as follows: (x '
11-x
11, y '
12-y
12), (x '
21-x
21, y '
22-y
22), (x '
31-x
31, y '
32-y
32), (x '
41-x
41, y '
42-y
42) (corresponding left hand).Then, by the above-mentioned relative position of the user's both hands calculated and the relative position { (x mapping two left ends
5, y
5), (x
6, y
6), (x
7, y
7), (x
8, y
8) mate, i.e., calculate (x '
11-x
11, y '
12-y
12), (x '
21-x
21, y '
22-y
22), (x '
31-x
31, y '
32-y
32), (x '
41-x
41, y '
42-y
42) (corresponding left hand) and { (x
5, y
5), (x
6, y
6), (x
7, y
7), (x
8, y
8) between similarity, suppose that the similarity calculated is 95%.In this example embodiment, if similarity threshold is 80%, so judge that the relative position of the user's both hands calculated mates with the relative position mapping two left ends.Thus, in this example embodiment, the result of gesture identification is " blast ".
Above step May Day and step 5 two, by the respective multiple action of both hands and mating between position with prearranged gesture table, determine the gesture of user, the precision of identification is higher; When needing to change (such as revise, increase or minimizing etc.) definition to gesture according to different application scene, do not need amendment algorithm or re-training sorter, the change that can realize definition of gesture by means of only modes such as the gesture title in adjustment prearranged gesture table or denomination of dive corresponding to gesture, substantially increases the adaptability of algorithm.
The above-mentioned gesture identification method according to the embodiment of the present invention, adopt and first identify that single-handed exercise and then the mode by double-handed exercise identification gesture realize, accurately can identify the gesture of user, accuracy of identification is higher, and recognition speed is very fast.
Above-mentioned gesture identification method of the present invention adopts Hierarchical Design algorithm, and algorithm complex is low, is convenient to realize.
In addition, apply above-mentioned gesture identification method of the present invention, when needing to change (such as revise, increase or minimizing etc.) definition to action and/or gesture, can realize (namely by means of only adjustment template and/or default gesture table, changing the definition of action by revising denomination of dive corresponding to default characteristic point sequence, presetting characteristic point sequence and respective action title thereof and increase by increasing or reducing, subtract action; The definition of gesture is changed by revising multiple actions that in default gesture table, gesture is corresponding, gesture in gesture table is preset and respective action increases, subtracts gesture) by increasing or reducing, and do not need to change algorithm or re-training sorter, substantially increase the adaptability of algorithm.
In addition, above-mentioned gesture identification method of the present invention real-time, can be applicable to the occasion of real-time interaction demand.
Although the embodiment according to limited quantity describes the present invention, benefit from description above, those skilled in the art understand, in the scope of the present invention described thus, it is contemplated that other embodiment.In addition, it should be noted that the language used in this instructions is mainly in order to object that is readable and instruction is selected, instead of select to explain or limiting theme of the present invention.Therefore, when not departing from the scope and spirit of appended claims, many modifications and changes are all apparent for those skilled in the art.For scope of the present invention, be illustrative to disclosing of doing of the present invention, and nonrestrictive, and scope of the present invention is defined by the appended claims.
Claims (7)
1. a gesture identification method, is characterized in that, described gesture identification method comprises:
The testing image sequence of step one, the user hand of acquisition containing depth information;
Step 2, according to image depth information and image color information, in every two field picture of described testing image sequence, detect the hand profile of described user;
Step 3, every hand for described user, utilize the hand structure template preset, determine the characteristic point sequence to be measured of this hand in every two field picture of described testing image sequence;
Step 4, every hand for described user, determine the matching sequence of the characteristic point sequence to be measured of this hand, to determine denomination of dive and the position of this hand according to described matching sequence in multiple default characteristic point sequence;
Step 5, the gesture that selection matches with denomination of dive and the position of described user's both hands in default gesture table, as the gesture identification result of described testing image sequence.
2. gesture identification method according to claim 1, is characterized in that, step 3 comprises:
Step 3 one, every hand for described user, utilize the hand structure template preset, determine a predetermined number unique point of this hand in the hand profile of every two field picture of described testing image sequence;
Step 3 two, every hand for described user, utilize the predetermined number unique point that this hand is corresponding in each two field picture of described testing image sequence, obtain the characteristic point sequence to be measured of this hand.
3. gesture identification method according to claim 1 and 2, is characterized in that, step one comprises:
By catching the image of the described user's hand in predetermined imaging region, obtain Detection Method in Optical Image Sequences
and infrared image sequence
for the pixel value at described Detection Method in Optical Image Sequences i-th two field picture coordinate (x, y) place, and
for the pixel value at described infrared image sequence i-th two field picture coordinate (x, y) place, obtain the image sequence extracting user's both hands information according to following formula:
Wherein, α, β, λ are parameter preset threshold value,
for the image sequence containing user's both hands of depth information obtained, as described testing image sequence, i=1,2 ..., the number of image frames that M, M comprise for described testing image sequence.
4. gesture identification method according to claim 1 and 2, is characterized in that, step 2 comprises:
For described testing image sequence
in every two field picture
this two field picture of color combining information deletion
in noise spot and non-area of skin color, utilize edge detection operator E () to the image obtained after deleting described noise spot and described non-area of skin color
carry out rim detection, obtain edge image
Described edge image
be the image only comprising described user's hand profile.
5. gesture identification method according to claim 2, is characterized in that, step 3 one comprises:
Every two field picture for described testing image sequence performs following process: according to the profile curvature of a curve in this image, finds the finger tip point in this outline line and refers to root articulation point; Using described finger tip point as setting base, mate each finger root articulation point singly referred to, obtain the benchmark of each length singly referred to as scaling; Based on the position of described finger tip point and described finger root articulation point and each length singly referred to, convergent-divergent and deformation are carried out to corresponding described hand structure template, obtained each articulations digitorum manus unique point and the wrist mid point unique point of every hand by coupling;
Wherein, described hand structure template comprises left-handed configuration template and right hand configurations template, and described left-handed configuration template and right hand configurations template comprise separately: the fingertip characteristic point of each finger, each articulations digitorum manus unique point, topological relation respectively between finger root joint characteristic point, wrist mid point unique point and each unique point.
6. gesture identification method according to claim 1 and 2, is characterized in that, step 4 comprises:
Step 4 one, characteristic point sequence to be measured for every hand, be divided into multiple subsequence according to schedule time window by this characteristic point sequence to be measured, and obtain mean place corresponding to each subsequence;
Step 4 two, for each subsequence corresponding to every hand, this subsequence is mated respectively with each in described multiple default characteristic point sequence, to select in described multiple default characteristic point sequence with the matching degree of this subsequence higher than the matching threshold preset and maximum default characteristic point sequence, as the matching sequence of this subsequence;
Step 4 three, the denomination of dive that mean place corresponding for each subsequence is corresponding with the matching sequence of this subsequence to be associated;
Step 4 four, for every hand, using the matching sequence of each subsequence corresponding for this hand as multiple matching sequences corresponding to this hand, and using the multiple denominations of dive of each for the plurality of matching sequence self-corresponding denomination of dive as this hand.
7. gesture identification method according to claim 1 and 2, is characterized in that, step 5 comprises:
Step May Day, pre-set following map listing and be used as described default gesture table: the left end of each mapping in this map listing be set title to and the right position of each denomination of dive; The right-hand member of each mapping in this map listing is a gesture;
Step 5 two, the left end of each mapping in described default gesture table to be mated with the denomination of dive of described user's both hands and position, wherein, the coupling of denomination of dive performs strict coupling, position is then calculate relative position information by user's both hands mean place separately, and then the similarity calculated between this relative position information and the position mapping left end realizes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510563293.9A CN105160323B (en) | 2015-09-07 | 2015-09-07 | A kind of gesture identification method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510563293.9A CN105160323B (en) | 2015-09-07 | 2015-09-07 | A kind of gesture identification method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105160323A true CN105160323A (en) | 2015-12-16 |
| CN105160323B CN105160323B (en) | 2018-11-27 |
Family
ID=54801175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510563293.9A Expired - Fee Related CN105160323B (en) | 2015-09-07 | 2015-09-07 | A kind of gesture identification method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105160323B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648068A (en) * | 2016-11-11 | 2017-05-10 | 哈尔滨工业大学深圳研究生院 | Method for recognizing three-dimensional dynamic gesture by two hands |
| CN106650628A (en) * | 2016-11-21 | 2017-05-10 | 南京邮电大学 | Fingertip detection method based on three-dimensional K curvature |
| CN106791763A (en) * | 2016-11-24 | 2017-05-31 | 深圳奥比中光科技有限公司 | A kind of application specific processor shown for 3D with 3D interactions |
| CN106971135A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of slip gesture recognition methods |
| CN106971130A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of gesture identification method using face as reference |
| CN106971131A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of gesture identification method based on center |
| CN107038424A (en) * | 2017-04-20 | 2017-08-11 | 华中师范大学 | A kind of gesture identification method |
| WO2018032700A1 (en) * | 2016-08-16 | 2018-02-22 | 广州视源电子科技股份有限公司 | Method for tracking finger web position and apparatus thereof |
| CN109685037A (en) * | 2019-01-08 | 2019-04-26 | 北京汉王智远科技有限公司 | A kind of real-time action recognition methods, device and electronic equipment |
| CN110298314A (en) * | 2019-06-28 | 2019-10-01 | 海尔优家智能科技（北京）有限公司 | The recognition methods of gesture area and device |
| WO2020078319A1 (en) * | 2018-10-15 | 2020-04-23 | 华为技术有限公司 | Gesture-based manipulation method and terminal device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030156756A1 (en) * | 2002-02-15 | 2003-08-21 | Gokturk Salih Burak | Gesture recognition system using depth perceptive sensors |
| CN102426480A (en) * | 2011-11-03 | 2012-04-25 | 康佳集团股份有限公司 | Human-computer interaction system and real-time gesture tracking processing method thereof |
| CN104281265A (en) * | 2014-10-14 | 2015-01-14 | 京东方科技集团股份有限公司 | Application program control method, application program control device and electronic equipment |
| CN104571482A (en) * | 2013-10-22 | 2015-04-29 | 中国传媒大学 | Digital device control method based on somatosensory recognition |
| CN104598915A (en) * | 2014-01-24 | 2015-05-06 | 深圳奥比中光科技有限公司 | Gesture recognition method and gesture recognition device |
| CN104750397A (en) * | 2015-04-09 | 2015-07-01 | 重庆邮电大学 | Somatosensory-based natural interaction method for virtual mine |
-
2015
- 2015-09-07 CN CN201510563293.9A patent/CN105160323B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030156756A1 (en) * | 2002-02-15 | 2003-08-21 | Gokturk Salih Burak | Gesture recognition system using depth perceptive sensors |
| CN102426480A (en) * | 2011-11-03 | 2012-04-25 | 康佳集团股份有限公司 | Human-computer interaction system and real-time gesture tracking processing method thereof |
| CN104571482A (en) * | 2013-10-22 | 2015-04-29 | 中国传媒大学 | Digital device control method based on somatosensory recognition |
| CN104598915A (en) * | 2014-01-24 | 2015-05-06 | 深圳奥比中光科技有限公司 | Gesture recognition method and gesture recognition device |
| CN104281265A (en) * | 2014-10-14 | 2015-01-14 | 京东方科技集团股份有限公司 | Application program control method, application program control device and electronic equipment |
| CN104750397A (en) * | 2015-04-09 | 2015-07-01 | 重庆邮电大学 | Somatosensory-based natural interaction method for virtual mine |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106971135A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of slip gesture recognition methods |
| CN106971130A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of gesture identification method using face as reference |
| CN106971131A (en) * | 2016-01-14 | 2017-07-21 | 芋头科技（杭州）有限公司 | A kind of gesture identification method based on center |
| WO2018032700A1 (en) * | 2016-08-16 | 2018-02-22 | 广州视源电子科技股份有限公司 | Method for tracking finger web position and apparatus thereof |
| CN106648068A (en) * | 2016-11-11 | 2017-05-10 | 哈尔滨工业大学深圳研究生院 | Method for recognizing three-dimensional dynamic gesture by two hands |
| CN106650628B (en) * | 2016-11-21 | 2021-03-23 | 南京邮电大学 | Fingertip detection method based on three-dimensional K curvature |
| CN106650628A (en) * | 2016-11-21 | 2017-05-10 | 南京邮电大学 | Fingertip detection method based on three-dimensional K curvature |
| CN106791763A (en) * | 2016-11-24 | 2017-05-31 | 深圳奥比中光科技有限公司 | A kind of application specific processor shown for 3D with 3D interactions |
| CN107038424A (en) * | 2017-04-20 | 2017-08-11 | 华中师范大学 | A kind of gesture identification method |
| WO2020078319A1 (en) * | 2018-10-15 | 2020-04-23 | 华为技术有限公司 | Gesture-based manipulation method and terminal device |
| CN109685037B (en) * | 2019-01-08 | 2021-03-05 | 北京汉王智远科技有限公司 | Real-time action recognition method and device and electronic equipment |
| CN109685037A (en) * | 2019-01-08 | 2019-04-26 | 北京汉王智远科技有限公司 | A kind of real-time action recognition methods, device and electronic equipment |
| CN110298314A (en) * | 2019-06-28 | 2019-10-01 | 海尔优家智能科技（北京）有限公司 | The recognition methods of gesture area and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105160323B (en) | 2018-11-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105160323A (en) | Gesture identification method | |
| CN105045398A (en) | Virtual reality interaction device based on gesture recognition | |
| CN105045399A (en) | Electronic device with 3D camera assembly | |
| Zhou et al. | A novel finger and hand pose estimation technique for real-time hand gesture recognition | |
| CN108496142B (en) | Gesture recognition method and related device | |
| JP6079832B2 (en) | Human computer interaction system, hand-to-hand pointing point positioning method, and finger gesture determination method | |
| US8254627B2 (en) | Method for automatically following hand movements in an image sequence | |
| CN205080499U (en) | Mutual equipment of virtual reality based on gesture recognition | |
| CN105068662A (en) | Electronic device used for man-machine interaction | |
| CN110443148B (en) | Action recognition method, system and storage medium | |
| CN105302295A (en) | Virtual reality interaction device having 3D camera assembly | |
| CN105302294A (en) | Interactive virtual reality presentation device | |
| CN110796018B (en) | Hand motion recognition method based on depth image and color image | |
| CN105069444B (en) | A kind of gesture identifying device | |
| CN103294996A (en) | 3D gesture recognition method | |
| Jalab et al. | Human computer interface using hand gesture recognition based on neural network | |
| CN105046249A (en) | Human-computer interaction method | |
| CN103995595A (en) | Game somatosensory control method based on hand gestures | |
| Störring et al. | Computer vision-based gesture recognition for an augmented reality interface | |
| CN105335711A (en) | Fingertip detection method in complex environment | |
| EP2786312A1 (en) | A gesture recognition method, an apparatus and a computer program for the same | |
| CN205080498U (en) | Mutual equipment of virtual reality with 3D subassembly of making a video recording | |
| CN104866826A (en) | Static gesture language identification method based on KNN algorithm and pixel ratio gradient features | |
| CN106529480A (en) | Finger tip detection and gesture identification method and system based on depth information | |
| Liang et al. | Egocentric hand pose estimation and distance recovery in a single RGB image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 150016 Heilongjiang Province, Harbin Economic Development Zone haping Road District Dalian road and Xingkai road junction Applicant after: HARBIN YISHE TECHNOLOGY Co.,Ltd. Address before: 150016 Heilongjiang City, Harbin province Daoli District, quiet street, unit 54, unit 2, layer 4, No. 3 Applicant before: HARBIN YISHE TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20181127 |