CN106529480A - Finger tip detection and gesture identification method and system based on depth information - Google Patents
Finger tip detection and gesture identification method and system based on depth information Download PDFInfo
- Publication number
- CN106529480A CN106529480A CN201610998600.0A CN201610998600A CN106529480A CN 106529480 A CN106529480 A CN 106529480A CN 201610998600 A CN201610998600 A CN 201610998600A CN 106529480 A CN106529480 A CN 106529480A
- Authority
- CN
- China
- Prior art keywords
- hand
- broken line
- evolution
- gesture
- contour curve
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000001514 detection method Methods 0.000 title claims abstract description 40
- 210000000245 forearm Anatomy 0.000 claims abstract description 24
- 210000000707 wrist Anatomy 0.000 claims abstract description 18
- 230000011218 segmentation Effects 0.000 claims abstract description 9
- 230000000007 visual effect Effects 0.000 claims description 64
- 238000004364 calculation method Methods 0.000 claims description 9
- 239000000203 mixture Substances 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 238000012360 testing method Methods 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 7
- 238000000354 decomposition reaction Methods 0.000 description 6
- 238000005286 illumination Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/117—Biometrics derived from hands
Landscapes
- Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a finger tip detection and gesture identification method based on depth information. The method comprises the following steps: obtaining a depth image comprising gesture information, performing coarse segmentation on a hand portion by use of a threshold method to obtain a hand shape with a part of a forearm area; realizing accurate segmentation of a hand area through detecting a wrist feature identification, and extracting a hand contour curve; obtaining a hand simplified broken line by simplifying the hand contour curve by use of a discrete curve evolution method; detecting finger tips on the hand simplified broken line by use of the threshold method; and identifying a gesture according to the quantity of the finger tips included in the hand simplified broken line and a set gesture model. The method can accurately segment the hand area, and at the same time, detects the finger tips and identifies the gesture. The invention further provides a corresponding finger tip detection and gesture identification system based on depth information.
Description
Technical Field
The invention belongs to the technical field of image recognition, and particularly relates to a fingertip detection and gesture recognition method and system based on depth information.
Background
Fingertip detection and gesture recognition are important means for human-computer interaction. In recent years, with the rise of mobile internet technologies, fingertip detection and gesture recognition technologies based on vision have been widely used in the fields of home entertainment, smart driving, smart wearing, and the like, due to the advantages of non-contact and low cost.
The key of fingertip detection and gesture recognition is to segment the acquired gesture image to obtain the shape characteristics of the gesture. Traditional vision-based fingertip detection and gesture recognition systems employ optical sensing devices to collect gesture information. However, due to the sensitive characteristic of the optical sensor, the image acquired by the sensor is easily interfered by factors such as illumination, background scattering and the like, so that accurate segmentation is difficult to achieve by using the existing method. The advent of the Kinect sensor has facilitated image segmentation. On top of the conventional optical sensor, the Kinect sensor adds two depth sensors to detect the depth information of the image. By combining the depth information, the defect of environmental illumination can be overcome, and the gesture can be accurately segmented from the stray background. However, due to the effect of the depth sensor resolution and noise, the segmented gesture shape contour contains a large amount of deformation and noise. These pose challenges to subsequent fingertip detection and gesture recognition efforts.
In view of the above problems, the scholars propose different solutions, which can be roughly summarized into two categories:
one is a conventional matching method for general shapes. The conventional matching method firstly extracts shape features and then carries out template matching by using a dynamic programming method. Where representative shape features include shape context and skeleton path. The shape context characterizes contour points using a log-polar space whose polar angle can be divided into an absolute polar angle and a relative polar angle. The absolute polar angle cannot guarantee constant rotation. While the relative polar angles require contour tangent alignment of the polar angles. Because the gesture shape contains a large amount of deformation and contour noise, the contour tangent line is difficult to accurately calculate, so that the feature description is not accurate enough, and the matching precision is not high. The skeleton has the advantage that the topology of a general shape can be intuitively depicted. However, for the shape of the hand, on one hand, the deformation of the hand is easy to cause the generation of redundant skeleton, which increases the difficulty of subsequent matching; on the other hand, a considerable number of shape classes have extremely similar topological structures, and further extracted skeletons are extremely similar, and thus are difficult to distinguish by a skeleton map matching method.
Another class is part-based matching methods for gesture shapes. The part-based approach is to first decompose the segmented gesture shape into a palm and fingers, and then to use the finger earth movement distance metric for template matching. Representative decomposition methods include a circle-based method, an approximately convex decomposition-based method, and a perceptual shape decomposition-based method. The method based on the circle is simple to realize and can meet the real-time requirement, but the decomposed fingers are not accurate enough, and the matching precision is not high. The accuracy of the finger decomposition of the approximate convex decomposition and the perception shape decomposition is improved, and the matching precision is effectively improved, but the improvement is at the cost of sacrificing the real-time performance. In addition, because the expression mode based on part does not contain fingertip characteristics, the method can not effectively detect fingertips, and the application of the method in the fields of home entertainment, intelligent driving, intelligent wearing and the like is limited.
Disclosure of Invention
The invention aims to acquire a hand contour curve corresponding to a hand region and simplify the hand contour curve into a broken line by acquiring a depth image containing gesture information, detect fingertips on the hand simplified broken line, and recognize gestures according to the number of the detected fingertips and a template matching method, thereby solving the technical problems that the gesture region cannot be accurately segmented, the gesture curve contains a large amount of deformation and noise to influence the recognition accuracy, and the fingertip detection and the gesture recognition are difficult to be simultaneously performed in the prior art.
To achieve the above object, according to one aspect of the present invention, there is provided a fingertip detection and gesture recognition method based on depth information, including the steps of:
(1) obtaining a depth image containing gesture information, and roughly segmenting a hand by adopting a threshold value method to obtain a hand shape with a partial forearm area;
(2) for the hand shape with partial forearm area, the accurate segmentation of the hand area is realized by detecting the wrist characteristic mark, and a hand contour curve is extracted;
(3) simplifying the hand contour curve by using a discrete curve evolution method to obtain a hand simplified broken line containing fingertip characteristics;
(4) detecting finger tips in the hand simplified fold lines by a threshold method;
(5) and performing gesture recognition according to the number of fingertips contained in the hand simplified broken line and a set gesture model.
In an embodiment of the present invention, the step (3) specifically includes the following sub-steps:
(3.1) enabling the original hand contour curve C obtained in the step (2) to be an initial hand evolution broken line; whereinRepresenting the original hand contour curve, c0Represents the starting point of the hand contour curve C, CiThe ith pixel point representing the hand contour curve C,represents the end point, N, of the hand contour curve CCThe number of pixel points representing the hand contour curve C;representing hand evolution polylines, where p0＝c0，pjJ-th vertex, N, representing hand evolution polyline PPRepresenting the number of vertexes of the hand evolution broken line P; kjIs a vertex pjCorresponding Δ pj- 1pjpj+1For characterizing the vertex pjThe magnitude of the contribution, Δ p, to the composition of the hand contour curve Cj-1pjpj+1Is a vertex pjThe corresponding related triangle is provided with
(3.2) sequentially calculating the visual saliencyAnd arranging the points in descending order to obtain visual saliency arrangement of the vertexes of the hand evolution broken line P
(3.3) deletion is ranked inThe minimum visual saliency at the tail end and the vertex P corresponding to the hand evolution broken line P are connected with two vertexes adjacent to the original vertex in the hand evolution broken line P, the visual saliency of the related triangles corresponding to the two vertexes is updated and the related triangles are reinserted into the hand evolution broken line P in a descending orderPerforming the following steps;
(3.4) repeating step (3.3) until ranked inThe minimum visual saliency at the end is larger than a set visual saliency threshold value KTThe hand evolution folding line P obtained at this time is taken as the final hand simplification folding line.
In one embodiment of the invention, the visual saliency KjThe calculation method is as follows:
wherein l (p)j-1,pj) And l (p)j,pj+1) Respectively represent the j (P) th edge of the hand evolution broken line Pj-1,pj) And j +1 th side (p)j,pj+1) Length of side of α (p)j) Is a vertex pjAngle of rotation of, i.e. line segment (p)j-1,pj) Around the vertex pjRotate to (p)j,pj+1) The angle swept.
In an embodiment of the present invention, the step (5) is specifically:
firstly, performing gesture recognition on the gestures according to the number of fingertips contained in the hand simplified broken line to obtain one or more gestures corresponding to the number of the fingertips;
and if the number of the fingertips corresponds to a plurality of gestures, comparing the hand simplified broken line with a plurality of gesture models corresponding to the number of the fingertips to obtain the gesture corresponding to the hand simplified broken line.
In one embodiment of the present invention, in the step (1):
acquiring a gesture image by using an image acquisition device comprising a depth sensor to obtain a depth image comprising gesture information; the hand and forearm are closer to the image capture device than other objects in the image are to the image capture device when capturing the gesture image.
In an embodiment of the present invention, the roughly segmenting the hand by using a threshold method in the step (1) to obtain the hand shape with a partial forearm region specifically includes:
segmenting an image area with depth information smaller than a set depth threshold value to obtain a hand shape with a partial forearm area;
the step (2) specifically comprises:
removing redundant forearm areas according to preset wrist characteristic marks to obtain more accurate hand areas;
and extracting the outline of the hand area to obtain a hand outline curve corresponding to the hand area.
In an embodiment of the present invention, the step (4) is specifically:
and (4) arranging the corners of each vertex on the simplified hand broken line obtained in the step (3) according to a descending order, detecting the corners larger than a set corner threshold value in the corners before the fifth ranking, and taking the corresponding vertex as the fingertip.
In an embodiment of the present invention, the image capturing device is a Kinect sensor or a RealSense sensor, and the predetermined wrist feature identifier is an annular jewelry with a significant distinction from skin color.
According to another aspect of the present invention, there is also provided a fingertip detection and gesture recognition system based on depth information, including a hand shape acquisition module, a hand contour curve generation module, a hand simplified broken line generation module, a fingertip detection module, and a gesture recognition module, wherein:
the hand shape acquisition module is used for acquiring a depth image containing gesture information and roughly segmenting a hand by adopting a threshold value method to obtain a hand shape with a partial forearm area;
the hand contour curve generating module is used for accurately dividing the hand region of the hand shape with partial forearm region by detecting the wrist feature identifier and extracting a hand contour curve;
the hand simplifying broken line generating module is used for simplifying the hand contour curve by utilizing a discrete curve evolution method to obtain a hand simplifying broken line containing fingertip characteristics;
the fingertip detection module is used for detecting the fingertips in the hand simplified fold line by a threshold method;
and the gesture recognition module is used for recognizing gestures according to the number of fingertips contained in the hand simplified broken line and a set gesture model.
In an embodiment of the present invention, the hand simplified broken line generating module specifically includes an initial hand evolvement broken line setting sub-module, a visual saliency calculation sub-module, a visual saliency updating sub-module, and a hand simplified broken line obtaining sub-module, where:
the initial hand evolution broken line setting submodule is used for enabling an original hand contour curve C obtained by the hand contour curve generating module to be an initial hand evolution broken line; whereinRepresenting the original hand contour curve, c0Represents the starting point of the hand contour curve C, CiThe ith pixel point representing the hand contour curve C,represents the end point, N, of the hand contour curve CCThe number of pixel points representing the hand contour curve C;representing hand evolution polylines, where p0＝c0，pjJ-th vertex, N, representing hand evolution polyline PPRepresenting the number of vertexes of the hand evolution broken line P; kjIs a vertex pjCorresponding Δ pj-1pjpj+1For characterizing the vertex pjThe magnitude of the contribution, Δ p, to the composition of the hand contour curve Cj-1pjpj+1Is a vertex pjThe corresponding related triangle is provided with
The visual saliency calculation operator module is used for calculating the visual saliency in sequenceAnd arranging the points in descending order to obtain visual saliency arrangement of the vertexes of the hand evolution broken line P
The visual saliency update submodule is used for deleting the ranksThe minimum visual saliency at the tail end and the vertex P corresponding to the minimum visual saliency in the hand evolution broken line P are connected with two vertexes adjacent to the original vertex in the hand evolution broken line P, the visual saliency of the relevant triangle corresponding to the two vertexes is updated and the relevant triangle is reinserted into the hand evolution broken line P in a descending orderPerforming the following steps;
the hand simplified broken line obtaining submodule is used for repeatedly executing the visual saliency updating submodule until the hand simplified broken line obtaining submodule is arranged in the rowThe minimum visual saliency at the end is larger than a set visual saliency threshold value KTThe hand evolution folding line P obtained at this time is taken as the final hand simplification folding line.
In general, compared with the prior art, the invention has the following beneficial effects:
(1) due to the sensitive characteristic of the optical sensor, the image acquired by the optical sensor is easily interfered by factors such as illumination, background scattering and the like, so that accurate segmentation of gestures is difficult to achieve only according to the color information of the image; according to the invention, the depth information and the color information of the image are utilized, and the depth information of the image is combined on the basis of the color information of the image, so that the interference of environmental factors such as insufficient illumination is effectively avoided, the accurate segmentation of the gesture is realized, and convenience is provided for subsequent fingertip detection and gesture recognition;
(2) for the hand shape segmented from the depth image, the shape features extracted by the existing method are not robust enough or do not contain fingertip features, so that the fingertip detection cannot be well realized. The method removes the contour noise through a discrete curve evolution method, and distinguishes the fingertip characteristics and the contour deformation in the hand shape from the most intuitive angle, so that the robust and accurate fingertip detection can be realized, and the application range of the method in the fields of home entertainment, intelligent driving, intelligent wearing and the like is expanded;
(3) the hand shape segmented from the depth image contains a large amount of deformation and noise, so that the gesture recognition is performed by adopting the existing method, and the accuracy and efficiency of the recognition are difficult to ensure at the same time; on the basis of a robust and accurate fingertip detection result, the method fully considers the intuitive characteristics of the number of fingertips of different gestures, the fluctuation of contour segments between adjacent fingertips and the like, so that the method has higher identification accuracy. In addition, due to the low computational complexity of the discrete curve evolution algorithm, the method has high computational efficiency, and can realize real-time detection of fingertips and real-time gesture recognition.
Drawings
FIG. 1 is a schematic flow chart of a fingertip detection and gesture recognition method based on depth information according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of image changes of a simplified hand broken line obtained by processing a depth image containing gesture information according to an embodiment of the present invention;
FIG. 3 is a simplified flowchart of the hand contour curve generated by the discrete curve evolution method according to the embodiment of the present invention;
FIG. 4 is a schematic diagram of images of ten types of gestures included in a gesture database according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a fingertip detection and gesture recognition system based on depth information in an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a simplified hand polyline generation module in an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
As shown in fig. 1, the present invention provides a fingertip detection and gesture recognition method based on depth information, which includes the following steps:
and S1, obtaining a depth image containing gesture information, and roughly dividing the hand by adopting a threshold value method to obtain the shape of the hand with a partial forearm area.
Due to the sensitive characteristic of the optical sensor, the image acquired by the sensor is easily interfered by factors such as illumination, background scattering and the like, so that accurate segmentation of the gesture is difficult to achieve only according to the color information of the image. And accurate segmentation of gestures can be realized by utilizing the depth information and the color information of the image.
Therefore, the image acquisition device comprising the depth sensor can be used for acquiring the gesture image to obtain the depth image comprising the gesture information; the hand and forearm are closer to the image capture device than other objects in the image are to the image capture device when capturing the gesture image.
For example, the image acquisition device can be a Kinect sensor, and the Kinect sensor brings convenience to image segmentation. On top of the conventional optical sensor, the Kinect sensor adds two depth sensors to detect the depth information of the image. By combining the depth information, the defect of environmental illumination can be overcome, and the gesture can be accurately segmented from the stray background. It should be noted that, in the embodiment of the present invention, the Kinect sensor is used to collect an image, and other types of depth cameras in the market, such as a Real sensor, may also implement this function.
S2, accurately dividing the hand region by detecting the wrist feature identifier for the hand shape with partial forearm region, and extracting a hand contour curve;
to further remove the forearm area, a predetermined wrist feature identifier, typically a ring-shaped jewelry having a distinctive degree of skin color, such as a black wrist band, bracelet or ribbon, may be used.
For example, a black ribbon may be tied to the wrist of the right hand of the tester (in this embodiment, the right hand is taken as an example, and the left hand is similar) and the distance between the right hand and the Kinect sensor is kept closer than the distance between other objects in the field of view, as shown in fig. 2 (a). These requirements are easily met in practical applications.
Since the depth value of the object is smaller as the object is closer to the camera, the collected depth image information is as shown in fig. 2 (b). By setting a depth threshold, an image region having depth information smaller than the threshold is divided, and a hand shape including a partial forearm region is generated as shown in fig. 2 (c).
Further, according to the preset wrist characteristic mark, removing the forearm area, and obtaining a relatively accurate hand area; as shown in fig. 2, by using the color image 2(c), the black ribbon at the wrist is detected, and the precise hand shape can be obtained, as shown in fig. 2 (d); in the embodiment of the invention, the right hand is taken as an example, the gesture curve is an eight-neighborhood open curve, starts from the right side of the wrist of the right hand, and ends from the left side of the wrist of the right hand along the counterclockwise direction.
Further, contour extraction is performed on the hand region, and a hand contour curve corresponding to the hand region is obtained, as shown in fig. 2 (e).
S3, simplifying the hand contour curve by using a discrete curve evolution method to obtain a hand simplified broken line containing fingertip characteristics;
due to the influence of the resolution and noise of the depth sensor, the collected hand contour curve contains a large amount of noise and deformation, which increases the difficulty of fingertip detection. Therefore, the collected hand contour curve is simplified, so that the simplified curve can eliminate the influence of noise and deformation on fingertip detection, and can keep better visual appearance, which is very important. In the embodiment of the invention, a method based on discrete curve evolution is adopted for simplification.
For convenience, useRepresenting the hand contour curve, c0Represents the starting point of the hand contour curve C, CiThe ith pixel point representing the hand contour curve C,represents the end point, N, of the hand contour curve CCThe number of pixel points representing the hand contour curve C;representing hand evolution polylines, where p0＝c0，pjJ-th vertex, N, representing hand evolution polyline PPRepresenting the number of vertexes of the hand evolution broken line P; kjIs a vertex pjCorresponding Δ pj-1pjpj+1For characterizing the vertex pjThe magnitude of the contribution, Δ p, to the composition of the hand contour curve Cj-1pjpj+1Is a vertex pjThe corresponding relevant triangle; by P*Representing a hand simplified broken line after discrete curve evolution; by usingRepresents a detected fingertip, wherein tkIs shown at P*The k-th fingertip detected above, NTIndicating the number of detected fingertips,NT∈ {0,1, …,5 }. The fingertip detection method based on curve evolution in the embodiment of the invention is to obtain a hand simplified broken line P keeping the main characteristics of the original curve by continuously deleting the pixel points in the hand contour curve C*Then on P*And detecting the fingertip T.
Specifically, as shown in fig. 3, the step (3) specifically includes the following sub-steps:
s31, setting the original hand contour curve C (shown in FIG. 2 (e)) obtained in the step S2 as an initial hand evolvement broken line;
so as to ensure the invariance of the starting point and the end point in the evolution process.
S32, sequentially calculating the visual saliencyAnd arranging the points in descending order to obtain visual saliency arrangement of the vertexes of the hand evolution broken line P
The key point for realizing curve evolution in the embodiment of the invention is to determine a relevant triangle delta pj-1pjpj+1Degree of visual saliency Kj. The vertex contributing a large amount to the composition of the hand contour curve C is required, and the value of the visual saliency thereof is also large. The visual saliency KjThe calculation method is as follows:
wherein l (p)j-1,pj) And l (p)j,pj+1) Respectively represent the j (P) th edge of the hand evolution broken line Pj-1,pj) And j +1 th side (p)j,pj+1) Length of side of αjIs a vertex pjAngle of rotation of, i.e. line segment (p)j-1,pj) Around the vertex pjRotate to (p)j,pj+1) The angle swept. It can be shown that the above-mentioned visual saliency KjIs between the size of the relevant triangle Δ pj-1pjpj+1Side (p) ofj-1,pj+1) Corresponding to the height and the middle line, it can also be intuitively considered as the vertex pjCorresponding protrusion size.
S33, delete and arrange inThe minimum visual saliency at the tail end and the vertex P corresponding to the minimum visual saliency in the hand evolution broken line P are connected with two vertexes adjacent to the original vertex in the hand evolution broken line P, the visual saliency of the relevant triangle corresponding to the two vertexes is updated and the relevant triangle is reinserted into the hand evolution broken line P in a descending orderPerforming the following steps;
s34, repeating the step S33 until the rows areThe minimum visual saliency at the end is larger than a set visual saliency threshold value KTThe hand evolution broken line P obtained at this time is taken as the final hand simplification broken line (as shown in 2 (f)); generally, the noise and distortion of the hand contour curve causes less protrusion than the fingertip. Experiments prove that the noise deformation and the fingertip characteristics can be well distinguished by setting the visual saliency threshold value to be 40% of the maximum inscribed circle radius of the hand shape.
The time complexity of the hand contour curve evolution described above is mainly composed of sub-step S32 and sub-step S33. For sub-step S32, the visual saliency of the relevant triangle corresponding to the pixel point on the hand contour curve needs to be sorted, and the time complexity is O (N)Clog NC) Wherein N isCThe number of pixels representing the hand contour curve. For sub-step S33, deleting one vertex each time, and updating the visual saliency of the relevant triangle corresponding to its neighboring vertex, with the temporal complexity of O (1); sequentially inserting it to length NCAmong the sequences of-L, their temporal complexity is O (log (N)C-L)), wherein L represents the number of cycles. The total time complexity is therefore:
in a practical system, using a 3.4GHz Intel i7-4770 processor with 8GB memory and Matlab to implement the above algorithm, the required time for a hand contour curve with 500 pixels is usually not more than 50 ms.
S4, detecting finger tips in the hand simplified fold line through a threshold value method;
and D, arranging the corners of all the vertexes on the simplified hand broken line obtained in the step S3 according to a descending order, detecting the corners larger than a set corner threshold value in the corners before the fifth ranking, and taking the corresponding vertexes as fingertips. Generally, the rotation angle threshold value of 3 pi/5 can meet the requirement of the accuracy of actual fingertip detection.
And S5, performing gesture recognition according to the number of fingertips included in the hand simplified broken line and the set gesture model.
The step S5 specifically includes:
firstly, performing first gesture recognition on the gesture according to the number of fingertips contained in the hand simplified broken line to obtain one or more gestures corresponding to the number of the fingertips;
in particular, a gesture recognition method based on segment matching may be employed. The gesture recognition method based on the segment matching is to perform segment matching on the simplified broken lines and the broken line templates of the test hand by using fingertips, and then recognize.
For convenience, with P*Representing a hand simplified broken line for testing after discrete curve evolution; by usingRepresents a detected fingertip, wherein tkIs shown at P*The k-th fingertip detected above, NTIndicating the number of detected fingertips,NT∈ {0,1, …,5 }. Here, we will refer to P*Is shown asWherein p isjRepresents P*The j-th fold line of (2), two end points of the fold line are fingertips t respectivelyjAnd tj+1。
By usingThe representation includes NTA set of hand broken line templates of individual fingertips, wherein PnRepresenting the nth hand polyline template, N, in the set P of hand polyline templatespIndicates the number of templates included in the hand broken line template set P. By usingRepresenting the nth hand polyline template P in PnThe set of fingertips of (a), wherein,represents PnThe jth fingertip of (1), and haveSimilarly, P isnIs shown asWherein,represents PnThe j section of the broken line template, two end points of the broken line template are fingertips respectivelyAnd
for facilitating template matching, use DnRepresenting hand reduction polyline P for testing*And the hand partFold line template PnA distance therebetween, byThe jth broken line P representing PjAnd PnThe j-th section of broken line templateThe distance between them. At this time, we have
Namely, the distance between the hand simplified broken line used for testing and the finger tip of the hand broken line template is the sum of the distances of the corresponding broken line segments. Thus, the template matching method based on discrete curve evolution is to find out the hand simplified broken line P in the hand broken line template set P*And the gesture type corresponding to the hand broken line template with the shortest distance.
The key for realizing the template matching method is to calculate the jth segment of the broken line P of the PjAnd PnThe j-th section of broken line templateThe distance betweenHere, the polyline similarity measure is used to calculate as follows:
wherein α (p)j(s)) andrespectively representing a broken line segment pjAndparameterized corner function of l (p)j) Andrespectively representing a broken line segment pjAndthe normalized lengths of the polylines are simplified with respect to the respective corresponding hands. The first two terms of the above formula respectively depict broken line segments pjAndthe differences in shape and size, while the last term characterizes their difference in proportion to their respective hand shapes. That is, the distanceIs not only dependent on the polyline pjAndthe absolute difference between them also depends on their respective relative differences from the whole.
The gesture recognition method takes the Kinect gesture database of Microsoft research institute, Nanyang university as an example to perform gesture recognition. The database contains ten types of gestures, each type of gesture containing 100 samples of gestures for a total of 1000 samples as shown in FIG. 4.
By using a discrete curve evolution method, the hand simplified broken line and the number of fingertips of the test gesture can be obtained. According to the number of detected fingertips, the gesture can be recognized for the first time, wherein the comparison table of the number of the fingertips as the gesture is shown in table 1.
TABLE 1 hand gesture-fingertip number comparison table
If the number of the fingertips of the test gesture is detected to be 0, the corresponding gesture category is gesture 1; if the number of the fingertips of the test gesture is detected to be 4, the corresponding gesture category is gesture 5; if the number of the fingertips of the test gesture is detected to be 5, the corresponding gesture category is gesture 6;
if the gesture corresponds to one gesture, the gesture is a recognition result; and if the number of the fingertips corresponds to a plurality of gestures, performing secondary gesture recognition by adopting a template matching method, namely comparing the hand simplified fold line with a plurality of hand fold line models corresponding to the number of the fingertips to obtain the gesture corresponding to the hand simplified fold line.
For example, in the embodiment of the present invention, if the number of fingertips detecting the test gesture is 1, 2, or 3, the secondary gesture recognition is performed by using a method based on template matching.
Further, as shown in fig. 5, the present invention further provides a fingertip detection and gesture recognition system based on depth information, which includes a hand shape acquisition module, a hand contour curve generation module, a hand simplified broken line generation module, a fingertip detection module, and a gesture recognition module, wherein:
the hand shape acquisition module is used for acquiring a depth image containing gesture information and roughly segmenting a hand by adopting a threshold value method to obtain a hand shape with a partial forearm area;
the hand contour curve generating module is used for accurately dividing the hand region of the hand shape with partial forearm region by detecting the wrist feature identifier and extracting a hand contour curve;
the hand simplifying broken line generating module is used for simplifying the hand contour curve by utilizing a discrete curve evolution method to obtain a hand simplifying broken line containing fingertip characteristics;
the fingertip detection module is used for detecting the fingertips in the hand simplified fold line by a threshold method;
and the gesture recognition module is used for recognizing gestures according to the number of fingertips contained in the hand simplified broken line and a set gesture model.
Further, the hand simplified broken line generation module specifically comprises an initial hand evolution broken line setting sub-module, a visual saliency calculation sub-module, a visual saliency updating sub-module and a hand simplified broken line acquisition sub-module, wherein:
the initial hand evolution broken line setting submodule is used for enabling an original hand contour curve C obtained by the hand contour curve generating module to be an initial hand evolution broken line; whereinRepresenting the original hand contour curve, c0Represents the starting point of the hand contour curve C, CiThe ith pixel point representing the hand contour curve C,represents the end point, N, of the hand contour curve CCThe number of pixel points representing the hand contour curve C;representing hand evolution polylines, where p0＝c0，pjJ-th vertex, N, representing hand evolution polyline PPRepresenting the number of vertexes of the hand evolution broken line P; kjIs a vertex pjCorresponding Δ pj-1pjpj+1For characterizing the vertex pjThe magnitude of the contribution, Δ p, to the composition of the hand contour curve Cj-1pjpj+1Is a vertex pjThe corresponding related triangle is provided with
The visual saliency calculation operator module is used for calculating the visual saliency in sequenceAnd arranging the points in descending order to obtain visual saliency arrangement of the vertexes of the hand evolution broken line P
The visual saliency update submodule is used for deleting the ranksThe minimum visual saliency at the tail end and the vertex P corresponding to the hand evolution broken line P are connected with two vertexes adjacent to the original vertex in the hand evolution broken line P, the visual saliency of the related triangles corresponding to the two vertexes is updated and the related triangles are reinserted into the hand evolution broken line P in a descending orderPerforming the following steps;
the hand simplified broken line obtaining submodule is used for repeatedly executing the visual saliency updating submodule until the hand simplified broken line obtaining submodule is arranged in the rowThe minimum visual saliency at the end is larger than a set visual saliency threshold value KTThe hand evolution folding line P obtained at this time is taken as the final hand simplification folding line.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A fingertip detection and gesture recognition method based on depth information is characterized by comprising the following steps:
(1) obtaining a depth image containing gesture information, and roughly segmenting a hand by adopting a threshold value method to obtain a hand shape with a partial forearm area;
(2) for the hand shape with partial forearm area, the accurate segmentation of the hand area is realized by detecting the wrist characteristic mark, and a hand contour curve is extracted;
(3) simplifying the hand contour curve by using a discrete curve evolution method to obtain a hand simplified broken line containing fingertip characteristics;
(4) detecting finger tips in the hand simplified fold lines by a threshold method;
(5) and performing gesture recognition according to the number of fingertips contained in the hand simplified broken line and a set gesture model.
2. The fingertip detection and gesture recognition method based on depth information as claimed in claim 1, wherein the step (3) specifically comprises the following sub-steps:
(3.1) enabling the original hand contour curve C obtained in the step (2) to be an initial hand evolution broken line; whereinRepresenting the original hand contour curve, c0Represents the starting point of the hand contour curve C, CiThe ith pixel point representing the hand contour curve C,represents the end point, N, of the hand contour curve CCThe number of pixel points representing the hand contour curve C;representing hand evolution polylines, where p0＝c0，pjJ-th vertex, N, representing hand evolution polyline PPRepresenting the number of vertexes of the hand evolution broken line P; kjIs a vertex pjCorresponding Δ pj- 1pjpj+1For characterizing the vertex pjThe magnitude of the contribution, Δ p, to the composition of the hand contour curve Cj-1pjpj+1Is a vertex pjThe corresponding related triangle is provided with
(3.2) sequentially calculating the visual saliencyAnd arranging the points in descending order to obtain visual saliency arrangement of the vertexes of the hand evolution broken line P
(3.3) deletion is ranked inThe minimum visual saliency at the tail end and the vertex P corresponding to the hand evolution broken line P are connected with two vertexes adjacent to the original vertex in the hand evolution broken line P, the visual saliency of the related triangles corresponding to the two vertexes is updated and the related triangles are reinserted into the hand evolution broken line P in a descending orderPerforming the following steps;
(3.4) repeating step (3.3) until ranked inThe minimum visual saliency at the end is larger than a set visual saliency threshold value KTThe hand evolution folding line P obtained at this time is taken as the final hand simplification folding line.
3. The fingertip detection and gesture recognition method based on depth information of claim 2, wherein the visual saliency K isjThe calculation method is as follows:
wherein l (p)j-1，pj) And l (p)j，pj+1) Respectively represent the j (P) th edge of the hand evolution broken line Pj-1，pj) And j +1 th side (p)j，pj+1) Length of side of α (p)j) Is a vertex pjAngle of rotation of, i.e. line segment (p)j-1，pj) Around the vertex pjRotate to (p)j，pj+1) The angle swept.
4. The fingertip detection and gesture recognition method based on depth information according to any one of claims 1 to 3, wherein the step (5) is specifically as follows:
firstly, performing gesture recognition on the gestures according to the number of fingertips contained in the hand simplified broken line to obtain one or more gestures corresponding to the number of the fingertips;
and if the number of the fingertips corresponds to a plurality of gestures, comparing the hand simplified broken line with a plurality of gesture models corresponding to the number of the fingertips to obtain the gesture corresponding to the hand simplified broken line.
5. A method for fingertip detection and gesture recognition based on depth information according to any one of claims 1 to 3, characterized in that in the step (1):
acquiring a gesture image by using an image acquisition device comprising a depth sensor to obtain a depth image comprising gesture information; when the gesture image is collected, the distance between the hand and the arm and the image collecting device is shorter than the distance between other objects in the image and the image collecting device.
6. The method for fingertip detection and gesture recognition based on depth information as claimed in any one of claims 1 to 3, wherein the step (1) of roughly segmenting the hand by using a threshold method to obtain a hand shape with a partial forearm area specifically comprises:
segmenting an image area with depth information smaller than a set depth threshold value to obtain a hand shape with a partial forearm area;
the step (2) specifically comprises:
removing redundant forearm areas according to preset wrist characteristic marks to obtain more accurate hand areas;
and extracting the outline of the hand area to obtain a hand outline curve corresponding to the hand area.
7. The method for finger detection and gesture recognition based on depth information as claimed in any one of claims 1 to 3, wherein the step (4) is specifically as follows:
and (4) arranging the corners of each vertex on the simplified hand broken line obtained in the step (3) according to a descending order, detecting the corners larger than a set corner threshold value in the corners before the fifth ranking, and taking the corresponding vertex as the fingertip.
8. The method of any one of claim 5, wherein the image capturing device is a Kinect sensor or a Real sensor, and the predetermined wrist feature identifier is a ring-shaped jewelry with a significant distinction from skin color.
9. The utility model provides a fingertip detection and gesture recognition system based on depth information, its characterized in that obtains module, hand contour curve generation module, the broken line generation module is simplified to the hand, fingertip detection module and gesture recognition module, wherein:
the hand shape acquisition module is used for acquiring a depth image containing gesture information and roughly segmenting a hand by adopting a threshold value method to obtain a hand shape with a partial forearm area;
the hand contour curve generating module is used for accurately dividing the hand region of the hand shape with partial forearm region by detecting the wrist feature identifier and extracting a hand contour curve;
the hand simplifying broken line generating module is used for simplifying the hand contour curve by utilizing a discrete curve evolution method to obtain a hand simplifying broken line containing fingertip characteristics;
the fingertip detection module is used for detecting the fingertips in the hand simplified fold line by a threshold method;
and the gesture recognition module is used for recognizing gestures according to the number of fingertips contained in the hand simplified broken line and a set gesture model.
10. The depth information-based fingertip detection and gesture recognition system of claim 9, wherein the hand simplified polyline generation module specifically comprises an initial hand evolution polyline setting sub-module, a visual saliency calculation sub-module, a visual saliency update sub-module, and a hand simplified polyline acquisition sub-module, wherein:
the initial hand evolution broken line setting submodule is used for enabling an original hand contour curve C obtained by the hand contour curve generating module to be an initial hand evolution broken line; whereinRepresenting the original hand contour curve, c0Represents the starting point of the hand contour curve C, CiThe ith pixel point representing the hand contour curve C,represents the end point, N, of the hand contour curve CCThe number of pixel points representing the hand contour curve C;representing hand evolution polylines, where p0＝c0，pjJ-th vertex, N, representing hand evolution polyline PPRepresenting the number of vertexes of the hand evolution broken line P; kjIs a vertex pjCorresponding Δ pj-1pjpj+1For characterizing the vertex pjThe magnitude of the contribution, Δ p, to the composition of the hand contour curve Cj-1pjpj+1Is a vertexpjThe corresponding related triangle is provided with
The visual saliency calculation operator module is used for calculating the visual saliency in sequenceAnd arranging the points in descending order to obtain visual saliency arrangement of the vertexes of the hand evolution broken line P
The visual saliency update submodule is used for deleting the ranksThe minimum visual saliency at the tail end and the vertex P corresponding to the minimum visual saliency in the hand evolution broken line P are connected with two vertexes adjacent to the original vertex in the hand evolution broken line P, the visual saliency of the relevant triangle corresponding to the two vertexes is updated and the relevant triangle is reinserted into the hand evolution broken line P in a descending orderPerforming the following steps;
the hand simplified broken line obtaining submodule is used for repeatedly executing the visual saliency updating submodule until the hand simplified broken line obtaining submodule is arranged in the rowThe minimum visual saliency at the end is larger than a set visual saliency threshold value KTThe hand evolution folding line P obtained at this time is taken as the final hand simplification folding line.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610998600.0A CN106529480A (en) | 2016-11-14 | 2016-11-14 | Finger tip detection and gesture identification method and system based on depth information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610998600.0A CN106529480A (en) | 2016-11-14 | 2016-11-14 | Finger tip detection and gesture identification method and system based on depth information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106529480A true CN106529480A (en) | 2017-03-22 |
Family
ID=58351456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610998600.0A Pending CN106529480A (en) | 2016-11-14 | 2016-11-14 | Finger tip detection and gesture identification method and system based on depth information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106529480A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107450672A (en) * | 2017-09-19 | 2017-12-08 | 曾泓程 | A kind of wrist intelligent apparatus of high discrimination |
| CN107515714A (en) * | 2017-07-27 | 2017-12-26 | 歌尔股份有限公司 | A kind of finger touch recognition methods, device and touch projection equipment |
| CN108985242A (en) * | 2018-07-23 | 2018-12-11 | 中国联合网络通信集团有限公司 | The method and device of images of gestures segmentation |
| CN111354029A (en) * | 2020-02-26 | 2020-06-30 | 深圳市瑞立视多媒体科技有限公司 | Gesture depth determination method, device, equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294996A (en) * | 2013-05-09 | 2013-09-11 | 电子科技大学 | 3D gesture recognition method |
| CN103488972A (en) * | 2013-09-09 | 2014-01-01 | 西安交通大学 | Method for detection fingertips based on depth information |
| CN103500010A (en) * | 2013-09-29 | 2014-01-08 | 华南理工大学 | Method for locating fingertips of person through video |
-
2016
- 2016-11-14 CN CN201610998600.0A patent/CN106529480A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294996A (en) * | 2013-05-09 | 2013-09-11 | 电子科技大学 | 3D gesture recognition method |
| CN103488972A (en) * | 2013-09-09 | 2014-01-01 | 西安交通大学 | Method for detection fingertips based on depth information |
| CN103500010A (en) * | 2013-09-29 | 2014-01-08 | 华南理工大学 | Method for locating fingertips of person through video |
Non-Patent Citations (8)
| Title |
|---|
| ZHONGYUAN LAI 等: "Perceptual Distortion Measure for Polygon-Based Shape Coding", 《IEICE TRANS.INF.&SYST.》 * |
| 刘鑫辰 等: "基于RGB-D摄像头的实时手指跟踪与手势识别", 《计算机科学》 * |
| 张星成: "基于骨架化方法的手势识别若干问题研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 张毅 等: "《移动机器人技术基础与制作》", 31 January 2013 * |
| 徐鹏飞: "基于Kinect的手势识别研究与应用", 《万方学位论文》 * |
| 范为 等: "基于改进型形状上下文描述子的字母手势识别", 《电子技术》 * |
| 谈家谱: "基于指尖信息的手势识别与人机交互应用研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 贺芳姿: "基于Kinect深度信息的手势识别研究", 《万方学位论文》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107515714A (en) * | 2017-07-27 | 2017-12-26 | 歌尔股份有限公司 | A kind of finger touch recognition methods, device and touch projection equipment |
| CN107515714B (en) * | 2017-07-27 | 2020-08-28 | 歌尔股份有限公司 | Finger touch identification method and device and touch projection equipment |
| CN107450672A (en) * | 2017-09-19 | 2017-12-08 | 曾泓程 | A kind of wrist intelligent apparatus of high discrimination |
| CN107450672B (en) * | 2017-09-19 | 2024-03-29 | 曾泓程 | Wrist type intelligent device with high recognition rate |
| CN108985242A (en) * | 2018-07-23 | 2018-12-11 | 中国联合网络通信集团有限公司 | The method and device of images of gestures segmentation |
| CN108985242B (en) * | 2018-07-23 | 2020-07-14 | 中国联合网络通信集团有限公司 | Gesture image segmentation method and device |
| CN111354029A (en) * | 2020-02-26 | 2020-06-30 | 深圳市瑞立视多媒体科技有限公司 | Gesture depth determination method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110232311B (en) | Method and device for segmenting hand image and computer equipment | |
| Zhou et al. | A novel finger and hand pose estimation technique for real-time hand gesture recognition | |
| CN106682598B (en) | Multi-pose face feature point detection method based on cascade regression | |
| Dominio et al. | Combining multiple depth-based descriptors for hand gesture recognition | |
| Jiang et al. | Multi-layered gesture recognition with Kinect. | |
| JP6079832B2 (en) | Human computer interaction system, hand-to-hand pointing point positioning method, and finger gesture determination method | |
| CN104063059B (en) | A kind of real-time gesture recognition method based on finger segmentation | |
| Nai et al. | Fast hand posture classification using depth features extracted from random line segments | |
| Matilainen et al. | OUHANDS database for hand detection and pose recognition | |
| CN109685013B (en) | Method and device for detecting head key points in human body posture recognition | |
| Zhu et al. | Vision based hand gesture recognition using 3D shape context | |
| CN103971102A (en) | Static gesture recognition method based on finger contour and decision-making trees | |
| CN104899600A (en) | Depth map based hand feature point detection method | |
| CN101901350A (en) | Characteristic vector-based static gesture recognition method | |
| Krejov et al. | Multi-touchless: Real-time fingertip detection and tracking using geodesic maxima | |
| CN106529480A (en) | Finger tip detection and gesture identification method and system based on depth information | |
| CN107832736B (en) | Real-time human body action recognition method and real-time human body action recognition device | |
| CN101794374A (en) | Method and system for identifying a person using their finger-joint print | |
| CN111444764A (en) | Gesture recognition method based on depth residual error network | |
| CN103455794A (en) | Dynamic gesture recognition method based on frame fusion technology | |
| CN111460976B (en) | Data-driven real-time hand motion assessment method based on RGB video | |
| Dinh et al. | Hand number gesture recognition using recognized hand parts in depth images | |
| CN105335711A (en) | Fingertip detection method in complex environment | |
| JP2016014954A (en) | Method for detecting finger shape, program thereof, storage medium of program thereof, and system for detecting finger shape | |
| CN112749646A (en) | Interactive point-reading system based on gesture recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170322 |