CN104299004A - Hand gesture recognition method based on multi-feature fusion and fingertip detecting - Google Patents

Abstract

The invention discloses a hand gesture recognition method based on multi-feature fusion and fingertip detecting. The method comprises a training process and a recognition process. In the training process, for a complex hand gesture, reasonable hand gesture features are selected, a multi-feature fusion feature extracting algorithm is used, the hand gesture is subjected to support vector machine training, and a training model is formed. In the recognition process, for an input video image sequence, hand gesture detecting is carried out first, then multi-feature extracting and fusion are carried out, and multiple features are input into the support vector machine to obtain a recognition results. Meanwhile, the hand gesture is subjected to fingertip detecting based on defects, through a defect screener, the positions of fingertips of fingers are located, then two-time recognition and detecting results are subjected to synthesized, and the final hand gesture recognition results are obtained. The problem that in a complex scene, the hand gesture recognition rate is not high can be effectively solved, the requirement of real-time performance is met, and the method can be well used in human-machine interaction.

Description

A kind of gesture identification method detected based on multiple features fusion and finger tip

Technical field

The present invention relates to a kind of method of gesture identification, relate to a kind of gesture identification method detected based on multiple features fusion and finger tip.

Background technology

Development along with computing machine is applied more and more extensively with rapid in modern society, the demand of human-computer interaction technology also becomes more and more higher in human lives, in these interaction techniques, gesture is a kind of natural and meet the interactive mode of the behavioural habits of people, it is directly perceived with it, convenient, naturally feature receive everybody concern, be one of the ideal chose as novel human-machine interaction technology.And gesture identification is as one of the step of most critical in interactive system, its recognition effect directly has influence on the communication capability between people and computing machine.

In conjunction with all kinds of research and practical application, the contradiction that technical difficult points that current gesture identification field exists is between system real time and gesture identification rate can be analyzed.In order to obtain higher gesture identification rate, usual researcher can select to retain feature as much as possible and go to characterize gesture, and by the algorithm identification gesture of complexity, and will certainly reduce the speed of identification like this, the real-time of system is not being met; And in order to be applicable to real-time system, usually calculated amount can only be reduced by the dimension reducing feature again, and the minimizing of characteristic dimension can make the impact of noise increase, and to early stage Hand Gesture Segmentation result have very high requirement, the discrimination of gesture decreases, the gesture kind that can be identified becomes simplification, lacks certain application.

Summary of the invention

In order to solve the problems of the prior art, the invention discloses a kind of gesture identification method detected based on multiple features fusion and finger tip, the method is by selecting rational gesture feature, comprise Hu moment characteristics, defect characteristic and ratio characteristic, utilize the feature extraction algorithm of multiple features fusion, make the calculated amount of feature little, validity is high, and the Fingertip Detection combined based on defect, improve the accuracy rate identified further.Under effectively solving complex scene in this way, the problem that gesture identification rate is not high, and the requirement meeting real-time.

The present invention is by the following technical solutions: a kind of gesture identification method detected based on multiple features fusion and finger tip, comprises the following steps:

Step 1): training process: rational gesture feature is selected to complicated gesture, and utilize the feature extraction algorithm of multiple features fusion, support vector machine training is carried out to gesture, forms training pattern;

Step 2): identifying: gestures detection is carried out to input image sequence, multi-feature extraction and fusion are carried out to the gesture detected, and is input in support vector machine the recognition result obtaining SVM; Meanwhile, carry out detecting based on the finger tip of defect to gesture, comprehensive two times result exports final recognition result.

Further, step 1) described in training process, select rational gesture feature to complicated gesture, and utilize the feature extraction algorithm of multiple features fusion, carry out support vector machine training to gesture, form training pattern, its detailed process is as follows:

Step 1.1): Hu Moment Feature Extraction:

Calculate the Hu moment characteristics of each images of gestures, and be normalized it, Hu moment characteristics can represent by formula:

Hu＝(φ ₁,φ ₂,φ ₃,φ ₄,φ ₅,φ ₆,φ ₇)

Wherein, φ ₁~ φ ₇represent each component of Hu moment characteristics respectively.

Step 1.2): defect characteristic extracts:

The defect part of gesture is exactly the part that gesture convex closure deducts gesture profile.The method calculating defect number is as follows:

Obtained the profile of gesture by eight neighborhood search procedure, judge whether the peripheral polygon of this gesture profile is convex, if convex, then this convex polygon is gesture convex closure; Again polygonal approximation carries out to gesture convex closure level and smooth; According to the defect number of gesture profile, gesture convex hull computation gesture.Each defective packets is containing three points: starting point, end point, depth point and the distance between depth point and convex closure, be defined as ptStart, ptEnd, ptFar and Depth respectively.

Step 1.3): ratio characteristic extracts:

Profile girth and area ratio:

Definition gesture profile girth is ConLenght, and definition gesture contour area is ConArea.The profile girth of definition gesture and area ratio feature:

$\mathrm{ConLA}=\frac{\mathrm{ConLenght}}{\mathrm{ConArea}}$

Profile girth and boundary rectangle girth ratio:

The boundary rectangle girth RectLenght of definition gesture, the girth of definition profile girth and boundary rectangle is than feature:

$\mathrm{LenCR}=\frac{\mathrm{ConLenght}}{\mathrm{RectLenght}}$

Contour area and boundary rectangle area ratio:

The boundary rectangle area of definition gesture is RectArea, contour area and boundary rectangle area ratio feature:

$\mathrm{AreaCR}=\frac{\mathrm{ConArea}}{\mathrm{RectArea}}$

Boundary rectangle the ratio of width to height:

The wide W of definition boundary rectangle, height is H, boundary rectangle the ratio of width to height:

$\mathrm{\α}=\frac{W}{H}$

Center of gravity and boundary rectangle up-and-down boundary ratio of distances constant:

O is expressed as the centre of gravity place of gesture, H ₁, H ₂represent the distance of gesture center of gravity to the upper and lower border of boundary rectangle respectively, definition center of gravity and boundary rectangle up-and-down boundary ratio of distances constant:

$\mathrm{\β}=\frac{H_1}{H_2}$

Center of gravity and boundary rectangle right boundary ratio of distances constant:

W ₁, W ₂represent the distance of gesture center of gravity to the left and right border of boundary rectangle respectively, definition center of gravity and boundary rectangle left and right lower boundary ratio of distances constant:

$\mathrm{\η}=\frac{W_1}{W_2}$

Step 1.4): multiple features fusion:

By step 1.1) ~ 1.3) in the Hu moment characteristics, defect characteristic and six ratio characteristic that extract to permeate a proper vector feature, for characterizing the characteristic information of images of gestures:

feature＝{Hu,numDefects,ConLA,LenCR,AreaCR,α,β,η}

Wherein, numDefects is defect number, Hu=(φ ₁, φ ₂, φ ₃, φ ₄, φ ₅, φ ₆, φ ₇).

Step 1.5): support vector machine is trained:

Be input in support vector machine together with classification mark by the proper vector feature of above-mentioned gesture sample image and train, wherein classification is labeled as the mark of gesture kind.

Further, step 2) described in identifying, gestures detection is carried out to input image sequence, multi-feature extraction and fusion is carried out to the gesture detected, and be input in support vector machine the recognition result obtaining SVM; Meanwhile, carry out detecting based on the finger tip of defect to gesture, comprehensive two times result exports final recognition result, and its detailed process is as follows:

Step 2.1): gestures detection:

The sequence of video images of input is carried out to the mixed Gaussian background modeling improved based on space time information, and the Face Detection result combined based on multiple color spaces, obtain the Hand Gesture Segmentation image of binaryzation after filtering process and morphological operation are carried out to both synthesis result.

Step 2.2): feature extraction:

To the binary map of gesture according to step 1.1) ~ 1.4) calculate the proper vector of this images of gestures;

Step 2.3): SVM identifies:

By in the proper vector of images of gestures input support vector machine, export SVM recognition result.

Step 2.4): finger tip detects:

The number of finger tip is exactly the number of finger, the detection of finger tip is comprised to the positional information of finger tip number and finger tip.According to step 1.2) obtain three defect point ptStart that gesture defect and each defect comprise, ptEnd, ptFar, finger tip point is contained in these defect points, so set up an effective defect point screening washer, filter out effective defect point and be finger tip point.Screening washer rule is as follows:

I. the starting point of defect and depth point distance are greater than a certain proportion of gesture boundary rectangle height H

Lenght (ptStart, ptFar) > α H, wherein α is scale-up factor.

Ii. the depth point of defect and end point distance are greater than a certain proportion of gesture boundary rectangle height H

Lenght(ptEnd,ptFar)＞αH

Iii. the angle that the starting point of defect, depth point, end point are formed is less than threshold value T _angle

angle(ptStart,ptFar,ptEnd)＜T _angle

Iv. the starting point of defect, depth point, end point are in the certain limit of gesture boundary rectangle

y _bounding＜y _ptStar＜y _bounding+βH

y _bounding＜y _ptEnd＜y _bounding+βH

y _bounding＜y _ptFar＜y _bounding+βH

Wherein β is scale-up factor, and H is that gesture boundary rectangle is high.

V. when defect point spacing is less than T _distime, two defect points are approximate to be overlapped, and can be judged to be same defect point

Lenght(pt _i,pt _j)＜T _dis

When defect point meets above condition simultaneously, then can be judged to be effective defect point by screening washer, record effective defect point number and position thereof, be finger tip point.

Step 2.5): comprehensive recognition result:

The result and the finger tip that compare support vector machine output detect the finger tip point number exported, and when both are consistent, then export recognition result.

The present invention adopts above technical scheme compared with prior art, has following technique effect:

1) adopt the feature extracting method of multiple features fusion, feature calculation amount is little, and each feature can describe gesture from different perspectives, can the effective mistake brought of modifying factor single features flase drop, under calculated amount little as far as possible, bring higher discrimination.

2) Fingertip Detection based on defect is visual and understandable, meet the priori of gesture and the understanding custom of the mankind, calculated amount is little, compare depend on template template matching method, need to calculate in a large number curvature, distance profile analysis method and relate to the heuristics directly perceived of complicated thinning process, Fingertip Detection based on defect to the tip portion of gesture, the basis ensureing accuracy rate can further increase the real-time of system by quick position in a straightforward manner.

3) by support vector machine recognition result based on multi-feature fusion and finger tip association, the common recognition result determining final output, further increases gesture identification rate, reduces the flase drop risk that single recognition methods brings.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of gesture identification of the present invention;

Fig. 2 is the process flow diagram of training process;

Fig. 3 is the process flow diagram that finger tip detects;

Embodiment

Below in conjunction with accompanying drawing also by specific embodiment, technical scheme of the present invention is described in further detail.

Following examples are implemented under premised on technical solution of the present invention, give detailed embodiment and concrete operating process, but protection scope of the present invention are not limited to following embodiment.

Embodiment

The present embodiment processes the video sequence (640X480 pixel, 30ftps) that a section is taken by Logitech C710 IP Camera.This video random shooting in indoor scene, comprises complicated background, has the background object of the class colour of skin to occur there is the change of illumination in scene, the kind of gesture comprises 0,1,2,3,4,5,8 seven kind of gesture.The present embodiment comprises following steps:

Step 1): training process: all gesture sample images are input in tranining database one by one, select Hu moment characteristics, defect characteristic and ratio characteristic, and utilize the feature extraction algorithm of multiple features fusion, support vector machine training is carried out to gesture, forms training pattern;

In the present embodiment, step 1) described in training process, Fig. 2 is the process flow diagram of training process, and its detailed process is as follows:

Step 1.1): training sample prepares:

The gesture sample image of seven kinds and its classification mark are input to one by one in tranining database, have 1369 width gesture sample images in database, seven kinds of gestures are labeled as 0,1,2,3,4,5,8 respectively.

Step 1.2): Hu Moment Feature Extraction:

Calculate the Hu moment characteristics of each images of gestures, and be normalized it, Hu moment characteristics can represent by formula:

Hu＝(φ ₁,φ ₂,φ ₃,φ ₄,φ ₅,φ ₆,φ ₇)

Wherein, φ ₁~ φ ₇represent each component of Hu moment characteristics respectively.

Step 1.3): defect characteristic extracts:

The defect part of gesture is exactly the part that gesture convex closure deducts gesture profile.The method calculating defect number is as follows:

Obtained the profile of gesture by eight neighborhood search procedure, judge whether the peripheral polygon of this gesture profile is convex, if convex, then this convex polygon is gesture convex closure; Again polygonal approximation carries out to gesture convex closure level and smooth; According to the defect number of gesture profile, gesture convex hull computation gesture.Each defective packets is containing three points: starting point, end point, depth point and the distance between depth point and convex closure, be defined as ptStart, ptEnd, ptFar and Depth respectively.

Step 1.4): ratio characteristic extracts:

Profile girth and area ratio:

Definition gesture profile girth is ConLenght, and definition gesture contour area is ConArea.The profile girth of definition gesture and area ratio feature:

$\mathrm{ConLA}=\frac{\mathrm{ConLenght}}{\mathrm{ConArea}}$

Profile girth and boundary rectangle girth ratio:

The boundary rectangle girth RectLenght of definition gesture, the girth of definition profile girth and boundary rectangle is than feature:

$\mathrm{LenCR}=\frac{\mathrm{ConLenght}}{\mathrm{RectLenght}}$

Contour area and boundary rectangle area ratio:

The boundary rectangle area of definition gesture is RectArea, contour area and boundary rectangle area ratio feature:

$\mathrm{AreaCR}=\frac{\mathrm{ConArea}}{\mathrm{RectArea}}$

Boundary rectangle the ratio of width to height:

The wide W of definition boundary rectangle, height is H, boundary rectangle the ratio of width to height:

$\mathrm{\α}=\frac{W}{H}$

Center of gravity and boundary rectangle up-and-down boundary ratio of distances constant:

O is expressed as the centre of gravity place of gesture, H ₁, H ₂represent the distance of gesture center of gravity to the upper and lower border of boundary rectangle respectively, definition center of gravity and boundary rectangle up-and-down boundary ratio of distances constant:

$\mathrm{\β}=\frac{H_1}{H_2}$

Center of gravity and boundary rectangle right boundary ratio of distances constant:

W ₁, W ₂represent the distance of gesture center of gravity to the left and right border of boundary rectangle respectively, definition center of gravity and boundary rectangle left and right lower boundary ratio of distances constant:

$\mathrm{\η}=\frac{W_1}{W_2}$

Step 1.5): multiple features fusion:

By step 1.2) ~ 1.4) in the Hu moment characteristics, defect characteristic and six ratio characteristic that extract to permeate a proper vector feature, for characterizing the characteristic information of images of gestures:

feature＝{Hu,numDefects,ConLA,LenCR,AreaCR,α,β,η}

Wherein, numDefects is defect number, Hu=(φ ₁, φ ₂, φ ₃, φ ₄, φ ₅, φ ₆, φ ₇).

Step 1.6): support vector machine is trained:

The proper vector feature of above-mentioned gesture sample image is input in support vector machine together with classification mark and trains.

Step 2): identifying: gestures detection is carried out to input image sequence, multi-feature extraction and fusion are carried out to the gesture detected, and is input in support vector machine the recognition result obtaining SVM; Meanwhile, carry out detecting based on the finger tip of defect to gesture, comprehensive two times result exports final recognition result.

In the present embodiment, step 2) described in identifying, Fig. 1 is the process flow diagram of gesture identification of the present invention, and its detailed process is as follows:

Step 2.1): gestures detection:

Face Detection is carried out to the sequence of video images of input, adopt the skin color detection method of multiple color spaces component, set up a new color space HLS-CbCr color space, image is transformed on HLS-CbCr color space, by the colour of skin Sample Establishing complexion model extracted in advance, according to the complexion model distribution situation on HLS-CbCr color space, detect the area of skin color in image; Meanwhile, carry out the mixed Gaussian background modeling improved based on space time information, by setting up a mixture gaussian modelling for each background pixel, judging the background parts in image, thus extracting foreground area further.And detection zone R (x is set according to the result of Face Detection, y), for different learning rates is distributed in detection zone and non-detection zone, and record each pixel and be judged as background number of times, distribute different learning rates according to this number of times, thus detect the foreground area in image more quickly; After two kinds of testing results are comprehensively carried out filtering process and morphological operation, obtain the Hand Gesture Segmentation image of binaryzation.

Step 2.2): feature extraction:

To the binary map of gesture according to step 1.2) ~ 1.5) calculate the proper vector of this images of gestures;

Step 2.3): SVM identifies:

By in the proper vector of images of gestures input support vector machine, export SVM recognition result, recognition result be 0,1,2,3,4,5, the type of 8 seven kind of gesture.

Step 2.4): finger tip detects:

Fig. 3 is the process flow diagram that finger tip detects, and the number of finger tip is exactly the number of finger, the detection of finger tip is comprised to the positional information of finger tip number and finger tip.According to step 1.3) obtain three defect point ptStart that gesture defect and each defect comprise, ptEnd, ptFar, finger tip point is contained in these defect points, so set up an effective defect point screening washer, filter out effective defect point and be finger tip point.Screening washer rule is as follows:

I. the starting point of defect and depth point distance are greater than a certain proportion of gesture boundary rectangle height H

Lenght (ptStart, ptFar) > α H, wherein α is scale-up factor.

Ii. the depth point of defect and end point distance are greater than a certain proportion of gesture boundary rectangle height H

Lenght(ptEnd,ptFar)＞αH

Iii. the angle that the starting point of defect, depth point, end point are formed is less than threshold value T _angle

angle(ptStart,ptFar,ptEnd)＜T _angle

Iv. the starting point of defect, depth point, end point are in the certain limit of gesture boundary rectangle

y _bounding＜y _ptStar＜y _bounding+βH

y _bounding＜y _ptEnd＜y _bounding+βH

y _bounding＜y _ptFar＜y _bounding+βH

Wherein β is scale-up factor, and H is that gesture boundary rectangle is high.

V. when defect point spacing is less than T _distime, two defect points are approximate to be overlapped, and can be judged to be same defect point

Lenght(pt _i,pt _j)＜T _dis

When defect point meets above condition simultaneously, then can be judged to be effective defect point by screening washer, record effective defect point number and position thereof, be finger tip point.

Step 2.5): comprehensive recognition result:

The result and the finger tip that compare support vector machine output detect the finger tip point number exported, and when both are consistent, then export recognition result.

All experiments all realize on PC, and COMPUTER PARAMETER is: central processing unit Intel (R) Core (TM) i5CPU750@2.67GHz, internal memory 4.00GB.

Claims (3)

1., based on the gesture identification method that multiple features fusion and finger tip detect, it is characterized in that, comprise the following steps:

Step 1): training process: rational gesture feature is selected to complicated gesture, and utilize the feature extraction algorithm of multiple features fusion, support vector machine training is carried out to gesture, forms training pattern;

Step 2): identifying: gestures detection is carried out to input image sequence, multi-feature extraction and fusion are carried out to the gesture detected, and is input in support vector machine the recognition result obtaining SVM; Meanwhile, carry out detecting based on the finger tip of defect to gesture, the result that the recognition result of comprehensive SVM and finger tip detect exports final recognition result.

2. a kind of gesture identification method detected based on multiple features fusion and finger tip according to claim 1, is characterized in that, step 1) described in training process, its detailed process is as follows:

Step 1.1): feature extraction:

Feature extraction is carried out to images of gestures, extract the Hu moment characteristics of gesture, defect characteristic and ratio characteristic, described ratio characteristic comprises following six kinds of features: profile girth and contour area ratio, profile girth and boundary rectangle girth ratio, contour area and boundary rectangle area ratio, boundary rectangle the ratio of width to height, center of gravity and boundary rectangle up-and-down boundary ratio of distances constant, center of gravity and boundary rectangle right boundary ratio of distances constant;

Step 1.2): multiple features fusion:

By step 1.1) in the Hu moment characteristics, defect characteristic and six ratio characteristic that extract to permeate a proper vector, for characterizing the characteristic information of images of gestures;

Step 1.3): support vector machine is trained:

The proper vector of above-mentioned gesture sample image be input in support vector machine together with classification mark and train, wherein classification is labeled as the mark of gesture kind.

3. a kind of gesture identification method detected based on multiple features fusion and finger tip according to claim 1, is characterized in that, step 2) described in identifying, its detailed process is as follows:

Step 2.1): gestures detection:

Foreground detection and Face Detection are carried out to the sequence of video images of input, by comprehensive for both testing results, then carries out filtering process and morphological operation, obtain the Hand Gesture Segmentation image of binaryzation;

Step 2.2): feature extraction:

To the binary map of gesture according to step 1.1) ~ 1.2) carry out feature extraction and merge the proper vector obtaining this images of gestures;

Step 2.3): SVM identifies:

By in the proper vector of images of gestures input support vector machine, export SVM recognition result;

Step 2.4): finger tip detects:

Adopt the Fingertip Detection based on gesture defect, according to step 1.1) obtain three defect point ptStart that gesture defect and each defect comprise, ptEnd, ptFar, set up an effective defect point screening washer, filter out effective defect point and be finger tip point; Screening washer rule is as follows:

I. the starting point of defect and depth point distance are greater than a certain proportion of gesture boundary rectangle height H

Lenght (ptStart, ptFar) > α H, wherein α is scale-up factor;

Ii. the depth point of defect and end point distance are greater than a certain proportion of gesture boundary rectangle height H

Lenght(ptEnd,ptFar)＞αH

Iii. the angle that the starting point of defect, depth point, end point are formed is less than threshold value T _angle

angle(ptStart,ptFar,ptEnd)＜T _angle

Iv. the starting point of defect, depth point, end point are in the certain limit of gesture boundary rectangle

y _bounding＜y _ptStar＜y _bounding+βH

y _bounding＜y _ptEnd＜y _bounding+βH

y _bounding＜y _ptFar＜y _bounding+βH

Wherein β is scale-up factor, and H is that gesture boundary rectangle is high;

V. when defect point spacing is less than T _distime, two defect points are approximate to be overlapped, and can be judged to be same defect point

Lenght(pt _i,pt _j)＜T _dis

When defect point meets above condition simultaneously, then can be judged to be effective defect point by screening washer, record effective defect point number and position thereof, be finger tip point;

Step 2.5): comprehensive recognition result:

The result and the finger tip that compare support vector machine output detect the finger tip point number exported, and when both are consistent, then export recognition result.