CN101661556A - Static gesture identification method based on vision - Google Patents

Abstract

The invention provides a static gesture identification method based on vision, comprising the following steps of S1, gesture image pretreatment: separating a hand region from an environment accordingto the complexional characteristic of a human body and obtaining a gesture profile through image filtering and image morphological operation; S2, gesture characteristic parameter extraction: extracting an Hu invariable moment characteristic, a gesture region characteristic and a Fourier description subparameter so as to form a characteristic vector; and S3, gesture identification, using a multi-layer sensor classifier having self-organizing and self-studying abilities, capable of effectively resisting noise and treating incomplete mode, and having mode generalization ability. The static gesture identification method based on vision in the invention firstly carries out pretreatment and binarizes the original gesture image according to the complexional characteristic of the human body. The extracted gesture characteristic parameters are in three groups, namely the Hu invariable moment characteristic, the gesture region characteristic and the Fourier description subparameter, which form the characteristic vector together. The characteristic has better recognition rate.

Description

Static gesture identification method based on vision

Technical field

The present invention relates to a kind of control method, all many-sides such as particularly a kind of research, stunt in the film making that is applied to the teaching of area of computer aided sign language, the bilingual broadcast of TV programme, visual human handled, the making of animation, medical research, Entertainment, simultaneously also help to improve deaf-mute's life study and work condition, be their static gesture identification method based on vision for better service.

Background technology

The identification work to 46 sign language symbols had been finished in the Fujitsu laboratory in 1991

In Dec, 2003, the Cybernet system house of Michigan, USA develops the system of cover gesture storm by name, is that company develops at weather predicting program, and the host can control the process of forecast by simple gesture.

2008, the scientist that Toshiba Corp is located at a research laboratory of univ cambridge uk invented " gesture interaction technology " (gesture interface technology), and it can make the people who sees TV control TV by simple gesture.

There is following technical difficulty in the prior art:

(1) difficulty of gesture target detection

The real-time detection of target is meant that detecting target under with the complicated background condition the people from image stream comes, and this is one of problem of mainly studying of machine vision.

(2) difficulty of gesture Target Recognition

Gesture identification is to explain the implication that it is high-level according to the attitude of staff and change procedure, because gesture has following characteristics:

A) be elastomeric objects, so very big with difference between a kind of gesture;

B) hand has bulk redundancy information, is identification finger feature because the people discerns the gesture key, so the palm feature is a redundant information;

Therefore c) position is at three dimensions, is difficult to the location, and the image that computing machine obtains is three-dimensional projection to two dimension, so projecting direction is very crucial;

D) the surperficial right and wrong of hand are smooth, therefore easily produce shade.

Summary of the invention

The present invention the invention provides the good static gesture identification method based on vision of a kind of recognition effect in order to overcome above-mentioned the deficiencies in the prior art.

The technical solution adopted for the present invention to solve the technical problems is: a kind of static gesture identification method based on vision is provided, it may further comprise the steps: the pre-service of S1 images of gestures, from environment, cut apart the zone of selling according to the features of skin colors of human body, obtain the gesture profile by figure filtering and morphological image operation then; S2 gesture feature parameter extraction has extracted the Hu invariant moment features, gesture provincial characteristics and Fourier descriptors parameter, composition characteristic vector; The S3 gesture identification is used the multilayer perceptron sorter, and this sorter has self-organization and self-learning capability, and effectively antinoise is promoted ability with handling imperfect pattern and having pattern.

The scheme that the present invention solves further technical matters is: described step S1 comprises S11 binaryzation images of gestures, and denoising of S12 smothing filtering and S13 profile extract.

The scheme that the present invention solves further technical matters is: described S11 binaryzation images of gestures is according to the colour of skin information of human body original images of gestures to be carried out binary conversion treatment, obtains the staff zone.

The scheme that the present invention solves further technical matters is: described S12 smothing filtering denoising uses medium filtering and linear smoothing filtering adversary zone binary map to carry out denoising, obtains clearly gesture zone.

The scheme that the present invention solves further technical matters is: described S13 profile extracts and adopts Laplce's edge extracting algorithm.

The scheme that the present invention solves further technical matters is: described gesture provincial characteristics comprises the ratio of gesture region area and gesture zone rectangular area, the length breadth ratio in gesture zone, the ratio of the two sections that the center of gravity of gesture binary map is cut apart the gesture boundary rectangle, the girth of hand region area and gesture profile.

Compared to prior art, static gesture identification method based on vision of the present invention is before carrying out characteristic parameter extraction to original images of gestures, carry out pre-service earlier, according to the features of skin colors of human body with original images of gestures binaryzation, and to the method denoising of the binary map of images of gestures by the figure image intensifying, obtain not having the binary map of noise, then binary map is carried out edge extracting, obtain the profile of gesture, the gesture feature parameter of extraction has three groups, be respectively the Hu invariant moment features, gesture provincial characteristics and Fourier descriptors are formed proper vector jointly, and this stack features has the better recognition rate, make recognition effect good, the discrimination height.

Description of drawings

Fig. 1 is the principle module diagram of the static gesture identification method based on vision of the present invention.

Embodiment

As shown in Figure 1, the invention provides a kind of static gesture identification method based on vision, it may further comprise the steps: the pre-service of S1 images of gestures, from environment, cut apart the zone of selling according to the features of skin colors of human body, and obtain the gesture profile by figure filtering and morphological image operation then; S2 gesture feature parameter extraction has extracted the Hu invariant moment features, gesture provincial characteristics and Fourier descriptors parameter, composition characteristic vector; S3 gesture identification part is used the multilayer perceptron sorter, and this sorter has self-organization and self-learning capability, and effectively antinoise is promoted ability with handling imperfect pattern and having pattern.

Described step S1 comprises S11 binaryzation images of gestures, and denoising of S12 smothing filtering and S13 profile extract.

S11 binaryzation images of gestures: the colour of skin information according to human body is carried out binary conversion treatment to original images of gestures, obtains the staff zone, and algorithm is as follows:

Each pixel for original image

{

Calculate the Hue and the Satisfaction value of this pixel by rgb value;

Judge that the Hue of this pixel and Satisfaction value are whether all in this people's colour of skin interval;

If this pixel would be become black;

Otherwise this pixel is become white;

}

Through binary conversion treatment, can obtain the binary map in gesture zone.

The denoising of S12 smothing filtering: because mainly there is the influence of salt-pepper noise in the binary map after the binaryzation, native system uses medium filtering and linear smoothing filtering adversary zone binary map to carry out denoising, obtains clearly gesture zone.

The S13 profile extracts: adopt Laplce's edge extracting algorithm, Laplace operator is a kind of second derivative scalar operator that two-dimensional function is carried out computing, therefore quite responsive to the noise in the image, and when handling, can produce a precipitous zero crossing in edge, therefore if, can detect by Laplace operator for noiseless and have the image of brink.

S2 gesture feature parameter extraction: to the gesture model that obtains after pretreated, be gesture zone binary map and gesture profile, carry out the gesture model characteristic parameter extraction, the gesture feature parameter that the present invention will extract has three groups, be respectively the Hu invariant moment features, gesture provincial characteristics and Fourier descriptors, form proper vector jointly:

＜1〉gesture provincial characteristics:

struct?REGION_FEATURE

{

Double area_ratio; The ratio of // gesture region area and gesture zone rectangular area

Double aspect_ratio; The length breadth ratio in // gesture zone

Double bary_ratio; The center of gravity of // gesture binary map is with gesture boundary rectangle gesture zone square

The ratio of the two sections that shape is cut apart

Double hand_area; // hand region area is represented with number of pixels

Double hand_perimeter; The girth of // gesture profile

}；

＜2〉Hu invariant moment features

struct?CvHuMoments

{

double?hu1，hu2，hu3，hu4，hu5，hu6，hu7

}；

＜3〉Fourier descriptors

struct?FOURIER_FEATURE

{

vector<double> feature_data；

}；

The calculating of gesture zone rectangle

int?cvFindContours(

CvArr*image，

CvMemStorage*storage，

CvSeq**first_contour，

int?header_size＝sizeof(CvContour)，

int?mode＝CV_RETR_LIST，

int?method＝CV_CHAIN_APPROX_SIMPLE，

CvPoint?offset＝cvPoint(0，0))；

The major parameter explanation:

The image:8-bit, the single channel image. nonzero element is left 0 as 1,0 pixel value

Storage: the storage container of the profile that obtains

First_contour: output parameter: the pointer that comprises first output profile

Function cvFindContours extracts profile from bianry image, and returns the number that extracts profile.The content of pointer first_contour is filled in by function.It comprises the pointer of first outermost layer profile, if pointer is NULL, does not then detect profile (is complete black such as image).Other profile can from first_contour utilize h_next and v_next links and accesses to.

Then by CvRect gesture zone rectangle=((CvContour*) contour)-＞rect, just can obtain the regional rectangle of gesture.

Computing method about the hand_area feature:

After gesture zone rectangle calculates, below the calculating of all feature all be in the rectangular area, gesture zone of image, to carry out.The area hand_area of gesture is the sum of all pixels that the hand zone comprises, and by rectangular area, gesture zone in the scanning binary map, the number of calculating black color dots can draw.

area＝∑∑f(x，y) (4-1)

Wherein

The built-in function cvContourArea that we can call OpenCV calculates hand_area, and function prototype is as follows:

double?cvContourArea(

const?CvArr*contour，

CvSlice?slice＝CV_WHOLE_SEQ)；

Parameter c ontour is an image outline, and slice then is the starting point of interest outline portion, and default is the area that calculates whole profile.The rreturn value of function is the area of whole profile.

Computing method about the area_ratio feature:

Area_ratio=hand_area/ (gesture zone rectangle .width* gesture zone rectangle .height)

Computing method about the aspect_ratio feature:

Rectangle .widht/ (float) gesture zone, aspect_ratio=(float) gesture zone rectangle .height

Computing method about the hand_perimeter feature:

The girth hand_perimeter of gesture profile is the sum of all pixels that the gesture contour images comprises in rectangular area, gesture zone, by rectangular area, gesture zone in the scanning gesture profile diagram, the number of calculating black color dots can draw:

hand_perimeter＝∑∑f(x，y)

Wherein

The built-in function cvArcLength that we can call OpenCV calculates hand_perimeter, and function prototype is as follows:

double?cvArcLength(

const?void*curve，

CvSlice?slice＝CV_WHOLE_SEQ，

int?is_closed＝-1)；

Parameter c urve is curve sequence of point sets or array, and slice then is the starting point of curve, and default is the length of calculating entire curve, and parameter is_closed represents whether curve is closed.The rreturn value of function is to calculate profile girth or length of curve.

The computing method of Hu invariant moment features:

1962, M.K.Hu has at first proposed invariant moments, he has provided the definition of continuous function square and about the fundamental property of square, the character such as translation invariance, rotational invariance and constant rate of relevant square have been proved, specifically provided the expression formula of seven invariant moments, embodied formula such as formula with translation invariance, rotational invariance and constant rate

φ ₁＝μ ₂₀+μ ₀₂

φ ₂＝(μ ₂₀-μ ₀₂) ²+(2μ ₁₁) ²

φ ₃＝(μ ₃₀-3μ ₁₂) ²+(3μ ₂₁-μ ₀₃) ²

φ ₄＝(μ ₃₀+μ ₁₂) ²+(μ ₂₁+μ ₀₃) ²

φ ₅＝(μ ₃₀-3μ ₁₂)(μ ₃₀+μ ₁₂)[(μ ₃₀+μ ₁₂) ²-3(μ ₂₁+μ ₀₃) ²]+(3μ ₂₁-μ ₀₃)(μ ₂₁+μ ₀₃)[3(μ ₃₀+μ ₁₂) ²-(μ ₂₁+μ ₀₃)

φ ₆＝(μ ₂₀-μ ₀₂)[(μ ₃₀+μ ₁₂) ²-(μ ₂₁+μ ₀₃) ²]+4μ ₁₁(μ ₃₀+μ ₁₂)(μ ₂₁+μ ₀₃)

φ ₇＝(3μ ₂₁-μ ₀₃)(μ ₃₀+μ ₁₂)[(μ ₃₀+μ ₁₂) ²-3(μ ₂₁+μ ₀₃) ²]-(μ ₃₀-3μ ₁₂)(μ ₂₁+μ ₀₃)[3(μ ₃₀+μ ₁₂) ²-(μ ₂₁+μ ₀₃)

Normalized central moment η _PqBe defined as:

${\mathrm{\&eta;}}_{\mathrm{pq}}=\frac{{\mathrm{\&mu;}}_{\mathrm{pq}}}{{\mathrm{\&mu;}}_{00}^r}$

Wherein $r=\frac{p+q+2}{2},$ p+q＝2，3，.....

The main code of computed image Hu square

void?CFeature::CalcHuMoments(const?IplImage?*img，CvHuMoments

&humoments)

{

CvMoments moments；

cvMoments(img，&moments，0)；

cvGetHuMoments(&moments，&humoments)；

}

The computing method of Fourier descriptors

Fourier transform is a kind of of line style conversion, and it provides a kind of solution that solves the line style system problem.Fourier transform plays an important role in the theory of a lot of subjects, although can be as treating other conversion, Fourier transform is regarded as the function of pure mathematics, but in a lot of fields, Fourier transform produces their the same clear physical meaning of function too.

Individual with N apart from one another by the sampling function f of Δ x unit (x), make it become series

{f(x ₀)，f(x ₀+Δx)，f(x ₀+2Δx，....f(x ₀+[N-1]Δx)}

Regulation f (x)=f (x ₀+ Δ x), at this moment x is discrete value x=0,1,2 ..., N 1, and an above-mentioned sequence table is shown

{f(0)，f(1)，f(2)，...，f(N-1)}

At this moment Fourier transform is to being

$F\left(u\right)=\frac{1}{N}\underset{x=0}{\overset{N-1}{\mathrm{\&Sigma;}}}f\left(x\right)\exp [-j2\mathrm{\&pi;ux}/N]$

$f\left(x\right)=\underset{u=0}{\overset{N-1}{\mathrm{\&Sigma;}}}F\left(u\right)\exp [j2\mathrm{\&pi;ux}/N]$

F in the formula (u)=F (u Δ u) u=0,1,2 ... N-1, and $\mathrm{\&Delta;u}=\frac{1}{\mathrm{N\&Delta;x}},$ Corresponding to 0, Δ u, 2 Δ u .... (N-1) Δ u

Fourier transform is right $f\left(x\right)\&DoubleLeftRightArrow;F\left(u\right)$

Discrete Fourier transform (DFT) satisfies orthogonality condition

$\frac{1}{N}\underset{x=0}{\overset{N-1}{\mathrm{\&Sigma;}}}\exp [j2\mathrm{\&pi;}{u}_{1}x/N]\exp [-j2\mathrm{\&pi;}{u}_{2}x/N]=\left\{\begin{array}{cc}1& \mathrm{if}(u_1=u_2)\\ 0& \mathrm{elsewhere}\end{array}\right.$

In fact, often make N=2 ^m

Fourier descriptors (Fourier DescriPtors) is the fourier transform coefficient of body form boundary curve, and it is the result of the frequency-domain analysis of object boundary curve signal, is the method that a kind of description is not subjected to moving of initial point and the curve that rotation influences.

The basic thought of Fourier descriptors is that the shape of supposition object is that the curve of a sealing is along the sequence of being had a few on the boundary curve: and x (l), y (l): l=0,1 ..., n-1} is expressed as with plural form:

P(l)＝x(l)+jy(l) (l＝0，1，....，n-1， $j=\sqrt{-1}$ )

Like this, the border just can be represented on the one-dimensional space.

The discrete Fourier coefficient of one-dimensional sequence is defined as:

$z\left(k\right)=\frac{1}{n}\underset{l=0}{\overset{n-1}{\mathrm{\&Sigma;}}}p\left(l\right)\exp (-j\frac{2\mathrm{\&pi;lk}}{n})$ (k＝0，1，....n-1)

Z is the Fourier transform of p, is the expression of point sequence in frequency domain.Its inverse fourier transform is

$P\left(l\right)=p\left(l\right)=\underset{k=0}{\overset{n-1}{\mathrm{\&Sigma;}}}z\left(k\right)\exp \left(j\frac{2\mathrm{\&pi;}\mathrm{lk}}{n}\right)$ (l＝0，1，....n-1)

Utilize character z (the k)=z* (n-k) (z* is the conjugate complex number of Z) of fourier coefficient, in coefficient Z, cancellation from $K+l(0\&le;K<\frac{n}{2}-1)$ To the interior radio-frequency component of n-K-1 scope.Carry out inverse fourier transform again, will obtain approximate curve with virgin curve, but the part of suddenling change in the virgin curve will become smoothly, and this curve that is similar to is called the K curve of approximation of virgin curve.{ z (k): k≤K} is called Fourier descriptors (Fourier Descriptor) to fourier coefficient subclass under this meaning.Because the characteristic that fourier coefficient has energy to concentrate to low frequency, so just can reach the purpose on differentiation difformity border with less coefficient.

Fourier descriptors is relevant with the initial point position of yardstick, direction and the curve of shape.Have rotation in order to discern, the shape of translation and yardstick unchangeability, need carry out normalization to Fourier descriptors.According to Fourier transform character, with body form border initial point position translation a length, object amplifies r doubly, anglec of rotation duty and translational displacement (x0, y0) after, the fourier transform coefficient z ' of new shape (k):

K=0 wherein, 1 ... .n-1, x ' (l)+iy ' (l)=x (l+a)+iy (l+a)

As can be seen from the above equation, when describing shape with fourier coefficient, coefficient amplitude Z (k), k=0,1 ... .n-1, have rotational invariance and translation invariance (Z (0) does not have translation invariance), and irrelevant with the selection of spring of curve.When object translation, only change its Z (0) component F (x ₀+ iy ₀) value.Amplitude Z (k) of each coefficient (except the Z (0)) divided by Z (l), then

Do not change with change of scale, so

K=1,2 ... .., n-1, represents delivery Have rotation simultaneously, translation and yardstick unchangeability, and irrelevant with the selection of spring of curve position, we it as Fourier descriptors, so normalized Fourier descriptors d (k) is defined as:

K=1,2 ... n-1, represents delivery

Native system is got preceding 12 coefficients except that z (0), obtains the proper vector of one 12 dimension, and this characteristic sentence measurer has rotation, translation and yardstick unchangeability, and flat and irrelevant with the selection of the start position of curve, specific algorithm is as follows:

CFeature::CalcFourierFeature(const?IplImage*img，FOURIER_FEATURE

&fourier_feature)

{

assert(img！＝NULL)；

Int MAX_POINT_NUMBER=1024; // suppose that the quantity of point mostly is most

1024

IplImage *clone_img＝cvCloneImage(img)；

CvMemStorage*storage＝cvCreateMemStorage(0)；

CvSeq *contour；

int?contour_count＝cvFindContours(

clone_img，

storage，

&contour，

sizeof(CvContour)，

CV_RETR_EXTERNAL，

CV_CHAIN_APPROX_NONE，

cvPoint(0，0))；

double?img_area＝img-＞width*img-＞height；

for(；contour；contour＝contour-＞h_next)

{

double?contour_area＝fabs(cvContourArea(contour，

CV_WHOLE_SEQ))；

if(contour_area/img_area＞＝0.05)

break；

}

Int point_count=contour-＞total; The quantity of // point

//assert(point_count＜MAX_POINT_NUMBER)；

MAX_POINT_NUMBER＝point_count；

if(point_count＞MAX_POINT_NUMBER)

{

fourier_feature.feature_data.clear()；

return?false；

}

CvPoint*pt＝new?CvPoint[point_count]；

cvCvtSeqToArray(contour，pt，CV_WHOLE_SEQ)；

CvMat?*real_input＝cvCreateMat(1，MAX_POINT_NUMBER，

CV_64F)；

CvMat?*imag_input＝cvCreateMat(1，MAX_POINT_NUMBER，

CV_64F)；

CvMat?*comp_input＝cvCreateMat(1，MAX_POINT_NUMBER，

CV_64FC2)；

cvZero(real_input)；

cvZero(imag_input)；

cvZero(comp_input)；

for(int?i＝0；i＜point_count；i++)

{

CvSetReal2D (real_input, 0, i, pt[i] .x); // x is set to real part

CvSetReal2D (imag_input, 0, i, pt[i] .y); // y is set to imaginary part

}

cvMerge(real_input，imag_input，NULL，NULL，comp_input)；

cvDFT(comp_input， comp_input，

CV_DXT_FORWARD|CV_DXT_SCALE, comp_input-＞heigbt); // Fourier becomes

Change

CvMat?*real_output＝cvCreateMat(1，MAX_POINT_NUMBER，

CV_64F)；

CvMat?*imag_output＝cvCreateMat(1，MAX_POINT_NUMBER，

CV_64F)；

CvMat*spectrum＝cvCreateMat(1，MAX_POINT_NUMBER，

CV_64F); // frequency spectrum

cvSplit(comp_input，real_output，imag_output，NULL，NULL)；

cvPow(real_output，real_output，2.0)；

cvPow(imag_output，imag_output，2.0)；

cvAdd(real_output，imag_output，spectrum，NULL)；

cvPow(spectrum，spectrum，0.5)；

double?factor＝1.0/(cvGetReal2D(spectrum，0，0))；

cvScale(spectrum，spectrum，factor，0)；

for(int?i＝1；i＜11；i++)

{

double?d＝cvGetReal2D(spectrum，0，i)；

fourier?feature.featuredata.push?back(d)；

}

cvReleaseMat(&real_input)；

cvReleaseMat(&imag_input)；

cvReleaseMat(&comp_input)；

cvReleaseMat(&real_output)；

cvReleaseMat(&imag_output)；

cvReleaseMat(&spectrum)；

cvReleaseImage(&clone_img)；

cvReleaseMemStorage(&storage)；

delete[]pt；

return?true；

}

Static gesture identification method based on vision of the present invention is before carrying out characteristic parameter extraction to original images of gestures, carry out pre-service earlier, according to the features of skin colors of human body with original images of gestures binaryzation, and to the method denoising of the binary map of images of gestures by the figure image intensifying, obtain not having the binary map of noise, then binary map is carried out edge extracting, obtain the profile of gesture, the gesture feature parameter of extracting has three groups, is respectively the Hu invariant moment features, gesture provincial characteristics and Fourier descriptors, the common proper vector of forming, this stack features has the better recognition rate, makes recognition effect good, the discrimination height.

Claims (9)

1. static gesture identification method based on vision, it may further comprise the steps: the pre-service of S1 images of gestures, from environment, cut apart the zone of selling according to the features of skin colors of human body, obtaining the gesture profile by figure filtering and morphological image operation then; S2 gesture feature parameter extraction has extracted the Hu invariant moment features, gesture provincial characteristics and Fourier descriptors parameter, composition characteristic vector; The S3 gesture identification is used the multilayer perceptron sorter, and this sorter has self-organization and self-learning capability, and effectively antinoise is promoted ability with handling imperfect pattern and having pattern.

2. the static gesture identification method based on vision according to claim 1 is characterized in that: described step S1 comprises S11 binaryzation images of gestures, and denoising of S12 smothing filtering and S13 profile extract.

3. the static gesture identification method based on vision according to claim 2 is characterized in that: described S11 binaryzation images of gestures is according to the colour of skin information of human body original images of gestures to be carried out binary conversion treatment, obtains the staff zone.

4. the static gesture identification method based on vision according to claim 2 is characterized in that: described S12 smothing filtering denoising uses medium filtering and linear smoothing filtering adversary zone binary map to carry out denoising, obtains clearly gesture zone.

5. the static gesture identification method based on vision according to claim 4 is characterized in that: described S13 profile extracts and adopts Laplce's edge extracting algorithm.

6. the static gesture identification method based on vision according to claim 1, it is characterized in that: described gesture provincial characteristics comprises the ratio of gesture region area and gesture zone rectangular area, the length breadth ratio in gesture zone, the ratio of the two sections that the center of gravity of gesture binary map is cut apart the gesture boundary rectangle, the girth of hand region area and gesture profile.

7. the static gesture identification method based on vision according to claim 6, it is characterized in that: after gesture zone rectangle calculates, the area of gesture (area) is the sum of all pixels that the hand zone comprises, by rectangular area, gesture zone in the scanning binary map, the number of calculating black color dots can draw

area＝∑∑f(x，y)

Wherein

8. the static gesture identification method based on vision according to claim 6, it is characterized in that: the girth (hand perimeter) of the girth gesture profile of gesture profile is the sum of all pixels that the gesture contour images comprises in rectangular area, gesture zone, by rectangular area, gesture zone in the scanning gesture profile diagram, the number of calculating black color dots can draw:

hand_perimeter＝∑∑f(x，y)

Wherein

9. the static gesture identification method based on vision according to claim 1 is characterized in that: described Fourier descriptors parameter will be carried out normalization when calculating.

Images