CN104808788A - Method for controlling user interfaces through non-contact gestures - Google Patents

Abstract

The invention relates to a method for controlling a computer, a touch screen, a projector and other user interfaces through Kinect non-contact gestures. The method comprises the main steps of acquiring actions of double arms of an operator and using a gesture recognition algorithm to recognize gestures, utilizing recognized gestures to achieve interaction between the operator and the computer, the touch screen, projector and other user interfaces. The method relates to six gestures including single click, right-key single click, double clicks, forward rolling, back rolling, zooming in, zooming out and dragging and basically covers the demands for interaction between the operator and the computer, the touch screen, projector and other user interfaces. The method is a novel recognition method, gesture recognition accuracy can be improved, and recognition rapidity can be improved. By the combination of the six gestures, man-machine interaction can be convenient and flexible.

Description

A kind of method of non-contact gesture manipulation user interface

Technical field

The present invention relates to a kind of non-contact gesture exchange method, the user interface such as a kind of gesture based on Kinect and computer, touch-screen and projection carries out contactless mutual method specifically.

Background technology

Enter 21 century, hardware performance and the popularity of the electronic equipment such as mobile phone, computing machine improve constantly, and touch-screen comes into vogue.Touch operation people the are departed from constraint of keyboard and mouse, the object that direct control screen shows, more meets cognition and the habits and customs of people, becomes the trend of recent man-machine interaction mode.But the appearance of screen along with more and more variety classes and specification, inconvenience and the limitation of touch operation also reveal gradually: small size touch-screen has only changed a kind of mouse and keyboard of form, does not allow user break away from the constraint of hardware yet; Hang touch-screen size on the wall and can reach 60 even 100 inches, screen must be walked close to during operation, make user cannot see object on full screen, the huge area that object occupies on screen in addition also makes dragging, convergent-divergent and the basic operation such as to choose to become very clumsy, so that cannot complete; In some sight, user does not allow to touch operating equipment, such as implementing perform the operation doctor can not touch any object that may carry disease germs, in order to operating electronic equipment, have to repeatedly after operation terminates opponent carry out disinfection.In Public Service Field, the application of touch-screen also encounters some insoluble bottlenecks.

Along with developing rapidly of human-computer interaction technology, people want to make man-machine interaction become more natural and direct, and traditional interactive mode, the such as hardware device such as keyboard, mouse, obviously our requirement has not been reached, it is active that the human-computer interaction technology that research meets the interchange custom of people more becomes very in recent years, man-machine interaction from before centered by computing machine, progressively transfer to that focus be put on man, this trend has been complied with in the man-machine interaction research of view-based access control model just.

Research shows, the acquisition more than 80% of human information comes from vision, and gesture has very strong visual effect, vivid and directly perceived.The Gesture Recognition of view-based access control model be also applied to gradually Sign Language Recognition, letter and Chinese Character Recognition, slide demonstration, light fixture switch control rule, game controls, TV remote, browse webpage and control, point multiple fields such as drawing.

Summary of the invention

In order to solve above technical matters, this invention takes following technical scheme: a kind of non-contact gesture control method based on Kinect, Kinect is adopted to gather the action of user's both arms skeletal joint, identify gesture by Gesture Recognition Algorithm, finally utilize the gesture identified to carry out operation to user interfaces such as computer, touch-screen and projections and control.

The gesture identification method that the present invention takes comprises following steps:

Step 1, by the action of camera collection operator both arms skeletal joint;

If controlled device for touch large-size screen monitors, then Kinect camera is arranged on touch large-size screen monitors center above; If controlled device is projection or computer, then Kinect is placed on the position of projection or computer side convenient operation.By the action of Kinect camera collection operator both arms skeletal joint.

Step 2, carry out Effective judgement to the image collected, concrete determination methods A is as follows:

The three-dimensional coordinate information data of the skeletal joint point of camera collection operator; When the skeletal joint point of the operator collected in every two field picture is greater than 14, and when containing the three-dimensional coordinate data of the middle shoulder of operator, right and left shoulders, left and right elbow, left and right wrist and right-hand man joint totally 9 articulation points in these 14 skeletal joint points, then regarded as effective image frame data, otherwise, it is invalid to be considered as, and gives up;

Wherein, Kinect camera can gather at most the three-dimensional coordinate information data of 20 the skeletal joint points appearing at its camera previous action person, and this three-dimensional data has three state value: null value, actual value, estimated value;

Then judge whether the left and right arm in effective image is valid function arm respectively, be specially: shoulder joint is less than d with the absolute value m of the difference of corresponding wrist joint y-axis coordinate and is effective arm, otherwise be invalid operation, wherein the coordinate of 9 skeletal joint points is expressed as follows: middle shoulder S _c(x _sc, y _sc, z _sc)=(0,0,0), left shoulder S _l(x _sl, y _sl, z _sl), right shoulder S _r(x _sr, y _sr, z _sr), left elbow E _l(x _el, y _el, z _el), right elbow E _r(x _er, y _er, z _er), left wrist W _l(x _wl, y _wl, z _wl), right wrist W _r(x _wr, y _wr, z _wr), left hand H _l(x _hl, y _hl, z _hl), right hand H _r(x _hr, y _hr, z _hr).

Wherein: m=|y _si-y _wi|,

$d=0.9*(\sqrt{{(x_{\mathrm{si}}-x_{\mathrm{ei}})}^2{(y_{\mathrm{si}}-y_{\mathrm{ei}})}^2+{(z_{\mathrm{si}}-z_{\mathrm{ei}})}^2}+\sqrt{{(x_{\mathrm{ei}}-x_{\mathrm{wi}})}^2+{(y_{\mathrm{ei}}-y_{\mathrm{wi}})}^2+{(z_{\mathrm{ei}}-z_{\mathrm{wi}})}^2})$

, i gets r or l;

Step 3, carry out gesture identification to the image of the effective operator arm of the existence collected, concrete recognition methods A is as follows:

3.1) three-dimensional coordinate data of the articulation point in image is converted to the three-dimensional data under human body coordinate system, wherein said human body coordinate system is for origin (0 with middle shoulder joint, 0,0), human body place plane is X-Y plane, direction is Y-axis positive dirction straight up, and human body is being just Z axis negative direction to direction, and X-Y-Z observes right-hand screw rule;

Wherein, when adopting Kinect camera collection image, the three-dimensional data three-dimensional data under Kinect space coordinates be converted under human body coordinate system is needed, as shown in Figure 4;

3.2) shoulder in the t time period is extracted, right and left shoulders, left and right elbow, three-dimensional coordinate data under the human body coordinate system of left and right wrist and right-hand man joint totally 9 articulation points, take in calculating, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, in these 9 articulation points of the right hand, elbow joint is to the vector U of corresponding shoulder joint node and the angle α of elbow joint to the vector V of corresponding wrist joint point, elbow joint is to the corresponding vector V of wrist joint point and the angle β of X-axis positive dirction, wrist joint to corresponding swivel of hand vector W and elbow joint to corresponding wrist joint point vector V between angle ω, pass through α, β, ω identifies gesture steering order,

The computing formula that wherein right hand is corresponding is as follows, and the computing formula that left hand is corresponding in like manner can obtain, namely only need by the coordinate in following formula all correspondence replace to left hand coordinate,

$\mathrm{\α}=\arccos \frac{(x_{\mathrm{wr}}-x_{\mathrm{er}})*(x_{\mathrm{sr}}-x_{\mathrm{er}})+(y_{\mathrm{wr}}-y_{\mathrm{er}})*(y_{\mathrm{sr}}-y_{\mathrm{er}})+(z_{\mathrm{wr}}-z_{\mathrm{er}})*(z_{\mathrm{sr}}-z_{\mathrm{er}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}*\sqrt{{(x_{\mathrm{sr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{sr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{sr}}-z_{\mathrm{er}})}^2}}$

$\mathrm{\β}=\arccos \frac{(z_{\mathrm{wr}}-z_{\mathrm{er}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}}$

$w=\arccos \frac{(x_{\mathrm{wr}}-x_{\mathrm{er}})*(x_{\mathrm{hr}}-x_{\mathrm{wr}})+(y_{\mathrm{wr}}-y_{\mathrm{er}})*(y_{\mathrm{hr}}-y_{\mathrm{wr}})+(z_{\mathrm{wr}}-z_{\mathrm{er}})*(z_{\mathrm{hr}}-z_{\mathrm{wr}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}*\sqrt{{(x_{\mathrm{hr}}-x_{\mathrm{wr}})}^2+{(y_{\mathrm{hr}}-y_{\mathrm{wr}})}^2+{(z_{\mathrm{hr}}-z_{\mathrm{wr}})}^2}}$

Identify that the method for gesture steering order is as follows by α, β, ω:

Time effective for single arm,

When diminishing after α first change in time t is large, be double-click gesture;

When α first diminish in time t become large afterwards time, be right-click gesture;

When diminishing after ω first change in time t is large, it is click gesture;

When ω occur to become in time t large after constant trend time, for choosing gesture, the trend until ω diminishes in another one time t, namely discharges gesture, and from choosing to, to discharge whole process be drag gesture;

When β occur first to become in time t large after the trend that diminishes, α does not change in time t simultaneously, for after roll gesture; When β occur first to diminish in time t become large trend afterwards time, simultaneously α does not change, for front rolling gesture in time t;

Time effective for two arms,

When right arm meet angle β occur first to become in time t large after the trend that diminishes, left arm meets angle β and occurs first to diminish when becoming large trend afterwards in time t, for reducing gesture simultaneously; Occur first to diminish in time t to become large trend afterwards when right arm meets angle β, simultaneously left arm meet angle β occur first to become in time t large after diminish trend time, be amplifying gesture.

Step 4, the outside input operation in the gesture steering order identified and controlled device system interface program to be mapped, realize the control of user interface.

Described gesture identification method A can also adopt gesture identification method B, specific as follows:

Extract the three-dimensional coordinate data under the human body coordinate system of shoulder in the t time, right and left shoulders, left and right elbow, left and right wrist and right-hand man joint totally 9 articulation points, using the three-dimensional coordinate data under the human body coordinate system of shoulder middle in the t time period, left shoulder, left elbow, left wrist, these 5 articulation points of left hand as M group, using the three-dimensional coordinate data under the human body coordinate system of shoulder middle in the t time period, right shoulder, right elbow, right wrist, these 5 articulation points of the right hand as N group, record the change of each articulation point x, y, z coordinate within the t time in two groups respectively, namely form test template; In in the t time of having deposited in advance in shoulder, these 9 articulation points of left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, right wrist, the right hand each skeletal joint point human body coordinate system under three-dimensional coordinate over time trend be reference template; Then utilize dynamic time warping DTW algorithm, calculate the similarity between test template and reference template, if calculated value is greater than given threshold value, then the match is successful for gesture, otherwise it fails to match;

Described reference template comprises following content:

Time effective for single arm:

When in arm, shoulder joint, shoulder joint, elbow joint, wrist joint coordinate do not change in time within the t1 time, and the y-axis coordinate of swivel of hand first diminishes in time and becomes large afterwards, the z-axis coordinate of swivel of hand first becomes large in time or diminishes simultaneously, and when variation range is less than b centimetre, i.e. click gesture;

When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and swivel of hand and carpal y-axis coordinate first diminish in time and become large afterwards, diminish after swivel of hand and carpal z-axis coordinate first change in time is simultaneously large, and variation range is when being less than a centimetre, namely double-click gesture;

When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and diminish after the first change greatly in time of swivel of hand and carpal y-axis coordinate, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, i.e. right-click gesture;

When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and swivel of hand and carpal x-axis coordinate first diminish in time and become large afterwards, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, namely roll gesture afterwards; When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t time, and diminish after the first change greatly in time of swivel of hand and carpal x-axis coordinate, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, namely before roll gesture;

When in arm, shoulder joint, shoulder joint, elbow joint, wrist joint coordinate do not change in time within the t1 time, and the y-axis coordinate of swivel of hand diminishes in time, the z-axis coordinate of swivel of hand becomes large in time simultaneously, and when variation range is less than b centimetre, namely chooses gesture; Until the y-axis coordinate of swivel of hand becomes large in time, the z-axis coordinate of swivel of hand diminishes in time simultaneously, and when variation range is less than b centimetre, namely discharges gesture; From choosing to, to discharge whole process be drag gesture;

Time effective for two arms:

When in two arms, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and the carpal x-axis coordinate in right hand joint and the right side first diminishes in time and becomes large afterwards, the carpal x-axis coordinate in left hand joint and a left side first becomes in time greatly and diminishes, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, namely reduce gesture;

When in two arms, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and diminish after the first change greatly in time of the carpal x-axis coordinate in right hand joint and the right side, the carpal x-axis coordinate in left hand joint and a left side first diminishes in time and becomes large afterwards, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, i.e. amplifying gesture;

The span of above-mentioned a, b is 10<a<30,5<b<10.

Described gesture identification method A can also adopt the mode be combined by method A and method B, is specially: first using method A, if identify successfully, stops; If unidentified go out gesture, then identify with method B again, if identify successfully, stop; If all unidentified go out gesture; re-start gesture identification.

As a kind of preferred version, operator described in step 1 stands or is seated at, 1.8 ~ 3.0 meters, Kinect video camera dead ahead, and optimum distance is about 2 meters.

As a kind of preferred version, the three-dimensional coordinate of described Kinect is: the camera axis parallel of Z axis and Kinect video camera, and the direction according to right-hand screw rule definition and horizontal direction parallel is X-axis, and vertical direction is Y-axis, as shown in Figure 3;

The gesture motion of single arm does not distinguish right-hand man, and what can complete with user interfaces such as computer, touch-screen and projections is alternately contactless.

The frame per second of described Kinect camera is 30 frames/second, and the scope of the execution time t of each gesture is 0.5s-1.5s.

Beneficial effect:

(1) the present invention be used for controlling user interfaces such as computer, touch-screen and projections, can operate on it without contact screen, interactive mode nature more directly perceived, also enhances the experience sense of user simultaneously.In the present invention, user does not need to wear any label.

(2) human hand and arm joint degree of freedom is combined with gesture design, by the restriction relation between shoulder joint 3 degree of freedom, elbow joint 1 degree of freedom, wrist joint 3 degree of freedom with click, double-click, right-click, Scalable, front roll after roll and drag 6 kinds of gestures and combine, truly achieve the target of human body as controller.The three degree of freedom of shoulder joint comprises anterior flexion and rear stretching, outreach adduction and inward turning outward turning; Elbow joint has one degree of freedom and anterior flexion and rear stretching; Carpal three degree of freedom comprises anterior flexion and rear stretching, outreach adduction and turns over interior turn of the palm.7 degree of freedom can meet the requirement of arm as controller completely.

(3) consider that operator needs significantly action just can be identified in gesture interaction system in the past, makes user in reciprocal process easily produce sense of fatigue.7 the degree of freedom power consumption situation in motion process of the present invention's design by considering human arm, the mobile position of gesture motion is concentrated on forearm, adopt the algorithm optimized, the unnecessary movement of arm joint can be reduced in identifying, have great importance in actual applications.In conjunction with human hand and arm joint limit of sports record, gesture design is more abundant.

(4) the present invention does not relate to the identification to finger-joint, so palm can arbitrarily open or close up when doing any gesture motion.

(5) only need follow the tracks of and process the coordinate data of middle shoulder in effective image frame, right and left shoulders, left and right elbow, left and right wrist and right-hand man joint totally 9 articulation points, avoiding the occlusion issue in tracing process, processing speed and accuracy of identification can be improved simultaneously.

(6) dynamic time warping (DTW) algorithm is retrained to combine with the distinctive limb motion of human body carry out the identification of gesture, improve the efficiency of identification, enhance the practicality of system.

Accompanying drawing explanation

The human body that the Kinect camera that accompanying drawing 1 adopts for the present invention can collect 20 articulation points;

Accompanying drawing 2 is technical solution of the present invention process flow diagram;

Accompanying drawing 3 is Kinect camera three-dimensional coordinate system schematic diagram of the present invention;

Accompanying drawing 4 is each joint position relation schematic diagram under human body coordinate system of the present invention;

The human hand and arm joint limit of sports record data that accompanying drawing 5 is used for reference for the present invention.

Embodiment

In order to make object of the present invention, technical scheme clearly understands, below with reference to drawings and Examples, the present invention is described in detail.The specific embodiment of this place statement, only for explaining the present invention, is not intended to limit the present invention.

The technical issues that need to address of the present invention are to provide the method that a kind of novel non-contact gesture control user interface replaces traditional finger touch, mouse and keyboard, and a kind of non-contact gesture based on Kinect controls the method for the user interfaces such as computer, touch-screen and projection specifically.

For above technical matters, take following technical scheme: utilize Kinect to gather the action of user's both arms, identify gesture by specific Gesture Recognition Algorithm, the user interfaces such as the gesture that utilization identifies and computer, touch-screen and projection are carried out alternately.Specifically, the computer installing Kinect driving is connected upper Kinect device, the depth data of collection, skeleton data are reached application program by Kinect camera, process is extracted effective arm joint coordinate data of operator by application program, by specific Gesture Recognition Algorithm identification 6 kinds of gestures, and corresponding gesture message order is sent to controlled device makes feedback, thus reach the object of manipulation user interface.

Figure 2 shows an embodiment of overall technical architecture in the method for a kind of non-contact gesture manipulation of the present invention user interface.Concrete gesture identification method comprises the following steps:

Kinect camera is arranged on and touches height 1.8 meters of directly over large-size screen monitors by step 1., and operator stands on, 2 ~ 2.5 meters, Kinect dead ahead;

Step 2, carry out Effective judgement to the image collected, concrete determination methods is as follows:

A. judge whether the image collected is effective image: Kinect camera can gather at most the three-dimensional coordinate information data of 20 the skeletal joint points appearing at its camera previous action person, this three-dimensional data has three state value: null value, actual value, estimated value; When the true three-dimension coordinate figure number of the skeletal joint point of operator in every two field picture is greater than 14, and when containing the three-dimensional coordinate data of shoulder in operator, right and left shoulders, left and right elbow, left and right wrist and right-hand man joint totally 9 articulation points in these 14 skeletal joint points, then regarded as effective image frame data, otherwise, give up invalid image frame data;

B. judge whether the left and right arm in effective image is valid function arm: shoulder joint is less than d with the absolute value m of the difference of corresponding wrist joint y-axis coordinate and is effective arm, otherwise is invalid operation, and wherein the coordinate of 9 skeletal joint points is expressed as follows: middle shoulder S _c(x _sc, y _sc, z _sc)=(0,0,0), left shoulder S _l(x _sl, y _sl, z _sl), right shoulder S _r(x _sr, y _sr, z _sr), left elbow E _l(x _el, y _el, z _el), right elbow E _r(x _er, y _er, z _er), left wrist W _l(x _wl, y _wl, z _wl), right wrist W _r(x _wr, y _wr, z _wr), left hand H _l(x _hl, y _hl, z _hl), right hand H _r(x _hr, y _hr, z _hr).

Wherein: i gets r or l; M=|y _si-y _wi|

$d=0.9*(\sqrt{{(x_{\mathrm{si}}-x_{\mathrm{ei}})}^2{(y_{\mathrm{si}}-y_{\mathrm{ei}})}^2+{(z_{\mathrm{si}}-z_{\mathrm{ei}})}^2}+\sqrt{{(x_{\mathrm{ei}}-x_{\mathrm{wi}})}^2+{(y_{\mathrm{ei}}-y_{\mathrm{wi}})}^2+{(z_{\mathrm{ei}}-z_{\mathrm{wi}})}^2})$

Step 3, three-dimensional data under Kinect space coordinates in effective image is converted to the three-dimensional data under human body coordinate system, extract shoulder in the t time period, right and left shoulders, left and right elbow, the three-dimensional coordinate data of left and right wrist and right-hand man joint totally 9 articulation points, calculate the middle shoulder in every two field picture, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, in these 9 articulation points of the right hand, elbow joint is to the vector U of corresponding shoulder joint node and the angle α of elbow joint to the vector V of corresponding wrist joint point, elbow joint is to the corresponding vector V of wrist joint point and the angle β of X-axis positive dirction, wrist joint to corresponding swivel of hand vector W and elbow joint to corresponding wrist joint point vector V between angle ω.As shown in Figure 4.Right arm computing formula is as follows:

$\mathrm{\&alpha;}=\arccos \frac{(x_{\mathrm{wr}}-x_{\mathrm{er}})*(x_{\mathrm{sr}}-x_{\mathrm{er}})+(y_{\mathrm{wr}}-y_{\mathrm{er}})*(y_{\mathrm{sr}}-y_{\mathrm{er}})+(z_{\mathrm{wr}}-z_{\mathrm{er}})*(z_{\mathrm{sr}}-z_{\mathrm{er}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}*\sqrt{{(x_{\mathrm{sr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{sr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{sr}}-z_{\mathrm{er}})}^2}}$

$\mathrm{\&beta;}=\arccos \frac{(z_{\mathrm{wr}}-z_{\mathrm{er}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}}$

$w=\arccos \frac{(x_{\mathrm{wr}}-x_{\mathrm{er}})*(x_{\mathrm{hr}}-x_{\mathrm{wr}})+(y_{\mathrm{wr}}-y_{\mathrm{er}})*(y_{\mathrm{hr}}-y_{\mathrm{wr}})+(z_{\mathrm{wr}}-z_{\mathrm{er}})*(z_{\mathrm{hr}}-z_{\mathrm{wr}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}*\sqrt{{(x_{\mathrm{hr}}-x_{\mathrm{wr}})}^2+{(y_{\mathrm{hr}}-y_{\mathrm{wr}})}^2+{(z_{\mathrm{hr}}-z_{\mathrm{wr}})}^2}}$

Angle α span is 0 ° ~ 180 °, and angle β span is 0 ° ~ 180 °, and angle ω span is 0 ° ~ 90 °.

If only there is an arm effective, namely left arm is effective or right arm effective, then α, β, ω gesture identification rule according to definition carries out gesture identification to single armed;

If two arms are all effective, then α, β, ω gesture identification rule according to definition carries out gesture identification to both arms.

Also the method for template matches can be adopted to carry out gesture identification, by the coordinate data of shoulder, left shoulder, left elbow, left wrist, these 5 articulation points of left hand in gathering in 3s stored in array M, by the coordinate data of shoulder, right shoulder, right elbow, right wrist, these 5 articulation points of the right hand in gathering in 3s stored in array N, to record in two arrays the x, y, z coordinate of each articulation point in 3s respectively over time, i.e. test template.In in the 1s time of having deposited in advance in shoulder, these 9 articulation points of left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, right wrist, the right hand each skeletal joint point human body coordinate system under three-dimensional coordinate over time trend be reference template.Then utilize dynamic time warping (DTW) algorithm, calculate the similarity between test template and reference template, if calculated value is greater than given threshold value, then the match is successful for gesture, otherwise it fails to match.This algorithm is the thought based on dynamic programming, can sequence settling time template matches problem different in size, and namely in the present embodiment, test template has adopted 3s, and reference template is only 1s, test template is divided into 3 1s, mates respectively with reference template during coupling.When the similarity of a certain gesture in the variation tendency calculating each body joint point coordinate in continuous multiple frames effective image and reference template is greater than 90%, then for identifiable design is this gesture.

Or first using method A, if identify successfully, stops; If unidentified go out gesture, then identify with method B again, if identify successfully, stop; If all unidentified go out gesture; re-start gesture identification.

Step 4, the gesture steering order after identifying and finger touch, mouse and keyboard etc. in controlled device system interface program operated map one by one, realize the control to user interfaces such as computer, touch-screen and projections.

The user action gesture that Kinect gathers by the present invention is applied to carries out alternately contactless to user interfaces such as computer, touch-screen and projections, and mode of operation intuitive and convenient more, can replace traditional interactive modes such as finger touch, mouse, keyboard; 6 kinds of gestures of the design of the present invention are simultaneously easy to learn and use, and substantially covers the demand to operating user interface; Gesture Recognition Algorithm is simple and practical, processing response speed, can meet real-time man-machine interaction experience.

In sum, the inventive method makes user free-handly to control effectively to user interfaces such as computer, touch-screen and projections, just can carry out mutual without the need to contact, and method is simply direct, more natural, convenient and efficient.

The above; be only preferred embodiment of the present invention; but protection scope of the present invention is not limited thereto; anyly be familiar with those skilled in the art in scope disclosed in this invention; be equal to according to technical scheme of the present invention and inventive concept thereof and replace or change, all belonged to protection scope of the present invention.

Claims (4)

1. a method for non-contact gesture manipulation user interface, is characterized in that comprising following steps:

1) by the action of camera collection operator both arms skeletal joint;

2) carry out Effective judgement to the image collected, concrete determination methods is as follows:

First judge whether the image collected is effective image, is specially, the three-dimensional coordinate information data of the skeletal joint point of camera collection operator; When the skeletal joint point of the operator collected in every two field picture is greater than 14, and when containing the three-dimensional coordinate data of the middle shoulder of operator, right and left shoulders, left and right elbow, left and right wrist and right-hand man joint totally 9 articulation points in these 14 skeletal joint points, then regarded as effective image frame data, otherwise, it is invalid to be considered as, and gives up;

Then judge whether the left and right arm in effective image is valid function arm respectively, be specially: shoulder joint is less than d with the absolute value m of the difference of corresponding wrist joint y-axis coordinate and is effective arm, otherwise be invalid operation, wherein the coordinate of 9 skeletal joint points is expressed as follows: middle shoulder S _c(x _sc, y _sc, z _sc)=(0,0,0), left shoulder S _l(x _sl, y _sl, z _sl), right shoulder S _r(x _sr, y _sr, z _sr), left elbow E _l(x _el, y _el, z _el), right elbow E _r(x _er, y _er, z _er), left wrist W _l(x _wl, y _wl, z _wl), right wrist W _r(x _wr, y _wr, z _wr), left hand H _l(x _hl, y _hl, z _hl), right hand H _r(x _hr, y _hr, z _hr).

Wherein: m=|y _si-y _wi|,

$d=0.9*(\sqrt{{(x_{\mathrm{si}}-x_{\mathrm{ei}})}^2+{(y_{\mathrm{si}}-y_{\mathrm{ei}})}^2+{(z_{\mathrm{si}}-z_{\mathrm{ei}})}^2}+\sqrt{{(x_{\mathrm{ei}}-x_{\mathrm{wi}})}^2+{(y_{\mathrm{ei}}-y_{\mathrm{wi}})}^2+{(z_{\mathrm{ei}}-z_{\mathrm{wi}})}^2})$ , i gets r or l;

3) carry out gesture identification to the image of the effective operator arm of the existence collected, concrete recognition methods A is as follows:

3.1) three-dimensional coordinate data of the articulation point in image is converted to the three-dimensional data under human body coordinate system, wherein said human body coordinate system is for origin (0 with middle shoulder joint, 0,0), human body place plane is X-Y plane, direction is Y-axis positive dirction straight up, and human body is being just Z axis negative direction to direction, and X-Y-Z observes right-hand screw rule;

3.2) shoulder in the t time period is extracted, right and left shoulders, left and right elbow, three-dimensional coordinate data under the human body coordinate system of left and right wrist and right-hand man joint totally 9 articulation points, take in calculating, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, in these 9 articulation points of the right hand, elbow joint is to the vector U of corresponding shoulder joint node and the angle α of elbow joint to the vector V of corresponding wrist joint point, elbow joint is to the corresponding vector V of wrist joint point and the angle β of X-axis positive dirction, wrist joint to corresponding swivel of hand vector W and elbow joint to corresponding wrist joint point vector V between angle ω, pass through α, β, ω identifies gesture steering order,

$\mathrm{\&alpha;}=\arccos \frac{(x_{\mathrm{wr}}-x_{\mathrm{er}})*(x_{\mathrm{sr}}-x_{\mathrm{er}})+(y_{\mathrm{wr}}-y_{\mathrm{er}})*(y_{\mathrm{sr}}-y_{\mathrm{er}})+(z_{\mathrm{wr}}-z_{\mathrm{er}})*(z_{\mathrm{sr}}-z_{\mathrm{er}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}*\sqrt{{(x_{\mathrm{sr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{sr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{sr}}-z_{\mathrm{er}})}^2}}$

$\mathrm{\&beta;}=\arccos \frac{(z_{\mathrm{wr}}-z_{\mathrm{er}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}}$

$w=\arccos \frac{(x_{\mathrm{wr}}-x_{\mathrm{er}})*(x_{\mathrm{hr}}-x_{\mathrm{wr}})+(y_{\mathrm{wr}}-y_{\mathrm{er}})*(y_{\mathrm{hr}}-y_{\mathrm{wr}})+(z_{\mathrm{wr}}-z_{\mathrm{er}})*(z_{\mathrm{hr}}-z_{\mathrm{wr}})}{\sqrt{{(x_{\mathrm{wr}}-x_{\mathrm{er}})}^2+{(y_{\mathrm{wr}}-y_{\mathrm{er}})}^2+{(z_{\mathrm{wr}}-z_{\mathrm{er}})}^2}*\sqrt{{(x_{\mathrm{hr}}-x_{\mathrm{wr}})}^2+{(y_{\mathrm{hr}}-y_{\mathrm{wr}})}^2+{(z_{\mathrm{hr}}-z_{\mathrm{wr}})}^2}}$ 4) the outside input operation in the gesture steering order identified and controlled device system interface program is mapped, realize the control of user interface.

2. the method for a kind of non-contact gesture manipulation user interface according to claim 1, is characterized in that:

Step 3.2) described in identify that the method for gesture steering order is as follows by α, β, ω:

Time effective for single arm,

When diminishing after α first change in time t is large, be double-click gesture;

When α first diminish in time t become large afterwards time, be right-click gesture;

When diminishing after ω first change in time t is large, it is click gesture;

When ω occur to become in time t large after constant trend time, for choosing gesture, the trend until ω diminishes in another one time t, namely discharges gesture, and from choosing to, to discharge whole process be drag gesture;

When β occur first to become in time t large after the trend that diminishes, α does not change in time t simultaneously, for after roll gesture; When β occur first to diminish in time t become large trend afterwards time, simultaneously α does not change, for front rolling gesture in time t;

Time effective for two arms,

When right arm meet angle β occur first to become in time t large after the trend that diminishes, left arm meets angle β and occurs first to diminish when becoming large trend afterwards in time t, for reducing gesture simultaneously; Occur first to diminish in time t to become large trend afterwards when right arm meets angle β, simultaneously left arm meet angle β occur first to become in time t large after diminish trend time, be amplifying gesture.

3. the method for a kind of non-contact gesture manipulation user interface according to claim 1, is characterized in that:

Step 3) described in gesture identification method A can also adopt gesture identification method B, specific as follows:

Extract the three-dimensional coordinate data under the human body coordinate system of shoulder in the t time, right and left shoulders, left and right elbow, left and right wrist and right-hand man joint totally 9 articulation points, using the three-dimensional coordinate data under the human body coordinate system of shoulder middle in the t time period, left shoulder, left elbow, left wrist, these 5 articulation points of left hand as M group, using the three-dimensional coordinate data under the human body coordinate system of shoulder middle in the t time period, right shoulder, right elbow, right wrist, these 5 articulation points of the right hand as N group, record the change of each articulation point x, y, z coordinate within the t time in two groups respectively, namely form test template; In in the t time of having deposited in advance in shoulder, these 9 articulation points of left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, right wrist, the right hand each skeletal joint point human body coordinate system under three-dimensional coordinate over time trend be reference template; Then utilize dynamic time warping DTW algorithm, calculate the similarity between test template and reference template, if calculated value is greater than given threshold value, then the match is successful for gesture, otherwise it fails to match;

Described reference template comprises following content:

Time effective for single arm:

When in arm, shoulder joint, shoulder joint, elbow joint, wrist joint coordinate do not change in time within the t1 time, and the y-axis coordinate of swivel of hand first diminishes in time and becomes large afterwards, the z-axis coordinate of swivel of hand first becomes large in time or diminishes simultaneously, and when variation range is less than b centimetre, i.e. click gesture;

When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and swivel of hand and carpal y-axis coordinate first diminish in time and become large afterwards, diminish after swivel of hand and carpal z-axis coordinate first change in time is simultaneously large, and variation range is when being less than a centimetre, namely double-click gesture;

When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and diminish after the first change greatly in time of swivel of hand and carpal y-axis coordinate, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, i.e. right-click gesture;

When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and swivel of hand and carpal x-axis coordinate first diminish in time and become large afterwards, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, namely roll gesture afterwards; When in arm, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t time, and diminish after the first change greatly in time of swivel of hand and carpal x-axis coordinate, swivel of hand and carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, namely before roll gesture;

When in arm, shoulder joint, shoulder joint, elbow joint, wrist joint coordinate do not change in time within the t1 time, and the y-axis coordinate of swivel of hand diminishes in time, the z-axis coordinate of swivel of hand becomes large in time simultaneously, and when variation range is less than b centimetre, namely chooses gesture; Until the y-axis coordinate of swivel of hand becomes large in time, the z-axis coordinate of swivel of hand diminishes in time simultaneously, and when variation range is less than b centimetre, namely discharges gesture; From choosing to, to discharge whole process be drag gesture;

Time effective for two arms:

When in two arms, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and the carpal x-axis coordinate in right hand joint and the right side first diminishes in time and becomes large afterwards, the carpal x-axis coordinate in left hand joint and a left side first becomes in time greatly and diminishes, right-hand man joint, carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, namely reduce gesture;

When in two arms, shoulder joint, shoulder joint, elbow joint coordinate do not change in time within the t1 time, and diminish after the first change greatly in time of the carpal x-axis coordinate in right hand joint and the right side, the carpal x-axis coordinate in left hand joint and a left side first diminishes in time and becomes large afterwards, right-hand man joint, carpal z-axis coordinate first diminish in time and become large afterwards simultaneously, and variation range is when being less than a centimetre, i.e. amplifying gesture;

The span of above-mentioned a, b is 10<a<30,5<b<10.

4. the method for a kind of non-contact gesture manipulation user interface according to claim 3, it is characterized in that: step 3) described in gesture identification method A can also adopt the mode be combined by method A and method B, be specially: first using method A, if identify successfully, stop; If unidentified go out gesture, then identify with method B again, if identify successfully, stop; If all unidentified go out gesture; re-start gesture identification.