CN103839040B - Gesture identification method and device based on depth image - Google Patents
Gesture identification method and device based on depth image Download PDFInfo
- Publication number
- CN103839040B CN103839040B CN201210490622.8A CN201210490622A CN103839040B CN 103839040 B CN103839040 B CN 103839040B CN 201210490622 A CN201210490622 A CN 201210490622A CN 103839040 B CN103839040 B CN 103839040B
- Authority
- CN
- China
- Prior art keywords
- gesture
- hand
- motion
- depth
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- User Interface Of Digital Computer (AREA)
- Image Analysis (AREA)
Abstract
There is provided a kind of gesture identification method based on depth image, gesture identifying device, the method and apparatus for opening man-machine interactive system.The gesture identification method, can include：Based on the range image sequence including hand region, the 3 D motion trace in detection hand candidate region；And according to the 3 D motion trace in hand candidate region, recognize prearranged gesture.The above-mentioned gesture identification method and device of the embodiment of the present invention take full advantage of the movable information in continuous identification image；Due to complexion model is not used, but the time-space domain motion information and continuous depth value change information of consecutive image is used to carry out gesture identification, therefore the gesture identification method performance is compared with robust, suffered illumination condition influence is smaller；Due to carrying out gesture identification based on movement locus, it can be used in distance range relatively far away from.
Description
Technical field
The present invention relates generally to gesture identification, relates more specifically to gesture identification method and dress based on depth image
Put.
Background technology
Some Gesture Recognitions based on image procossing have been proposed.
In United States Patent (USP) " US7340077B2 ", it is proposed that a kind of gesture recognition system based on depth transducer.This method
By in the range of certain time posture combination carry out gesture identification, mainly for the shape of institute's identification division, position and
Direction carries out gesture identification, and related electric equipment is controlled finally by the gesture recognized.This method is mainly used
Static image information is identified, and has lost a large amount of movable informations in continuous videos image.In addition, the identification of gesture is mainly
Based on assembled gesture, therefore user must do multiple gesture modes and go to complete a gesture, and this is not very square for user's operation
Just.
U.S. Patent Publication " US20120069168A1 " proposes a kind of gesture recognition system being controlled to TV.
Gesture (palm open with closure) be used for TV " selection " and " it is determined that " feature operation.First, hand posture ("ON" or
" conjunction ") it is detected based on the calculating distance between palm center and bottom palm, then gesture (palm opens and closure) can quilt
Identification, the State Transferring relation between "ON" and " conjunction " based on hand.Within the system, in order to effectively judge the folding shape of hand
The distance of state, user and TV must be in effective distance range.Therefore this method is poorly suitable for remote-controlled operation.Simultaneously
The system carries out the detection of hand using complexion model, and testing result will be highly susceptible to the influence of ambient lighting change.
Entitled " Hand Gesture Recognition for Human-Machine Interaction ",
Journal of WSCG, 2004 article proposes a kind of vision utility system based on gesture identification in real time.First, adopt
Carried out moving hand region segmentation with complexion model；It is then based on the gesture recognition that Hausdorff distances carry out hand.This method is same
Easily influenceed by illumination.
In addition, some articles carry out gesture identification using 2D movement locus, generally they are all based on still image progress
Feature extraction carries out foreground segmentation using complexion model.
The content of the invention
Embodiments in accordance with the present invention can include there is provided a kind of gesture identification method based on depth image：It is based on
Range image sequence including hand region, the 3 D motion trace in detection hand candidate region；And according to hand candidate area
The 3 D motion trace in domain, recognizes prearranged gesture.
According to another embodiment of the present invention there is provided a kind of method for opening man-machine interactive system, including：Including
The range image sequence of hand region；Based on the range image sequence including hand region, the three-dimensional in detection hand candidate region
Movement locus；According to the 3 D motion trace in hand candidate region, identification lift hand gesture；And if recognizing lift hand gesture,
Man-machine interactive system is then opened, into man-machine interaction state.
There is provided a kind of side that human body predetermined action is recognized based on depth image according to another embodiment of the present invention
Method, can include：Based on the range image sequence in the human body region is included, three maintenance and operations of human body candidate region are detected
Dynamic rail mark；And according to the 3 D motion trace of human body candidate region, recognize the predetermined action of human body.
According to another embodiment of the present invention there is provided a kind of gesture identifying device based on depth image, it can include：
3 D motion trace detection part, for based on the range image sequence including hand region, the three of detection hand candidate region
Tie up movement locus；And gesture identification part, for the 3 D motion trace according to hand candidate region, recognize prearranged gesture.
According to embodiments of the present invention gesture identification method and device based on depth image are because by the fortune in Depth Domain
Dynamic rail mark includes gesture identification process, therefore takes full advantage of the movable information in continuous identification image；Due to the colour of skin is not used
Model, but use the time-space domain motion information and continuous depth value change information of consecutive image to carry out gesture identification, therefore
The gesture identification method performance is compared with robust, and suffered illumination condition influence is smaller；Due to carrying out gesture identification based on movement locus, it
It can be used in distance range relatively far away from.Gesture identification method according to embodiments of the present invention takes short and robustness is high.
The technology of unlatching man-machine interactive system according to embodiments of the present invention is opened there is provided one kind facility, reliable system
Dynamic control mode, prompts the user whether to come into system control state, prevents user's unconscious movement is erroneously identified as can
Operating gesture, so that there is provided a kind of man-machine interaction mode of more user friendly.
Brief description of the drawings
Fig. 1, which schematically shows Gesture Recognition according to an embodiment of the invention, is used for the scene of man-machine interaction
Schematic diagram.
Fig. 2 shows the overview flow chart of gesture identification method according to a first embodiment of the present invention.
Fig. 3 shows and according to an embodiment of the invention need not recognize user's hand to detect user's hand candidate region
3 D motion trace illustrative methods flow chart.
Fig. 4 shows the method according to an embodiment of the invention that lift hand gesture is recognized based on 3 D motion trace
Overview flow chart.
Fig. 5 is the movement locus exploded view in different dimensional according to the 3 D motion trace of one embodiment of the invention.
Fig. 6 (a) schematically shows the form of 3 D motion trace and the schematic diagram of Motion feature extraction to 6 (c).
Fig. 7 shows movement locus of the sliding window according to an embodiment of the invention based on variable-size from input
Feature recognizes the flow chart of the method for gesture.
Fig. 8 shows the flow chart of gesture identification method according to a second embodiment of the present invention.
Fig. 9 shows the flow of the method for anthropometry model checking prearranged gesture according to an embodiment of the invention
Figure.
Figure 10 (a1) to (a3), (b1) to (b3) and (c) show it is according to an embodiment of the invention, using Nogata
The positioning of head center line, the positioning of shoulder horizontal direction and the hand and head as example lift hand gesture end of figure analysis method
The schematic diagram of the mutual alignment relation in portion.
Figure 11 shows a kind of flow chart of method for opening man-machine interactive system according to an embodiment of the invention.
Figure 12 shows the functional configuration block diagram of the gesture identifying device according to embodiments of the present invention based on depth image.
Figure 13 is to show the general hardware block diagram according to the gesture recognition system of the embodiment of the present invention.
Embodiment
In order that those skilled in the art more fully understand the present invention, with reference to the accompanying drawings and detailed description to this hair
It is bright to be described in further detail.
It will be described in the following order：
1st, Application Scenarios-Example
2nd, the first embodiment of gesture identification method
The overall procedure of 2.1 gesture identification methods
2.2nd, the acquisition of 3 D motion trace
2.3rd, based on 3 D motion trace identification lift hand gesture
2.4th, the Motion feature extraction of 3 D motion trace
2.5th, the motion feature identification prearranged gesture based on 3 D motion trace
2.6th, the gesture identification of the variable sliding window of window size is utilized
3rd, the second embodiment of gesture identification method
The overall procedure of the gesture identification method of 3.1 second embodiments
3.2nd, prearranged gesture is verified according to anthropometry model
4th, the open method of man-machine interactive system
5th, the gesture identifying device based on depth image
6th, system hardware configuration
7th, summarize
1st, Application Scenarios-Example
Fig. 1, which schematically shows Gesture Recognition according to an embodiment of the invention, is used for the scene of man-machine interaction
Schematic diagram.As shown in figure 1, subscriber station is before the human-computer interaction device of such as computer, the solid of such as binocular camera is taken the photograph
Camera, shoots such as sequence of left-right images of people or directly obtains range image sequence, and issued such as individual calculus
The gesture identification equipment of machine, personal computer analysis depth image sequence and carry out gesture identification, and the knot based on gesture identification
Fruit is responded, if it is a lift hand gesture for being used to start for example to recognize this, draws this effective enabling signal,
And send enabling signal；If instead recognizing this lift hand gesture for being not intended to start, then it is invalid startup letter to draw this
Number, do not send enabling signal.Certainly, this is a schematic example, can for recognizing that the equipment of gesture is not limited to computer
To be such as game machine, projecting apparatus, television set etc..
As known to those skilled in the art, depth image (Depth image) is that the value of the pixel in image is depth
Image.Compared to gray level image, depth image has the depth of object（Distance）Information, is therefore particularly suited for needing three-dimensional letter
The various applications of breath.
In addition, as it is well known, have simple conversion relation between the depth value and parallax value of pixel, therefore this
The depth image of invention is meant that broad sense, including anaglyph.
2nd, the first embodiment of gesture identification method
The overall procedure of 2.1 gesture identification methods
Fig. 2 shows the overview flow chart of gesture identification method 100 according to a first embodiment of the present invention.
As shown in Fig. 2 in step 110, based on the range image sequence including hand region, detection hand candidate region
3 D motion trace.
The range image sequence can be that the camera that can obtain depth image from any one of such as binocular camera is passed
Pass, or can locally calculate and obtain from gray level image in real time, or by network from outside deep etc..
Here 3 D motion trace, refer to it is different from the two-dimentional track of traditional two dimensional image, with depth（Distance）
The movement locus of information, namely each movement locus point on track have both plane (x, y) coordinate informations, it may have characterize deep
Spend the Z coordinate information of (distance).
The method of the 3 D motion trace in detection hand candidate region can be roughly classified into method based on hard recognition and
The method filtered based on hand.
In the method based on hard recognition, for example, from initial depth image, first according to the spy of hand region
Levy, hand region is recognized by matching treatment etc. and the position of hand region is determined, then using motion tracking technology follow-up
Depth image in track hand, so as to obtain the 3 D motion trace of hand.Here, in order to from hand is identified, if also
There is corresponding gray level image, then can also combine the technology of the hard recognition based on complexion model.
In the method filtered based on hand, hand region identification is not carried out first, but detect depth image first
In moving mass region, then the extraordinary feature for example based on hand come from moving mass region select（Or filtering）Hand exercise
Block region.This mode can need not recognize hand, so as to faster carry out gesture identification, be particularly suitable for real-time people
Machine interaction process.The 3 D motion trace in detection hand candidate region filtered based on hand is described later with reference to Fig. 3.
But, the 3 D motion trace detection mode in above-mentioned hand candidate region is merely illustrative, the invention is not limited in
This, any to be based on range image sequence, the technology for obtaining the 3 D motion trace of certain object can apply to the present invention.
In the step 120, according to the 3 D motion trace in hand candidate region, prearranged gesture is recognized.
In general, different gestures corresponds to different 3 D motion traces, therefore can be by analyzing above-mentioned acquisition
Hand candidate region 3 D motion trace, and carry out the identification of prearranged gesture.
For example, for lift hand gesture, with people for as object of reference, its 3 D motion trace be start from below to
The parabola that lordosis is terminated to above；If be decomposed into Depth Domain and the two dimensional surface domain vertical with Depth Domain on two motion
Track, then be distance from as far as closely again to remote parabola for the video camera in face of in Depth Domain, and two
It is linear motion from top to bottom on dimensional plane domain.
For another example as the let go gesture relative with lift hand gesture, with people from for as object of reference, thirdly maintenance and operation dynamic rail
Mark is started from above to lordosis to the parabola terminated below；If being decomposed into Depth Domain and the two dimension vertical with Depth Domain
Two movement locus on plane domain, then in Depth Domain, for the video camera in face of, be distance from as far as closely again to
Remote parabola, and be linear motion from top to bottom on two dimensional surface domain.By the way, on lifting hand gesture and putting
Difference between hand gesture, when can also be according to the direction of motion indicated by the relation between chronological location point, terminal
The hand position at quarter etc. judges.
For another example for drawing circle gesture, its 3 D motion trace is approximate circle.For gesture of waving, its movement locus
For reciprocal pendular movement track.
For another example for being naturally drooped from hand to the hand exercise pushed away forward, its three-dimensional track can be considered as half lift hand
Plus the combination pushed away forward, wherein the action pushed away forward is linear motion in Depth Domain, and it is approximate motionless in plane domain.
Subsequently, Fig. 4,5,6 will be referred to, exemplified by lifting hand gesture, will be illustrated by the movement locus and two on analysis depth domain
Both movement locus on dimensional plane domain lift the process of hand gesture to recognize.
The method according to embodiments of the present invention that prearranged gesture is recognized based on depth image is illustrated above in conjunction with Fig. 1.No
Cross, the invention is not limited in gesture identification, the action recognition of other human bodies can also be applied to, such as applied to foot
Action recognition, for example forward skirt footwork identification.Obviously, the present invention is also not restrictively applied to the mankind, also may be used
With the object active applied to other, for example, animal, robot etc..
2.2nd, the acquisition of 3 D motion trace
Fig. 3 shows and according to an embodiment of the invention need not recognize user's hand to detect user's hand candidate region
3 D motion trace illustrative methods 110 flow chart.The illustrative methods 110 can apply to the step shown in Fig. 2
Rapid S110.
As shown in figure 3, in step S111, obtaining the moving mass region in depth image.
As an example, the detection and acquisition in moving mass region can be realized by conventional interframe difference method.Specifically,
For example present frame and former frame are subtracted each other, by about the difference in region compared with predetermined motion differential threshold, if current
Region difference is more than predetermined differential threshold, then current region is detected as into moving region block.Area above can for example pass through
Connected domain detection is obtained.
As alternative exemplary, moving mass region can be obtained based on background subtraction, specifically, for example, is subtracted by present image
Remove Background to carry out the acquisition in moving mass region, by the accumulation of above multiple image and can be asked on being formed for Background
It is worth to.In step S112, at least one in the position in moving mass region, area and shape, from moving mass area
Hand candidate region is chosen in domain.
For example, can according to the general position of the position in moving mass region, area and shape etc. and the human hand learnt in advance,
The knowledge of area and shape, chooses hand candidate region from moving mass region.Specifically, if the moving mass region may
For the moving region of hand, then the region area size should close to the average hand of the mankind size.If the moving mass
Region area is excessive or too small, can reject the region.Similarly, there is also one on Aspect Ratio for the shape of the hand of the mankind
Whether fixed proportion, can also meet the proportionate relationship come filtering motions block region according to moving mass region.For another example human hand and
The distance between human body is general within limits, if moving mass region not herein within the scope of, will can also move
Reject outside hand candidate moving region in block region.If a moving mass region meets position, area and shape bar simultaneously
Part, then the moving mass is hand region.Certainly, it should to two moving mass regions of more options as hand candidate moving region, because
This meets the filter condition according to the position of human hand, area and shape if more than two moving mass regions, then can comment
Estimate the confidence level that they meet human hand region（Or possibility）, therefrom select at most two moving mass regions to be used as hand candidate
Moving region.Hereinafter, for ease of description, it is subject to exemplified by a hand candidate moving mass region is selected from all moving mass
Explanation.
In addition, to select from moving mass region above based on the anticipatory knowledge of the position of human hand, area and shape
Hand candidate moving region.But be only for example, can also according to various other factors, use various techniques to selection most may be used
It can be the moving mass region of hand, such as in the case where there is gray level image, complexion model can be combined to select hand
Moving mass region.
In step 113, the positional information in hand candidate region is calculated and recorded.As an example, hand time can be calculated
The barycenter dot position information of favored area, is used as the positional information in hand candidate region.The center of mass point position in so-called hand candidate region
Put for example to be obtained by being averaging to all positions in hand candidate region.
In step 114, the positional information sequence corresponding to range image sequence is obtained.
Specifically, for example, can be after the center of mass point that every subsystem gets current hand exercise block, by the center of mass point root
It is put into according to time order and function order in " motion dot position information sequence " memory, " motion dot position information sequence " is formed three
Dimension（3D）Movement locus.In addition, when motor point sequence length is more than the motion sequence length of setting, can be by deleting always
Motor point, is put into new motor point, carries out movement locus point renewal.
In one example, in motion dot position information sequence, it can store corresponding in association with positional information
Temporal information.
In one example, if motor point sequence length also not up to sets sequence length, system can not start
Follow-up 3D motion trajectory analysis.
2.3rd, based on 3 D motion trace identification lift hand gesture
Fig. 4 shows the method according to an embodiment of the invention that lift hand gesture is recognized based on 3 D motion trace
120 overview flow chart.This method 120 can apply to the step S120 shown in Fig. 2.
As shown in figure 4, in step S121, Motion feature extraction is carried out to the 3 D motion trace obtained.Below will
Exemplary specific descriptions are provided to this with reference to Fig. 5 and Fig. 6.
In step S122, based on the motion feature of acquired 3 D motion trace, gesture identification is carried out.
For example, can be by by the motion feature of the 3 D motion trace obtained motion model corresponding with prearranged gesture
Compare, to recognize whether the 3 D motion trace characterizes prearranged gesture.This is provided below with reference to Fig. 7 and exemplary specifically retouched
State.
But, recognize that the method for prearranged gesture is merely illustrative based on 3 D motion trace shown in Fig. 4.For example, at certain
, can be without Motion feature extraction in the case of a little, and only analyze whether the 3 D motion trace meets by numerical analysis
Specific mathematical form such as parabolic carries out gesture identification.
2.4th, the Motion feature extraction of 3 D motion trace
3 D motion trace can be by with the motion feature of oneself of each tracing point corresponding to each frame depth image
Portrayed with the feature of overall 3 D motion trace,
Fig. 5 is the movement locus exploded view in different dimensional according to the 3 D motion trace of one embodiment of the invention.
As shown in figure 5, exemplarily, the motion feature 121 of oneself of each tracing point can include：Two-dimension time-space domain is special
Levy the Depth Domain motion feature 1222 of 1211, tracing point, the two-dimension time-space characteristic of field and Depth Domain motion feature of each tracing point
The time point of the equal and tracing point is associated.Two-dimension time-space characteristic of field can the position including tracing point, speed, angle.Track
The Depth Domain motion feature of point can include the depth value of tracing point.
The form and Motion feature extraction of 3 D motion trace are schematically described below with reference to Fig. 6 (a) to 6 (c).
Fig. 6 (a) shows world coordinate system, and wherein Z axis is depth transducer direction depth direction in other words.Fig. 6 (b) is
From X-Y plane it was observed that 2D time-space domain motions track, corresponding to the part pointed out in Fig. 6 (a) by mark 601.Fig. 6 (c) is Z
The 1D depth value movement locus observed on axle, corresponding to the part pointed out in Fig. 6 (a) by mark 602.Depth value z in figure
Represent the horizontal range between depth transducer to moving mass.
The feature extraction of the 2D movement locus in X-Y plane is described first below.
The lift hand 2D movement locus observed from front view (X-Y plane) is similar to the linear motion in Fig. 6 (b).
Time range [ts, te], hand center of mass point P is from starting point PsMove to terminating point Pe, tm is between time range [ts, te]
Intermediate time point.Because different user's lift hand customs is different, therefore movement locus starting point may be different, and movement locus is whole
Stop may also be different, and possible a plurality of movement locus is for example as shown in Fig. 6 (b).
According to one embodiment of present invention, the linear motion feature in X-Y plane includes：The position of each tracing point,
Range of movement for movement velocity and the direction of motion, and overall track.
Each positions of the tracing point Pi in X-Y plane can use 2D coordinates (xi,yi) represent.
Movement velocity speed can be calculated by following formula (1) and obtained（Whether formula 1 should be the speed for calculating Y-direction
Degree）.
Wherein, dis (Pi,Pi-1) it is current motion block center of mass point PiWith previous frame center of mass point Pi-1Between space length, example
Such as Euclidean distance, tiAt the time of representing present frame, ti-1At the time of representing previous frame, it is assumed that the number of the continuous path to be analyzed point
Mesh（Equal to window size, the frame number to be analyzed in other words）For n, n is the integer more than or equal to 2, can be chosen as needed, example
Such as take 15,20.
The direction of motion can be represented with movement angle θ, and movement angle θ tangent tan θ can be represented with following formula (2).
In formula（2）In, i represents the numbering of present frame, xi, yiThe hand region center of mass point p in present frame is represented respectivelyi
X coordinate value and y-coordinate value, similarly, xi-1, yi-1The x coordinate value of the hand region center of mass point pi in former frame is represented respectively
And y-coordinate value.
Range of movement between 3 D motion trace starting point on an x-y plane and terminal is represented in hand exercise region
In the respective range of movement Range of X-direction and Y-directionxAnd Rangey, can be represented by formula (3)
Wherein, RangeYThe difference in height between movement locus terminating point and movement locus starting point is represented, Rangex is represented
Level error between movement locus terminating point and movement locus starting point
In summary feature, for X-Y plane, motion feature is { [2Dfeaturei],2Dfeaturetotal, wherein often
The feature of individual tracing point includes time, position, speed, angle, i.e. 2Dfeaturei=[ti,(xi,yi),speedi,θi], 2D is put down
The general characteristic X-direction of face track and the respective range of movement of Y-direction, 2Dfeaturetotal=[Rangex,Rangey]。
But, above-mentioned motion feature composition is merely illustrative, and mainly considers the identification of lift hand gesture and design.Base
In factors such as the gestures and desired precision of desired identification, the motion features different with considering can be designed and constituted, for example,
The general characteristic that the motion feature of each tracing point can also include track in acceleration, 2D planes can also include maximum speed
Degree, peak acceleration, extreme lower position point information, extreme higher position point information, leftmost position point information, least significant point information etc.
Deng.
Similar with above-mentioned plane domain for Depth Domain, motion feature can be divided into the Depth Domain fortune of each tracing point
Dynamic feature and the Depth Domain motion feature of whole track.The Depth Domain motion feature of each tracing point can include the time, position,
Speed and acceleration etc., the Depth Domain motion feature of whole track can for range of movement, maximum depth value, minimum depth value,
Maximal rate, minimum speed, peak acceleration, minimum acceleration.It is also possible to be designed according to the gesture difference for identification
Different Depth Domain motion features.According to an example, for lift hand gesture, it is considered to the motion feature of following Depth Domain
{[Zfeaturei],Zfeaturetotal}.The feature of wherein each tracing point includes time, position, speed, angle, i.e.,
Zfeaturei=[ti, zi,speedi], the range of movement of the general characteristic of Depth Domain track, i.e. Z-direction, Zfeaturetotal=
[Rangez]=[(Zmax-Zmin)]。
In addition, it is necessary to explanation, the motion feature of above-mentioned motion feature, either each tracing point or three-dimensional motion
Overall movement feature of the track in the overall movement feature and Depth Domain on 2D X-Y planes can be extracted together, then be entered again
The follow-up gesture identification of row；But Motion feature extraction can also be carried out while carrying out gesture identification in real time, it is so parallel
As long as the benefit for carrying out feature extraction and gesture identification is that the motion feature for finding 3 D motion trace does not meet prearranged gesture
Correspondence track and/or kinetic characteristic, it is possible to terminate the motion subtree and gesture identification of previous cycle and enter next follow
Ring.
2.5th, the motion feature identification prearranged gesture based on 3 D motion trace
After the motion feature of 3 D motion trace is obtained as described above, can according to the prearranged gesture to be recognized,
Motion feature based on 3 D motion trace recognizes whether user is made that prearranged gesture.
For example, if it is desired to recognize whether user is made that lift hand gesture, it can be determined that the relevant fortune of 3 D motion trace
The characteristics of whether dynamic feature meets corresponding with lift hand gesture.
The analysis can be carried out in 2D X-Y plane domain and Depth Domain respectively.
For example, for lift hand gesture, in 2D X-Y plane domain, it should meet：
（1）The speed of each tracing point in the Y direction is consistently greater than 0, i.e. hand and moved upwards all the time, such as formula (4)
It is shown；
Speedyi>0i=1,2 ..., n (4)
(2) because the movement locus on X-Y plane is similar to linear motion, therefore in whole motion process [ts, te],
Movement angle should keep approximate constant, i.e., the angle of each tracing point should approximately equal, such as formula (5) is shown.
θ1≈θ2≈...≈θn (5)
(3) range of movement on x directions and y directions should meet predetermined range threshold, shown in such as formula (6).
Wherein, hthresminAnd hthresmaxIt is predetermined lower threshold value and the upper threshold value in short transverse, Lthresmin, LthresmaxTable
Show the level error scope between starting point and terminating point, optionally, LthresminIt could be arranged to 0.hthresmin、hthresmax、
Lthresmin、LthresmaxValue it is related to mankind's forearm average length, can by collecting the lift hand operating habit of most users and
Analysis calculating is obtained.
（4）Maximum and minimum value on y directions should be the y-coordinate value of terminating point and the y-coordinate value of starting point, such as public affairs
Shown in formula (7).
In addition, for lift hand gesture, in Depth Domain, it should meet：
(5) the depth value changing rule of motion center of mass point is：First from big to small, after from small to large, and motion center of mass point exist
Motion in time range [ts, te] on Z axis is similar to parabolic motion, shown in such as Fig. 6 (c).Movement Locus Equation is approximate
As shown in formula (8), wherein d represents the coordinate value on Z axis, i.e. depth value.
d=at2+ bt+c wherein ts<t<te,dmin≤d≤dmax, D in additionthresmin<(dmax-dmin)<Dthresmax (8)
In formula (8), at the time of ts is track starting point correspondence, at the time of te is track end point correspondence, dmin tables
Show the minimum value of the depth value in the time range, dmaxThe maximum of the depth value in the time range is represented, depth value d exists
In time range [ts, te], [d should be fallen intomax,dmin] in the range of, and depth value scope (dmax-dmin) human body hand should be fallen into
Threshold range [the D of arm lengthsthresmin,Dthresmax] within, the threshold range of human arm length can be by many people's
Arm length is counted and obtained, in one embodiment, is arranged to [200mm, 500mm].
On whether coincidence formula (8) can pass through a numerical analysis such as most young waiter in a wineshop or an inn to the motion feature of track in Depth Domain
The method of multiplication is determined, that is, there is known each data point (ti, di), track fitting is carried out with quadratic function.
Above-mentioned formula (8) is general formulae, in one embodiment, it is believed that the parabola should be on tmThe track at moment
Almost symmetry is put, now formula (8) will be changed into d=a (t-tm)2+b(t-tm)+dmin, wherein tmFor intermediate time point, in moment point
Tm, the depth value of the center of mass point of hand region reaches the i.e. d=d of minimummin。
In addition, for prearranged gesture, between the movement locus on 2D X-Y plane on movement locus and 1D Depth Domains, generally
It is not completely self-contained, but there should be certain incidence relation.For example, for lift hand gesture, it should meet：
(6) in intermediate time point tm, on the movement locus of Depth Domain, the center of mass point depth value of hand region reaches most
It is small, meanwhile, on 2D X-Y plane on movement locus, the height value of the center of mass point of hand region（ym）Height should be approximately equal to
Hstart (that is, the y-coordinate y of starting points) and hend (that is, the y-coordinate ye of terminating point) average value, i.e., as shown in formula (9).
Thus, according to one embodiment, in order to recognize that the 3 D motion trace of a hand region has indicated whether one
Hand gesture is lifted, the 2D relevant with 3 D motion trace X-Y plane movement locus is analyzed（Motion feature）, in 1D Depth Domains
Movement locus（Motion feature）And above-mentioned condition (1)-(6) whether are met between the two.If it is satisfied, then judging that user does
Lift hand gesture is gone out, has otherwise judged that user does not make lift hand gesture.
It is above-mentioned to be applied to and lift hand gesture phase by simply changing for the various operations that lift hand gesture is carried out
To gesture of letting go identification.
In addition, above-mentioned gesture identification can be applied similarly to the identification of leg action, for example, kick and receive leg.
2.6th, the gesture identification of the variable sliding window of window size is utilized
It whether there is certain continuous motor pattern in continuous videos to detect, according to one embodiment, it may be considered that adopt
With sliding window, i.e., all frame of video in window are analyzed and recognize gesture, if do not identified in a window
Gesture, then according to a pre- fixed step size such as frame slip window, then analyzed and recognized to all frame of video in next window
Gesture.
Because the lift hand custom of different user is different, therefore many-sided each not phase such as the length of run duration
Together, and even same user, the lift hand gesture made when different is also not quite similar.For example, can for certain user
To make lift hand gesture in such as 15 frame ins, and other users may need to make lift hand gesture etc. in such as 20 frame ins.
In view of case above, in one embodiment, determine which is based on using the variable sliding window of window size
A little and how many depth images carry out 3 D motion trace detection and analysis, and the size of window represented with the depth of continuous how many frame
Image is spent as the input of gesture identification；If based on the three-dimensional motion of the range image sequence in the sliding window of predefined size
Track can not match motion model in Depth Domain corresponding with prearranged gesture simultaneously and the plane vertical with Depth Domain on two
Motion trajectory model is tieed up, then increases the size of the sliding window to regard more depth image frames as the defeated of gesture identification
Enter, and proceed the motion corresponding with the prearranged gesture of the corresponding 3 D motion trace of size of the sliding window after the increase
The matching of locus model.
For example, in one example, the variable sliding window series of size is as shown in Equation 10.
Sliding window size=[15,20,25,30,35,40,45,50,55](10)
Figuratively, such case is similar to for a window, gives initiating terminal, and starting is sized to 15 frames, such as
The movement locus of video analysis in window of the fruit based on 15 frame does not identify such as lift hand gesture, then passes through clearing end one
Secondary such as 5 frames that extend back are so that window size expands as 20 frames, and continues to carry using the video of 20 frame as analysis object
Take movement locus, carry out lift hand gesture identification；If recognizing lift hand gesture in some window size, window change stops simultaneously
No longer slide, except the identification of another gesture have to be proceeded；If generally repeating said process until window size is 55 frames
Lift hand gesture is not recognized yet, then window initiating terminal is moved rearwards by pre- fixed step size, for example, moves 2 frames, is then proceeded to using upper
The sliding window series that the size in face is variable carries out the new identifying processing of a wheel.
Fig. 7 shows movement locus of the sliding window according to an embodiment of the invention based on variable-size from input
Feature recognizes the flow chart of the method 122 of gesture.This method 122 can apply to the step S122 shown in Fig. 4.
In step S1221,3D motion track characteristic is inputted, wherein motion feature for example comes from the step shown in Fig. 4
S121 result.
In step S1222, the 3D motion track in the range of current window and prearranged gesture are for example lifted to the fortune of hand gesture
Movable model matches.
In step S1223, judge whether the movement locus matches the motion model of prearranged gesture, for example whether meeting 2D
The parabolic motion on linear motion and 1D Depth Domains in plane, such as judge whether to meet the condition in above-mentioned 2.5 trifle
(1) (6) are arrived.If judging that the movement locus meets the motion trajectory model in the present invention, 3D motion rail in step S1223
Mark identification process terminates, and otherwise proceeds to step S1224.
In step S1224, judge sliding window size whether for maximum.
If in step S1224, determining sliding window template not up to maximum, then proceeding to step S1225；Otherwise
Proceed to step S1226.
In step S1225, change the sizes values of sliding window, and return to step S1221, input newly increases the fortune of frame
Dynamic feature, to proceed movement locus template matches.
In step S1226, judge whether motion terminates.
If determining that motion does not terminate also, proceeds to step S1227 in step S1226, otherwise 3D motion track is known
Other process terminates.
In step S1227, sliding window is resetted, i.e., sliding window size is transferred to minimum value again, then advanced
To step S1228.
In step S1228, track is moved to next motion center of mass point, step S1221 is then returned to, input is newly-increased
Plus the motion feature of frame, to proceed movement locus matching.
It should be noted that above-mentioned formula（10）Shown in sliding window size series it is merely illustrative, can be according to image
The frame per second of collecting device, performance of computer etc. arbitrarily set suitable sliding window size series.
In above-described embodiment, prearranged gesture, Ke Yigeng are detected by using the variable sliding window of above-mentioned window size
The exercise habit difference of different people is adapted to well, more accurately carries out gestures detection.
3rd, the second embodiment of gesture identification method
The overall procedure of the gesture identification method of 3.1 second embodiments
Fig. 8 shows the flow chart of gesture identification method 200 according to a second embodiment of the present invention.According to second embodiment
Gesture identification method 200 according to the difference of the gesture identification method 100 of first embodiment with being anthropometry checking many
Step S230, and 3 D motion trace detecting step S210 therein, based on 3 D motion trace recognize prearranged gesture the step of
S220 is identical with step S110, S120 in first embodiment, and the repeated description to it is omitted here.
In step S230, if identifying prearranged gesture according to the 3 D motion trace in hand candidate region, it is determined that
Whether the position relationship between hand candidate region and other positions of human body meets the anthropological measuring made in the case of prearranged gesture
Model is learned, to verify the prearranged gesture.
The prearranged gesture that anthropometry model is used for such as lift hand gesture further to being identified is verified.People
Bulk measurement is to describe the subject of the constitutive character situation of the mankind with the method for measurement and observation.Anthropometry is tied
Share in the computer video such as image recognition field.
Gained knowledge according to anthropological measuring, it is generally the case that in lift hand gesture end, the height of hand is typically up to human body
A certain height region, by close to the height region of head, while there is also certain distance between hand central point and head center point
Value.
The specific illustrative implementation that prearranged gesture is verified according to anthropometry model is provided below in conjunction with Fig. 9
Example.
3.2nd, prearranged gesture is verified according to anthropometry model
Fig. 9 shows the method 230 of anthropometry model checking prearranged gesture according to an embodiment of the invention
Flow chart.This method can apply to the step S230 shown in Fig. 8.
As shown in figure 9, in step S231, foreground segmentation is carried out to depth image, to obtain human region.For example, such as
Preceding described use connected domain analysis method carries out foreground segmentation, association area region is then combined with, according to body region priori
Indicate, obtain correct foreground image, in this example i.e. human region.Shown in obtained foreground image such as Figure 10 (a1).
In step S232, head zone is detected from human region, the position of head zone is calculated.
According to one embodiment, we use histogram analysis method progress head positioning.But, histogram analysis method
Only it is example, any method that can carry out head positioning may be incorporated for the present invention, as utilized " Ω " head and shoulder detection model
Method.
Figure 10 show it is according to an embodiment of the invention, using histogram analysis method head center line position,
Shoulder horizontal direction positions and lifted as example the schematic diagram of the hand of hand gesture end and the mutual alignment relation on head.
How the positioning of head center line and shoulder water are carried out according to one embodiment of the invention below in conjunction with Figure 10 explanations
Square to positioning, and how the mutual alignment relation that should be met according to the hand of such as lift hand gesture end with head be carried out
Lift the checking of hand gesture.
According to one embodiment, the head positioning in step S232 can be realized by operations described below.
1) statistics with histogram in vertical direction is carried out to foreground image, or, in other words, each row of foreground image are united
The sum for the number that the pixel value of meter from top to bottom is not zero, to find head median vertical line.As shown in (a2) in Figure 10, vertical
In statistic histogram, the statistic histogram value of head zone is maximum, therefore can obtain figure by searching local maxima statistical value
Head median vertical line in 10 (a3).
2) median vertical line based on head, to lift hand on one side（In this example, i.e., on the right of）Body region carry out level
Statistics with histogram on direction, or, in other words, the number that each row statistical pixel values of transverse direction of image are not zero sum, with
The position (being indicated by H2) of neck area in the horizontal direction is positioned by finding minimum statistics value location point.Such as Figure 10 (b1)
Shown, the left part in Figure 10 (b1) represents horizontal histogram, right part represent be people foreground image, the water in left side
Flat histogram is that the histogrammic result of level statistic is done to the human body foreground picture of lift hand side, and such statistics with histogram mode will
It will not be influenceed by another side hands movement.
3) alternatively, in one embodiment, in order to more accurately navigate to the position of neck area in the horizontal direction
Put, line translation, transformation for mula such as formula (11), shown in experimental result such as Figure 10 (b2) can be entered to level statistic histogram.Becoming
In level statistic histogram after changing, neck area is more more obvious than the neck area in Figure 10 (b1), is conducive to neck to position.
Based on Figure 10 (b2), neck area can be positioned by finding maximum statistical value location point.
In formula (11), histiRepresent level statistic histogram, i be histogrammic index value, hist (i-c) and
Hist (i+c) is initial level statistic histogram, sees Figure 10 (b1).Hist_new is that obtain new straight is converted by formula (11)
Fang Tu, c are step-length constant value, and n is maximum index value.
4) position of position (being indicated by H2) and head coboundary in the horizontal direction according to neck in the horizontal direction
（Indicated by H1）, obtain head zone.After head zone is oriented, head width and head center of mass point can be calculated
Position.
In step S233, whether the first predetermined model is fallen into based on the vertical range between head position and hand position
Enclose, and whether the space length between head position and hand position falls into the second preset range, whether checking gesture is pre-
Determine gesture.
Typically, for prearranged gesture, according to anthropometry, the vertical range between head position and hand position should
Fall into preset range.Exemplified by lifting hand gesture, according to the position relationship between head and hand, it can interpolate that out that the position of motion hand is high
Whether degree meets the final height of lift hand gesture.This will be described in more detail with reference to Figure 10 (c).
In Figure 10 (c), P1 is head center of mass point, and P2 is hand center of mass point.Head width/and highly it is h, shoulder width is
2h.Under normal circumstances, the final height of lift hand gesture is between height H1 and H3.
, can be using formula (12) by checking head center of mass point P1 and hand center of mass point P2 according to an exemplary embodiment
Whether the distance on y-coordinate axle is less than certain altitude scope, to verify whether the final height of hand meets the requirements.
abs(p1.y-p2.y)<a*h (12)
In formula (12), p1.y represents head center of mass point P1 y-coordinate value, and p2.y represents the y-coordinate value of hand center of mass point,
A is constant coefficient, and a value can be obtained by collecting the lift hand custom progress data analysis statistical of user
In addition, for prearranged gesture, according to anthropometry, the space length between head position and hand position should fall
Enter preset range.
, can be using hand center of mass point P2 and head center of mass point P1 in actual spatial coordinates system according to an exemplary embodiment
In Euclidean distance verify the validity of lift hand gesture, such as shown in formula (13).
In formula (13), Dp1-p2For the Euclidean distance between head center of mass point P1 and hand center of mass point P2.c1And c2To be pre-
Determine threshold value, the threshold value can be by gathering user profile, and experimental calculation is obtained repeatedly.Euclidean distance Dp1-p2Position for judging hand
Put a little, if in the range of human region.But Euclidean distance is merely illustrative, can weigh hand using other distance metrics
Space length between head.
Further optionally, in one embodiment, have to lift residence time of hand at hand gesture end as lift hand gesture
One verification condition of effect property.Usually, as a lift hand gesture, at the end of lift hand gesture, it should stop and at least make a reservation for
Time, for example, it can be judged according to following formula (14) in given time range, whether hand remains static.
In formula (14), Dp2Represent position P2 of the hand center of mass point in last moment lastlastWith at current time
Current position P2currentBetween space length；c3It is a smaller value for predetermined threshold, if hand center of mass point and upper
Space length D between one moment and the position at current timep2Less than the threshold value, it is determined that hand remains static.Numerical value c3
It can be needed to set different numerical value according to systematic function, such as could be arranged to 1cm or 5cm, in the case of 1cm, system will
The hand of operator is asked to be kept essentially stationary；If provided as 5cm, the hand of user can keep light during anthropometry is analyzed
Micro- movement, can so prevent system point jitter conditions.For the robustness of strengthening system, the meter of residence time can be set
When CountstayingtimeIt should be greater than scheduled time threshold value Tthres, time threshold TthresCan suitably it be set according to application, for example
1 second.
Utilize the above-mentioned gesture verification process based on anthropometry model, if it is determined that the lift hand gesture meets people's body examination
Amount learns model, then it is an effective lift hand gesture to determine the lift hand gesture.
4th, the open method of man-machine interactive system
Above-mentioned gesture identification method and device based on depth image can have many application-specifics, for example, can answer
For the open method as man-machine interactive system.
At present, many systems are all that gesture special to some in certain circumstances is identified, and to user whether
State is can control into system gesture, corresponding prompt message is seldom provided.This causes some of user to be in operation not intended to
Knowledge behavior is identified as operable gesture, and the control to man-machine interactive system brings very big inconvenience, reduces man-machine interactive system
Friendly.
If in man-machine interactive system, adding a kind of startup control signal to prompt the user whether to have been enter into system control
State, prevents the appearance of user's unconscious movement, then man-machine interaction mode will be more friendly.Therefore system starts control signal
It is very important in man-machine interaction, and a kind of convenient, reliable system starts control mode and will improve user's body well
Test.
One simple, natural and stabilization initiation gesture is very important in man-machine interactive system.Lifting hand gesture is
A kind of very convenient effective user's operating gesture, can be used in man-machine interactive system, start as system and control, improve
The Consumer's Experience of man-machine interactive system.
Describe according to an embodiment of the invention a kind of by recognizing that lift hand gesture opens man-machine friendship with reference to Figure 11
The method of mutual system.
Figure 11 shows a kind of flow of method 300 for opening man-machine interactive system according to an embodiment of the invention
Figure.
As shown in figure 11, in step S310, acquisition includes the range image sequence of hand region, such as by such as double
Mesh camera come shoot obtain, from the outside through cross wired connection or wireless connection transmission obtain.
In step s 320, based on the range image sequence including hand region, three maintenance and operations in detection hand candidate region
Dynamic rail mark
In step S330, according to the 3 D motion trace in hand candidate region, identification lift hand gesture.
The implementing of above-mentioned steps S320 and S330 may be referred to the step S110 and S120 for being previously with regard to Fig. 2 and
The realization of figure, only, what is specifically recognized herein is lift hand gesture.
In step S340, if recognizing lift hand gesture, man-machine interactive system is opened, into man-machine interaction state
In.
Equally similarly, in the open method of this man-machine interactive system, people's body examination can also be applied for gesture identification
Amount is learned model and verified, specifically may be referred to the description of above-mentioned combination Fig. 8,9,10, repeats no more here.
By said process so that after lifting hand gesture activation system, user can carry out system control behaviour using other gestures
Make.
In above-mentioned scene, quickly, stably, in real time, robustly identify that lift hand gesture is very important.The present invention
Embodiment is based on 3D motion trajectory analysis and alternatively also based on anthropometry model analysis, there is provided a kind of quick, robust
Gesture identification method.
Complexion model is not used in the invention, but uses the time-space domain motion information and continuous depth value of consecutive image to become
Change information and carry out gesture identification.Therefore the gesture identification method performance is compared with robust, and suffered illumination condition influence is smaller.The other hair
Bright to carry out gesture identification based on movement locus, it can be used in distance range relatively far away from.The gesture identification method takes
It is short and robustness is high, in man-machine interactive system, user experience will to be effectively improved.
5th, the gesture identifying device based on depth image
Gesture identifying device based on depth image according to embodiments of the present invention is described below with reference to Figure 12.
Figure 12 shows the functional configuration frame of the gesture identifying device 400 according to embodiments of the present invention based on depth image
Figure.
As shown in figure 12, gesture identifying device 400 can include：3 D motion trace detection part 410, for based on bag
Include the range image sequence of hand region, the 3 D motion trace in detection hand candidate region；And gesture identification part 420,
For the 3 D motion trace according to hand candidate region, prearranged gesture is recognized.
Above-mentioned 3 D motion trace detection part 410 and gesture identification part 420, concrete function and operation may be referred to
Description related above-mentioned Fig. 1 to Fig. 3.Here relevant repeated description is omitted.
6th, system hardware configuration
The present invention can also improve hardware system to implement by a kind of disparity map.Figure 13 is shown according to present invention implementation
The general hardware block diagram of the gesture recognition system 1000 of example.As shown in figure 13, disparity map improves system 1000 and can included：Input
Equipment 1100, for left image and right image that from the relevant image of outside input or information, such as video camera is shot, video camera
Parameter or depth map, initial parallax figure etc., for example can include keyboard, Genius mouse and communication network and its connected it is remote
Journey input equipment etc.；Processing equipment 1200, for implementing the above-mentioned gesture based on depth map according to the embodiment of the present invention
Recognition methods, is either embodied as above-mentioned gesture identifying device or implements the open method of above-mentioned man-machine interactive system, for example
Central processing unit or other chips with disposal ability of computer etc. can be included, it may be connected to such as internet
Network（It is not shown）, according to image after being handled to teletransmission the need for processing procedure etc.；Output equipment 1300, is used for
Implement the result obtained by the opening process of above-mentioned gesture identification process or man-machine interactive system to outside output, for example, can wrap
Include display, printer and communication network and its remote output devices connected etc.；And storage device 1400, use
It is such as deep involved by the unlatching of above-mentioned gesture identification process or man-machine interactive system in being stored in volatile and nonvolatile mode
Spend figure, foreground picture, Background, motion center of mass point position and correspondence moment, 3 D motion trace, the spy of 2D plane motions track
Levy, the data such as movement locus feature in Depth Domain, can for example include random access memory（RAM）, read-only storage
（ROM）, hard disk or semiconductor memory etc. various volatile and nonvolatile property memories.
7th, summarize
Embodiments in accordance with the present invention can include there is provided a kind of gesture identification method based on depth image：It is based on
Range image sequence including hand region, the 3 D motion trace in detection hand candidate region；And according to hand candidate area
The 3 D motion trace in domain, recognizes prearranged gesture.
According to another embodiment of the present invention there is provided a kind of method for opening man-machine interactive system, including：Including
The range image sequence of hand region；Based on the range image sequence including hand region, the three-dimensional in detection hand candidate region
Movement locus；According to the 3 D motion trace in hand candidate region, identification lift hand gesture；And if recognizing lift hand gesture,
Man-machine interactive system is then opened, into man-machine interaction state.
There is provided a kind of side that human body predetermined action is recognized based on depth image according to another embodiment of the present invention
Method, can include：Based on the range image sequence in the human body region is included, three maintenance and operations of human body candidate region are detected
Dynamic rail mark；And according to the 3 D motion trace of human body candidate region, recognize the predetermined action of human body.
According to another embodiment of the present invention there is provided a kind of gesture identifying device based on depth image, it can include：
3 D motion trace detection part, for based on the range image sequence including hand region, the three of detection hand candidate region
Tie up movement locus；And gesture identification part, for the 3 D motion trace according to hand candidate region, recognize prearranged gesture.
According to another embodiment of the present invention there is provided a kind of device for opening man-machine interactive system, including：Depth image
Sequence obtains part, includes the range image sequence of hand region for obtaining；3 D motion trace detection part, for based on
Range image sequence including hand region, the 3 D motion trace in detection hand candidate region；Hand gesture identification part is lifted, is used
In the 3 D motion trace according to hand candidate region, identification lift hand gesture；And man-machine interactive system turned parts, for such as
Fruit recognizes lift hand gesture, then man-machine interactive system is opened, into man-machine interaction state.
There is provided a kind of dress that human body predetermined action is recognized based on depth image according to another embodiment of the present invention
Put, can include：3 D motion trace detection part, for based on the range image sequence in the human body region is included, examining
Survey the 3 D motion trace of human body candidate region；And human body motion identification component, for being waited according to human body
The 3 D motion trace of favored area, recognizes the predetermined action of human body.
Using gesture identification method and device according to embodiments of the present invention based on depth image, because by Depth Domain
Movement locus include gesture identification process, so as to take full advantage of the movable information in continuous identification image；Due to being not used
Complexion model, but use the time-space domain motion information and continuous depth value change information of consecutive image to carry out gesture identification,
Therefore the gesture identification method performance is compared with robust, and suffered illumination condition influence is smaller；Due to carrying out gesture knowledge based on movement locus
Not, it can be used in distance range relatively far away from.Gesture identification method according to embodiments of the present invention takes short and robust
Property it is high.
According to embodiments of the present invention lifts hand gesture so as to open man-machine interactive system by being recognized based on depth image
Technology there is provided one kind is convenient, reliable system starts control mode, prompt the user whether to come into system control shape
State, prevents user's unconscious movement to be erroneously identified as operable gesture, so that there is provided a kind of people of more user friendly
Machine interactive mode.
It is described above only illustrative, much it can be changed and/or be replaced.
To be illustrated exemplified by lifting the identification of hand gesture in accompanying drawing above and description, but the invention is not limited in
This, recognizes that the technology of gesture can apply to the identification of other gestures, example based on the 3 D motion trace in hand candidate region
The gesture such as let go downwards, from letting go to by the gesture for being manually placed into front, the gesture waved.Further, it is of the invention
Be not limited to the identification of hand motion, but can apply to the identification of the action of other human bodies, for example step, leg,
Buttocks, head etc..Yet further, the method for the invention based on 3 D motion trace identification maneuver is not limited to only recognize
The action of people, can also be applied to the identification of the identification such as action of animal, robot, mechanical hand etc. movable object.
In addition, hereinbefore, the application of gesture identification is illustrated exemplified by opening man-machine interactive system, but the present invention is not
It is confined to this.Man-machine interaction that gesture identification based on depth image can be used in projecting apparatus control, game machine etc..
In addition, being hereinbefore broken down into the motion in Depth Domain when the 3 D motion trace to hand is analyzed
Track and the 2D movement locus vertical with Depth Domain, but it is only for example.Can be without decomposing, in Direct Analysis 3d space
Movement locus.Or, it can also further be decomposed, for example, be decomposed into the movement locus in Depth Domain, x in 2D planes
Movement locus on movement locus and y-axis on axle etc..
In addition, the depth map in being described above is construed as generalized concept, that is, include the image of range information, it contains
Justice covers usually said disparity map, because it will be apparent to those skilled in the art that can be by between parallax and depth simple
Mutually it is converted to.
In addition, in being described above, the position of hand region is characterized in gesture identification with the position of hand center of mass point, but
Example is only for, point such as artis can be represented using other as needed.In addition, being clicked through here only with a barycenter
Row analysis, but example is only for, it will be appreciated that in some cases, for complicated gesture, can both analyze the matter of hand
Heart point, also artis, artis of ancon of analysis wrist etc..
In addition, the Gesture Recognition in description above, is based only on depth image, but should be based on depth image
Gesture identification can combine the technology based on gray level image, such as carrying out hard recognition based on complexion model.
The general principle of the present invention is described above in association with specific embodiment, however, it is desirable to, it is noted that to this area
For those of ordinary skill, it is to be understood that the whole or any steps or part of methods and apparatus of the present invention, Ke Yi
Any computing device（Including processor, storage medium etc.）Or in the network of computing device, with hardware, firmware, software or
Combinations thereof is realized that this is that those of ordinary skill in the art use them in the case where having read the explanation of the present invention
Basic programming skill can be achieved with.
Therefore, the purpose of the present invention can also by run on any computing device a program or batch processing come
Realize.The computing device can be known fexible unit.Therefore, the purpose of the present invention can also be included only by offer
Realize that the program product of the program code of methods described or device is realized.That is, such program product is also constituted
The present invention, and the storage medium for such program product that is stored with also constitutes the present invention.Obviously, the storage medium can be
Any known storage medium or any storage medium developed in the future.
It may also be noted that in apparatus and method of the present invention, it is clear that each part or each step are to decompose
And/or reconfigure.These decompose and/or reconfigured the equivalents that should be regarded as the present invention.Also, perform above-mentioned series
The step of processing can order naturally following the instructions perform in chronological order, but and need not necessarily sequentially in time
Perform.Some steps can be performed parallel or independently of one another.
Above-mentioned embodiment, does not constitute limiting the scope of the invention.Those skilled in the art should be bright
It is white, depending on design requirement and other factors, can occur various modifications, combination, sub-portfolio and replacement.It is any
Modifications, equivalent substitutions and improvements made within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (8)
1. a kind of gesture identification method based on depth image, including：
Based on the range image sequence including hand region, the 3 D motion trace in detection hand candidate region；And
According to the 3 D motion trace in hand candidate region, prearranged gesture is recognized,
Wherein, if the 3 D motion trace in hand candidate region meets parabolic motion model, Yi Ji in the depth direction
Two dimensional motion track in the plane of depth direction meets linear motion model, then identify upwards lift hand gesture or to
Under let go gesture, and
The direction of motion, end of time according to indicated by the relation between location point on chronological 3 D motion trace
Hand position distinguishes lift hand gesture upwards and gesture of letting go downwards.
2. gesture identification method according to claim 1, in addition to：
If identifying prearranged gesture according to the 3 D motion trace in hand candidate region, it is determined that hand candidate region and human body
Whether the position relationship between other positions meets the anthropometry model made in the case of prearranged gesture, to verify that this makes a reservation for
Gesture.
3. gesture identification method according to claim 1, based on range image sequence, the three-dimensional motion in detection hand candidate region
Track includes：
Obtain the moving mass region in depth image；
Hand candidate region is chosen from moving mass region；Calculate and record the positional information in hand candidate region；And
Obtain the positional information sequence corresponding to range image sequence.
4. gesture identification method according to claim 3, wherein 3 D motion trace are included corresponding to each of each frame depth image
The motion feature of oneself of individual tracing point,
The motion feature of oneself of each tracing point includes：The two-dimension time-space domain of position, speed, angle including tracing point is special
Levy including tracing point depth value Depth Domain motion feature, each tracing point two-dimension time-space characteristic of field and Depth Domain motion
The feature equal and tracing point time point is associated.
5. gesture identification method according to claim 4, wherein, determine which is based on using the variable sliding window of window size
A little and how many depth images carry out 3 D motion trace detection and analysis, and the size of window represented with the depth of continuous how many frame
Image is spent as the input of gesture identification；
If based on the range image sequence in the sliding window of predefined size 3 D motion trace can not match simultaneously with advance
Determine motion model in the corresponding Depth Domain of gesture and the plane vertical with Depth Domain on two dimensional motion locus model, then increase
The size of the sliding window is so that more depth image frames, as the input of gesture identification, and are proceeded after the increase
The matching of the corresponding 3 D motion trace of the size motion trajectory model corresponding with the prearranged gesture of sliding window.
6. gesture identification method according to claim 2, wherein determining the position between hand candidate region and other positions of human body
Putting relation and whether meeting the anthropometry model made in the case of prearranged gesture includes：
Foreground segmentation is carried out to depth image, to obtain human region；
Head zone is detected from human region, the position of head zone is calculated；
Whether the first preset range, and head position and hand are fallen into based on the vertical range between head position and hand position
Whether the space length between portion position falls into the second preset range, and whether checking gesture is prearranged gesture.
7. a kind of method for opening man-machine interactive system, including：
Acquisition includes the range image sequence of hand region；
Based on the range image sequence including hand region, the 3 D motion trace in detection hand candidate region；
According to the 3 D motion trace in hand candidate region, identification lift hand gesture；And
If recognizing lift hand gesture, man-machine interactive system is opened, into man-machine interaction state,
Wherein, if the 3 D motion trace in hand candidate region meets parabolic motion model, Yi Ji in the depth direction
Two dimensional motion track in the plane of depth direction meets linear motion model, then identify upwards lift hand gesture or to
Under let go gesture, and
The direction of motion, end of time according to indicated by the relation between location point on chronological 3 D motion trace
Hand position lifts hand gesture and gesture of letting go downwards upwards to distinguish.
8. a kind of gesture identifying device based on depth image, including：
3 D motion trace detection part, for based on the range image sequence including hand region, detection hand candidate region
3 D motion trace；And
Gesture identification part, for the 3 D motion trace according to hand candidate region, recognizes prearranged gesture,
Wherein, if the 3 D motion trace in hand candidate region meets parabolic motion model, Yi Ji in the depth direction
Two dimensional motion track in the plane of depth direction meets linear motion model, then the gesture identification part is identified
Hand gesture or gesture of letting go downwards are lifted upwards, and
The direction of motion, end of time according to indicated by the relation between location point on chronological 3 D motion trace
Hand position distinguishes lift hand gesture upwards and gesture of letting go downwards.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210490622.8A CN103839040B (en) | 2012-11-27 | 2012-11-27 | Gesture identification method and device based on depth image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210490622.8A CN103839040B (en) | 2012-11-27 | 2012-11-27 | Gesture identification method and device based on depth image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103839040A CN103839040A (en) | 2014-06-04 |
| CN103839040B true CN103839040B (en) | 2017-08-25 |
Family
ID=50802519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210490622.8A Expired - Fee Related CN103839040B (en) | 2012-11-27 | 2012-11-27 | Gesture identification method and device based on depth image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103839040B (en) |
Families Citing this family (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105279526B (en) * | 2014-06-13 | 2019-11-29 | 佳能株式会社 | Divide the method and apparatus of track |
| CN104200491A (en) * | 2014-08-15 | 2014-12-10 | 浙江省新华医院 | Motion posture correcting system for human body |
| CN105528060B (en) * | 2014-09-30 | 2018-11-09 | 联想(北京)有限公司 | terminal device and control method |
| CN105718037A (en) * | 2014-12-05 | 2016-06-29 | 乐视致新电子科技（天津）有限公司 | Method and device for identifying states of target object |
| CN104714650B (en) * | 2015-04-02 | 2017-11-24 | 三星电子（中国）研发中心 | A kind of data inputting method and device |
| KR102370678B1 (en) * | 2015-06-25 | 2022-03-07 | 삼성전자주식회사 | Method and Apparatus for Controlling A Touch Sensing Module of Electronic Device, Method and Apparatus for Operating A Touch Sensing Module of Electronic Device |
| US9702722B2 (en) * | 2015-09-26 | 2017-07-11 | Volkswagen Ag | Interactive 3D navigation system with 3D helicopter view at destination |
| CN106886741A (en) * | 2015-12-16 | 2017-06-23 | 芋头科技（杭州）有限公司 | A kind of gesture identification method of base finger identification |
| CN106909871A (en) * | 2015-12-22 | 2017-06-30 | 江苏达科智能科技有限公司 | Gesture instruction recognition methods |
| CN106325509A (en) * | 2016-08-19 | 2017-01-11 | 北京暴风魔镜科技有限公司 | Three-dimensional gesture recognition method and system |
| US10095315B2 (en) * | 2016-08-19 | 2018-10-09 | Otis Elevator Company | System and method for distant gesture-based control using a network of sensors across the building |
| WO2018069981A1 (en) * | 2016-10-11 | 2018-04-19 | 富士通株式会社 | Motion recognition device, motion recognition program, and motion recognition method |
| CN106570482B (en) * | 2016-11-03 | 2019-12-03 | 深圳先进技术研究院 | Human motion recognition method and device |
| CN108073851B (en) * | 2016-11-08 | 2021-12-28 | 株式会社理光 | Grabbing gesture recognition method and device and electronic equipment |
| CN106846403B (en) * | 2017-01-04 | 2020-03-27 | 北京未动科技有限公司 | Method and device for positioning hand in three-dimensional space and intelligent equipment |
| CN107239728B (en) * | 2017-01-04 | 2021-02-02 | 赛灵思电子科技(北京)有限公司 | Unmanned aerial vehicle interaction device and method based on deep learning attitude estimation |
| CN106886750A (en) * | 2017-01-04 | 2017-06-23 | 沈阳工业大学 | Extracting tool movement locus recognition methods based on Kinect |
| CN108268181A (en) * | 2017-01-04 | 2018-07-10 | 奥克斯空调股份有限公司 | A kind of control method and device of non-contact gesture identification |
| CN106933352A (en) * | 2017-02-14 | 2017-07-07 | 深圳奥比中光科技有限公司 | Three-dimensional human body measurement method and its equipment and its computer-readable recording medium |
| CN107065599B (en) * | 2017-06-12 | 2021-05-07 | 山东师范大学 | Motion simulation system and method of wheeled robot based on somatosensory interaction |
| CN107590794A (en) * | 2017-09-11 | 2018-01-16 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
| CN108563995B (en) * | 2018-03-15 | 2019-04-26 | 西安理工大学 | Human computer cooperation system gesture identification control method based on deep learning |
| CN108815845B (en) * | 2018-05-15 | 2019-11-26 | 百度在线网络技术（北京）有限公司 | The information processing method and device of human-computer interaction, computer equipment and readable medium |
| CN108830891B (en) * | 2018-06-05 | 2022-01-18 | 成都精工华耀科技有限公司 | Method for detecting looseness of steel rail fishplate fastener |
| CN109033972A (en) * | 2018-06-27 | 2018-12-18 | 上海数迹智能科技有限公司 | A kind of object detection method, device, equipment and storage medium |
| CN109407842A (en) * | 2018-10-22 | 2019-03-01 | Oppo广东移动通信有限公司 | Interface operation method, device, electronic equipment and computer readable storage medium |
| CN111107278B (en) * | 2018-10-26 | 2022-03-01 | 北京微播视界科技有限公司 | Image processing method and device, electronic equipment and readable storage medium |
| CN111563401A (en) * | 2019-02-14 | 2020-08-21 | 上海汽车集团股份有限公司 | Vehicle-mounted gesture recognition method and system, storage medium and electronic equipment |
| CN110347260B (en) * | 2019-07-11 | 2023-04-14 | 歌尔科技有限公司 | Augmented reality device, control method thereof and computer-readable storage medium |
| TWI704499B (en) * | 2019-07-25 | 2020-09-11 | 和碩聯合科技股份有限公司 | Method and device for joint point detection |
| CN110834331A (en) * | 2019-11-11 | 2020-02-25 | 路邦科技授权有限公司 | Bionic robot action control method based on visual control |
| CN113139402B (en) * | 2020-01-17 | 2023-01-20 | 海信集团有限公司 | A kind of refrigerator |
| CN111401166A (en) * | 2020-03-06 | 2020-07-10 | 中国科学技术大学 | Robust gesture recognition method based on electromyographic information decoding |
| CN111651038A (en) * | 2020-05-14 | 2020-09-11 | 香港光云科技有限公司 | Gesture recognition control method based on ToF and control system thereof |
| CN112089595A (en) * | 2020-05-22 | 2020-12-18 | 未来穿戴技术有限公司 | Login method of neck massager, neck massager and storage medium |
| CN111913574B (en) * | 2020-07-15 | 2024-04-30 | 抖音视界有限公司 | Method, apparatus, electronic device, and computer-readable medium for controlling device |
| CN112099634A (en) * | 2020-09-17 | 2020-12-18 | 珠海格力电器股份有限公司 | Interactive operation method and device based on head action, storage medium and terminal |
| CN112306235B (en) * | 2020-09-25 | 2023-12-29 | 北京字节跳动网络技术有限公司 | Gesture operation method, device, equipment and storage medium |
| CN112835449A (en) * | 2021-02-03 | 2021-05-25 | 青岛航特教研科技有限公司 | Virtual reality and somatosensory device interaction-based safety somatosensory education system |
| CN113282164A (en) * | 2021-03-01 | 2021-08-20 | 联想(北京)有限公司 | Processing method and device |
| CN112686231B (en) * | 2021-03-15 | 2021-06-01 | 南昌虚拟现实研究院股份有限公司 | Dynamic gesture recognition method and device, readable storage medium and computer equipment |
| CN113158912B (en) * | 2021-04-25 | 2023-12-26 | 北京华捷艾米科技有限公司 | Gesture recognition method and device, storage medium and electronic equipment |
| CN114415830A (en) * | 2021-12-31 | 2022-04-29 | 科大讯飞股份有限公司 | Air input method and device, computer readable storage medium |
| CN114639172B (en) * | 2022-05-18 | 2022-09-30 | 合肥的卢深视科技有限公司 | High-altitude parabolic early warning method and system, electronic equipment and storage medium |
| CN116627260A (en) * | 2023-07-24 | 2023-08-22 | 成都赛力斯科技有限公司 | Method and device for idle operation, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253713A (en) * | 2011-06-23 | 2011-11-23 | 康佳集团股份有限公司 | Display system orienting to three-dimensional images |
| CN102509074A (en) * | 2011-10-18 | 2012-06-20 | Tcl集团股份有限公司 | Target identification method and device |
| CN102789568A (en) * | 2012-07-13 | 2012-11-21 | 浙江捷尚视觉科技有限公司 | Gesture identification method based on depth information |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE602004006190T8 (en) * | 2003-03-31 | 2008-04-10 | Honda Motor Co., Ltd. | Device, method and program for gesture recognition |
-
2012
- 2012-11-27 CN CN201210490622.8A patent/CN103839040B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253713A (en) * | 2011-06-23 | 2011-11-23 | 康佳集团股份有限公司 | Display system orienting to three-dimensional images |
| CN102509074A (en) * | 2011-10-18 | 2012-06-20 | Tcl集团股份有限公司 | Target identification method and device |
| CN102789568A (en) * | 2012-07-13 | 2012-11-21 | 浙江捷尚视觉科技有限公司 | Gesture identification method based on depth information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103839040A (en) | 2014-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103839040B (en) | Gesture identification method and device based on depth image | |
| Singh et al. | Video benchmarks of human action datasets: a review | |
| CN110738101B (en) | Behavior recognition method, behavior recognition device and computer-readable storage medium | |
| Raheja et al. | Robust gesture recognition using Kinect: A comparison between DTW and HMM | |
| Zhu et al. | Evaluating spatiotemporal interest point features for depth-based action recognition | |
| CN105022982B (en) | Hand motion recognition method and apparatus | |
| CN103718175B (en) | Detect equipment, method and the medium of subject poses | |
| CN105260726B (en) | Interactive video biopsy method and its system based on human face posture control | |
| CN107908288A (en) | A kind of quick human motion recognition method towards human-computer interaction | |
| US9117138B2 (en) | Method and apparatus for object positioning by using depth images | |
| CN109299659A (en) | A kind of human posture recognition method and system based on RGB camera and deep learning | |
| Bhuyan et al. | Fingertip detection for hand pose recognition | |
| CN110633004B (en) | Interaction method, device and system based on human body posture estimation | |
| JP6066093B2 (en) | Finger shape estimation device, finger shape estimation method, and finger shape estimation program | |
| CN102567716B (en) | Face synthetic system and implementation method | |
| JP2014165660A (en) | Method of input with virtual keyboard, program, storage medium, and virtual keyboard system | |
| CN114241379B (en) | Passenger abnormal behavior identification method, device, equipment and passenger monitoring system | |
| CN103777748A (en) | Motion sensing input method and device | |
| CN113065505B (en) | Method and system for quickly identifying body actions | |
| JP2016014954A (en) | Method for detecting finger shape, program thereof, storage medium of program thereof, and system for detecting finger shape | |
| JP6906273B2 (en) | Programs, devices and methods that depict the trajectory of displacement of the human skeleton position from video data | |
| Prathap et al. | Gait recognition using skeleton data | |
| CN106406507B (en) | Image processing method and electronic device | |
| US11361467B2 (en) | Pose selection and animation of characters using video data and training techniques | |
| Vo et al. | Dynamic gesture classification for Vietnamese sign language recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170825 Termination date: 20191127 |