Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/20—Movements or behaviour, e.g. gesture recognition
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/20—Movements or behaviour, e.g. gesture recognition
  • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language

Definitions

• This invention relates to data processing.
• BACKGROUND Interacting with computers is not limited to mouse and keyboard. Sensing the movement of a person to recognize his/her gesture is the subject of a wide spectrum of research in Human Computer Interaction and Computer Vision. Recognizing human hand gestures in particular provides computers with a natural method of communication. Applications from medical to surveillance and security may use the technology described herein. Learning and recognizing hand movements are significant components of such technologies.
• Bimanual movements in general form a large subset of hand movements in which both hands move simultaneously in order to do a task or imply a meaning. Clapping, opening a bottle, typing on a keyboard and drumming are some common bimanual movements. Sign Languages also use bimanual movements to accommodate sets of gestures for communication.
• Objects may be tracked using stereo imaging.
• Particle filtering may be used for tracking and resolving occlusion problems.
• Other tracking algorithms may use techniques such as, for example, Bayesian Networks, object model matching based on probabilistic tracking functions, minimization of cost functions, and analytic model matching.
• Several tracking algorithms include non-linear optimizations.
• One or more described implementations allow two hands to be tracked before an occlusion, the occlusion to be identified as such, and the separate hands to be reacquired and tracked after the occlusion.
• the tracking is independent of camera view, of hand shape, and of a changing hand shape such as occurs, for example, when fingers are moving.
• a gesture being performed by the hands may be recognized, including portions ofthe gesture being performed before, during, and after the occlusion.
• One or more tracking algorithms are able to deal with occlusions in real-time, to track non-rigid objects such as human hands, and are tolerant of changes caused by moving the position of a camera.
• one or more described systems is able to reacquire the hands when occlusion ends, and can do so without requiring the hands to be wearing different color gloves.
• One or more disclosed systems handles the variability of an object's shape due to the object's non-rigid nature. Such a system does not necessarily lose its tracking clue when the shape ofthe object changes quickly.
• One or more disclosed systems use a tracking algorithm that is independent of the camera view direction. Therefore, a change in the view direction may be tolerated by the algorithm.
• An interesting application of this is tracking hands while the camera moves. Dynamic changes in camera position are often inevitable in active vision applications such as mobile robots.
• Neural Networks are used for recognition in one or more systems, as are Bayesian Networks and in particular Hidden Markov Models (HMM).
• HMM Hidden Markov Models
• One or more disclosed implementations uses a recognition technique that tolerates hand-hand occlusion. During a bimanual movement one hand may cover the other hand partially or completely.
• One or more disclosed implementations uses a recognition technique that tolerates a hand temporarily moving out ofthe region of interest. In such a case, two hands are not present over the whole period of a bimanual gesture. A disclosed recognition technique also tolerates a hand being completely occluded by some other object like the body of person.
• One or more implementations was a recognition technique that recognizes continuous (concatenated) periodic bimanual movements. A periodic bimanual movement like clapping typically includes a short cycle of movement of two hands repeated several times. In many Virtual Reality applications, a few bimanual movements are concatenated in order to interact with the virtual environment, and these movements should be recognized and movement transitions should be detected.
• a Cognitive System for tracking the hands of a person, resolving left hand and right hand in the presence of occlusion, and recognizing bimanual movements is presented.
• the two hands of a person are tracked by a novel tracking algorithm based on one or more neuroscience phenomena.
• a gesture recognition algorithm recognizes the movement of each hand and combines the results in order to recognize the performed bimanual movement.
• the system may be useful in tracking and recognizing hand movements for interacting with computers, helping deaf people to communicate with others, and security applications.
• movement is tracked of two occluded hands during an occlusion period, and the two occluded hands are tracked as a unit.
• a type of synchronization is determined that characterizes the two occluded hands during the occlusion period.
• the type of synchronization is based, at least in part, on the tracked movement ofthe two occluded hands.
• Based at least in part on the determined type of synchronization it is determined whether directions of travel for each ofthe two occluded hands change during the occlusion period.
• Implementations may include one or more ofthe following features. For example, determining whether directions change may be further based on the tracked movement ofthe two occluded hands. Determining whether directions change may include determining whether the two hands pass each other during the occlusion period, pause during the occlusion period, or collide with each other during the occlusion period.
• Determining whether directions change may include determining whether each ofthe two hands go, after the occlusion period, to directions from which they came, or to directions opposite from which they came.
• the directions may include one or more of a vertical direction, a horizontal direction, and a diagonal direction.
• Determining a type of synchronization may include determining whether the two hands are positively or negatively synchronized, and determining whether directions change may be further based on whether the two hands are negatively synchronized.
• Determining a type of synchronization may include determining a measure ofthe occluded hands' velocities. The measure may include a standard deviation of a difference of velocities of parallel sides of a rectangle formed to surround the occluded hands.
• Tracking movement ofthe two occluded hands may include tracking movement of a rectangle formed to surround the occluded hands, and determining whether directions change may include determining a measure ofthe occluded hands' velocities based on velocities of one or more sides ofthe rectangle. Determining whether directions change may be based on whether the measure goes below a threshold.
• the measure may be a function of a square root of a sum of squares of velocities of parallel sides ofthe rectangle. Determining whether directions change may be based on one or more probability distributions ofthe measure.
• the measure may be a function of a difference of velocities of parallel sides ofthe rectangle.
• the one or more probability distributions may include a first set of distributions associated with a first velocity pattern and a second set of distributions associated with a second velocity pattern.
• the first velocity pattern may be indicative ofthe two hands passing each other during the occlusion period
• the second velocity pattern may be indicative ofthe two hands not passing each other during the occlusion period.
• Determining whether directions change may further include determining a first and a second probability, and comparing the first probability with the second probability.
• the first probability may be based on the first set of distributions, and be the probability that the first velocity pattern produced the measure ofthe occluded hands' velocities.
• the second probability may be based on the second set of distributions, and be the probability that the second velocity pattern produced the measure ofthe occluded hands' velocities. Based on a result obtained during the comparing, it may be determined whether the two occluded hands passed each other during the occlusion period.
• a first hand and a second hand are occluded, the first hand having come from a first direction and the second hand having come from a second direction.
• the movement ofthe occluded hands is tracked as a unit.
• a type of synchronization is determined that characterizes the occluded hands.
• the type of synchronization is determined, at least in part, based on the tracked movement ofthe occluded hands. It is determined that the first hand and the second hand are no longer occluded and, after this determination, the first hand is distinguished from the second hand based at least in part on the determined type of synchronization.
• FIGS. l(a)-(b) shows three main components of a particular system and a hierarchy for recognizing bimanual movements.
• FIG. 2 shows a rectangle around each of two hands.
• FIG. 3 shows the rectangles of FIG. 2 overlapping with no hand-hand occlusion.
• FIG. 4 shows a progression of movement ofthe rectangles of FIG. 2 creating a hand-hand occlusion.
• FIG. 5 shows the rectangles of FIG. 2 modeled by their sides.
• FIG. 6 illustrates a prediction of the intersection of two rectangles.
• FIG. 7 illustrates a scenario in which two hands may be labeled interchangeably in two consecutive images.
• FIGS. 8(a)-8(n) illustrate 14 models of bimanual movements.
• HI and H2 represent hand number one and hand number two.
• the thick ellipses represent the occlusion areas (a, c, d, e,f, h, i,j, and n), and the solid small rectangles represent collision (b, g, k, and /).
• FIG. 9 illustrates an occlusion-rectangle formed around the big blob of hands.
• FIG. 10 shows a progression of images in which the vertical sides ofthe occlusion-rectangle are pushed back because hands pass each other and push the vertical sides in opposite directions.
• FIGS. 1 l(a)-(b) illustrate the velocity changes for movements in which hands (a) pause/collide and return, or (b) pass each other.
• FIGS. 12(a)-(b) illustrate sequences of Gaussian distributions to model an occlusion-rectangle sides' velocities during the two categories of (a) hand-pause, and (b) hand-pass.
• FIG. 13 illustrates hand movements being separated and projected into blank sequences of images.
• FIG. 14 shows an image frame divided into 8 equal regions to represent direction of movement.
• FIG. 15 includes a series of images illustrating hand movement and an extracted vector for the movement.
• FIG. 16 illustrates the segmentation of a bimanual movement over a period of time.
• the separate lines at segments A, C, and D show the separated hands.
• segments B the overlaped lines show hand-hand occlusion.
• FIG. 17 shows a Bayesian network for fusing Hidden Markov Models for the recognition of bimanual movements.
• FIG. 18 shows an abstracted Bayesian network, based on FIG. 17, for the recognition of bimanual movements.
• FIG. 19 shows a 2-state left-to-right Hidden Markov Model assigned to partial gestures.
• FIG. 20(a) graphs the local belief of the root node for three concatenated bimanual movements.
• FIGS. 20(b)-(e) isolate various graphs from FIG. 20(a) associated with particular gestures.
• FIG. 21 graphs the local belief of the root node with limited memory for the three concatenated bimanual movements of FIG. 20.
• FIG. 22 shows a hardware implementation
• FIG. 23 illustrates a process for recognizing a bimanual gesture.
• one or more disclosed implementations includes a cognitive system 100 for learning and understanding bimanual movements that entails three fundamental components: low-level processing 110 to deal with sensory data, intelligent hand tracking 120 to recognize the left hand from the right hand, and bimanual movement recognition 130 for recognizing the movements.
• the hands are to be extracted from the images.
• the second component 120 includes hand tracking, which may be complicated by hand-hand occlusion. When one hand covers the other hand partially or completely, the two hands should be reacquired correctly at the end of occlusion.
• An implementation uses a Kalman filtering based technique to monitor hands' velocities, to detect pauses and to recognize synchronization between the hands. By detecting the synchronization and pauses, particularly during a hand-hand occlusion period, the tracking algorithm of an implementation recognizes the right hand from the left hand when occlusion ends.
• the tracking algorithm of one implementation is also used for segmenting a bimanual movement.
• each part ofthe movement receives a label that indicates whether the part is an occlusion or non-occlusion segment.
• a non-occlusion category may include three different segments, namely beginning, middle, and ending segments. Therefore, the tracking algorithm ofthe implementation divides a bimanual movement into up to four different segments depending on the nature of the movement.
• the tracking algorithm takes a general view ofthe tracking problem. For example, from a pure pattern recognition point of view, a movement can be recognized differently when it is seen from different viewing directions.
• a general set of movement models that are generally independent of view direction are defined so that a model can be found for a bimanual movement when it is seen from different viewing angles.
• bimanual synchronization may also make the tracking algorithm of one or more described implementations independent ofthe hand shapes. independence of hand shape and view direction may make a tracking algorithm useful in mobile vision applications (e.g., Active Vision in Robotics).
• the tracking algorithm of one implementation contains a model that is independent ofthe actual positions and velocities ofthe hands. Consequently, this tracking algorithm can be used in applications where the visual system moves or turns. For instance, assuming that a camera is installed on a mobile robot, the tracker can track the hands of a subject while the robot moves.
• the third component 130 includes gesture recognition, and, referring to FIG. 1(b), may be represented by a hierarchical cognitive system 140.
• System 140 analyzes hand shapes at a bottom level 150, which may use image analysis and pattern recognition for hand shape extraction and detection.
• System 140 learns the individual partial movement of each hand at an intermediate level 160, using, for example, spatio- temporal single-hand gesture recognition.
• System 140 combines the partial movements at a top level 170 to recognize the whole movement.
• Statistical and spatio-temporal pattern recognition methods such as Principal Component Analysis and Hidden Markov Models may be used in the bottom 150 and intermediate 160 levels ofthe system 140.
• a Bayesian inference network at the top level may perceive the movements as a combination of a set of recognized partial hand movements.
• a bimanual movement may be divided into individual movements ofthe two hands. Given that the hands may partially or completely occlude each other or a hand can disappear due to occlusion by another object, the fusion network at the bottom level may be designed to be able to deal with these cases.
• the occlusion and non-occlusion parts of a movement, which are treated as different segments, may be recognized separately.
• Individual Hidden Markov Models at the intermediate level may be assigned to the segments ofthe gestures ofthe hands.
• partial movements are recognized at the intermediate level.
• the hand shapes and the movement of each hand in each frame of a given image sequence are recognized and labeled.
• the recognition and labeling may be done at the bottom level ofthe hierarchy using Principal Component Analysis and motion vector analysis.
• system 140 has been developed so that it learns single movements and recognizes both single and continuous (concatenated) periodic bimanual movements. As mentioned earlier, recognizing continuous movements may be particularly useful in interacting with a virtual environment through virtual reality and immersive technologies.
• Recognition of hand gestures may be more realistic when both hands are tracked and any overlapping is taken into account.
• the gestures of both hands together typically make a single gesture. Movement of one hand in front ofthe other is one source of occlusion in bimanual movements. Also, for the bimanual movements where there is no occlusion in the essence ofthe movement, changing the view direction ofthe camera can cause one hand to be occluded by the other occasionally.
• By using pixel grey-level detection hands from a dark background may be extracted. In an extracted image, only the pixels with a non-zero value can belong to the hands.
• the Grassfire algorithm may be used in order to extract the hands.
• Grassfire may be described as a region-labelling or blob-analysis algorithm, and the Grassfire algorithm may scan an image from left to right, top to bottom to find the pixels of connected regions with values belonging to the range ofthe hands' grey-level. For the first pixel found in that range the algorithm turns around the pixel to find other pixels. The algorithm attempts to find all the connected regions and label them. In order to track hands, we detect occlusion. Two types of occlusion are considered here. First, the case where one hand occludes the other, which we call hand-hand occlusion. Second, the case in which something else occludes a hand or the hand hides behind another object, e.g., the body, partially or completely.
• another object e.g., the body
• a rectangle 210, 220 is constructed around each hand in an image.
• the sides of a rectangle represent the top, bottom, left, and right edges of the corresponding hand's blob. Therefore, by moving a hand its rectangle moves in the same way.
• By tracking these rectangles we detect the start and end points of a hand- hand occlusion.
• To detect the beginning point we look at the movement ofthe rectangles. If at some stage there is any intersection between the rectangles it could be recognized as occlusion.
• FIG. 2 a rectangle 210, 220 is constructed around each hand in an image.
• the sides of a rectangle represent the top, bottom, left, and right edges of the corresponding hand's blob. Therefore, by moving a hand its rectangle moves in the same way.
• By tracking these rectangles we detect the start and end points of a hand- hand occlusion.
• To detect the beginning point we look at the movement ofthe rectangles. If at some stage there is any intersection between the rectangles it could be recognized as o
• x k the state vector of process at time t k k : a matrix relating x k to x k+l v? k : a white noise sequence with known covariance structure
• z k measurement vector at time t k ⁇ .
• k matrix giving the noiseless connection between the measurement and the state vector at time t k
• ⁇ k measurement error - assumed to be a white noise sequence with known covariance structure.
• rectangle 220 includes two vertical sides and , and two horizontal sides and .
• rectangle 210 includes two vertical sides and two horizontal sides and .
• the movement of a rectangle can be modelled by the movement of its sides (see Figure 5). Therefore, Equation 3 is expanded to,
• x k , x 2 ' k , y k and y 2 ' k are the sides ofthe rectangle i at time k, that is, x[ k , x 2 ' k , y ⁇ >k and y 2 ' k describe the positions ofthe sides ofthe rectangle i at time k.
• JC W is position
• the first and second derivatives of x (l) are the velocity and acceleration respectively.
• Equation 4 is expanded to Equation 7 for z-1, 2.
• x[ , x 2 , y[ , y are assumed to have continuous first and second order derivatives denoted by one-dot and double-dot variables, and h>0 is the sampling time.
• Equation 11 predicts the next state of vector x one step in advance. In other words, equation 11 predicts the position ofthe rectangle i one step in advance. The prediction can also be performed for more than one step by increasing the power of ⁇ .
• the occlusion alarm is set 4. In the next captured image if only one hand is detected by Grassfire and the occlusion alarm is already set the hand-hand occlusion is assumed to have happened. Otherwise, if we see one hand in the image and the occlusion alarm is not set, the other type of occlusion (e.g., occlusion by apart of body or leaving the scene) is assumed to have happened. One or more variables may be set to indicate that occlusion of a particular type has been detected 5. Image capturing is continued
• the first shape found in an image is labelled as the first hand.
• a window at time "t” shows a hand 720 labeled "1" because the search finds hand 720 first, and a hand 730 labeled "2" because the search finds hand 7305 second.
• a window 740 shows that at time "t+1," hand 720 has moved down slightly, and hand 730 has moved up slightly, such that the left to right, top to bottom search finds hand 730 first and hand 720 second — as indicated by labeling hand 730 with "1" and labeling hand 720 with "2.” Such re-labeling of hands 720 and 730 may cause confusion, but may be avoided if hands 720 and 730 are tracked.
• Another implementation uses the centroid ofthe hands to track them in a sequence of images. The centroid-based algorithm finds the centroids ofthe hands and compares them in two consecutive frames. By using this technique we are able to track the hands correctly even when something else occludes them.
• the hands For example, if one of the hands is occluded or get totally hidden by the body for some moments and then5 reappears, it can be tracked correctly by keeping records of its last position before occlusion and the position of the other hand. This is expected because when a hand moves behind another object like the body or moves out of the image frame it most probably appears in an area close to the last position before the occlusion. We also have the other hand tracked over the occlusion period. Therefore, if at some point0 there is only one hand in the image the algorithm may keep tracking the hands properly without any confusion. Other implementations may track the hands using an indicator other than the centroid. In a bimanual movement, when one hand, completely or partially, covers the other hand the hand extraction algorithm detects one big blob in the images.
• a tracking system recognizes these classes and identifies the hands correctly at the end of occlusion.
• clapping can be represented by model g, tying a knot by model j, etc.
• model g tying a knot by model j, etc.
• Temporal coordination implies that the hands' velocities are synchronized in bimanual movements. Also, the hands' pauses happen simultaneously. We may exploit the hands' temporal coordination to track the hands in the presence of occlusion.
• a rectangle is constructed around each hand.
• a rectangle 910 around the big blob is formed.
• rectangle 910 the occlusion-rectangle.
• V c x c : velocity of side c
• the subscript k indicates the discrete time index, and the defined terms are referred to as "velocities.”
• the hands In the movements where the hands either collide or pause (for example, classes 2 and 3), the hands return to the same sides that the hands were on prior to the occlusion period. In these movements the parallel sides ofthe rectangle in either horizontal or vertical directions pause when the hands pause or collide. For example, in the models of e,/and / the hands horizontally pause and return to their previous sides. In the models g andy they pause and return in both horizontal and vertical directions. The horizontal pauses ofthe hands are captured by the pauses ofthe vertical sides ofthe occlusion-rectangle, and the vertical pauses ofthe hands are captured by the pauses ofthe horizontal sides.
• the pauses ofthe parallel sides are typically simultaneous.
• the parallel sides associated with the horizontal and vertical movements of hands typically pause simultaneously.
• the horizontal sides ofthe occlusion-rectangle typically pause simultaneously when the hands pause or collide vertically during occlusion.
• the velocities ofthe horizontal sides ofthe occlusion-rectangle reach zero. This is captured by v v>k in the hand-pause model.
• a small threshold ⁇ > 0 can provide a safe margin because we are working in discrete time and our images are captured at discrete points in time.
• v v> ⁇ or V h>k falls below the threshold we conclude that the hands have paused vertically or horizontally. By detecting the pauses in the horizontal or vertical direction we may conclude that the hands have paused or collided and returned to the same sides prior to occlusion in that direction.
• a window 1010 shows two hands 1020 and 1030 approaching each other, resulting in vertical sides "c” and “d” approaching each other.
• a window 1040 shows, at a point in time later than window 1010, hands 1020 and 1030 pushing past each other such that vertical sides "c” and “d” are pushing away from each other. Therefore, the sign ofthe velocities are changed without passing through zero. If no hand pause is detected we conclude that the hands have passed each other.
• the hand shapes may change during an occlusion period.
• the fingers may also move simultaneously so that the shape ofthe hand changes.
• the movement of fingers may be considered in an attempt to detect simultaneous pauses ofthe hands.
• Research shows that fingers and hand are coordinated too in the movement of one hand. In other words, the hand and fingers are temporally synchronized.
• Our experiment shows that the velocity ofthe hand and the velocity of the fingers are highly synchronized with almost no phase difference. Therefore, the pauses ofthe hand and the pauses ofthe fingers that change the hand shape may be expected to happen simultaneously.
• the hand-finger coordination typically guarantees that the velocities ofthe parallel sides ofthe rectangle are synchronized and the pauses happen simultaneously, regardless of whether finger movement causes the hands to change shape. This phenomenon typically makes the algorithm independent of the changing hand shape.
• an unwanted pause may be detected in the vertical or horizontal directions because the velocity ofthe static direction (vertical or horizontal) will be small according to equation 12. For example, when the hands move only horizontally (see FIG. 8(d)) a vertical pause may be detected because vertically the hands do not have much movement and the speed ofthe vertical sides may reach zero.
• N is the number of images (frames) during the occlusion period
• i andj are the frame indices
• v k are the velocities of sides a , b, c, and d at the k frame during hand-hand occlusion.
• the tracking is performed based on the pauses ofthe other sides ofthe occlusion-rectangle. For example, if a small s v is observed we base the tracking on the pauses ofthe other sides, c and d.
• a small standard deviation in the velocity-synchronization model means that a pair of parallel sides ofthe rectangle has been positively synchronized with quite similar velocities during occlusion.
• the above algorithm tracks the hands during a hand-hand occlusion and makes a decision on the positions ofthe hands at the end of occlusion with respect to their positions prior to occlusion.
• the above algorithm 2 may be modified in various ways to provide information on the position ofthe hands after occlusion.
• the form of algorithm 2 presented above typically provides enough information to distinguish the left and right hands after occlusion.
• Implementations of algorithm 2, and other algorithms may provide increased robustness by verifying that (1) the vertical sides are negatively synchronized in step 1, and/or (2) the horizontal sides are negatively synchronized in step 2.
• Another implementation uses a tracking algorithm having a different hand-pause and hand-pass detection methodology.
• the number of images should ideally be large enough so that the velocities converge to zero in the cases of hand collisions and pauses.
• the algorithm should have enough time and images so that the rectangle's sides' velocities reach zero in the cases that a collision or pause occurs.
• the proposed Kalman filter is based on the Kinematics equations of motion. Therefore, in a fast movement (with an insufficient number of images), the sides ofthe occlusion-rectangle have the potential to move further rather than to stop quickly. That is, if the samples are too far apart, the velocities below the threshold may be missed.
• the algorithm may not detect collisions and pauses accurately. Also, in some applications where the visual system moves (e.g., active vision) the velocities may not exactly reach zero. Therefore, we develop a technique to make the algorithm independent ofthe actual velocities, and investigate the speed changes of the occlusion-rectangle's sides.
• this function results in a velocity equal to the sum ofthe individual velocities.
• An important feature of this function is that it makes the algorithm independent ofthe actual velocities. Therefore, in some applications (e.g., active vision) the effect of a constant value added to the both velocities is eliminated.
• FIG. 12(a) shows distributions 1205-1240 in the movements where a pause is detected.
• FIG. 12(b) shows distributions 1245-1280 for the movements where the hands pass each other.
• each ellipse 1205- 1280 represents a 2-dimensional gaussian distribution.
• a decision on whether the hands have passed each other or paused and returned is made based on the probabilities that Function 14 for a given movement matches each ofthe two patterns in FIGS. 12(a) and (b).
• the probabilities are calculated using the following equation,
• v 0 ⁇ v 0 2 ,... ⁇
• v 0 stands for the set of observed velocities over a given occlusion period calculated by Function 14
• v 0 J v(j) is the observed velocity at time j during occlusion
• H is the k th gaussian distribution in the pattern H
• Hf) is calculated using the multidimensional gaussian probability density function
• ⁇ k l stands for the standard deviation of distribution H on the / th principal axis ofthe /c th distribution
• ⁇ k l is the mean ofthe distribution on the Z th principal axis ofthe k th distribution
• VQ Vector Quantization
• the beginning and the end ofthe occlusion period is detected 1. If the horizontal sides ofthe rectangle are positively synchronized
• each hand is tracked and separately projected into a blank sequence of images. For example, two hands 1310 and 1320 on an image 1330 are separately projected onto individual images 1340 and 1350, respectively.
• the direction of movement of each hand is recorded.
• FIG. 14 to record direction of movement, we divide a 2-dimensional space of an image frame 1410 into 8 equal regions 1420-1455. We call the divided frame 1410 the regional-map. The index (1- 8) of each region represents the direction of movement in that region.
• An index of zero (not shown in frame 1410) represents a stationary hand.
• a vector representing the movement is extracted for every single frame. This vector represents the movement from the last image to the present one.
• a hand 1510 is shown at time “t” in frame 1520 and at time “t+1" in frame 1530.
• the movement of hand 1510 from time “t” to time “t+1” is represented by a vector 1540 in window 1550.
• the angle of the vector with respect to the horizontal axis determines the region in the regional-map in which the vector maps onto.
• the region index is recorded for the movement at each time t.
• Implementations may consider the speed ofthe gesture, for example, by determining and analyzing an appropriate magnitude for vector 1540.
• a bimanual movement is constituted from two groups of parts, the occlusion parts in which one hand is occluded, and the other parts.
• the parts in which the hands are recognizable separately are called non-occlusion parts. Since a bimanual movement can be a periodic movement like clapping we separate different parts, which we call segments. Four segments are obtained as following,
• FIG. 16 an example of a segmented bimanual movement is illustrated in window 1610 over the time axis.
• the movement starts and ends in non-occlusion segments
• other implementations extend the algorithm to other cases.
• the process is the same with only one segment (a beginning segment) for the whole gesture.
• FIG. 16 there are 3 occlusion segments labelled "B,” and 2 middle segments labelled “C,” as well as a beginning segment labelled "A” and an ending segment labelled “D".
• the implementation is able to deal with multiple occlusion and middle segments as well as the beginning and the ending segments in order to understand the whole bimanual movement.
• the movement of a hand within a segment is treated as a single movement appearing in the sequence of images ofthe segment.
• These movements are modelled and recognized by Hidden Markov Models, although other models may be used. Therefore, for a bimanual movement we get a set of recognized movements of each ofthe two hands, and the recognized movements ofthe occlusion parts. This information is combined to recognize the bimanual movement.
• One implementation uses a Bayesian network in which the whole gesture is divided into the movements ofthe two hands. Referring to FIG. 17, the movement of each hand is also divided into the four segments through the evidence nodes of BEG, MID, OCC, and END.
• the occluded part of a gesture is a common part for both hands.
• a tree 1700 includes a top node "Bimanual Gesture" 1705, that includes a left-hand gesture node 1710 and a right-hand gesture node 1715.
• Left-hand gesture node 1710 and right-hand gesture node 1715 include BEG evidence nodes 1720 and 1750, respectively, MID evidence nodes 1725 and 1745, respectively, and END evidence nodes 1730 and 1740, respectively, and share a common OCC node 1735.
• each node in this tree represents a multi-valued variable.
• every node is a vector with length g, as shown with vectors 1720a, 1735a, and 1750a.
• the three top nodes of Bimanual Gesture, Left Hand Gesture, and Right Hand Gesture are non-evidence nodes updated by the messages communicated by the evidence nodes.
• the evidence nodes are fed by the Hidden Markov Models of different segments separately, as shown with models 1755a, 1755g, 1760a, 1760g, 1765a, and 1765g.
• the causal tree 1700 can be abstracted to tree 1800 that includes non- occlusion segment nodes (NS nodes) 1810 and 1820, and occlusion segment node (OS node) 1830.
• Node 1810 is associated with vector 1810a, and with models 1840a through 1840g.
• node 1830 is associated with vector 1830a and with models 1850a through 1850g.
• the NS nodes 1810 and 1820 represent the evidences of the beginning, middle, and ending segments at different times for each hand.
• an eigenspace for each hand.
• An eigenspace is made by using a set of training images of a hand in a given segment and Principal Component Analysis.
• the covariance matrix ofthe set of images is made and the eigenvalues and eigenvectors ofthe covariance matrix are calculated.
• the set of eigenvectors associated with the largest eigenvalues are chosen to form the eigenspace.
• the projection ofthe set of training images into the eigenspace is the Principal Components.
• a separate eigenspace is created, also, for the occlusion segments. These eigenspaces are made by the movements in the training set. By projecting all the images of one hand into its own eigenspace a cloud of points is created. Another dimension is also added to the subspaces which is the motion vector extracted using the regional-map.
• a set of codewords is extracted for each eigenspace using Vector Quantization.
• the set of extracted codewords in each eigenspace is used for both training and recognition.
• By projecting a segment of a gesture into the co ⁇ esponding eigenspace a sequence of codewords is extracted.
• a 2-state left-to- right Hidden Markov Model 1900 is assigned to each hand in a non-occlusion segment. Due to the fact that a partial movement of a hand in a segment is normally a short movement, a 2-state HMM is typically suitable to capture the partial movement. Every segment of a gesture has its individual HMMs. Thus, for every gesture in the vocabulary of bimanual movements seven HMMs are assigned, two for the beginning segments for the two hands, one for the occlusion segments, two for the middle segments, and two for the ending segments. By using the extracted sequence of codewords the HMM of each hand in a segment is trained.
• the HMMs ofthe occlusion segments are trained by the extracted sequence of codewords ofthe projected images into the corresponding eigenspace. For example, for a vocabulary of 10 bimanual movements 70 HMMs are created and trained. In the recognition phase the same procedure is performed. A given gesture is segmented. Images of each segment are projected into the corresponding eigenspace and the sequences of codewords are extracted. By employing the trained HMMs, the partial gesture of each hand presented in a segment is recognized. However, we use the HMMs to calculate the likelihoods that a given partial gesture is each ofthe co ⁇ esponding partial gestures in the vocabulary.
• a normalized vector ofthe likelihoods for a given partial gesture in a segment is passed to one ofthe evidence nodes in the Bayesian network of FIG. 18.
• the second scalar in the NS vector 1810a ofthe left hand is the likelihood that: • In a beginning segment: the given partial gesture is the beginning segment of gesture number 2 in the vocabulary, calculated by the HMM ofthe beginning segment ofthe left hand of gesture number 2
• the given partial gesture is the middle segment of gesture number 2 in the vocabulary, calculated by the HMM ofthe middle segment of the left hand of gesture number 2 and so on.
• the occlusion vector which is fed by the likelihoods of the HMMs ofthe occlusion segments, is a shared message communicated to the LH and RH nodes and, ultimately, the BG node, as evidences for the two hands.
• the LH, RH, and BG nodes calculate their beliefs, that is, their vectors of the likelihoods of the possible gestures, using, for example, the well-known belief propagation algorithm.
• three sets of training images are extracted from videos of gestures.
• Each image may contain, for example, 1024 pixels.
• eigenspaces of lower dimensionality are determined for the training data.
• the training data is projected into the eigenspace to produce reduced dimensionality training data.
• codewords are determined for the eigenspaces.
• HMMs are then developed using the sequences of codewords corresponding to appropriate segments ofthe training data for given gestures.
• Images of a given gesture are then projected into the appropriate eigenspace and the closest codewords are determined, producing a sequence of codewords for a given set of images corresponding to a segment of a gesture.
• the sequence of codewords is then fed into the appropriate HMMs (segment and gesture specific) to produce likelihoods that the segment belongs to each ofthe trained gestures. These likelihoods are then combined using, for example, the belief propagation algorithm.
• the network looks loopy (containing a loop).
• the nodes of BG, LH, OS, and RH form a loop. Therefore, the network does not seem to be singly connected and a message may circulate indefinitely.
• the node OS is an evidence node. Referring to the belief propagation rules of Bayesian networks the evidence nodes do not receive messages and they always transmit the same vector. Therefore, the NS and OS nodes are not updated by the messages ofthe LH and RH nodes. In fact, the LH and RH nodes do not send messages to the evidence nodes. Therefore, although the network looks like a loopy network, the occlusion node of OS cuts the loop off and no message can circulate in the loop. This enables us to use the belief propagation rules of singly connected networks in this network.
• a bimanual gesture is segmented by a tracking algorithm
• sequence of codewords is extracted for each hand using, for example, the Principal Components and the motion vectors
• the vectors of likelihoods are passed into the corresponding NS nodes while the vector of occlusion node is set to a vector of all Is.
• the image sequence ofthe segment is projected into the eigenspace of the occlusion segments 3.2.
• a sequence of codewords is extracted using the Principal Components and the motion vectors
• the vector of likelihoods is calculated and normalized by using the corresponding HMMs 3.4.
• the vector is passed to the OS node
• the nodes ' beliefs are updated by the belief propagation algorithm
• the vectors of likelihoods are calculated and normalized by using the corresponding HMMs 4.4.
• the vectors of likelihoods are passed to the corresponding NS nodes
• the nodes ' belief are updated by the belief propagation algorithm
• Tlie nodes ' belief are updated by the belief propagation algorithm
• Hie sequence of codewords are extracted using the Principal Components and the motion vectors 7.3.
• the vectors of likelihoods are calculated and normalized by using the
• Tfie vectors are passed to the corresponding NS nodes
• the nodes ' beliefs are updated by the belief propagation algorithm
• bimanual movements are periodic in essence. Clapping and drumming are some examples. In the environments where the bimanual movements are used as a communication method, e.g., Virtual Reality, concatenated periodic movements should be recognized.
• the HMM of every segment of a gesture was trained by the 5 samples in the training set. Three bimanual gestures were selected to create concatenated periodic bimanual movements. From the 15 movements, first gesture number 3 was repeated 5 times. It was followed by gesture number 2 repeated 30 times, and followed by gesture number 5 repeated 41 times. Therefore, the first gesture is divided into 11 segments, including a beginning segment, and 5 occluded segments separated by 4 middle segments, and an end segment. The second gesture is divided into 61 segments, including a beginning segment, 30 occluded segments, 29 middle segments, and an end segment. The third gesture is divided into 83 segments, including a beginning segment, 41 occluded segments, 40 middle segments, and an end segment. Given the fact that the first segment in the graph of local beliefs represents the belief of th initialization, the first gesture transition should appear in the 13 segment (the beginning segment associated with the second gesture) and the second transition in the 74 th segment(the beginning segment associated with the third gesture).
• a plot 2010 shows multiple graphs (15 graphs) including a first graph 2020 for the first gesture, rising at approximately segment 2 to a belief of approximately 1, and falling at approximately segment 12 to a belief of approximately 0.
• Plot 2010 also shows a second graph 2030 for the second gesture, rising at approximately segment 13 to a belief of approximately 1, and falling at approximately segment 73 to a belief of approximately 0.
• Plot 2010 also shows a third graph 2040 for the third gesture, rising at approximately segment 74 to a belief of approximately 1, and stopping at approximately segment 156.
• Plot 2010 shows a fourth graph 2050 having a positive belief around, for example, segment 40.
• Second graph 2030 also includes several dips, particularly around segment 40.
• the belief is higher for the gesture associated with fourth graph 2050 than for the second gesture.
• the gestures are correctly recognized most ofthe time.
• the gesture transitions are detected properly.
• the belief is not very stable and it varies such that at some points it falls below the graph of other gestures. This happens when the partial gestures of one or two hands are recognized incorrectly. Although the confusion can be treated as temporary spikes, an algorithm may determine that the gesture has changed at some points.
• Each ofthe graphs 2020, 2030, 2040, and 2050 is isolated in one of FIGS.
• An implementation avoids these confusing spikes by changing the belief propagation algorithm. Specifically, the previous belief of the root node is given greater weight so that temporary confusing evidence does not change the belief easily. To give greater weight to a previous belief, we add memory to the root node of the network. This is done, for example, by treating the current belief of the root node as the prior probability ofthe node in the next step. When a hypothesis (that one of the gestures in the vocabulary is the correct gesture) is strengthened multiple times by the messages received from the HMMs, many strong pieces of evidence are needed to change this belief.
• FIG. 21 shows a first graph 2120, a second graph 2130, and a third graph 2140, corresponding to the first, second, and third gestures, respectively.
• an imaging device 2240 e.g., a CCD camera captures sequences of images of a person doing a bimanual movement. The images are transferred to a computing device 2210 running the algorithms described.
• the storage device 2230 such as, for example, a database, contains the training information required by the tracking and recognition algorithms. Storage device 2230 may also store the code for the algorithms. During a training phase the training information of the tracking algorithm including the threshold values and distributions are stored in the storage device 2230.
• a process 2300 may be used to recognize bimanual gestures, and includes many Operations discussed in this disclosure.
• Process 2300 includes receiving or otherwise accessing a series of images of a bimanual gesture (2310). Left and right hands are extracted and tracked from the received images (2320) and a hand-hand occlusion is predicted (2330).
• the hand-hand occlusion is detected (2340) and a single blob including both hands is extracted and tracked from the images in which the occlusion exists (2345).
• the synchronization ofthe left and right hands during the occlusion is determined (2350), the behavior ofthe hands (whether they passed each other or they paused/collided and returned) is recognized (2355), and the left and right hands are identified after the occlusion ends (2360).
• the left and right hands are extracted and tracked post-occlusion (2365).
• the movements in each ofthe segments pre-occlusion, occlusion, and post-occlusion
• the overall gesture is recognized (2370).
• Determining the synchronization ofthe left and right hands (2350) may generally involve determining any relationship between the two hands.
• the relationship may be, for example, a relationship between component-velocities of parallel sides of a rectangle su ⁇ ounding a blob, as described earlier. In other implementations, however, the relationship relates to other characteristics ofthe hands, or the single blob.
• One variation of process 2300 may be performed by a plug-in to a bimanual gesture recognition engine.
• the plug-in may perform some variation of tracking a blob (2345), determining a type of synchronization (2350), and determining whether the two hands change their direction of travel during the occlusion period.
• a plug-in may be used with a gesture recognition engine that is unable to deal with hand-hand occlusion.
• the gesture recognition engine may track the left and right hands until a hand-hand occlusion occurs, then call the plug-in.
• the plug-in may track the blob, determine if the two hands changed direction during the occlusion, and then transfer control ofthe recognition process back to the gesture recognition engine. In transferring control back to the gesture recognition engine, the plug-in may tell the gesture recognition engine whether the two hands changed direction during the occlusion.
• the gesture recognition engine can reacquire the left and right hands and continue tracking the two hands.
• Implementations may attempt to discern whether two occluded hands have passed each other, have collided with each other, or have merely paused.
• the result of a pause may typically be the same as the result of a collision; that the two hands return to the directions from which they came.
• the velocity profile of a "pause” may be similar to the velocity profile of a "collision,” and any differences may be insignificant given expected noise. However, implementations may attempt to separately detect a "collision” and a "pause.”
• the directions referred to with respect to various implementations may refer, for example, to the direction ofthe velocity vector or the direction of a component of the velocity vector.
• the direction of a velocity vector may be described as being, for example, a left direction, a right direction, a top direction, a bottom direction, and a diagonal direction.
• Components of a velocity vector may include, for example, a horizontal component and a vertical component.
• Implementations may be applied to tracking bimanual gestures performed by a single person using the person's left and right hands. Other implementations may be applied to gestures being performed by, for example, two people each using a single hand, one or more robots using one or more gesturing devices, or combinations of people and robots or machines, particularly if a coordination similar to the bimanual coordination exists between the hands.
• Implementations may include, for example, a process, a device, or a device for carrying out a process.
• implementations may include one or more devices configured to perform one or more processes.
• a device may include, for example, discrete or integrated hardware, firmware, and software.
• a device may include, for example, computing device 2210 or another computing or processing device, particularly if programmed to perform one or more described processes or variations thereof.
• Such computing or processing devices may include, for example, a processor, an integrated circuit, a programmable logic device, a personal computer, a personal digital assistant, a game device, a cell phone, a calculator, and a device containing a software application..
• Implementations also may be embodied in a device that includes one or more computer readable media having instructions for carrying out one or more processes.
• the computer readable media may include, for example, storage device 2230, memory 2220, and formatted electromagnetic waves encoding or transmitting instructions.
• Computer readable media also may include, for example, a variety of non- volatile or volatile memory structures, such as, for example, a hard disk, a flash memory, a random access memory, a read-only memory, and a compact diskette. Instructions may be, for example, in hardware, firmware, software, and in an electromagnetic wave.
• computing device 2210 may represent an implementation of a computing device programmed to perform a described implementation
• storage device 2230 may represent a computer readable medium storing instructions for carrying out a described implementation.

Landscapes

• Engineering & Computer Science (AREA)
• Human Computer Interaction (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Health & Medical Sciences (AREA)
• Psychiatry (AREA)
• Social Psychology (AREA)
• Health & Medical Sciences (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Theoretical Computer Science (AREA)
• Image Analysis (AREA)
• User Interface Of Digital Computer (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=35197616

Family Applications (1)

Country Status (5)

Families Citing this family (442)

Family Cites Families (44)

• 2005
  • 2005-04-15 CN CNB200580019474XA patent/CN100573548C/zh not_active Expired - Fee Related
  • 2005-04-15 US US11/106,729 patent/US7379563B2/en active Active
  • 2005-04-15 JP JP2007508608A patent/JP4708422B2/ja not_active Expired - Fee Related
  • 2005-04-15 WO PCT/US2005/013033 patent/WO2005104010A2/en active Application Filing
  • 2005-04-15 EP EP05733722A patent/EP1743277A4/de not_active Withdrawn
• 2007
  • 2007-10-31 US US11/932,766 patent/US8259996B2/en active Active
• 2012
  • 2012-08-01 US US13/564,492 patent/US8515132B2/en active Active

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events