Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
  • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
  • G06F18/2137—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
  • G06F18/21375—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps involving differential geometry, e.g. embedding of pattern manifold
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/20—Movements or behaviour, e.g. gesture recognition
  • G06V40/23—Recognition of whole body movements, e.g. for sport training

Definitions

• the invention relates to tracking 3D human motion. More particularly, the invention relates to a system and method for tracking 3D articulated human motion in a dimensionality- reduced space given monocular video sequences.
• Tracking articulated human motion is of interest in numerous applications including video surveillance, gesture analysis, human computer interface, and computer animation.
• 3D motion tracking is important in analyzing and solving problems relating to the movement of human joints.
• traditional 3D motion tracking subjects wear suits with special markers and perform motions recorded by complex 3D capture systems.
• motion capture systems are expensive due to the required special equipment and significant studio time.
• conventional 3D motion capture systems require considerable post-processing work which adds to the time and cost associated with traditional 3D tracking methods.
• the present invention provides a method for efficiently and accurately tracking 3D human motion from a 2D video sequence, even when self-occlusion, motion blur and large limb movements occur.
• 3D motion capture data is acquired using conventional techniques.
• a prediction model is then generated based on the learned motions.
• 3D tracking is performed without requiring any special equipment, clothing, or markers. Instead, 3D motion can be tracked from a monocular video sequence based on the prediction model generated in the offline stage.
• the motion is tracked in a dimensionality-reduced state. Human motion is limited by many physical constraints resulting from the limited angles and positions of joints.
• a low-dimensional latent model can be derived from the high-dimensional motion capture data.
• a probabilistic algorithm performs non-linear dimensionality reduction to reduce the size of the original pose state space.
• a mixture of factor analyzers is learned.
• Each factor analyzer can be thought of as a local dimensionality reducer that locally approximates the pose state.
• Global coordination between local factor analyzers is achieved by learning a set of linear mixture functions that enforces agreement between local factor analyzers.
• the formulation allows easy bidirectional mapping between the original body pose space and the low-dimensional space.
• the projected data forms clusters within the globally coordinated low-dimensional space. This makes it possible to derive a multiple hypothesis tracking algorithm based on the distribution modes.
• FIG. 1 is an example computer system for executing the methods of the present invention.
• FIG. 2 is a block diagram illustrating one embodiment of the present invention.
• FIG. 3 a is an offline learning algorithm for generating a prediction model used in 3D motion tracking.
• FIG. 3b is an online tracking algorithm for tracking 3D human motion given a monocular video sequence and the prediction model generated in the offline learning stage.
• FIG. 4 is a dimensionality reduction algorithm according to one embodiment of the present invention.
• FIG. 5 is a block diagram illustrating a learning process for a dimensionality reduction model.
• FIG. 6 illustrates clustering in a low dimensional space as a result of the dimensionality reduction algorithm.
• FIG. 7 is a flow diagram illustrating the computation performed during online tracking according to one embodiment of the present invention.
• Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
• the present invention also relates to an apparatus for performing the operations herein.
• This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.
• a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
• the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
• the algorithms and displays presented herein are not inherently related to any particular computer or other apparatus.
• Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps.
• the required structure for a variety of these systems will appear from the description below, hi addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references below to specific languages are provided for disclosure of enablement and best mode of the present invention.
• FIG. 1 is a computer system according to one embodiment of the present invention.
• the computer system 100 comprises an input device 102, a memory 104, a processor 106, an output device 108, and an image processor 110.
• the input device 102 is coupled to a network 120, a database 130, and a video capture unit 140.
• the output device 108 is coupled to a database 150, a network 160, and a display 170.
• the input device is connected to only one or two of a network 120, a database 130, and a video capture unit 140.
• the input device may be connected to any device configured to input data to the computer system.
• FIG. 2 is a block diagram illustrating one embodiment of the present invention.
• the embodiment comprises an offline learning algorithm 210 and an online tracking algorithm 220.
• the offline learning algorithm 210 uses 3D motion capture data 212 to produce a prediction model 215 utilized by the online tracking algorithm 220.
• the online tracking algorithm 220 uses a 2D image sequence 222 and the prediction model 215 to generate the 3D tracking data 224.
• 3D motion capture data 212 may be acquired by a variety of conventional techniques during the offline stage.
• a subject wears a special suit with trackable markers and performs motions captured by video cameras.
• the subject may perform a series of different motions which are captured and processed.
• 3D motion capture data may be acquired from multiple subjects performing similar sets of motions. This provides statistical data from which the prediction model 215 can be derived.
• FIG. 3a summarizes one embodiment of the offline learning algorithm 210.
• a computer system 100 receives 302 3D motion capture data 212.
• the pose state is then extracted 304 from the 3D motion capture data.
• the unfiltered pose state resides in a high dimensional state space and it is desirable to reduce the dimensionality of the state space to decrease memory requirements and increase processing efficiency.
• a dimensionality reduction model is learned 306 to reduce the dimensionality of the pose state from a high dimensional space to a low dimensional space.
• a dynamic model is learned 308.
• the dynamic model if learned, may optimize the prediction model 215 for more efficient tracking.
• the prediction model 215 is formed by generating 310 hypotheses based on the dimensionality reduction model and in some embodiments, the learned dynamic model.
• the motion capture data 212 may be received from a video capture unit 140 interfaced to an input device 102 of a computer system 100.
• the 3D motion capture data 212 may be received by the input device 102 from a database 130 or through a network 120.
• the 3D motion capture data 212 is processed by the computer system 100 to extract 304 the pose states.
• the pose states comprise data which completely represent the positions of the subject throughout a motion.
• the extracted pose state comprises a vector of joint angles.
• the pose state may comprise any set of data that completely describes the pose. This may include angles, positions, velocities, or accelerations of joints, limbs, or other body parts or points of interest.
• the 3D motion capture data 212 may be processed by a standard computer processor 106 or by a specialized image processor 110, for example.
• the pose state may be stored in memory 104 or outputted by an output device 108.
• the output device 108 interfaces to an external database 150 for storage or sends the data to a network 160 or a display 170.
• a dimensionality reduction model is learned 306 based on the extracted pose states.
• the dimensionality reduction model takes advantage of the physical constraints of human motion to generate a low-dimensional latent model from high-dimensional motion capture data.
• Many algorithms for dimensionality reduction are known including Principal Component Analysis (PCA), Locally Linear Embedding (LLE) described in Ro Stamm, et al., Nonlinear Dimensionality Reduction by Locally Linear Embedding, Science 290, 2000, 2323- 2326; Isomap described in Tenenbaum, et al., A Global Geometric Framework for Nonlinear Dimensionality Reduction, Science 290, 2000, 2319-2323; and Laplacian Eigenmaps described in Belkin, et al., Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering, Advances in Neural Information Processing Systems (NIPS), 2001, 585-591 all of which are incorporated by reference herein in their entirety.
• PCA Principal Component Analysis
• LLE Locally
• regression methods are used to learn the mapping back from the low dimensional space to the high dimensional space.
• an invertible dimensionality reduction method is used. Inverse mapping of particles back to the original human pose space allows for re- weighting of the particles given the image measurements during online tracking without using a regression method.
• Examples of dimensionality reduction techniques that provide inverse mapping include Charting described in Brand, Charting a Manifold, NIPS, 2001, 961-968; Locally Linear Coordination (LLC) described in Teh, et al., Automatic Alignment of Local Representations, NTPS, 2002, 841-848; and Gaussian Process Latent Variable Model (GPLVM) described in Lawrence, Gaussian Process Models for Visualization of High Dimensional Data, NIPS, 2003 all of which are incorporated by reference herein in their entirety.
• LLC Locally Linear Coordination
• GPLVM Gaussian Process Latent Variable Model
• the dimensionality reduction model is based on an LLC algorithm.
• a probabilistic algorithm is employed to perform non-linear dimensionality reduction and clustering concurrently within a global coordinate system.
• the projected data forms clusters within the globally coordinated low-dimensional space.
• a mixture of factor analyzers is learned with each factor analyzer acting as a local dimensionality reducer.
• a GPLVM algorithm or other dimensionality reduction algorithm is used.
• a model which performs a global coordination of local coordinate systems in a mixture of factor analyzers (MFA) is known is the art, for example, in Ro Stamm, et al. Global Coordination of Local Linear Models, NIPS, 2001, 889-896 which is incorporated by reference herein in its entirety.
• MFA factor analyzer
• Each factor analyzer (FA) can be regarded as a local dimensionality reducer.
• Both the high-dimensional data y and its global coordinate g are generated from the same set of latent variables s and z s , where each discrete hidden variable s refers to the s-th FA and each continuous hidden variable z s represents the low-dimensional local coordinates in the s-th FA ⁇
• data generated from s-th FA with prior probability P(s) and the distribution of z s are Gaussian: Zs l tS ⁇ * ⁇ ⁇ - ⁇ where I is the identity matrix.
• y and the global coordinate g are generated by the following linear equations
• T LS and TQ S are the transformation matrices
• ⁇ s and ⁇ s are uniform translations between the coordinate systems
• u " " " " ⁇ ⁇ ° : ⁇ u - > and v» ⁇ ⁇ ' ' ' 0> ' 4 ⁇ are independent zero mean Gaussian noise terms.
• the following probability distributions can be derived from Eq. 1 : 7i, ⁇ r[T L ⁇ z ⁇ + ⁇ ⁇ . A 11 J) g
• s, y n ) are Gaussian distributions
• y n ,s) also follows a Gaussian distribution. Since p(s
• an efficient two stage learning algorithm leverages on the mixture of local models to collapse large groups of points together as described by Teh, et al. referenced above. This algorithm works with the groups rather than individual data points in the global coordination.
• a graphical representation of the two stage dimensionality reduction model is depicted in FIG. 4.
• a data point in the original space, y n 402 is characterized by S factor analyzers.
• First the MFA between y 402 and (s, z s ) 406 is learned using the method set forth in Ghahramani, et al., The EM Algorithm for Mixtures of Factor Analyzers, Technical Report CRG-TR-96-1, University of Toronto, 1996 which is incorporated by reference herein in its entirety.
• z ns 406 is the expected local coordinate in the s-th FA for each data point y n .
• r lls 404 denotes the likelihood, p(y n
• the set of Z n 406 acts as a local dimensionality reducer while the set of r n 404 gives the responsibilities of each local dimensionality reducer.
• the weighted combination, U n 408 is formed from r n and z n as
• the alignment parameters L 410 provide the mapping from the weighted combination, U n 408 to the global coordinates, g n , 412 in the global coordinated latent space from Eq. 6.
• G [U 1 , U 2 , . . . , U N ] T .
• G [U 1 , U 2 , . . . , U N ] T .
• G [U 1 , U 2 , . . . , U N ] T .
• G [U 1 , U 2 , . . . , U N ] T .
• G [U 1 , U 2 , . . . , U N
• FIG. 5 represents an embodiment of a method for learning 306 a dimensionality reduction model which computes the alignment parameters, L, and the global coordinates, G.
• Local linear construction weights are first computed 502.
• a mixture of factor analyzers are trained 504 as local dimensionality reducers.
• the local linear construction weights are combined to form 506 the weighted combination matrix.
• Optimal alignment parameters are determined 508 to map the weighted combination matrix to the global coordinate system.
• the global coordinates are determined 510 from the weighted combination matrix and alignment parameters.
• the local linear reconstruction weights are computed 502 using equation 7 and as described below. For each data point y n , its nearest neighbors are denoted as y m (me N n ) and following is minimized:
• Both the cost function (Eq. 8) and the constraints (Eq. 10) are quadratic and the optimal alignment parameters, L, is determined 408 by solving a generalized eigenvalue problem.
• d « D be the dimensionality of the underlying manifold that y is generated from.
• D may typically be around 50 and d may typically have a value around 3. However, these values may vary depending on the specific problem of interest.
• clusters are obtained in the globally coordinated latent space 600 as illustrated in FIG. 6.
• Each cluster is modeled as a Gaussian distribution in the latent space with its own mean vector and covariance matrix.
• Each ellipsoid 602 represents a cluster in the latent space 600, where the mean of the cluster is the centroid 604 and the covariances are the axes of the ellipsoids 602.
• This cluster-based representation leads to a straightforward algorithm for multiple hypothesis tracking.
• a dynamic model is optionally learned 308 for specific motions to be tracked. The dynamic model predicts how individual particles move over time. In one embodiment, a different dynamic model may be learned for each motion.
• the online tracking algorithm 220 tracks a pose state in 3D by utilizing a modified multiple hypothesis tracking algorithm. Examples of such techniques are set forth in Isard, et al, CONDENSATION: Conditional Density Propagation for Visual Tracking, International Journal of Computer Vision (UCV) 29, 1998, 5-28; Cham, et al., A Multiple Hypothesis Approach to Figure Tracking, Proc.
• a multiple hypothesis tracker (MHT) together with the learned LLC model provides the 3D motion tracker.
• MHT multiple hypothesis tracker
• As LLC provides clusters in the latent space as a step in the global coordination, it is natural to make use the centers of the clusters as the initial modes in the MHT ( p(g
• z s , s) follows a Gaussian distribution.
• a simple dynamic model may be applied in the prediction step of the filtering algorithm.
• the modes are passed through a simple constant velocity predictor in the latent space.
• the dynamic model is not used.
• FIG. 3b summarizes one embodiment of the online tracking method 220.
• the pose state at the next time frame is predicted 322 based on the prediction model 215. In one embodiment, this prediction generates several of the most likely pose states based on the prediction model.
• the 2D image corresponding to the predicted time frame is then received 324 from a video sequence.
• the predicted pose state is then updated 326 based on the 2D image information. In one embodiment, this update comprises selecting the pose state of the several predicted possible pose states that best matches the data in the 2D image.
• the time frame advances 328 and the process repeats for each frame of 2D video.
• FIG. 7 summarizes the computations performed in the online tracking stage 220.
• a prior probability density funption is computed 702. This function is based on the prediction model 215 and all past image observations, hi one embodiment, the modes of the prior probability density function are passed through a simple constant velocity predictor to predict 322 the pose state at the next time frame. In equation 11 , the prior probability density function is represented by p(X t
• the likelihood function is computed 704 based on receiving the 2D image from the 2D image sequence 324. In order to compute the likelihood for the current prediction and the input video frame, the silhouette of the current video frame is extracted through background subtraction. The predicted model is then projected onto the image and the chamfer matching cost between the projected model and the image silhouettes is considered to be proportional to the negative log-likelihood.
• the projected model consists of a group of cylinders as described by Sigal, et al., Tracking Loose-limbed People., CVPR, 2004, 421-428.
• the predicted pose state is updated 326.
• the likelihood function is represented by p(z t
• the posterior probability density function is computed 706 through equation 11, where the posterior probability density function is represented by p(X t
• the time frame advances 708 and the calculation is repeated for each time frame of video.
• the MHT algorithm proposed here differs from conventional techniques in a variety of ways.
• the present invention uses the latent space to generate proposals in a principled way. This is in contrast with conventional techniques, where the modes are selected empirically and the distributions are assumed to be piecewise Gaussian.
• the output from the off-line learning algorithm (LLC) forms clusters (each cluster is described by a Gaussian distribution in latent space), the samples generated from the latent space are indeed drawn from a piecewise Gaussian distribution.
• LLC off-line learning algorithm

Landscapes

• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Multimedia (AREA)
• Social Psychology (AREA)
• Human Computer Interaction (AREA)
• Health & Medical Sciences (AREA)
• Psychiatry (AREA)
• General Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Artificial Intelligence (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Bioinformatics & Computational Biology (AREA)
• Evolutionary Biology (AREA)
• Evolutionary Computation (AREA)
• General Engineering & Computer Science (AREA)
• Image Analysis (AREA)

Priority Applications (1)

Applications Claiming Priority (4)

Publications (2)

Family

ID=38007031

Family Applications (1)

Country Status (3)

Cited By (4)

Families Citing this family (291)

Family Cites Families (11)

• 2006
  • 2006-10-26 US US11/553,382 patent/US7450736B2/en active Active
  • 2006-10-27 WO PCT/US2006/042135 patent/WO2007053484A2/en not_active Ceased
  • 2006-10-27 JP JP2008538039A patent/JP4625129B2/ja not_active Expired - Fee Related

Also Published As

Similar Documents

Legal Events