JP2008140101A - Unconstrained and real-time hand tracking device using no marker - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for acquiring a personal hand's motion in real time without constraint and without using a marker. <P>SOLUTION: A hand tracker 90 includes a storage part of a three-dimensional hand model 116; an image processor 104 for extracting a skin color binary map 140 and an edge image 142; particle filter units 108 and 110 each estimating a posture and a position of a hand in a current frame 102 by applying particle filtering to the hand model 116 based on the skin color map 140 and the edge image 142; and a combination and updating modules 112 and 114 for updating the hand model 116 using the estimation result. The particle filter units 108 and 110 include z-buffers and calculate probabilities of the particle configurations by comparing the z-buffer projection images of the particle configurations with the skin color map 140 and the edge image 140. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to human hand tracking systems, and more particularly to an effective, unconstrained real-time, marker-free, human hand tracking system.

  Any robot designed to interact with people in a natural way requires an unconstrained hand tracker. Humanoids' ability to demonstrate using marker-free hand trackers is about interactions with people, such as skills learning, handing over and exchanging things through eye-hand coordination. Imitation and understanding of gestures are possible. A person can also use this system to draw the robot's attention by pointing. The hand tracker system seeks to provide a solution to what is needed in these challenges.

  A hand tracker that uses a marker and a glove has been well studied. These techniques usually give good tracking results. In our application, the robot must be able to track any random hand in its natural environment, so it cannot rely on any provisions such as fixed markers. For this reason, we decided to take a solution that does not use markers.

  Marker-free hand tracking is usually done by interpreting the video input stream. Look for clues from the frames taken. A commonly used clue is the contour of the hand, which can be found by applying an edge filter to the input frame. Another clue is the surface of the visible hand, which can be extracted by filtering the skin color. The hand shape estimated by the tracker is compared with the clue of the input image to obtain a predetermined gesture probability. For comparison with the clue, a posture database or a 3D hand model projection model is used.

  The problem of recognizing a predefined hand posture is usually solved by a system that uses a posture database. An example is appearance-based gesture recognition disclosed in Non-Patent Document 1, which detects a gesture in a sign language database using a hidden Markov model.

  Closer to the object of the present invention is a multi-degree-of-freedom hand tracker recently studied by Errol in Non-Patent Document 2. In this review, recent developments in this area are shown, distinguished by three categories. A single frame estimator, a single hypothesis estimator and a multiple hypothesis estimator.

[Single frame estimator]
The first category is single frame pose estimators that do not rely on prior knowledge of previous frames. One possible solution is a global search on a set of templates labeled with kinematic parameters. In Non-Patent Document 3, these templates are generated using a 3D hand model and stored in a tree structure. To detect the pose, the tree is searched without prior knowledge.

  Since these systems are independent of the previous frame, they do not require repair from the tracking error of the previous frame, resulting in robust behavior. However, because it is necessary to build a template database, the performance of the system representation is limited to a discrete subset of poses, which contradicts the goal of unconstrained tracking of the present invention. Global search is very computationally expensive. Using a continuous joint space for tracking is not practical in a real-time system. By tracking over a large number of frames, only a local search is performed in the shape space, so that a system that uses fewer resources can be realized.

[Single hypothesis estimator]
The second category includes hand trackers that use a single hypothesis obtained from the previous estimation as prior knowledge. The most commonly used strategy to fit a model in a clue extracted from an image is to use standard optimization techniques.

  Non-Patent Document 4 proposes a divide-and-conquer policy. First, the total hand movement is estimated, and then the joint angle is estimated. This procedure is applied repeatedly until convergence. A Kalman filter is also used to solve the single hypothesis tracking problem. In Non-Patent Document 5, a hand is tracked using an unscented Kalman filter (UKF).

  Since these systems use gradient search, they can only reach local minima and there is always a risk of tracking errors due to not reaching global minima.

[Multiple hypothesis estimator]
In contrast to a single hypothesis tracker, the multiple hypothesis tracker attempts to continue multiple pose estimations while tracking the sequence. This increases the chance of reaching the minimum value. This idea is best realized in a framework called Bayesian filter processing that holds the probability distribution of states up to the current frame. One well-known technique for implementing a recursive Bayes filter is a particle filter.

  Particle filtering is also known as the Condensation algorithm introduced by Michael Isard and Andrew Blake in [6]. The condensation algorithm is designed to track a curve in scatter. The Kalman filter is unsuitable for such work because it is based on a Gaussian density that cannot simultaneously represent alternative hypotheses.

A particle is a pair (s _n , Π _n ), where s _n is the shape vector of particle n and Π _n is the probability of this shape. For a hand tracker, each particle consists of a shape vector and its probability, and each value of the shape vector represents an angle or translation of the kinematic hand model. Therefore, the dimension of the vector is equal to the number of DOF (Degree-Of-Freedom) of hand movement. Π _n is calculated by evaluating the state of s _n . This is done by searching for different cues such as visible surfaces or edges of the video frame. In this embodiment, both the visible surface and the edge of the hand image are used. The probability distribution is modeled by a set of particles S = (s ₁ , _{１ 1} ), ..., (s _N , _{Ｎ N} ).

The main steps for each iteration of the particle filter comprises the steps of sampling particles set S _t from the set S _t-1 of the last iteration, and obtaining the probability [pi _n by evaluating each s _n, Calculating a weighted average s _mean . s _mean represents the currently assumed hand posture estimate.
P. Drew, “Appearance-based gesture recognition,” Diploma Thesis, RWTH Aachen University, Aachen, Germany, January 2005 .) A. Errol, G. Bavis, M.M. Nicolesque, R.D. D. Boyle and X. Tom Buri, “A Study on Vision-Based Full Dof Hand Motion Estimation” CVPR '05, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR '05)-Workshop. Washington DC, USA, IEEE Computer Society, 2005, p. 75 (A. Erol, G. Bebis, M. Nicolescu, RD Boyle, and X. Twombly, “A review on vision-based full dof hand motion estimation,” in CVPR '05: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) -Workshops. Washington, DC, USA: IEEE Computer Society, 2005, p. 75.) B. Stenger, A.M. Tayanantan, P.A. Toru, and R. Sipora, “Hand Posture Estimation Using Hierarchical Detection” 2004. [Online], available at citizene.ist.psu.edu/stenger04hand.html (B. Stenger, A. Thayananthan, P. Torr, and R. Cipolla, “Hand pose estimation using hierarchical detection,” 2004. [Online] Available: citeseer.ist.psu.edu/stenger04hand.html.) Y. Wu and T.W. S. Huan, “Capturing Hands of Articulated People: Strategies for Dividing and Exploiting”, ICCV (International Conference on IEEE Computer Vision) '99, Volume 1, page 606, 1999 (Y. Wu and TS Huang, “Capturing articulated human hand motion: A divide-and-conquer approach, ”ICCV (IEEE International Conference On Computer Vision) '99, vol. 01, p. 606, 1999.) B. Stenger, P.A. R. S. Mendonka and R.M. Cipora, “Model-based 3d tracking of articulated hands”, CVRP (IEEE Conference on Computer Vision and Pattern Recognition), Vol. 2, page 310, 2001 (B. Stenger, PRS Mendonca, and R. Cipolla, “ Model-based 3d tracking of an articulated hand, ”CVPR (IEEE Conference on Computer Vision and Pattern Recognition), vol. 02, p. 310, 2001.) M.M. Isard and A.I. Blake, "Condensation-Conditional Concentration Propagation for Visual Tracking," Computer Vision International Journal, Vol. 29, No. 1, pp. 5-28, 1998. [Online], available at citizene.ist.psu.edu/isard98condensation.html (M.Isard and A. Blake, “Condensation-conditional density propagation for visual tracking,” International Journal of Computer Vision, vol. 29, no. 1, pp. 5-28, 1998. [Online]. Available: citeseer.ist.psu.edu/isard98condensation.html.) J. et al. Lee and T. L. Kunii, “Model-Based Analysis of Hand Posture,” IEEE Computer Graphics and Applications, Vol. 15, No. 5, pp. 77-86, 1995 (J. Lee and TL Kunii, “Model-based analysis of hand posture, ”IEEE Computer Graphics and Applications, vol. 15, no. 5, pp. 77-86, 1995.) N. Shimada, K.M. Kimura and Y.K. Shirai, “Real-time 3-d hand posture estimation based on 2-d appearance acquisition using a monocular camera”, RATFG-RTS (International Workshop on Face, Gesture Recognition, Analysis and Tracking in Real-Time Systems), Volume 00 , 0023, 2001 (N. Shimada, K. Kimura, and Y. Shirai, “Real-time 3-d hand posture estimation based on 2-d appearance retrieval using monocular camera,” RATFG-RTS (International Workshop on Recognition , Analysis, and Tracking of Faces and Gestures in Real-Time Systems), vol. 00, p. 0023, 2001.) A. Ude, V. Y Art, M.C. Phosphorus and G. Chen, “Distribution of visual attention of humanoid robots”, International Conference on IEEE-RAS / RSJ Humanoid Robots (Humanoids 2005), 2005 (A. Ude, V. Wyart, M. Lin, and G. Cheng, “Distributed visual attention on a humanoid robot, ”in IEEE-RAS / RSJ International Conference on Humanoid Robots (Humanoids 2005), 2005.) J. et al. Deutscher, A. Break and I. Reed, "Articulated body motion capture captured by annealed particle filtering," CVPR, Vol. 02, 2126, 2000 (J. Deutscher, A. Blake, and I. Reid, “Articulated body motion capture by annealed particle filtering, ”CVPR, vol. 02, p. 2126, 2000.)

  Due to the need for a reliable hand tracker, the third category method appears to be more promising than the single hypothesis tracker. However, these are more expensive for calculations. Since any robot designed to interact with people in a natural way requires real-time tracking capabilities, a third category of effective hand tracking devices is needed.

  Accordingly, one of the objects of the present invention is an unconstrained, real-time, marker-free, multi-hypothesis estimator-type hand tracking system that captures human hand movements effectively and reliably in real time. Is to provide a device.

  According to the first aspect of the present invention, a hand tracking device for tracking a hand in a video sequence using particle filter processing, a storage means for storing a three-dimensional hand model, and a current video sequence A clue extracting means for extracting a clue of the posture of the hand from the frame of the image, and an estimation for estimating the posture of the hand in the current frame by performing particle filtering on the clue and the current three-dimensional hand model shape And means for updating the three-dimensional hand model shape using the posture estimated by the estimating means.

  The estimation means adds a random noise to the three-dimensional hand model to generate a predetermined number of particles representing the shape of the three-dimensional hand model, and generates a z-buffer image for each of the particles. Z-buffer processing means. A z-buffer image is a projection of a three-dimensional hand model onto an image plane of an image capture device that outputs a video sequence. The estimation means further uses a probability calculation means for calculating the probability of each of the hand shapes based on the clues and the z-buffer image, and uses the respective probabilities of the particles as the respective weights of the hand shapes, Weighted sum means for calculating a hand shape weight sum calculated for the hand shape.

  The three-dimensional hand model is stored in advance in the storage means. A clue of the posture of the hand is extracted from the current frame of the video sequence by the clue extraction means. Skin-colored surfaces and / or edges are used as cues. The estimation means performs particle filtering on the three-dimensional hand model and the clue to estimate the hand posture in the current frame. The three-dimensional hand model shape is updated with the estimated posture.

  In the estimation means, a predetermined number of particle shapes are generated. For each hand shape, a projected z-buffer image is generated. By comparing the cues with the z-buffer image, the estimation means can calculate the hand shape probabilities. The three-dimensional hand model shape is updated by the weighted sum of the estimated hand postures.

  Since the z-buffer image is used for the probability calculation, only the visible part, not the hidden part, is searched. Therefore, tracking can be performed quickly and real-time tracking is possible.

  Preferably, the three-dimensional hand model includes a palm portion and a finger portion. The probability calculation means includes first evaluation means for calculating the probability of the first part of the finger particle. Each of the particle shaped finger portions of the first portion is replaced with the current finger portion of the three-dimensional model. The probability calculation means further includes second evaluation means for calculating the probability of the second portion of the particles. Each palm part of the second part of the particle is replaced with the current palm part of the three-dimensional model. The weighted sum means uses the respective probabilities as weights, the first sum means for calculating the weighted sum of the first part of the finger shape, and the respective probabilities as weights for the particles representing the hand shape. Second summing means for calculating a weighted sum of the second portion. The combining means includes means for combining the weighted sum palm portion output by the first summing means and the weighted sum finger portion output by the second summing means.

  The first evaluation means calculates the probability of the first part of the particle representing the finger shape. Since the first finger part of the particle shape of the first part is replaced with the current finger part of the hand model, only the palm part is considered in the calculation of the probability. Similarly, the second evaluation means calculates the probability of the second part of the particle. Since the palm part of the particle shape of the second part is replaced with the current palm part of the hand model, only the finger part is considered in the calculation of the probability. In this way, the weighted sum of the palm portion postures is calculated based on the output of the first evaluation means. A weighted sum of finger postures is calculated based on the output of the second evaluation means. By combining these results, the posture of the entire hand is estimated.

  Since the search space is divided into two spaces (palm part and finger part), the cost of calculation is substantially reduced and real-time tracking can be realized.

  More preferably, the first evaluation means is realized by a plurality of first calculation units operable in parallel. Each of the plurality of first calculation units is programmed to calculate a probability of a portion of the first part of the hand shape (finger shape).

  More preferably, the second evaluation means is realized by a plurality of second calculation units operable in parallel. Each of the plurality of second calculation units is programmed to calculate a probability of a portion of the second portion of the particle shape.

  Since the calculation cost for each particle is independent of all other particles, these can be performed in parallel by a plurality of computers. Therefore, the time required for each repetition can be reduced, and real-time tracking can be realized.

<Features>
[Example of hand tracker implementation]
A. Unconstrained tracking Particle filters seem promising because the computational cost can be explained by parallel execution with balanced particles. This is because when the cost increases, the cost is generated in the form of an independent probability calculation. Therefore, the first embodiment of the present invention uses a particle filter to estimate the hand posture. Since the DOF of a human hand is relatively large, the search space tends to be wide. Therefore, in this embodiment, the search space is decomposed into two. That is, hands and fingers. After estimating the hand posture and the finger posture separately, the device combines them to obtain a new hand posture.

  A gesture recognition system that learns how to see gestures is an example of a specialized hand tracker. Such a system performs well in its domain, but it is not designed to track arbitrary hand shapes. I want humanoid robots to interact naturally with people without introducing artificial restrictions. Therefore, the strategy here is to be able to represent the tracked hand shape without bias, so that our humanoid behavior module can react to and interpret the hand movement. Become. This can be used as the basis for a variety of tasks with different requirements, such as imitation, drawing attention and estimating intent, emphasizing eyes and hands, and recognizing gestures.

  In order to achieve this goal, a model-based strategy using a model similar to that disclosed in Non-Patent Document 7 is selected in this embodiment. By using the 3D hand model as a means for shape representation, it is not limited to a predetermined set of hand postures. In Non-Patent Document 8, Shimada builds a labeled database of hand posture templates using a 3D hand model, which is then used to track hand movements. The main result of this strategy is a hand posture with discrete kinematic parameters.

  In contrast, in this embodiment, the 3D hand model is projected directly onto the scene, so that accurate kinematic parameters can be known over a range of joint angles. Using a database is generally inexpensive in terms of computation. However, in this case, a considerable amount of memory and search time are required instead of reducing the calculation amount.

  There are several advantages to using the model directly. Its structural parameters can be changed dynamically and the model can be completely exchanged to fit different hands. Also, the details of the projection can be freely changed in a moving state, which is particularly important for humanoid robot heads in which central vision and wide-angle camera systems with different projection characteristics are used. In contrast, a system using a template would require a separate database for each camera. In addition, the degree of freedom of the model can be adjusted for different tasks. Since all the postures are generated at the time of execution, the system according to this embodiment does not increase the required memory amount related to the posture database exponentially, and is not limited by dimensions.

B. Real-time requirements One of the conclusions of Non-Patent Document 2 was the lack of real-time 3D hand trackers. In order to interact with the environment and people, it is necessary to create a real-time system. Therefore, it is unacceptable to introduce a long delay in the tracking process. The goal is to build a real-time tracking system and establish a natural means for communication between robots and people.

  With this requirement in mind, the particle filter system seems promising. It is scaled very well because it is parallelized in such a way as to minimize the communication required between the distributed system parts. By utilizing this characteristic, it was possible to realize a distributed particle filter based on the distributed visual cluster proposed in Non-Patent Document 9. The most expensive task in calculation is to calculate the probability of each particle. Here, the same configuration as in Non-Patent Document 8 is selected, tracking is managed by one computer, image capturing and preprocessing are performed, and probability functions are calculated at several cluster nodes (see FIG. 15). ).

  One application that requires real-time is imitation. To achieve this goal, the estimated angle of the tracker system is converted to the joint angle of the robot's hand. A commercially available robotic hand was used to mimic the hand posture. The results are shown in FIG.

[Tracker system without marker]
A. Data Flow Overview FIG. 2 shows an overview of the data flow according to this embodiment. Referring to FIG. 2, an input image 50 of a human hand is filtered by a skin color filter, and a binary map 52 indicating a skin color surface is generated. Using the binary map 52, a hand image 54 cut out from the input image 50 is obtained. By applying an edge filter such as a Sobel filter to the image 54, an image 56 subjected to edge filter processing is obtained.

  By adding random noise 59 to the 3D hand model 58 prepared from the previous iteration, a predetermined number (3000 in this embodiment) of particles is generated. Each particle represents a possible hand posture.

  A z-buffer image 60 is generated for each of the particles. The model-shaped z-buffer image 60 is compared to the skin-filtered binary map 52 and the edge-filtered image 56, as indicated by blocks 62 and 64, respectively. Using these results, as shown in block 66, the hand posture probabilities of the particle model shape are calculated. How this is done will be described in detail later.

  The new shape of the 3D hand model 58 is estimated by the weighted sum of the particle shapes as shown in block 68. The shape probabilities are used as weights in this calculation.

  For each frame cycle, the above calculation is performed to estimate the hand posture for each iteration. Hereinafter, each of the above-described processes will be described in more detail.

B. Image Acquisition and Preprocessing Referring to FIG. 3, the system uses a calibrated stereo camera 32 with a small distance between the eyeballs as seen on the head of a humanoid. The stereo camera was selected because if there are two viewpoints for the hand, errors that are negligibly small in one field of view in depth estimation are visible in the second field of view. This is because is better.

  After photographing a stereo image of the hand 30, the skin color is detected in the HSV space, a binary map of the skin color surface in the image is generated, and this is also used for evaluation of the clues on the skin surface. Thereafter, using this mask, all irrelevant portions of the image are cut, and an edge filter is applied to the surface obtained from the original image to obtain an edge image of the hand. Based on these images, the robot hand 34 is controlled to mimic the posture of the hand 30.

C. 3D hand model A 3D hand model (Figs. 4 and 5) consisting only of a conical section is used. Referring to FIG. 4, this model 58 can project onto a frame very quickly to detect cues, while still being accurate enough to track hand and finger movements. . In FIG. 4, “DIP”, “PIP”, “MCP”, “CMC”, “IP” and “TM” are “distal interphalangeal joint”, “proximal interphalangeal joint”, “metacarpal finger”. It means “internode joint”, “carpal joint”, “interphalangeal joint” and “phalangeal joint”. In FIG. 4, white circles indicate 1 DOF joints, and black circles indicate 2 DOF joints.

The DOF of a human hand is 22, but to avoid complexity, this kinematic hand model 58 has been limited to one degree of freedom for flexion for each finger. As seen in Non-Patent Document 2, θ _DIP = θ _PIP was set. θ _PIP represents the PIP angle of each finger, and θ _DIP represents the DIP angle of each finger. In order to express the angle of the joint of the model 58, the same notation is used in the following.

In addition, this tracker assumes θ _PIP = θ _MCP to reduce the number of dimensions of the search space. The overall model consists of 6 degrees of freedom for hand posture and position and 2 degrees of freedom for each finger. In the current version of the system, the thumb is fixed in the standard position. Therefore, the shape vector _{s n} in each particle has a dimension 14.

D. Solution to Self-Overlap Deutscher proposed a method for evaluating image cues in Non-Patent Document 10. He directly evaluates each projected body part in the image. In order to avoid searching for a hidden part that overlaps with the self in the projection of the hand model, this embodiment modifies his method. In order to search only the visible part, not the hidden part, this embodiment proposes a z-buffer strategy.

  The idea of z-buffer processing is to draw a distant object first and then draw a hidden object that is overwritten by a closer, overlapping object. In order to be able to accurately project this 3D hand model using z-buffer processing, an ordered list of finger parts classified according to their distance from the projection plane is required. Since this projection plane is not the same in a stereo camera system, it is necessary to generate a separate z-buffer for each camera.

  To project 3D hand model elements, we use the inverse matrix of each camera's calibration matrix. As an approximation, the center of gravity of each finger converted to the coordinate system of the viewpoint is compared. Thereafter, the finger portions are sorted so as to start from the largest z coordinate according to the z axis perpendicular to the projection plane. After generating the list, each element of the finger is drawn in a buffer according to the order of the list, thereby encoding whether the finger portion represents an edge or surface cue. By doing so, the index of the clue that overlaps with the self is overwritten with the duplicate finger. Both are written at the same position in the buffer. Depth information is represented by edge and surface brightness.

  The z-buffer is then calculated as illustrated in FIG. 6, where we know whether an edge is predicted for each pixel, a hand plane is predicted, or nothing is predicted. Yes. These projections are ultimately used for comparison with the preprocessed image.

E. Shape Probability Evaluation The shape of each particle is evaluated based on cues between edges and faces found in the preprocessed frame. To calculate the edge probability [pi _e and surface probability [pi _a of each particle n, using the following equation.

ここでΦ^ａはバイナリの肌マスクであり、μ_ｎ ^ａ及びμ_ｎ ^ｍはパーティクルｎについて実際に見出された肌面の画素と欠落している肌面の画素とを含み、Φ^ｅはマスクによりカットされた勾配画像であり、Π_ｎ ^ａ／ｅはパーティクルｎについてｚバッファにより生じた全ての面／エッジを含む。肌色マスク２８０上とエッジフィルタ処理された画像２８２との上に描かれた投影モデルを図７に示す。

Where Φ ^a is a binary skin mask, μ _n ^a and μ _n ^m include the skin surface pixels actually found for the particle n and missing skin surface pixels, and Φ ^e is the mask Π _n ^{a / e} includes all the faces / edges generated by the z-buffer for particle n. FIG. 7 shows a projection model drawn on the skin color mask 280 and the edge-filtered image 282.

Furthermore, collisions between fingers are also detected by calculating the shortest distance between the two finger portions. If this distance is less than the sum of their radii, assume that there is a collision and penalize this state in the probability function by Π _coll . Next, the probability 各_{n of} each particle is calculated as follows:

λを設定することにより、指数関数の入力範囲を選択することができる。[ｅ^−λ,１]の範囲内での確率を得るために、全ての重みω[ｉ]を実験により決定して、合計が１となるように正規化する。また、全てのΠ_ｎを合計が１になるように正規化する。その後、全てのパーティクルの重み平均を計算して、現在の手の形状ｓ_ｍｅａｎを推定する。

By setting λ, the input range of the exponential function can be selected. In order to obtain a probability within the range of [e ^−λ , 1], all weights ω [i] are determined by experiment and normalized so that the sum is 1. Further, the sum of all [pi _n normalized to be 1. Thereafter, the weight average of all particles is calculated to estimate the current hand shape s _mean .

Ｆ．再サンプリングにおける検索空間の区分け
パーティクルフィルタの次の繰返しのために、パーティクルの組を再サンプリングしなければならない。基本的に、非特許文献６でイサードが説明しているように、古いパーティクルの組に対する要素分解した（ｆａｃｔｏｒｅｄ）再サンプリングを用いる。この方策を用いることにより、手のトラッキングが可能となるが、個々の指のトラッキングはできない。姿勢形状がうまくアライメントされていないパーティクルは、モデル中の指を画像中の隣の指と比較してしまうことがあるために、指トラッキングの評価を低下させるという問題に直面した。

F. Partitioning the search space in resampling For the next iteration of the particle filter, the set of particles must be resampled. Basically, factored resampling is used for the old set of particles, as Isard explains in [6]. Using this strategy, hand tracking is possible, but individual fingers cannot be tracked. Particles with poorly aligned pose shapes faced the problem of reducing finger tracking evaluation because they could compare the finger in the model with the next finger in the image.

  In order to improve performance, the particle set was divided into two separate sets. The first set is used to track the posture of the hand (palm) with the current finger shape, and therefore the average finger shape from the previous iteration is assigned to this particle set. That is, the particle-shaped finger element is replaced with the current finger shape obtained in the previous iteration. The pose is resampled from all particles, taking their weights into account. The second set is used to track the finger in the previously known hand (palm) posture. For this purpose, the average hand (palm) posture from the previous iteration is assigned to all particles in this set. That is, the particle-shaped hand (palm) element is replaced with the current hand (palm) shape obtained in the previous iteration. Thereafter, the finger shape is resampled from all the particles. After resampling, merge both sets.

  By using this measure, the advantage of concentrating the particles at the previously known position when searching for the finger shape in the second set is obtained, and the dimension of the search space can be actually reduced by 6 degrees of freedom. . At the same time, a new hand posture is searched using the previously known finger shape using the first set. By assigning the average of particles in this way, the particle filter attributes for representing a number of hypotheses are preserved.

G. Distributed real-time tracking Particle filters are well suited for parallel clustering. This is because the computationally expensive evaluation of each particle is independent of all other particles. However, communication is necessary to distribute the particles to all cluster nodes and collect the corresponding probabilities. Here, it is also necessary to distribute the preprocessed image.

  The cluster nodes of this embodiment are connected to each other via a standard gigabit network. Broadcast was used to minimize the traffic required to send images. In order to realize a distributed particle filter, a distributed vision cluster proposed in Non-Patent Document 9 is used. By starting a large number of evaluation threads in each node, it is possible to enjoy the benefits of a multiprocessor, multicore system. It was found that the performance can be almost doubled by interleaving the stereo image to only one image every time the particle filter is repeated. Experimentally determined 3000 particles were used for all tests and benchmarks. The tracking was not improved much with more particles, and the results were considerably degraded with fewer than 2000 particles.

  With the distributed visual cluster, the particle filter could be scaled up to 6.84 times using 9 clusters, which is 76.1% efficient. 10.2 FPS could be reached. For measurement, a dual 2.16 GHz CPU was used in the cluster node. The performance measure is shown in FIG.

<System structure>
FIG. 9 shows the overall structure of the hand model tracker 90 of this embodiment. Referring to FIG. 9, hand model tracker 90 has a frame photographing unit 100 for capturing a stereo image output output from stereo camera 32, and a frame for storing a stereo image captured by frame photographing unit 100. A memory 102; an image preprocessing module 104 for processing a stereo image stored in the frame memory 102 and outputting a hand skin color binary map 140 and an edge image 142; and a memory 106 for storing them. including.

  The hand model tracker 90 further includes a storage unit (memory) for storing the 3D hand model 116, and a random number generation module 118 for generating 3000 particles by adding random noise to the 3D hand model 116. And a particle memory 120 for storing the particle shape for estimating the posture of the hand, and a particle memory 122 for storing the particle shape for estimating the posture of the finger.

  The hand model tracker 90 further includes a particle filter unit 108 for hand posture, a particle filter unit 110 for finger posture, and a combination block 112 for combining the estimated hand posture and finger posture. And a model update module 114 that updates the 3D hand model 116 using the output of the combination block 112.

  FIG. 10 shows a block diagram of the image preprocessing module 104. Referring to FIG. 10, the image preprocessing module 104 detects a skin color surface from the hand image stored in the frame memory 102 and outputs a skin color binary filter 140 and a skin color filter 180 for outputting the skin color binary map 140. A masking module 184 for extracting a part of the image stored in the frame memory 102 by cutting an irrelevant portion of the original image by using as a mask, and a hand cut out output from the masking module 184 And an edge filter 188 for detecting an edge image from the image 186. The output of the edge filter 188 is an edge image 142.

  FIG. 11 is a block diagram showing details of the particle filter unit 108 for hand posture. Referring to FIG. 11, the particle filter unit 108 includes a skin color binary map 140, an edge image 142, a particle shape memory 200 for storing a hand particle model shape, and each of which is assigned to the evaluation unit. To output a new hand posture by calculating a weighted sum of hand shapes using a number of evaluation units 202A-202N for calculating model shape probabilities and the respective probabilities as weights. A weighted sum module 204.

  Each of the evaluation units 202A-202N receives only the part related to the hand posture of the particle model shape and combines them with the current finger shape. Using these shapes, the evaluation units 202A-202N calculate particle probabilities.

  Each of the evaluation units 202A-202N has the same structure. FIG. 12 shows details of the evaluation unit 202K which is one of the evaluation units 202A to 202N. Referring to FIG. 12, the evaluation unit 202K combines each of the hand posture particle model shapes assigned to this unit with the current finger shape and outputs a vector shape module 218 for outputting the resulting particle shape vector. And a memory 220 for storing the particle shape vector output from the vector forming module 218, and projecting the elements of the particle shape vector onto the image plane of the camera 32 using the inverse matrix of the calibration matrix of the camera 32. And a coordinate conversion module 222 for converting to a suitable coordinate system. This new coordinate system has an X axis and a Y axis on the image plane, and a Z axis perpendicular to the image plane.

The evaluation unit 202K further includes a center-of-gravity calculation module 224 for calculating the center of gravity of the hand part having the shape stored in the memory 220, and the respective parts of the hand in descending order of the Z-coordinate of each center of gravity of each part of the hand. A sorting module 226 for sorting and outputting an ordered list 228 of each part of the hand, a z buffer 232, and a hand part for rendering a particle-shaped projection image in the z buffer 232 in the order of the list 228 A drawing module 230; a collision detection module 234 for detecting a collision between parts of the hand and calculating a penalty for the collision; a memory 236 for storing a skin color binary map and an edge image; and a memory 236 The skin color binary map and edge image stored in the image, the image in the z buffer 232, and the collision detection module 2. 4 by using the calculated collision penalty by, and a probability calculation module 238 for calculating the probability of each particle shapes assigned to this unit 202K. The resulting hand-shaped j-th particle probability _{ｊ H, K, j} assigned to the K-th evaluation unit 202K is provided to the weighted sum module 204 shown in FIG.

  Details of the particle filter unit 110 for the finger are shown in FIG. Referring to FIG. 13, the particle filter unit 110 includes a skin color binary map 140, an edge image 142, a particle shape memory 240 for storing a finger particle model shape, and a model each assigned to the evaluation unit. It includes a number of evaluation units 242A-242N for calculating shape probabilities and a weighted sum module 244 for outputting new finger postures by adding finger shapes using each probability as a weight.

  Each of the evaluation units 242A-242N receives only the part related to the finger posture of the particle model shape and combines them with the current hand shape. Using these shapes, the evaluation unit 242A-242N calculates the probability of the particles.

  Each of the evaluation units 242A-242N has the same structure. FIG. 14 shows details of the evaluation unit 242K which is one of the evaluation units 242A-242N. Referring to FIG. 14, the evaluation unit 242K combines each of the finger particle model shapes assigned to this unit with the current hand shape and outputs a resulting hand shape vector 258. A memory 260 for storing the particle shape vector output from the vector forming module 258, and a coordinate conversion module for converting the elements of the particle shape vector into a coordinate system suitable for projecting onto the image plane of the camera 32 262. This new coordinate system has an X axis and a Y axis on the image plane, and a Z axis perpendicular to the image plane.

The evaluation unit 242K further sorts each part of the hand in descending order of the Z coordinate of each center of gravity of the hand part, and a center of gravity calculation module 264 for calculating the center of gravity of each part of the hand of the shape stored in the memory 260. Then, a sort module 266 for outputting an ordered list 268 of each part of the hand, a z buffer 272, and a hand part drawing module for drawing a projected image of the particle shape in the z buffer 272 in the order of the list 268 270, a collision detection module 274 for detecting a collision between each part of the hand and calculating a penalty for the collision, a memory 276 for storing the skin color binary map and the edge image, and storing in the memory 276 Skinned color binary map and edge image, image in z-buffer 272, and collision detection module 274 Therefore comprising using the calculated collision penalty, and a probability calculation module 278 for calculating the probability of each particle shapes assigned to this unit 242K. The resulting probability 指_{F, K, j of} the finger-shaped jth particle assigned to the Kth evaluation unit 242K is provided to the weighted sum module 244 shown in FIG.

<Operation>
The hand model tracker 90 operates as follows. Referring to FIG. 9, the 3D hand model 116 stores the initial 3D hand model 58 shown in FIGS. The camera 32 outputs a stereo image signal. Each stereo frame is shot by the frame shooting unit 100 and stored in the frame memory 102.

  Referring to FIG. 10, skin color filter 180 detects a skin color surface in an image in the HSV space and generates skin color binary map 140. This map is used by the masking module 184 to cut all unrelated portions of the image, thereby producing a cut hand image 186, resulting in the cut hand image 186. The edge filter 188 is applied to the edge image 142 to obtain the edge image 142.

  The resulting skin color binary map 140 and edge image 142 are provided to the particle filter unit 108 for hand posture and the particle filter unit 110 for finger posture.

  On the other hand, the random number generation module 118 generates random noise and adds these to the 3D hand model 116 to generate 3000 particles. Half of the particles are stored in the particle memory 120 and the remaining half are stored in the particle memory 122. The former is used to estimate the next hand posture, and the latter is used to estimate the next finger posture.

  Referring to FIG. 11, skin color binary map 140 and edge image 142 are stored in particle shape memory 200. The particle shape of the hand posture is stored in the particle shape memory 200. Both are provided to each of the evaluation units 202A-202N. The current finger shape is supplied to each of the evaluation units 202A-202N.

  Referring to FIG. 12, the particle shape vector is formed by combining the particle shape of the hand posture and the particle filter of the current finger shape by the vector forming module 218, and this is stored in the memory 220. The coordinate conversion module 222 converts the vector coordinate system into one suitable for z-buffer processing. The center-of-gravity calculation module 224 calculates the center of gravity of each hand part of the particle shape.

  Sort module 226 sorts the hand portions in descending order of their Z coordinates to generate list 228. The hand drawing module 230 draws each element of the hand in the z-buffer 232 so that the farther object is drawn first and overwritten by the closer, overlapping objects, and the finger part and the edge or Encodes whether to represent a clue to the surface. The hand part drawing module 230 overwrites the index of the clue that is hidden by itself with the overlapping finger.

  With reference to the z-buffer 232 illustrated in FIG. 6, it is known for each pixel whether an edge is predicted, a face is predicted, or nothing is predicted. These projections are ultimately used for comparison with the image preprocessed by the probability calculation module 238.

  The probability calculation module 238 calculates the probability of each particle shape based on the edge and surface cues found in the preprocessed frame and stored in the memory 236 using the above equations (1)-(4). To do.

The collision detection module 234 detects a collision between fingers by calculating the shortest distance between the two finger portions. If this distance is less than the sum of the finger radii, it is assumed that a collision has occurred and this state is given a penalty in the probability function by Π _coll . Thereafter, the probability _{ｎ n of} each particle is calculated by the probability calculation module 238 using equation (5).

Thus, the probabilities Π _{H, K, j} for each shape are calculated and provided to the weighted sum module 204 shown in FIG.

Given all of the probabilities Π _{H, K, j} , the weighted sum module 204 calculates a weighted average of all hand particles and uses equation (6) to determine the current hand shape s _{mean, H for} the hand pose. Is estimated. Similarly, the particle filter 110 for finger posture estimates the current finger shape s _{mean, F} for the finger posture using Equation (6). Given the hand shape s _{mean, H} for the hand pose and the finger shape s _{mean, F} for the finger pose, the combination block 112 shown in FIG. 9 has estimated the estimated hand pose. Can be combined with finger posture. The resulting estimated overall hand shape is provided to the model update module 114, which updates the 3D hand model 116 with the estimated overall hand shape.

  By repeating the above estimation for each captured stereo image, the hand model tracker 90 tracks the hand posture in real time. Since the particle filter is used to estimate the posture of the entire hand, it is unlikely that only the local minimum will be reached. Therefore, the hand model tracker 90 tracks the posture of the hand with reliability. Due to the decomposition of the search space (hand posture and finger posture), tracking can be performed in a short time for each iteration, and reliable real-time tracking is possible.

<Computer hardware configuration>
The hand model tracker 90 described above can be realized by a plurality of computers connected to each other via a high-speed network. Referring to FIG. 15, a computer system 320 for realizing the hand model tracker 90 captures a stereo image, and a personal computer (PC) 332 connected to the camera 32 for preprocessing the captured image. And PC 334A-334X, each having a core CPU configured to operate as evaluation units 202A-202N and 242A-242N, and evaluation units 202A-202N and 242A-242N, each having a dual-core CPU configuration. Using the output probability, the weight average of the hand shape and the finger shape is calculated, the PC 338 for combining the estimated hand posture with the estimated finger posture, the 3D hand model is stored, and the output of the PC 338 PC 336 to update this using the and the robot according to the estimated hand posture It includes a hand control unit 340 for controlling the hand, and a high-speed network 330 that connects PC332,334A-334X, the 336, 338 and 340.

  FIG. 16 shows a typical configuration of the PC 332. Referring to FIG. 16, the PC 332 stores a dual core CPU 376 having a dual calculation unit, a read only memory (ROM) 378 for storing a bootup program, and a program to be executed by the dual core CPU 376. It includes a random access memory (RAM) 380 for storing variables used in the program and a hard disk 374 for storing programs, initial values of variables, initial 3D hand models, and other data.

  The PC 332 further includes a digital versatile disk (DVD) drive 370 for reading and writing data from the DVD 382, a keyboard 366 and a mouse 368, a video capture board 388 to which the monitor 362 and the camera 32 are connected, and a portable. A memory port 372 for reading data from the memory 384 and writing data, and a network interface 396 for enabling the PC 332 to be connected to the network 330 are included.

  The other PCs 334A-334X, 336, 338 and 340 have the same configuration. However, since all of these PCs can be controlled from the PC 332, it is not necessary to provide a mouse, a keyboard, or a monitor on these PCs.

<Result>
It is difficult to evaluate the tracking performance for complex articulated objects. This is because the comparison data from the video sequence cannot usually be obtained easily. In hand tracking, more accurate methods such as data gloves or markers cannot be easily used. This is because they change the appearance of the hand.

  We have chosen to conduct a two-step evaluation. First, it was shown that the tracker of the present invention works well with the real world by tracking between one posture and another posture using natural images. To show the performance of the tracker, an artificial video sequence with known comparative data was generated to evaluate the tracking error of the system.

A. Natural video sequence Five hand poses were selected to track the transition between each pair. We focused on the number of frames required for the tracker to fully converge to the correct posture after the end of the transition. All video capture speeds were 30 FPS.

表１に、姿勢[列]から姿勢[行]への遷移に必要なフレームの数の概略をまとめた。収束に必要とされる平均のフレーム数は４．６５である。わずかに掌のオフセットがあるが、収束した指形状は常に正しかった。

Table 1 summarizes the number of frames required for transition from posture [column] to posture [row]. The average number of frames required for convergence is 4.65. There was a slight palm offset, but the converged finger shape was always correct.

  FIG. 17 shows the tracking sequence for synthetic hand stereo projection (frames 150 to 400). The first row shows input for surface cues and the second row shows input for edge cues.

B. Composite Video Sequence In the second experiment, a composite video produced by projecting a stereo animation sequence of a 3D hand model (FIG. 18) was used. Data for comparison of each sequence can be derived from the presented sequence. Accordingly, the tracked value can be compared with the comparison data, and the result of FIG. 19 is obtained.

  FIG. 19 shows a comparison between estimated values and comparison data for two DOFs of the MCP of the index finger. As can be seen from FIG. 19, the error between the tracked value and the comparison data is very small.

In FIG. 20, the posture error of the hand in tracking over 400 frames can be seen. As can be seen from FIG. 20, the average error (θ _MCP ) only reached the highest 0.06 radians in the frame 300.

<Conclusion>
In the present invention, an embodiment of a marker-free hand and finger tracking system using a particle filter has been presented. We have proposed a new method using a z-buffer that accurately evaluates the cues of objects that are multi-joint and hidden by overlapping themselves, such as human hands. In order to be able to accurately track the finger, we proposed particle resampling to segment the search space. This can be achieved using particle filters dispersed in PC clusters, and tracking results can be achieved at a high frame rate. The effectiveness of the tracking system has been demonstrated by controlling a sophisticated robot hand with 20 degrees of freedom in real time.

  Combining this tracking system with a tracker for the whole body of a person can provide tremendous benefits. The wrist position can be obtained by using the posture of the arm, and therefore the search space for the direction of the hand can be greatly reduced.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the sequence of the motion of the robot hand controlled by embodiment of this invention. It is a figure which shows the general view of the data flow according to this embodiment. It is a figure which shows the setup of one embodiment of this invention. It is a figure which shows roughly the 3D hand model used in one embodiment of this invention. It is a figure which shows the projection image of 3D hand model. It is a figure which shows the image of the 3D hand model projected on z buffer. It is a figure which shows the projection model drawn by the skin color mask 280 and the image 282 by which the edge filter process was carried out. It is a figure which shows the performance measure of the cluster node used by one embodiment of this invention. It is a figure which shows the whole structure of the hand model tracker 90 of this embodiment. 3 is a block diagram showing an image preprocessing module 104. FIG. It is a block diagram which shows the detail of the particle filter unit 108 for posture of a hand. It is a figure which shows the detail of the evaluation unit 202K of the particle filter unit 108 for hand postures. It is a figure which shows the detail of the particle filter unit 110 for fingers. It is a figure which shows the detail of the evaluation unit 242K of the particle filter unit 110 for fingers. It is a figure which shows the computer system structure for implement | achieving one embodiment of this invention. FIG. 16 is a diagram showing a typical configuration of a personal computer (PC) used in the computer system 320 shown in FIG. 15. FIG. 6 shows a tracking sequence of a combined hand stereo projection (frames 150 to 400), where the first row shows input for surface cues and the second row shows input for edge cues. FIG. It is a figure which shows the sequence of the synthetic | combination image | video used by the 2nd experiment produced by projecting the stereo animation sequence of 3D hand model. It is a figure which shows the comparison with the data for a comparison, and an estimated value about 2DOF of MCP of an index finger in experiment. It is a figure which shows the attitude | position error of the hand observed in experiment.

Explanation of symbols

30 hand 32 camera 34 robot hand 58, 116 3D hand model 90 hand model tracker 100 frame photographing unit 102 frame memory 104 image pre-processing module 106, 220, 236, 260, 276 memory 108 particle filter unit 110 for hand finger Particle filter unit 112 for combination block 114 Model update module 118 Random number generation module 120 and 122 Particle memory 200, 240 Particle shape memory 202A-202N and 242A-242N Evaluation unit 204, 244 Weighted sum module 218, 258 Vector formation module 222, 262 Coordinate transformation module 224, 264 Center of gravity calculation module 226, 266 Sort module 228, 268 List 30,270 hand of the drawing module 232,272 z buffer 234,274 collision detection module 238,278 probability calculation module

Claims (4)

A hand tracking device for tracking a hand in a video sequence using particle filter processing,
Storage means for storing a three-dimensional hand model;
A clue extraction means for extracting a hand posture clue from the current frame of the video sequence;
Estimating means for performing particle filtering on the clue and the current three-dimensional hand model shape to estimate the posture of the hand in the current frame;
Means for updating the model shape of the three-dimensional hand using the posture estimated by the estimating means,
The estimation means includes
Means for adding a random noise to a three-dimensional hand model to generate a predetermined number of particle shapes for the three-dimensional hand model;
Z-buffer processing means for generating a z-buffer image for each of the particles representing a hand shape, wherein the z-buffer image outputs the video sequence of the three-dimensional hand model A probability calculating means for calculating the probability of each of the hand shapes based on the clue and the z-buffer image;
Weighted summing means for calculating the total weight of the hand shape calculated for the shape of the particle, using each probability of the shape of the hand as the weight of each particle representing the shape of the hand Including hand tracking device. The three-dimensional hand model includes a palm part and a finger part, and the probability calculation means includes:
First evaluation means for calculating a probability of the first part of the particle, each of the finger parts of the particle shape of the first part being replaced with a current finger part of the three-dimensional model The probability calculation means further includes:
Second evaluation means for calculating the probability of the second part of the particle, each of the palm parts of the particle shape of the second part being replaced with the current palm part of the three-dimensional model ,
Said weighted summing means, using respective probabilities as weights, first summing means for calculating a weighted sum of said first portion of said particles;
Second summing means for calculating a weighted sum of the second portion of the particles using each probability as a weight;
The combination means includes means for combining the weighted sum palm portion output by the first summing means and the weighted sum finger portion output by the second summing means. The hand tracking device described in 1. The first evaluation means is realized by a plurality of first calculation units operable in parallel, and each of the plurality of first calculation units calculates a probability of a part of the first portion of the particle shape. The hand tracking device according to claim 2, programmed to: The second evaluation means is realized by a plurality of second calculation units operable in parallel, and each of the plurality of second calculation units calculates a probability of a part of the second portion of the particle shape. 4. A hand tracking device according to claim 2 or claim 3, programmed to:

Images