JP2000331170A - Hand motion recognizing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a hand motion recognizing device which can stable recognize the motions of a plurality of hands in real time. SOLUTION: This hand motion recognizing device takes the picture of a plurality of hands from different directions by means of a plurality of cameras. A visual point selecting section 4 selects the optimum visual point for processing out of a plurality of visual points (of the cameras). A feature extracting section 6 selects such an image that does not cause occlusion based on the feature point of the image. A chase processing section 8 predicts the positions of the hands by means of a Kalman filter. The visual point selecting section 4 operates based on the predicted results. The chase processing section 8 selects the images which recognize the shapes of the hands based on the angles of rotations of the hands and recognizes the shapes of the hands based on a P-type Fourier descripter. The hand motion recognizing device recognizes hand motion by integrating the shapes of the hands.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a hand gesture recognition device, and more particularly, to a hand gesture recognition device for recognizing a plurality of hand movements in real time.

[0002]

2. Description of the Related Art A number of systems having hand gestures have been proposed as intuitive and easy-to-use interfaces. Most of these systems require the user to wear special gloves with sensors, as represented by Data Gloves (registered trademark), which imposes a heavy burden on the user such as the complexity of putting on and taking off. It has not spread widely. On the other hand, in order to realize an equivalent system in a non-contact manner, a method of detecting a hand gesture by image processing has been proposed.

[0003]

However, in the conventional system based on image processing, the occlusion between hands and the self-occlusion (for example, the shape of the hand with the extended index finger and the shape of the hand with the extended index finger and middle finger) are considered. This means that deformation of the captured image due to a change in the posture of the hand with respect to the optical axis of the camera is a major problem.

In a system based on image processing, in order to reconstruct and track three-dimensional information, it is necessary to match a detected feature point with an image. However, when an object such as a hand is moved, it is difficult to continuously detect and match feature points due to the occurrence of occlusion and a change in shape described above.

The present inventor has sought to solve this problem.
A method for estimating the position and posture of a hand using a large number of cameras has been proposed (Japanese Patent Laid-Open Publication No. Hei 10-63864, Akira Utsumi, Tsutomu Miyazato, Fumio Kishino, Jun Ohtani, Ryohei Nakatsu: Hand posture estimation method using viewpoint images, Journal of the Institute of Image Information and Television Engineers, Vol.51, No.12, pp.2116-2125 (1997). According to this method, a three-dimensional position / posture can be stably obtained irrespective of a change in the shape of a hand, based on robust feature amounts obtained from a plurality of viewpoints. However, this method cannot handle multiple hands simultaneously. Also,
The shape of the hand cannot be recognized.

[0006] On the other hand, research on tracking a plurality of hands by a stereo camera using a prediction filter such as a Kalman filter has been conducted (Ali Azarbayejani and
Alex Pentland. Real-time self-calibrating stereo
person tracking using 3-dshape estimation from blo
d feature.In 13th International Conference onPatt
ern Recognition, pp. 627-632, 1996.). However, in this method, the position of both hands can be tracked, but the posture and shape cannot be recognized.

[0007] On the other hand, researches aimed at shape recovery using a three-dimensional model of a finger have been conducted (Rehg, JM
and Kanade, T .: Visual Tracking og High DOF Artic
ulated Structures: an Application to Human Hand Tr
acking, Computer Vision-ECCV '94, LNCS vol. 801, p
p.35-46 (1994)., Masayuki Nakajima, Yuyu Shiba: Finger Motion Detection Method for Constructing Virtual Reality World, Graphics and CAD
67-6, pp.41-46 (1994)., Y. Iwai, Y. Yagi, and M. Yauchida: Estimation of 3D hand movement and position from monocular video, IEICE (D-II), Vol. -D-II, No. 1, pp.44-55
(1997).), While these methods can be expected to provide precise posture information, they have the problem that the degree of freedom of finger joints is large and the calculation cost is enormous. Further, no consideration has been given to avoiding occlusion, and there is a problem that estimation becomes difficult when image features that are prerequisites for processing cannot be detected by occlusion.

Therefore, an object of the present invention is to provide a hand gesture recognition device capable of stably recognizing a plurality of hand movements.

[0009]

According to a first aspect of the present invention, there is provided a hand gesture recognition apparatus for recognizing a plurality of hand movements, wherein a plurality of hands are photographed from different directions to obtain a plurality of images. A plurality of image pickup means, a first selection means for selectively outputting an image for recognizing a plurality of hand movements among the plurality of images, and a feature point of the image selected by the first selection means, A feature extraction unit for selecting an image to be tracked based on a point; and a plurality of tracking processing units provided for each of the plurality of hands and tracking the movement of the corresponding hand. Each of the means is a prediction means for predicting the state of the hand using the feature points of the image to be tracked, and the normal direction of the hand is calculated based on the image to be tracked. Image obtained by shooting the hand from the direction closest to And a recognition unit for recognizing the shape of the hand based on the image selected by the second selection unit, wherein the first selection unit performs the prediction in the prediction unit among a plurality of images. An image that is determined to be less likely to occlude from the state is selected.

According to a second aspect of the present invention, in the first aspect of the present invention, the feature extracting unit determines occurrence of occlusion based on a ratio between the position of the extracted feature point and the width of the hand. An image for which occlusion has been detected is detected and is not selected.

A hand gesture recognition device according to a third aspect is the hand gesture recognition device according to the second aspect, wherein the recognition unit extracts a contour line of the hand in the image selected by the second selection unit. P-type Fourier description means for describing the contour of the hand extracted by the contour-line extraction means with a P-type Fourier descriptor, and specifying the hand shape based on the P-type Fourier descriptor from the P-type Fourier description means And a shape specifying means.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [Embodiment 1] A hand gesture recognition device according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding portions have the same reference characters allotted, and description thereof will not be repeated. The hand gesture recognition apparatus according to the embodiment of the present invention selects a plurality of viewpoints optimal for processing from among a large number of viewpoints for photographing a hand from different directions, thereby enabling a plurality of hand movements (position, posture, shape). ) In real time. Hereinafter, the hand gesture recognition device 1000 that tracks the right hand and the left hand will be described as an example.

FIG. 1 is a block diagram showing a configuration of a main part of a hand gesture recognition device 1000 according to Embodiment 1 of the present invention. The hand gesture recognition apparatus 1000 tracks the movement of the right hand and the left hand using continuous images obtained from a plurality of viewpoints. The hand gesture recognition device 1000 shown in FIG.
# 1 to #n, a viewpoint selection unit 4, a feature extraction unit 6, a tracking processing unit 8, and a gesture recognition unit 14.

The cameras 2 # 1 to 2 # n capture hands from different directions to obtain hand images. The viewpoint selection unit 4
An image optimal for processing is selected from a plurality of images obtained by a plurality of cameras. The feature extraction unit 6 calculates a distance conversion value (skeleton value) using an image received from the selected camera, and extracts a center of gravity (feature point). At this point, the image in which the occlusion between the hands has occurred is excluded from the target of shape recognition.

The tracking processor 8 includes a left-handed tracking unit 10 and a right-handed tracking unit 12. Each of the tracking units 10 and 12 predicts a corresponding hand movement by a Kalman filter as described later. The viewpoint selecting unit 4 selects a viewpoint (camera) in the next frame based on the position prediction result. This makes it possible to reduce the probability of occurrence of occlusion.

The tracking units 10 and 12 further select one viewpoint to be processed based on the rotation angle of the hand and recognize the corresponding hand shape. The gesture recognition unit 14 recognizes a gesture based on the results of the tracking units 10 and 12. A command corresponding to the recognition result is issued to the application program.

Here, the configuration of the feature extracting unit 6 will be described with reference to FIG. FIG. 2 shows the feature extraction unit 6 shown in FIG.
FIG. 2 is a block diagram showing an outline of the configuration of FIG. Referring to FIG. 2, feature extracting section 6 includes area dividing sections 20 # 1 to 20 # n,
Spindle detectors 22 # 1 to 22 # n, rotation converters 24 # 1
24 # n and feature point calculation units 26 # 1 to 26 # n. Area dividing units 20 # 1 to 20 # n, spindle detecting unit 22 #
1 to 22 # n, rotation conversion units 24 # 1 to 24 # n, and feature point calculation units 26 # 1 to 26 # n
It is provided corresponding to # 1 to #n.

Hereinafter, the area dividing units 20 # 1-2 will be generically referred to.
0 # n is assigned to the area dividing unit 20 and the spindle detecting units 22 # 1 to 22 # 22.
#N is the spindle detection unit 22, and the rotation conversion units 24 # 1 to 24 #
n is a rotation conversion unit 24 and feature point calculation units 26 # 1 to 26 #
n is described as a feature point calculator 26.

The area dividing section 20 divides a skin color area in an input image obtained by a corresponding camera using color information and luminance information. The spindle detection unit 22 is a sobel (So
bel) The average edge direction is obtained from the result of the filter, and this is defined as the hand direction (main axis) in two dimensions.

The rotation conversion unit 24 rotates the image based on the hand direction obtained by the main axis detection unit 22 so that the fingertip points upward. The feature point calculation unit 26 receives the output of the rotation conversion unit 24 and calculates a distance conversion value indicating the shortest distance from each of the pixels constituting the hand to the contour of the image of the hand.

For example, from the area dividing section 20, FIG.
An input binary image (silhouette image) as shown in FIG. The main axis detection unit 22 obtains an average edge direction of the hand region by applying a Sobel filter to the obtained silhouette image. This average edge direction is shown in FIG.
As shown in (b), it is regarded as the fingertip direction (Φ in the figure) (Koichi Ishibuchi, Keisuke Iwasaki, Haruo Takemura, Fumio Kishino: Real-time hand gesture estimation using image processing and application to human interface, IEICE (D-II), Vol.J79-DI
I, No. 7, pp. 1218-1229 (1996).).

The rotation converter 24 rotates the image so that the average edge direction (fingertip direction) Φ obtained in FIG. 3B is directed upward on the image. For example, the image is rotated by the rotation angle γi. Thus, the image shown in FIG. 3C is obtained.

The feature point calculator 26 receives the image shown in FIG. 3C and calculates the distance conversion image shown in FIG. 3D. In FIG. 3D, a pixel having a larger distance conversion value is represented as black, and a pixel having a smaller distance transformation value is represented as white. Therefore, the pixels gradually become darker as the distance from the hand outline increases. Then, a point (symbol COG in the figure) having the largest distance conversion value (skeleton value) is extracted. This is the center of gravity (feature point).

Assuming that the distance between the center of gravity points is s, if the ratio (w / s) of the distance s to the vertical width w of the silhouette of the hand is greater than a certain threshold value, the occlusion between the hands is reduced. It is determined that it has occurred. Images for which occlusion has not occurred are subjected to tracking processing in the tracking processing unit 8, and for images for which occlusion has occurred,
Remove from tracking processing.

Next, the tracking processing section 8 will be described with reference to FIG. FIG. 4 is a block diagram showing an outline of the configuration of the tracking processing unit 8 shown in FIG. Referring to FIG. 4, tracking processing unit 8 includes three-dimensional position / direction detection units 30 # 1 to 30 #.
2, rotation angle detectors 31 # 1-31 # 2, viewpoint selector 32
# 1 to 32 # 2 and hand shape recognition units 33 # 1 to 33 #
2 inclusive.

The three-dimensional position / direction detection unit 30 # 1, rotation angle detection unit 31 # 1, viewpoint selection unit 32 # 1, and hand shape recognition unit 33 # 1 provide a three-dimensional position / direction
The direction detecting unit 30 # 2, the rotation angle detecting unit 31 # 2, the viewpoint selecting unit 32 # 2, and the hand shape recognizing unit 33 # 2 are included in the tracking unit 12 corresponding to the right hand.

Hereinafter, the three-dimensional position / direction detecting units 30 # 1 to 30 # 2 are generally referred to as a three-dimensional position / direction detecting unit 30,
The rotation angle detectors 31 # 1-31 # 2 are replaced with the rotation angle detector 31.
And the viewpoint selecting units 32 # 1 to 32 # 2
And the hand shape recognition units 33 # 1 to 33 # 2
Write 3.

The three-dimensional position / direction detection unit 30 predicts a corresponding hand movement by a Kalman filter based on feature points obtained from an image to be tracked. The rotation angle detector 31 detects the rotation angle of the corresponding hand. The viewpoint selecting unit 32 selects an image (camera) for recognizing a corresponding hand shape from an image (camera) to be tracked based on the detection result of the rotation angle detecting unit 31. The hand shape recognition unit 33 recognizes the corresponding hand shape based on the selected camera and the corresponding hand rotation angle.

The three-dimensional position / direction detector 30 will be described with reference to FIG. FIG. 5 is a diagram for describing an observation model in the Kalman filter. In FIG. 5, symbols X, Y, and Z represent three axes of the world coordinate system, a subscript i represents a camera number, and a subscript j represents a left hand or a right hand, respectively. Note that the image 1 is obtained by observation of the camera 2 # i.

It is assumed that the hand (left hand or right hand) at the position (Xhj, Yhj, Zhj) is observed by the camera 2 # i (i = 1 to n) at the position (Xci, Yci). I do. Here, this observation is assumed to include a Gaussian error (covariance is represented by Σ).

Here, Lhj, i is the hand hj and the camera 2
距離 Let the distance to i, li be the focal length of the camera Ci.
Also, let whj, i be the angle between the epipolar line and the YZ plane, and let vhj, i be the angle between the Z axis and the projection result of the epipolar line on the YZ plane. Further, Rwhj, i, Rv
Let hj, i be a rotation matrix for rotating the corresponding epipolar line to make it parallel to the Z-axis.

Assuming that the hand moves at a constant speed,
The position Xhj, t of the point is represented by equations (1) to (4). Expressions (2) to (4) represent the speed of the hand hj on the X, Y, and Z axes, respectively. The symbol "'" added to the matrix means transposition.

[0033]

(Equation 1)

It is assumed that the hand hj is observed by N cameras. ∧Xhj, t-1 is an estimated value of the position of the hand hj at time (t-1), and ∧Shj, t-1 is a variance matrix of the estimated value ∧Xhj, t-1. The state at time t is expressed by equations (5) to (7). In the following formulas, the subscripts “∧” and “/” are respectively described above the corresponding variables.

[0035]

(Equation 2)

The matrix F represents a transition matrix, and the matrix Q represents a covariance matrix of errors occurring in transitions. Here, if the state Ci of the camera 2♯i is expressed by Expression (8), the state of the camera and the epipolar axes whj, i, vh
The observation formulas (9) to (10) are obtained from j and i. Note that e in equation (9) represents an observation error (the mean is [0 0] ', and the covariance matrix is Σhj, i). Considering that the covariance matrix Σhj, i increases as the distance from the camera to be observed increases, Equation (11)
It is expressed as follows. Since the actual distance Lhj, i between the camera 2 @ i and the hand hj is unknown, the camera 2 @ i and the predicted position /
The distance / Lhj, i, t from Hhj, t is used as an approximate value.

[0037]

(Equation 3)

In the observation formula (9), the left side represents the observation information, and the right side represents the result of projecting the position of the hand on the image. These observations and estimates (Equation (5),
According to (6)), the state of the hand hj is determined by the equations (12) and (1).
It is updated (predicted) as shown in 3). Note that the expression (1)
Σi in 2) and (13) indicates that the sum of all traceable images (images that do not cause occlusion) is calculated for one hand.

[0039]

(Equation 4)

Thus, the three-dimensional position and velocity of the observation model are updated. Note that the direction can be similarly predicted.

The Kalman filter gives a predicted value in a Gaussian distribution. The viewpoint selecting unit 4 shown in FIG. 1 associates a predicted position with a feature point. More specifically, the Gaussian distribution N (/ Xhj, t, / Shj, t) indicating the predicted position is weakly perspective projected on the image. Then, the proximity between the projected distribution and the feature point is calculated using the Mahalanobis distance. The filter is updated with the feature points closest to the projected distribution.

In order to explain the effectiveness of the tracking processing unit 8 and the viewpoint selecting unit 4, experiments shown in FIGS. FIG. 7 and FIG. 8 are diagrams showing experimental results of tracking the right hand and the left hand in a continuous sequence. Five cameras are arranged, and a scene in which the right hand and the left hand cross at a small distance in the X-axis direction is photographed. 7A shows a tracking result in the X-axis direction, and FIG. 7B shows a tracking result in the Y-axis direction.
(C) shows the tracking results in the Z-axis direction, respectively.
From FIG. 7A, it can be seen that the hand gesture recognition apparatus 1000 is tracking the right hand (symbol “+”) and the left hand (symbol “◇”) that are close to each other.

FIG. 8 is a diagram showing the processing result of the viewpoint selecting unit 4 with respect to FIG. The vertical axis in FIG. 8 indicates a camera, and the horizontal axis indicates a frame. In FIG. 8, each symbol is
The camera selected in each frame is shown. In FIG. 8, three viewpoints are selected in each frame. In the drawing, the symbol “x” and the symbol “□” indicate the viewpoints used for the shape recognition of the right and left hands, respectively, in later processing. By tracking the hand while switching the camera according to the predicted position of the hand, it can be seen that the two hands are properly tracked.

Next, the rotation angle detector 31 described with reference to FIG. 4 will be described. Since the position of the hand and the direction of the main axis have already been obtained, the rotation angle detection unit 31 estimates the rotation angle around the main axis of the remaining hand from the skeleton value of the center of gravity based on a predetermined palm model. FIG. 9 shows the rotation angle detector 3.
FIG. 2 is a diagram for describing detection of a rotation angle using a palm model in FIG. Referring to FIG. 9, an ellipsoid is used as a palm model. The skeleton value of the center of gravity is defined as the width s of the ellipsoid on the image captured by the camera 2 # i. Assuming the weak perspective transformation, the relationship between the angle θ between the optical axis and the long axis of the rotation axis of the palm model and the skeleton value s of the center of gravity follows Expression (14). L in equation (14) is the distance between the position of the hand and the camera 2;
a and b represent constants. Here, θ means the rotation angle of the palm. Note that the constants a and b are determined in advance using sample data. Assuming that the observations include Gaussian errors, the probability P (s1,..., S for the centroid skeleton values s1,.
The value θ that maximizes k | θ) is the rotation angle around the main axis of the hand (see equations (15) to (16)).

[0045]

(Equation 5)

This method eliminates the need for stereo matching in posture estimation. This increases the stability of the estimation. Note that the skeleton value at the center of gravity is hardly affected by shape changes such as finger bending. For this reason, the rotation angle around the main axis of the hand calculated by the rotation angle detection unit 31 is
It has the feature that it is stable against shape changes.

FIG. 10 is a diagram showing experimental results for explaining the effectiveness of the rotation angle detecting unit 31. In FIG. The rotation angles were detected for five different hand shapes shown in the black and white image of FIG. 10B. Each dot on the graph of FIG. 10A indicates the angle detected by the rotation angle detection unit 31.
The dotted lines indicate the rotation angles measured using a magnetic sensor for reference. As shown in FIG. 10A, it is understood that the rotation angle detection unit 31 can detect the rotation angle without being affected by the shape of the hand.

Next, the viewpoint selecting section 32 described with reference to FIG. 4 will be described. Based on the rotation angle output from the rotation angle detection unit 31, the viewpoint selection unit 32 selects a camera having an optical axis that is most perpendicular to the palm from viewpoints (cameras) in which occlusion has not occurred. . More specifically, the normal vector N of the palm as shown in FIG. 11 can be determined by determining the fingertip direction (main axis) of the hand and the rotation angle about the main axis of the hand. As shown in FIG. 11, when the optical axis vector of the camera 2 # i is Ci, the angle θi that the optical axis vector Ci forms with the normal vector N
Is selected. Since occlusion between fingers is most unlikely to occur when a palm is photographed from the front, occlusion can be avoided by selecting such a viewpoint.

FIG. 12 is a diagram showing experimental results for explaining the effectiveness of the viewpoint selecting unit 32. In the experiment shown in FIG. 12, three cameras were arranged at equal intervals, and the posture of the hand was changed around the center camera (the symbol camera 1 in the figure). FIG. 12A shows a rotation angle detected in each frame, and FIG. 12B shows a camera selected corresponding to FIG. 12A. From this result, it can be understood that the most appropriate one camera is sequentially selected from the three cameras as the rotation angle of the palm changes.

Next, the hand shape recognition section 33 described with reference to FIG. 4 will be described. The hand shape recognition unit 33 determines a corresponding hand shape by a P-type Fourier description element using the image selected by the corresponding viewpoint selection unit 32.

FIG. 13 is a block diagram showing the configuration of the hand shape recognition unit 33. As shown in FIG. 13, the hand shape recognition unit 33 includes a contour extraction unit 81 that extracts a contour of the hand in the image based on the image of the hand from the camera selected by the viewpoint selection unit 32, The outline of the hand extracted by the line extraction unit 81 is corrected to the outline of the hand that would have been obtained if the hand was photographed from a direction that matches the normal vector calculated by the rotation angle detection unit 31. Contour correction unit 82
And a P-type Fourier description unit 83 that describes the outline of the hand extracted by the outline extraction unit 81 and corrected by the outline correction unit 82 using a P-type Fourier descriptor. A feature vector calculation unit 84 that calculates a feature vector having a Fourier coefficient of a lower order than a predetermined order as a vector component among a plurality of Fourier coefficients included in the type Fourier descriptor, and a relation between the shape of the hand and the feature vector. Based on a number of sample images obtained by capturing a hand with multiple known shapes for learning, define the probability that the captured hand shape would have been obtained if it had a known shape Probability definition unit 85 and probability definition unit 8
And a shape selecting unit 86 that selects, as a hand shape, a known shape having the highest probability of obtaining the feature vector calculated by the feature vector calculating unit 84 among the plurality of known shapes based on the probability defined by the step S5. . That is, the feature vector calculation unit 84, the probability definition unit 85, and the shape selection unit 86 specify a hand shape based on the P-type Fourier descriptor from the P-type Fourier description unit 83, and output the shape as a recognition result. .

Next, the operation of the thus configured hand shape recognition unit will be described. The contour extraction unit 81 extracts a contour of the hand from the image of the selected camera 2 # i. Here, only the outline of the finger portion that particularly characterizes the hand shape is extracted. As described above, since the input binary image has been subjected to the rotation transformation of the angle γi so that the fingertip is directed upward as preprocessing for the detection of the center of gravity (feature point), as shown in FIG. By extracting the contour of only the portion above the center of gravity, the shape of the finger portion can be obtained.
More specifically, a pixel adjacent to a black pixel among white pixels located above the center of gravity on a silhouette image as shown in FIG. 14 is extracted as a contour line. Here, the outline is extracted clockwise, and the coordinates of each extracted pixel are set to (a (t), b (t)) (t = 0,..., M−
1, m is the total number of pixels constituting the contour line).

The contour line extracted in this way is composed of the normal vector N of the palm and the optical axis vector C of the camera 2♯i.
In response to a change in the angle θi made by i, the image is deformed by projection. Therefore, the contour correction unit 82 corrects the extracted contour of the hand to a contour that would have been obtained if the palm was photographed from the front. More specifically, assuming that θi << 90 ° as a result of the camera selection and assuming a weak perspective transformation, the observed image has a camera normal vector N as shown in FIG. Is reduced by cos θi times in the direction of Ni ′ projected on the image pickup surface (the angle between the horizontal axis x _Ci in the image and φi).
Therefore, the coordinates (a (t),
b (t)) is corrected by the following equation (17), and (a ′)
(T), b '(t)) (t = 0,..., M-1).
Here, R (α) is a rotation transformation matrix of the angle α.

[0054]

(Equation 6)

In the P-type Fourier description unit 83, the extracted contour is approximated by a polygonal line, and is described by a P-type Fourier descriptor. The P-type Fourier descriptor was proposed by Kamisaka (Yoshinori Uesaka: New Fourier descriptor applicable to open curves, IEICE (A), Vol.J67-A, No.3, PP.166-
173 (1984).), Which has excellent features in shape recognition, such as being able to describe various patterns with few parameters, being invariant in translation and scaling, and being applicable to open curves. (Toshiya Ito, Tatsuya Hirata, Naohiro Ishii: Shape Recognition and Processing of Parts Using Fourier Descriptors, IEICE (D), Vol.J71-D, No.6, pp.1065-1
073 (1988).), Character Recognition (Tomohiko Otomo, Kenichi Hara: Online Handwritten Kanji Character Recognition Using P-Type Fourier Descriptors, Theory of Information Processing, Vol.34, No.2, pp.281-288 (1993) ), Human Profile Recognition (Tsunehiro Aihara, Kenji Ohgami, Yasushi Matsuoka: Number of active components of P-type Fourier descriptor in human profile recognition, IEICE (D-II), Vol.J74-DII, No.10, pp.1486-
1487 (1991).).

In the P-type Fourier descriptor, a two-dimensional curve is considered as a sequence of points on a complex plane, and is approximated by a polygonal line composed of line segments having the same length. Each vertex of the polygonal line is represented by z (j) = x (j)
+ Iy (j) (| z (j + 1) -z (j) |
= Δ; j = 0,..., N−1), each polygonal line is normalized by its length δ, and the P of the polygonal line represented by the equation (18) is obtained.
Obtain the expression w (j). Then, a Fourier coefficient c (k) is obtained by Expression (19) by discrete Fourier expansion of w (j).

[0057]

(Equation 7)

Here, a set of coefficients ｛c (k); k = −
N,..., 0,..., N} are P-type Fourier descriptors at the time of N. In this embodiment, the magnitude of c (k) | c (k)
Is used to describe the outline.

The feature vector calculator 84 calculates (2N + 1) (for example, N = 4) orders lower than a predetermined order among the Fourier coefficients included in the P-type Fourier descriptor {c (k)}.
The feature vector V defined by equation (20) is calculated using the Fourier coefficients of 4 or 5). Here, A ^t represents the transposed vector of the vector A.

[0060]

(Equation 8)

Prior to recognizing the hand shape, the probability definition unit 85 learns the correspondence between some known shapes of the hand and the feature vector, and the probability P (V |) that the feature vector V is observed for each shape s. s) is defined by the normal distribution represented by equation (21).

[0062]

(Equation 9)

Here, Vms is an average vector (for example, a vector of a large number of feature vectors) of the feature vector V of the known shape s calculated based on a large number of sample images obtained by photographing a hand having a known shape. The average value of the components as a vector component). Σs is a row variance matrix. Such a probability function is prepared in advance for each known shape.

The shape selecting section 86 selects a known shape having the highest probability of obtaining the calculated feature vector V as a hand shape. That is, the calculated feature vector V
Is obtained for all the known shapes s, and the known shape s at which the obtained probability P (V | s) is maximized is recognized as the hand shape in this case. .

As described above, since the optimal camera used for shape recognition is selected based on the posture of the hand, occlusion can be avoided and the shape of the hand can be recognized stably. Also, in order to describe the outline of the hand extracted from the image, a P-type Fourier descriptor that is invariant to translation, enlargement / reduction, etc. in the image of the shape is used. Thereby, the hand shape can be stably recognized regardless of the posture of the hand.

Since the extracted contour of the hand is corrected to the contour of the hand that would have been obtained if the hand was photographed from a direction that coincides with the vertical direction of the palm, the selected camera is Can be accurately recognized even when the image is not taken directly in front of the camera.

Since the feature vector V is calculated using the Fourier coefficients of a relatively low order, high-frequency noise (appearing as higher-order Fourier coefficients) contained in the extracted hand contour is removed. Only true hand contours (appearing as low-order Fourier coefficients) can be accurately identified. As a result, the hand shape can be more stably recognized.

In the above equation, the difference between the front and back of the palm is not taken into account in the camera selection. Therefore, two types of image input can be performed for the same hand shape. Such determination of the left and right of the contour is represented by Expression (22) of the obtained P-type Fourier descriptor. Where c ′
(K) indicates the coefficient after inversion, and the upper line of c (-k) indicates its conjugate complex number. Therefore, considering the judgment of the front and back of the hand,
Define V ^{* in} which the components of the feature vector V are rearranged as represented by equation (23).

[0069]

(Equation 10)

In recognizing the hand shape, P (V | s) and
And P (V^*| S) to evaluate the input image
Corresponds to the difference between the front and back. That is, extracted from the selected image
V, V from the contour^*And calculate all the shapes s
P (V | s), P (V ^*| S)
Is selected, and the corresponding shape s is set as a recognition result.

[0071]

EXAMPLES For the following experiments, images were taken as follows. In the experiment, as shown in FIG. 6, five cameras are arranged at predetermined intervals on the concentric circle toward the center, and the hand is photographed. Images of about 600 frames were taken for each of the seven types of hand shapes 1 to 7 shown in FIG.
The captured image includes rotation and movement of the palm. Images obtained by the five cameras were recorded on five write-once video disks, and then processed frame by frame by frame advance.
Of all the frames, 300 frames were used for learning, and the rest were used for recognition experiments.

At the time of parameter learning, a feature vector V ₀ was extracted from the image of the processing frame for each shape by the method described in the above embodiment. For two frames later, feature vector V, of V ^*, the Euclidean distance is recorded by selecting those closer to the feature vector V _0. The probability distribution described in the above embodiment was determined based on the feature vectors of 300 samples obtained for each shape.

FIG. 16 shows a shape recognition result for an image of about 300 frames for each shape for a recognition experiment. As shown in FIG. 16, a stable recognition result of 87% or more was obtained for all the shapes, and particularly 99% or more for the shapes 1, 2, 6, and 7. Thus, it can be said that the effectiveness of the hand gesture recognition device according to the present invention has been demonstrated. In addition,
It is considered that erroneous determination due to self-occlusion is caused for shapes shape 3, 4, and 5, but such erroneous determination due to occlusion can be avoided by increasing the number of viewpoints and increasing the opportunity for viewpoint selection. .

Using the above-described hand gesture recognition device, a system capable of interactively operating a virtual space was constructed. By issuing a command by hand gesture, a user can create a three-dimensional virtual object having a basic shape and perform operations such as arranging, deleting, and scaling the virtual object.

In the experiment, the command shown in FIG.
Six types of hand shapes shown in FIG. The “hand position” in FIG. 17 indicates whether the user's hand is inside or outside the virtual object. “Shape transition” in FIG. 17 means a change in the shape of the hand. Shape shap
After the presentation of e7, the command is executed or started by presenting the other shapes 1-6.

In the virtual space displayed by the computer graphics, as shown in FIGS. 18 to 19, an instruction pointer indicating the three-dimensional position of the user's hand is drawn. By moving the pointer inside the virtual object to be operated by moving the object, the operation object can be directly designated.

The “create” (specify the shape of the virtual object along the trajectory of the hand) command closes the hand (shape 7 →
1). Then draw a trajectory by moving your hand. Opening the closed hand and ending the “create” command creates a virtual object.

In all commands except the “create” command, the hand is moved inside the virtual object in the three-dimensional space.

The three commands are a "delete" command (to delete a virtual object), a "combination" command (to combine a plurality of adjacent virtual objects), and a "division" (to release the combination of virtual objects).
The command is executed by moving the pointer into the operation target and presenting a shape transition corresponding to each command.

A “grasping / moving” command (grasping and moving the virtual object), an “enlarge / reduce” command (changing the size of the virtual object), a “color / texture changing” command (color / texture of the virtual object surface) For changing the texture), following the instruction by the pointer and the corresponding shape transition, moving the hand further changes the object parameters (position, size, color, etc.) to be changed according to the amount of movement. I do. To change the parameters, change the hand shape to shape 7
The process is terminated by making a transition again to.

All commands except the "stretch" (change the length of the virtual object) command can be issued independently of the right hand and the left hand. In the “extension” command, the virtual object is grasped by the right hand and the left hand, and the hand is relatively moved.

As described above, a hand gesture recognition device using a large number of cameras and a virtual scene generation system using the same have been described. The present system reconstructs hand movements (three-dimensional position, hand movements) from a small number of optimal images extracted from images of many viewpoints. More specifically, the best image that does not cause occlusion according to the tracking result is
Selected for each hand. With this viewpoint selection mechanism, the occurrence of self-occlusion and occlusion between hands can be effectively reduced without using a sophisticated three-dimensional model.

Further, this makes it possible to reduce the cost on the computer for generating the model and reconstructing the model.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[0085]

As described above, according to the present invention, an optimum camera image is selected from a plurality of cameras based on the prediction by the Kalman filter, and the hand movement is tracked. For this reason, it becomes possible to avoid occlusion and to track and recognize the movements of a plurality of hands stably and in real time.

Also, based on the feature points of the selected image,
The image to be processed is further narrowed down so as to avoid occlusion between hands. This makes it possible to track and recognize the movements of a plurality of hands stably and in real time.

Further, in recognizing the hand shape, a P-type Fourier descriptor which is invariant to translation, enlargement / reduction, etc. in the image of the shape is used, so that the hand shape can be stably recognized at low calculation cost. be able to.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a main part of a hand gesture recognition device 1000 according to Embodiment 1 of the present invention.

FIG. 2 is a block diagram showing an outline of a configuration of a feature extracting unit 6 shown in FIG.

FIG. 3 is a diagram for explaining the operation of a feature extraction unit 6;

FIG. 4 is a block diagram illustrating an outline of a configuration of a tracking processing unit 8 illustrated in FIG. 1;

FIG. 5 is a diagram for describing an observation model in a Kalman filter.

FIG. 6 is a diagram showing an experimental environment for explaining the effectiveness of the tracking processing unit 8 and the viewpoint selecting unit 4;

FIG. 7 is a diagram showing an experimental result of tracking a right hand and a left hand in a continuous sequence.

FIG. 8 is a diagram showing an experimental result of tracking a right hand and a left hand in a continuous sequence.

FIG. 9 is a diagram for explaining detection of a rotation angle using a palm model in a rotation angle detection unit 31;

FIG. 10 is a diagram showing experimental results for explaining the effectiveness of the rotation angle detection unit 31.

FIG. 11 is a diagram for describing an operation of a viewpoint selection unit 32.

FIG. 12 is a diagram showing experimental results for explaining the effectiveness of the viewpoint selecting unit 32.

13 is a block diagram illustrating a configuration of a hand shape recognition unit 33. FIG.

FIG. 14 is a diagram for explaining the operation of a contour line extraction unit 81.

FIG. 15 is a diagram for explaining a hand shape used in a recognition experiment.

16 is a diagram illustrating a recognition result for the hand shape illustrated in FIG. 15;

FIG. 17 is a diagram illustrating a relationship between a command and a hand in an interactive system based on a hand gesture recognition device.

FIG. 18 is a diagram illustrating a state of an instruction pointer in an interactive system based on a hand gesture recognition device.

FIG. 19 is a diagram illustrating a state of an instruction pointer in an interactive system based on a hand gesture recognition device.

[Explanation of symbols]

 2, 2♯1-2♯n Camera 4, 33 Viewpoint selection unit 6 Feature extraction unit 8 Tracking processing unit 10, 12 Tracking unit 14 Gesture recognition unit 20 # 1-20♯n Area division unit 22 # 1-22♯n Spindle detection unit 24 # 1 to 24 # n Rotation conversion unit 26 # 1 to 26 # n Feature point calculation unit 30 # 1 to 30 # 2 Three-dimensional position / direction detection unit 31 # 1 to 31 # 2 Rotation angle detection unit 32 ♯1-32♯2 viewpoint selection unit 33 ♯1-33♯2 hand shape recognition unit 81 contour extraction unit 82 contour correction unit 83 P-type Fourier description unit 84 feature vector calculation unit 85 probability definition unit 86 shape selection unit 1000 Hand gesture recognition device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06F 15/70 350H 460B F-term (Reference) 2F065 AA04 AA17 AA37 AA39 AA51 CC16 FF01 FF04 JJ03 JJ05 JJ19 JJ26 QQ16 QQ17 QQ27 QQ31 QQ42 5B057 AA20 BA02 CA01 CA12 CD03 DA08 DB02 DB05 DB08 DC05 DC09 DC17 DC34 5L096 AA07 BA08 CA05 EA16 FA06 FA60 FA66 FA67 HA05 HA09 JA11

Claims (3)

[Claims] 1. A hand gesture recognition device for recognizing a plurality of hand movements, comprising: a plurality of image pickup means for photographing the plurality of hands from different directions to obtain a plurality of images; First selecting means for selectively outputting an image for recognizing a plurality of hand movements; extracting feature points of the image selected by the first selecting means; and an image to be tracked based on the feature points And a plurality of tracking processing means provided for each of the plurality of hands and tracking the movement of the corresponding hand, wherein each of the plurality of tracking processing means Prediction means for predicting the state of the hand using the feature points of the target image; calculating the normal direction of the hand based on the image to be tracked; the direction closest to the normal direction Image obtained as a result of hand shooting from A second selecting unit for selecting, and a recognizing unit for recognizing a hand shape based on the image selected by the second selecting unit, wherein the first selecting unit includes the A hand gesture recognition device for selecting an image in which a feature point is located near a predicted state. 2. The method according to claim 1, wherein the feature extraction unit detects occurrence of occlusion based on a ratio between the position of the extracted feature point and a width of a hand, and deselects an image in which the occlusion is detected. The hand gesture recognition device according to 1. 3. The method according to claim 1, wherein the recognizing means extracts a contour of the hand in the image selected by the second selecting means, and extracts the contour of the hand extracted by the contour extracting means into P. 3. The hand gesture according to claim 2, further comprising: a P-type Fourier description unit described by a type-Fourier descriptor; Recognition device.