JP2934190B2 - Hand gesture recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a hand gesture recognition device, and more particularly, to a hand gesture recognition device for recognizing a posture of a hand in real time using a palm model in which the palm is modeled in advance.

[0002]

2. Description of the Related Art In recent years, research on a virtual space presentation system has been advanced with the aim of constructing a new man-machine interface. In many of the virtual space presentation systems proposed so far, it is necessary for a user to wear a special device such as an HMD (head mounted display) for displaying a stereoscopic image. Regarding hand gesture recognition, it is common to wear special gloves to which a sensor is added, and the burden on the user, such as the complexity of putting on and taking off, is large. In contrast,
Aiming at realizing a system equivalent to the above without wearing it, aiming at hand gesture recognition, a vision-based hand gesture recognition device has been proposed and its development is underway.

[0003] Nakajima et al. Tracked finger movement with a single eye using a model consisting of a fingertip and the base of a finger (Masayuki Nakajima, Yuu Shiba, Finger movement detection method for constructing a virtual reality world, Graphics and CAD67). -6, pp. 41-46, 199
4). However, a special marker needs to be attached to detect a necessary feature point, and since it is a single eye, the restoration accuracy is not high.

[0004] Iwai et al. Performed three-dimensional reconstruction with a single eye using time-series binary images (Yoshio Iwai, Yasushi Yagi, Masahiko Yauchida, Fist movement reconstruction for virtual reality, IEICE Technical Report PRU93). −57, pp. 23-30, 199
3). This method can recover the posture by associating the frames with each other, but increases the accumulated error.

[0005] Okamura et al. Prepared a feature amount such as the length of a finger for some gestures in advance, and performed gesture recognition by collating it with the features (Izumi Okamura, Akimoto Sumaki,
Non-contact hand shape recognition and its application, IEICE HC93-6
pp. 31-38, 1993). In this method, the main focus is on recognition of necessary gestures, and continuous detection of hand posture is not performed.

On the other hand, Ishibuchi et al. Used a relatively simple model of expressing the hand shape by the position of the center of gravity and the fingertip point in the plane, and used a plurality of cameras to track the hand gesture, thereby attaching a marker. We have constructed a system that can perform processing including motion reconstruction in real time without the need for realization (Ishibuchi et al., Real-time hand shape recognition using image processing and application to man-machine interface, Proceedings of IEICE Autumn Meeting, pp.
1-132-1-132, 1991, Ishibuchi et al., Real-time hand shape recognition using a pipeline type image processing device, Proceedings of IEICE Spring Conference, pp. 139-138. 1-21-1-1
291, 1992). Since this system uses a plurality of cameras, it is possible to cope with a hand rotation that was not taken into account by the methods of Nakajima et al., Iwai et al., And Okamura et al.

The present applicant has already proposed a hand gesture recognition device that can accurately recognize the posture of the hand in real time even when the user is wearing short-sleeved clothes (Akira Utsumi, Ts.
utomu Miyasato and Fumio Kishino, Multi-camera han
d pose recognition systemusing skeleton image ， In
4th IEEE International Workshop on Robot and Human
Communication, pp. 219-224, 1995, and Japanese Patent Application No. 7-334860. In this device, first, a skin color region is cut out from the color and luminance information of an image as a hand region,
Obtain a binary image. After distance conversion of the obtained binary image, a candidate for the center of gravity is selected from the local maximum point, and a candidate for the wrist part is selected from the local minimum point of the horizontal projection. The position of the center of gravity is determined by performing some confidence evaluations. The three-dimensional direction of the hand is obtained by associating the main directions of the edges obtained in the binary image.

[0008]

However, in the above-described conventional apparatus, since it is assumed that the fingertip points can be corresponded for posture detection, the posture is stabilized when the user bends all the fingers. There was a problem that it was not possible to ask for it.

The present invention has been made to solve such a problem, and an object of the present invention is to provide a hand gesture recognition device capable of stably recognizing a posture of a hand regardless of the shape of the hand.

[0010]

A hand gesture recognition apparatus according to the present invention recognizes a posture of a hand in real time using a palm model in which the palm is modeled in advance, and captures hands from different angles. A plurality of photographing means, a distance converting means for calculating a center-of-gravity skeleton value indicating a distance from the center of gravity of the hand to the contour of the hand on the image of the hand photographed by the photographing means, and a center-of-gravity skeleton calculated by the distance converting means Maximum likelihood estimating means for applying the value to the palm model and estimating the rotation angle about the main axis of the hand by the maximum likelihood method.

The maximum likelihood estimating means includes: a probability calculating means for applying a rotation angle candidate predicted as a rotation angle to a palm model to calculate a probability that the barycenter skeleton value can be observed; And a rotation angle determining means for determining a rotation angle at which the maximum value is obtained as a rotation angle.

[0012]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

First, a hand shape model used in the hand gesture recognition device according to the embodiment of the present invention will be described. In this device, the position and posture of the hand are expressed by the center of gravity and the direction of the palm, and the bending of each finger is expressed by the positional relationship between the center of gravity and each fingertip point. For details, see "Onishi et al., Hand Shape Description for Hand Gesture Recognition, Technical Report of the Institute of Television Engineers of Japan, Vol. 15, No. 25, pp. 25-29,
May 1991 ". In FIG. 1, Oh indicates the center of gravity of the hand 1, and Xh indicates the direction of the hand 1. These are approximated in the direction of the center of gravity of the palm area of the hand 1 in the image and the direction of the edge of the palm of the hand 1. By combining information from a plurality of cameras, the position and direction of the hand 1 can be calculated. Also, assuming that the characteristic points of the palm are in the same plane, the movement of the fingertip point is expressed as the movement of the projection point on the obtained plane.

The hand gesture recognition device according to this embodiment is:
As shown in FIG. 2, a plurality of cameras 11 to 1k for photographing hands from different angles and an area for dividing a skin color area in an input image obtained by the cameras 11 to 1k using color information and luminance information. An average edge direction is determined from the results of the dividing units 21 to 2k, a Sobel filter, and the main axis detecting units 31 to 3k, which use the average edge direction as a two-dimensional hand direction (main axis), and the main axis detecting units 31 to 3k. Rotation conversion units 41 to 4k that rotate the image so that the fingertips face upward based on the obtained hand direction, and a distance indicating the shortest distance from each of the pixels forming the hand image to the contour of the hand image Distance conversion units 51 to 5k for calculating conversion values (skeleton values);
Referencing the certainty that can be the center of gravity of the hand based on those skeleton values, determining the three-dimensional center of gravity Oh of the hand 1, and stereo-matching the hand directions obtained by the plurality of cameras 11 to 1k. The three-dimensional center-of-gravity point / direction detection unit 6 determines the direction Xh of the hand 1 on the three-dimensional basis, and the hand based on the distance conversion value (centroid skeleton value) at the center of gravity of the hand obtained by the plurality of cameras 11 to 1k. 1 spindle (Xh)
A rotation angle detection unit 7 for determining the rotation angle r around the camera, and a camera selection unit for selecting a camera having an optical axis that is most perpendicular to a plane (hand plane) having the obtained center of gravity position, hand direction and rotation angle. 8, a fingertip point detection unit 9 that extracts a fingertip candidate point by performing labeling processing on an image obtained from the selected camera and projects it on the hand plane, a rotation angle r and a fingertip point from the rotation angle detection unit 7 And a hand model restoring unit 10 for restoring the model of the hand 1 based on the

Next, the operation of the hand gesture recognition device thus configured will be described. The images obtained by the cameras 11 to 1k are provided to the area dividing units 21 to 2k, respectively. In the area division units 21 to 2k, the skin color area of the input image is separated from other areas using the information of color and intensity, and a binarized image as shown in FIG. 3A is obtained. Spindle detectors 31 to
In 3k, the main axis of the edge segment whose edge has been detected is detected as the direction of the hand 1. The rotation converters 41 to 4k rotate the image so that the axis of the image coincides with the vertical axis. In the distance conversion units 51 to 5k, a distance conversion image as shown in FIG. 3B and a projected image as shown in FIG. 3C are obtained.

The three-dimensional center-of-gravity point / direction detection unit 6 extracts the maximum point of the distance-converted image as a candidate for the center of gravity, and extracts the minimum point of the projection as a candidate for the wrist. In FIG. 3B, a pixel having a larger skeleton value is represented as black, and a pixel having a smaller skeleton value is represented as white.
Therefore, the pixels gradually become darker as the distance from the contour of the hand increases. From the correspondence between all sets of images of the obtained center of gravity candidate and the wrist candidate (for the wrist candidate, the midpoint of the hand region at the vertical position with respect to the obtained main axis is used as a candidate), three-dimensional processing is performed by confidence processing. The upper center of gravity is determined. When three-dimensional center of gravity is determined, the center of gravity skeleton values s ₁ ~s _k on each image is obtained.

[0017] In the rotation angle detecting unit 7, based on a predetermined palm model that models the palm of hand 1, the rotation angle r around the main axis of the hand 1 is estimated from the centroid skeleton values s ₁ ~s _k. Here, in order to simplify the problem, the ellipsoid model 2 shown in FIG. 4 is used to represent the palm of the hand 1 excluding the fingers when mounting. Here, the camera 11
The centroid skeleton value is used as the width s of the ellipsoid on the image captured at (（1k). Assuming that the weak perspective transformation is performed, an angle formed between the optical axis of the camera and the normal of the palm is θ, and the observed center-of-gravity skeleton value s follows Expression (1).

[0018]

(Equation 1)

Here, L is the distance from the center of gravity of the hand to the camera, and a and b are constants.

Assuming that the observation involves a Gaussian error, the probability p (s | θ) of observing the center of gravity skeleton value s is expressed by the following equation (2).

[0021]

(Equation 2)

Here, the constants a and b and the error variance σ _s are estimated from the learning data. The posture of the hand 1 can be represented by Euler angles (a, e, r) as shown in FIG. Here, a represents an azimuth angle, e represents an elevation angle, and r represents a rotation angle. Since a and e are determined by the main axis direction Xh of the hand 1, only the rotation angle r around the main axis is an unknown number.

The normal vector N of the palm is given by the following equation (3).
Is represented by _{N = R z (a) R} y (e) R x (r) e z ... (3) _{where, R z, R y, R} x each represent a rotation matrix z, y, the x-axis, e _z Indicates a unit vector in the z-axis direction.

Assuming that the rotation angle of the hand 1 about the main axis is r,
The angle θ _ci (r) formed by the optical axis vector C of the i-th camera and the normal vector N of the palm is represented by the following equation (4).

[0025] ^{_{θ ci (r) = cos -1}} N T C ... (4) In addition, N ^T in the equation (4) denotes the transpose vector of a normal vector N.

The probability P (W | r) that the set of centroid skeleton values W (s ₁ , s ₂ ,..., S _k ) is observed by _k cameras 11 to _{1 k} is given by the following equation from the above equation (2). It is represented by (5).

[0027]

(Equation 3)

Here, s _i indicates the center-of-gravity skeleton value observed by the i-th camera. Let r that maximizes P (W | r) be the estimated value of the rotation angle.

The rotation angle detector 7 for detecting the rotation angle r by the above-described method has a rotation angle r as shown in FIG.
Multiple calculation unit 711 for calculating a _{| (θ Ci (r) s} i) the center of gravity Skeleton value s ₁ ~s _k by applying the rotation angle candidate r _C to ellipsoidal model 2 is expected probability P may be observed as
S ₁ , s by k cameras 11 to ₁ k
_2, ..., s _k that centroid skeleton value probability P combination may be observed in | a calculating unit 72 for calculating the (W r), the calculated probability P | rotation angle candidate r _C to (W r) becomes maximal Is determined as a rotation angle r.

For example, the calculation unit 711 determines that the rotation angle r is r
_{If C} , the probability P (s ₁ | θ _C1 (r _C )) of obtaining the centroid skeleton value s ₁ from the camera 11 is calculated based on the ellipsoid model 2.

Referring now to FIG. 6, it will be described that there is a correspondence between the center of gravity skeleton value and the rotation angle r of the hand 1. As shown in FIG. 6A, when the normal line of the hand 1 coincides with the optical axis of the camera 11 (１1k),
(1k) on the image of the hand
Since the shortest distance from the center of gravity of the hand 1 to the contour of the hand 1 is maximized, the skeleton value of the center of gravity is also maximized. FIG. 6A shows an ellipsoidal model 2 of the palm of the hand 1 photographed in this manner. The ellipsoid model 2 in this case is observed as a perfect circle. Therefore, the shortest distance from the center of gravity Oe of the ellipsoidal model 2 to the contour is maximized,
The center of gravity skeleton value in this case is also maximized.

As shown in FIG. 6B, if the normal line of the hand 1 forms an acute angle with respect to the optical axis of the camera 11 (1k), the hand imaged by the camera 11 (１1k) is obtained. Since the shortest distance from the center of gravity of the hand 1 to the outline of the hand 1 on the image 1 is shorter than in the case of FIG. 7A, the skeleton value of the center of gravity in this case is smaller than in the case of FIG. The ellipsoid model 2 in this case is observed in a form compressed in the horizontal direction in the figure as compared with the case of (a). Accordingly, the shortest distance from the center of gravity Oe of the ellipsoid model 2 to the contour is shorter than in the case of (a), and the skeleton value of the center of gravity in this case is smaller than in the case of (a).

As shown in FIG. 6C, when the normal of the hand 1 is perpendicular to the optical axis of the camera 11 (〜1 k), the hand 1 photographed by the camera 11 (〜1 k) is used. Since the shortest distance from the center of gravity of the hand 1 to the contour is minimized on the image of (1), the skeleton value of the center of gravity in this case is also minimized. The ellipsoid model 2 in this case is observed in a more compressed form in the horizontal direction on the figure than in the case of (b), as shown in (c) of FIG. Therefore, the shortest distance from the center of gravity Oe of the ellipsoid model 2 to the contour is minimized,
The centroid skeleton value in this case is also minimized.

[0034] Is this way models the palm ellipsoidal model 2, the center of gravity skeleton values actually obtained from k cameras 11~1k s ₁ ~s _k is obtained when what rotation angle r Is estimated by the maximum likelihood method using the ellipsoid model 2.

[0035] As described above, according to the gesture recognition apparatus by using the ellipsoidal model 2 models the palm from the center of gravity skeleton values s ₁ ~s _k to estimate the rotation angle r of the hand, the hand 1 The posture can be stably recognized regardless of the shape of the finger.

[0036]

EXAMPLES In order to clarify the effectiveness of the present invention, the following simulation experiments were performed. As an experimental environment, three cameras were used for image input, and a dark curtain was used as a background so that a hand region could be cut out only by luminance. As the learning data, an image of about 300 frames including the rotation of the hand was used together with the rotation angle data detected by the magnetic sensor. FIG. 7 shows the distribution of the learning data. In the figure, the solid line indicates the estimated ellipsoid model, and the distance between the solid line and the broken line indicates the magnitude of the variance σ _{s in} the above equation (2). The input images used in the experiment were taken with long-sleeved clothes so that only the palms were divided into areas, and with the sleeveless clothes so that only the upper arm was included. Using. The hand was rotated in three ways: one with the main axis of the hand kept horizontal, one with the hand turned up about 30 °, and one with the hand turned up about 60 °.

The experimental results are shown in FIGS. FIG.
Is an image obtained by processing the condition of keeping the main axis horizontal in the image of wearing long-sleeved clothes. In the drawing, a solid line shows a rotation angle obtained by a magnetic sensor as a comparative example, and a broken line shows a rotation angle obtained by this embodiment. Hand rotation
This was performed for the six shapes shown in the lower part of FIG. It can be seen that the rotation angle is appropriately detected for any hand shape.

FIG. 9 also shows the results for other cases. As can be seen from this example, it has been clarified that the rotation angle can be detected stably irrespective of differences in clothes and changes in the direction of the main shaft.

[0039]

According to the hand gesture recognition apparatus of the present invention, the center of gravity skeleton value obtained from the hand image is applied to the palm model in which the palm is pre-modeled, and the rotation angle around the main axis of the hand is determined by the maximum likelihood method. Therefore, the rotation angle can be detected regardless of the shape of the hand, and as a result, the posture of the hand can be stably recognized.

[Brief description of the drawings]

FIG. 1 is a diagram showing a hand shape model used in a hand gesture recognition device according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an overall configuration of a hand gesture recognition device according to the embodiment of the present invention.

3A and 3B are schematic diagrams illustrating distance conversion and projection processing in the hand gesture recognition device in FIG. 1, wherein FIG. 3A is a binarized image, FIG. 3B is a distance conversion image, and FIG. It is.

4 is a diagram showing an ellipsoidal model of a palm used in the hand gesture recognition device of FIG. 2;

FIG. 5 is a block diagram illustrating a configuration of a rotation angle detection unit in the hand gesture recognition device of FIG. 2;

FIG. 6 is an explanatory diagram showing a principle in which a hand rotation angle can be detected by an ellipsoid model in the hand gesture recognition device of FIG. 2;

FIG. 7 is a graph showing a distribution of learning data used in the embodiment of the hand gesture recognition device of FIG. 2;

8 is a graph showing a result of recognition of a rotation angle by the embodiment of the hand gesture recognition device of FIG. 2;

9 is a graph showing a result of recognition of a rotation angle by the embodiment of the hand gesture recognition device of FIG. 2;

[Explanation of symbols]

 Reference Signs List 1 hand 2 ellipsoid model 6 3D barycentric point / direction detection unit 7 rotation angle detection unit 11-1k camera 51-5k distance conversion unit 711-71k, 72 calculation unit 73 rotation angle determination unit

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-8-263629 (JP, A) JP-A-7-239218 (JP, A) JP-A-7-105371 (JP, A) JP-A-3- 195909 (JP, A) JP-A-7-311849 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G06T 1/00 G06T 7/00

Claims (2)

(57) [Claims] 1. A hand gesture recognition device for recognizing a posture of a hand in real time using a palm model in which a palm is pre-modeled, a plurality of photographing means for photographing the hand at different angles from each other; Distance conversion means for calculating a center of gravity skeleton value indicating a distance from the center of gravity of the hand to the contour of the hand on the image of the hand taken by the means; and a palm model for calculating the center of gravity skeleton value calculated by the distance conversion means. And a maximum likelihood estimating means for estimating a rotation angle of the hand about a main axis by a maximum likelihood method. 2. The maximum likelihood estimating means: a probability calculating means for applying a rotation angle candidate predicted as the rotation angle to the palm model to calculate a probability that the barycenter skeleton value can be observed; The hand gesture recognition device according to claim 1, further comprising: a rotation angle determining unit that determines, as the rotation angle, a candidate having the maximum probability among the candidates.