JPH09245178A - Gesture moving image recognizing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize the same kind of gestures by a single standard pattern without receiving influence from a position by regarding plural frame images constituting a moving image as dots in a vector space and regarding an operation locus formed by the points as a feature parameter. SOLUTION: Based on an idea that the shape of a triangle does not change even if its position varies or rotates, e.g. the operation locus arranging frame image data time sequentially is handled as the feature parameter. At the time of regarding image data at the point of a time as one dot in the vector space, two line segments are formed by the locus of adjacent three points and an angle formed by these two line segments hardly changes even if the position and the angle of the gesture change. Namely at the time of setting an angle formed by P, Q and R in a pattern space to be θPQR when the pattern on the frame image varies P-Q-R, an angle θ'PQR formed by PQ and QR becomes equal with θPQR when the pattern is linear-transformed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gesture moving image recognition method for recognizing a human gesture or hand gesture shown in a captured moving image by an image processing device.

[0002]

2. Description of the Related Art As a research on a method of capturing a human gesture and recognizing the captured moving image by an image processing device (1) Takahashi, Seki, Kojima and Oka: "Spotting
Recognition of HumanGesture from Time-Varying Ima
ges ", Trans. of IEICE (D-II), J77-DII, 8, pp.1552-1
561. (1994). (2) Seki, Kojima, Nagaya and Oka: "Efficient g
esture recognition algorithm based of Continuous D
ynamic Programming ", Proc. of RWC Symposium Technic
al Report, pp47-48. (1995). (3) Nagaya, Seki and Oka: "Gesture Recognition
Using Multiple Resolution Feature ", Technical Repo
rt of IEICE, PRU95-99, pp.121-126, (1995). (4) Yamato, Ohya and Ishii: "Recognizing Human
Action in Time-Sequential Images Using Hidden Mar
kov Models ", Trans. of IEICE (D-II), J76-DII, 12, p
p.2556-2563. (1993). (5) Murase and Nayar: "Learning and Recognitio
n of 3D Object from Appearance ", Technical Report
of IEICE, PRU93-120, pp.31-38, (1994). As a typical example, the system configuration shown in the above document 1 is shown in FIG. In FIG. 1, the TV camera 13 images a gesture of a subject, and a frame grabber 11 acquires a plurality of still images (hereinafter, referred to as frame images) from a moving image obtained as a result of the imaging. The contour line image is extracted by the image processing device 10 so that the image processing device 10 can identify the subject of the frame image. From the image data of the contour line image, various characteristic parameters indicating the characteristics of the gesture are extracted. The moving image can be stored in the real-time disc 12 and the rewritable LD 14 and can be reproduced by the VTR 15.

Various proposals have been made on how to determine the types of characteristic parameters. As an example, it is known that a person as a subject wears gloves of a specific color and the position of the pixel data of the specific color included in the contour line image data is used as the specific parameter. When the reference pattern is created, the characteristic parameters extracted from each still image are arranged in the order in which the frame images are acquired (this is referred to as a characteristic parameter series), and are stored in the image processing apparatus 10 together with identification information indicating the content of the gesture at this time. It is stored as a standard pattern. When the characteristic parameter series extracted from the moving image to be recognized and the characteristic pattern series of the reference pattern are compared using the continuous DP method, and when they match or the most similar one is obtained, the identification information is recognized. As a result, it is output from the image processing apparatus 10.

By performing such image processing, it has become possible for a computer to recognize gestures that were not possible by machine recognition until now.

[0005]

However, the proposals published so far have a problem that the recognition rate is lowered when the position of the hand or the like to be recognized is different from the position of the hand when the standard pattern is sampled in advance. It was More specifically, when a standard pattern is created from a moving image when the hand is shaken horizontally at the head position, and the user tries to recognize the moving image when the hand is shaken near the body, the hand is shaken in the same way. However, since the positions of the hands are different, the characteristic parameter series obtained from the moving image are different, and in the worst case, the recognition result cannot be obtained. Therefore, conventionally, when trying to increase the recognition rate, there is a problem that a plurality of standard patterns must be created by imaging the same type of gesture with different hand positions.

Therefore, an object of the present invention is to provide a gesture moving image recognition method capable of recognizing the same type of gesture with a single standard pattern without being affected by the position.

[0007]

In order to achieve such an object, the invention of claim 1 creates a reference pattern by extracting a characteristic parameter from a moving image obtained by capturing a gesture by an image processing device. In the recognition method of the gesture moving image, the characteristic parameter is extracted from the moving image to be recognized, and the gesture is recognized by comparing the extracted characteristic parameter with the characteristic parameter of the reference pattern. Each of the plurality of frame images that make up the moving image is regarded as a point on the vector space,
It is characterized in that the motion locus formed by the points is used as the characteristic parameter.

According to a second aspect of the present invention, in addition to the first aspect of the invention, a plurality of representative points that characterize the locus shape of the motion locus are selected, and a polygon having the representative points as vertices, It is characterized in that the motion locus is approximated.

The invention of claim 3 is characterized in that, in addition to the invention of claim 2, the representative point is determined based on the maximum and minimum points of the curvature of the motion locus.

The invention of claim 4 is, in addition to the invention of claim 3, continuous DP for comparing the characteristic parameters of claim 1.
It is characterized by using the method of.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

In the present embodiment, the operation locus in which the frame image data is arranged in time series is treated as a characteristic parameter under the idea that, for example, the shape of a triangle does not change even if the position changes or rotates. That is, if the image data at a certain time point is regarded as one point on the vector space, two line segments are formed by the loci of three adjacent points.
The inventors of the present application have found that the angle formed by two line segments hardly changes even if the position or angle of the gesture changes, and based on this finding, the inventors invented the gesture moving image recognition method described below. Hereinafter, the description will be made. 1: Definition of motion locus 1.1 Definition and properties of pattern space Finite area I = {(x, y) | 0 ≦ x <
W, 0 ≦ y <H} in the real-valued function defined above

[0013]

## EQU1 ## Consider a set P of functions satisfying ∫ _I | f (→ x) | ² d → x <∞.

(Function definition of pattern) When f, gεP and α are real numbers, f + g and αf are f (→ x) + g, respectively.
When defined as (→ x) and αg (→ x), P forms a linear space. Although x is a vector, the vector x is expressed as → x for the sake of notation. For a frame image of size W × H, the image pattern can be regarded as a P element by setting the brightness value of this image area coordinate → x to f (→ x). That is, the pattern can be considered as a vector. Therefore, P is called a pattern space.

(Definition of Inner Product) Arbitrary 2 in the pattern space
The dot product of two patterns

[0016]

[Number 2] _{(f, g) = ∫ I} f (→ x) g (→ x) d → x and can be defined, ‖f‖ the norm of f = √ (f,
f) can be introduced naturally. In addition, considering the angle θ between f and g, Formula 3 is defined from the definition of the inner product.

[0017]

(Equation 3)

Next, the properties of the pattern space will be examined for the case where the object of interest is reflected without protruding from the frame image.

(Congruent transformation) As the congruent transformation for preserving the shape and size of the object on the frame image, there are parallel movement and rotation (including mirror image). Joint conversion (→ x) ⇒ (→
x ′) can be described by the equation 4 by the translation → t and the orthogonal matrix O.

[0020]

[Expression 4] → x ′ = O → x + (→ t) At this time, the transformation f ′ = Λ (f) in the pattern space
Is

[0021]

F ′ (→ x) = f (→ x ′) = f (O → x + (→ t)) Also, for an arbitrary real number α, Λ (f + g) = Λ
(F) + Λ (g) becomes Λ (αf) = αΛ (f), and Λ
It can be seen that is a linear transformation in the pattern space.

For any two patterns f and g, assuming that they do not jump out of the image by the congruential transformation,

[0023]

(Equation 6)

Therefore, the inner product is stored in the pattern space. That is, it can be seen that the congruential conversion to the pattern on the frame image corresponds to the orthogonal conversion in the pattern space.

Next, the above discussion is generalized and the congruential transformation is extended to the affine transformation. Instead of the orthogonal matrix O, using a quadratic matrix A whose determinant is non-O, → x ′ = A →
Consider x + (→ t). At this time, the inner product in the pattern space is the expression (7) and is not generally stored.

[0026]

(Equation 7)

However, the angle θ'between f'and g'is
And the angle θ between f and g is preserved.

[0028]

(Equation 8)

That is, it can be seen that the angle formed by two arbitrary patterns (see FIG. 3) as shown in FIG. 2 is preserved for the similarity transformation on the frame image.

1.2 Properties of Moving Image Next, consider a case where a temporal change like a moving image occurs. The brightness value at time t and coordinate → x is f (→ x, t). When the time t is fixed, the brightness value becomes a function of → x and is considered as one point in the pattern space. Than this,
The pattern at time t is f (t).

When the real world is photographed with a sufficiently high time resolution Δt, f (→ x, t) is considered to be continuous temporally and spatially. Therefore, the temporal change of the luminance at an arbitrary coordinate → x is calculated using | f (→ x, t
+ Δt) -f (→ x, t) | <ε.
This means that if ε is small enough in the pattern space, f (t
It means that the norm of + Δt) -f (t) approaches O as follows.

[0032]

[Equation 9]

As described above, the time variation of the pattern can be treated as a continuous locus drawn in the pattern space.

As shown in FIG. 4, when the pattern on the frame image changes from P to Q to R, P in the pattern space is changed.
It is assumed that the angle formed by Q and QR is θPQR (see Fig. 5).

When the pattern is linearly converted on the frame image, the angle θ'PQR formed by PQ and QR is equal to θPQR because the conversion is similar in the corresponding pattern space.

1.3 Definition of Motion Trajectory As shown in FIG. 6, if the cause of the time variation on the frame image is a gesture of one person, the trajectory in the pattern space corresponding to this is the position of the person or It is invariant to size and has a shape unique to the type of gesture.

Therefore, in the present embodiment, the locus (see FIG. 7) drawn by the gesture in the pattern space (the vector space of the present invention) is defined as the "movement locus", and the gesture is identified using this. This makes it possible to identify the type of gesture without correcting the position, orientation, or size of the person on the frame image.

2: Segmentation of motion 2.1 Fluctuation of the same motion FIG. 8 shows an actual gesture, and FIG. 9 shows the motion locus projected on a two-dimensional plane by principal component analysis. If the motions of each time are exactly the same, the shapes of the motion loci are exactly the same. However, in reality, as shown in FIG. 9, a minute fluctuation occurs every time, and the shape of the motion locus gradually changes. Therefore, in a motion that absorbs such a variation and is recognized as the same by humans, some measure is required so that the motion loci have the same shape.

If the motion is divided into minute times, the motion can be considered as a series of poses that change little by little. Even if there is some variation in one type of motion, humans can identify that motion by dividing the motion into a number of typical constituent units (pauses) and using this to determine the motion determination results. It is considered to be corrected.

Therefore, a similar approach is adopted for the motion locus. That is, some representative points that characterize the trajectory shape are selected, and the similarity is determined using an approximate polygon having these points as vertices to absorb the variation. The selected vertices are considered to be a representative constituent element of the motion, and are therefore referred to as motion elements in this embodiment. In the next section, we explain in detail the algorithm that determines motion elements from motion loci.

2.2 Division of Motion Trajectory by Curvature Let the motion locus in the pattern space be f (s). here,
s represents the length of the locus from a certain reference point on the locus and is a function of time t. At this time, the unit vector t in the tangential direction of the locus is represented by Formula 10.

[0042]

(Equation 10)

Further, the curvature is K, the radius of curvature is ρ (≧ 0),
If the unit vector in the normal direction is ν, then Equation 11 is obtained.

[0044]

[Equation 11]

The linear transformation A in the two-dimensional image is a similarity transformation in the pattern space, and its curvature K'is

[0046]

(Equation 12)

Is proportional to the original curvature.

From the expression (12), it is understood that the maximum and the minimum of the curvature appear at the same position of the locus shape regardless of the size of the person on the frame image in any gesture. Therefore, the motion locus is divided based on the maximum and minimum of the curvature.

2.3 Motion Trajectory Segmentation Algorithm FIG. 10 shows an algorithm for dividing the motion trajectory (processing contents executed by the image processing apparatus 10 in FIG. 1). It is necessary to find the quadratic difference in order to find the curvature, but since the difference between simple frame images is easily affected by noise,
A method of waiting until the magnitude of the inter-frame difference serving as the denominator becomes an appropriate unit length S _th is adopted.

The latest frame image I _Now is taken in (step S2), and the path length S _In to the reference image I _B is obtained (step S2 → S3). S _In> S _th if updating the reference image (I _B ⇒I _Now, step S4⇒S5), curvature K
Ask for.

Next, the time at which the maximum value / minimum value is reached for the K time series is determined, and the frame image is stored as an action element (N _now = I _now , step S7 → S8).

Finally, the angle θ formed in the pattern space is obtained from the latest three motion elements, and the above procedure is repeated again. Through the above processing, the gesture is divided into time series having a series of motion elements (specific poses) as elements. This can be considered as an operation of segmenting one gesture into a plurality of constituent elements. Therefore, in the present embodiment, the operation of dividing the motion locus into polygonal components by approximating the motion locus is called segmentation.

2.4 Gesture Recognition Method Based on Motion Trajectory In order to determine the shape of the motion trajectory approximated to a polygon for each frame, continuous DP, which is a typical method often used for recognition of time series patterns, is used. Match the model created in advance with the video input.

The local distance between the model and the input used for continuous DP is defined as follows.

[0055]

(Equation 13)

Here, l (alphanumeric character L) is the ratio of the lengths of the vectors connecting the three adjoining motion elements. FIG. 11 shows the definition of the local distance dlocal. For each model and input, the Euclidean distance of two points obtained by mapping on the polar coordinates is obtained from the angle formed by the ratio of the Euclidean distance between the latest three motion elements.

The local distance is defined in this way because the shape of the motion locus is a curve having the same dimension number as that of the original pattern space, whereas the polygonal approximation of the motion locus having N vertices is at most. This is because only the (N-1) dimension becomes indefinite, and from the viewpoint of real-time recognition, it is desirable to obtain the degree of local shape matching by using only the information available at that time. is there.

2.5 Application of Continuous DP Using the local distance defined in Section 2.4 above, continuous DP is performed by the following procedure to determine the shape of the motion locus.

First, using the latest three motion elements obtained from the segmentation algorithm in Section 2.3, the angle θ with the length ratio l (ell) is obtained. Next, the local distances to the input at that time are obtained for all the action elements of the model. This is weighted and added for each operating element.

At this time, of the three elements including the front and back of the target operating element, the one having the smallest total sum at the time of addition is adopted. The total distance obtained for all motion elements is normalized by the total number of motion elements of the model.

The above procedure is repeated for all models and the one with the smallest total distance is adopted. If the minimum distance is greater than or equal to the threshold, it is determined that there is no match.

3: Preliminary Experiment for Gesture Recognition As a preparation before conducting the recognition experiment, in order to confirm the characteristics of the motion trajectory, 1) segmentation of the gesture, 2) confirmation of trajectory shape preservation, 3) angle and length We conducted three preliminary experiments on the characteristics of the ratio.

[0063]

EXAMPLE 3.1 Gesture segmentation experiment The result of applying the segmentation algorithm of Section 2.4 to the operation "ByeBye" shown in FIG. 13 is shown in FIG. FIG. 12 is a result of displaying the curvature of the movement trajectory as a time series. FIG. 13 shows the result of segmenting the operation with the maximum value and the minimum value of this curvature time series. Each is a frame image in each motion element.

It can be seen that the motion "ByeBye" has a repeating cycle composed of four motion elements. Further, with respect to the position where the arm is swung, both end points correspond to the maximum value points in the curvature time series, and the two middle points correspond to the minimum value points.

3.2 Storage of Trajectory Shape FIGS. 14 and 15 show a pair of length and angle for the bye-bye (FIG. 13) and plotted as a polygon on a two-dimensional plane. One by-bye motion is composed of four motion elements and corresponds to a pentagon. 14 and 1
5 is the same bye bye, but FIG. 14 is when the elbow is almost at the same height as the shoulder, and FIG. 15 is when the elbow is lowered to the lowest position. Since both are almost congruent pentagons, it is considered that the trajectory shape is preserved even if the position of the gesture moves.

3.3 Properties of Angle-to-Length Ratio As a preliminary experiment for gesture recognition, the properties of the angle-to-length ratio formed by the three motion elements used to calculate the local distance were investigated. FIG. 16 is a graph in which the variations of the angle and the length ratio used for the calculation of the local distance are summarized. From this graph, these two features are
It can be seen that it fluctuates in the same cycle as the operating element.

A scatter plot showing the relationship between the two is
FIG. It can be seen that there is a correlation, but since it is not linear, the effectiveness of combining the angle and length ratio can be expected.

In this embodiment, the motion locus, which is a new feature amount (feature parameter) for gesture recognition, was proposed. From the results of some preliminary experiments, even if the position and size of the subject in the frame image cannot be corrected accurately, we believe that we could show the possibility of dealing with this.

Further, it was shown that the gesture can be segmented into motion elements which are its constituent elements, from the geometrical property of the motion locus. As a result, it is possible to not only adapt to changes in movements themselves, but also realize new functions such as the computer operating in time with human movements (eg, playing music in time with a command). it is conceivable that.

[0070]

As described above, according to the first aspect of the invention, when the time series change of the frame image is considered as a vector, the frame image can be treated as a point. The motion loci of the above points represent the features of the same type of gesture in correspondence with the frame images that make up the moving image, without being affected by the position thereof. As a result, the number of standard gesture patterns can be reduced, which can contribute to speeding up of recognition processing and improvement of recognition accuracy.

According to the second aspect of the present invention, by approximating the motion locus to a polygon, it is possible to reduce the recognition accuracy and simplify the process of acquiring the motion locus.

According to the third aspect of the present invention, by setting the representative points as the maximum and minimum points of curvature, the motion locus can be automatically divided without relying on human hands.

According to the fourth aspect of the present invention, by comparing the characteristic parameters of the motion loci by the continuous DP method, it is possible to perform pattern matching even for a gesture moving image that changes in three time periods, such as a moving image. .

[Brief description of drawings]

FIG. 1 is a block diagram showing a typical system configuration of a gesture recognition device.

FIG. 2 is an explanatory diagram showing the principle of the present invention.

FIG. 3 is an explanatory diagram showing the principle of the present invention.

FIG. 4 is an explanatory diagram showing the principle of the present invention.

FIG. 5 is an explanatory diagram showing the principle of the present invention.

FIG. 6 is an explanatory diagram showing a relationship between a moving image and a motion locus.

FIG. 7 is an explanatory diagram showing a relationship between a moving image and a motion locus.

FIG. 8 is an explanatory diagram showing gestures used in an experiment.

FIG. 9 is a waveform diagram showing an operation locus.

FIG. 10 is a flowchart showing the processing contents of this embodiment.

FIG. 11 is an explanatory diagram showing a local distance.

FIG. 12 is a characteristic diagram showing the locality of the motion locus.

FIG. 13 is an explanatory diagram showing the content of a gesture.

FIG. 14 is an explanatory diagram showing an operation locus.

FIG. 15 is an explanatory diagram showing an operation locus.

FIG. 16 is a characteristic diagram showing the result of recognition of two motion loci.

FIG. 17 is a scatter diagram showing a result of recognizing a motion trajectory twice.

[Explanation of symbols]

 10 Image Processing Device 11 Frame Grabber 12 Real Time Disc 13 TV Camera 14 Rewritable LD 15 VTR

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Shigeki Nagaya 1-280, Higashi Koikeku, Kokubunji, Tokyo Inside the Central Research Laboratory, Hitachi, Ltd. (72) Inventor Susumu Kan 22 22-22 Nagaike-cho, Abeno-ku, Osaka Incorporated (72) Inventor Ryuichi Oka 1-6-1, Takezono, Tsukuba City, Ibaraki Prefecture Tsukuba Mitsui Building Technology Research Association New Information Processing Development Organization Tsukuba Research Center

Claims (4)

[Claims] 1. A feature parameter is extracted from a moving image obtained by capturing a gesture by an image processing apparatus to create a reference pattern, and the feature parameter is extracted from a moving image to be recognized, and the extracted feature parameter is extracted. In a gesture moving image recognition method for recognizing a gesture by comparing a characteristic parameter with a characteristic parameter of the reference pattern, each of a plurality of frame images forming the moving image is regarded as a point on a vector space, and the point A method of recognizing a gesture moving image, characterized in that the motion locus formed by the above is used as the characteristic parameter. 2. The motion locus is approximated by selecting a plurality of representative points that characterize the locus shape of the motion locus and approximating the motion locus with a polygon having the representative points as vertices. The gesture moving image recognition method described in 1. 3. The gesture moving image recognition method according to claim 2, wherein the representative point is determined based on the maximum and minimum points of the curvature of the movement trajectory. 4. The gesture moving image recognition method according to claim 3, wherein a continuous DP method is used for comparing the characteristic parameters according to claim 1.