JP2007272839A - Gesture recognition device, online operation system using the same, gesture recognition method, and computer readable medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a gesture recognition device for recognizing gesture by a simple method without designating the start and the end of the gesture by any means. <P>SOLUTION: This gesture recognition device is constructed of a photographing means 10, a frame image processing means 20 acquiring and accumulating the operation direction of a mobile object as a vector from frame images inputted continuously from the photographing means 10, a gesture factor registration means 30 storing status feature map data and status weight map data, a status position initialization part 41 initializing respective status positions, a vector feature quantity matching part 42 converting an outputted vector into a vector feature quantity, inputting the status map, and temporarily storing and outputting a comparison value between a predetermined status and statuses before and after it based on both of the vector feature quantity and the status map, a status transition determination part 43 determining and outputting transition of the respective statuses based on the comparison value, and a gesture determination part 44 determining whether a gesture recognition flag is established or not. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a gesture recognition device, an online operation system using the same, a gesture recognition method, and a computer-readable medium.

  A gesture recognition device is a device that recognizes a gesture by sequentially capturing images from cameras into a processing device, detecting hand movements by digital image processing.

  In recent years, with the development of computers, more sophisticated and intelligent appliances have been developed. Therefore, what is important is the human-computer interface (hereinafter referred to as HCI (human-computer interaction). For example, a keyboard and mouse are used to operate a computer. These are indirect but essential tools for HCI. In recent years, as an HCI tool, a system for recognizing human ability such as voiceprints and fingerprints has been developed.Hand gestures are regarded as one of them.Originally, hand gestures are communication between people. Used as a means, gesture recognition systems can be an important HCI tool.

As a technique often used when performing gesture recognition, there is a hidden Markov model (hereinafter referred to as HMM (hidden Markov model)). The HMM is effective for sequentially checking gestures, but it requires an extremely large amount of training for tuning parameters before recognition, and it is impractical to add a new gesture to be recognized (for example, non-patent literature). 1).
As a method other than the HMM, there is dynamic programming (hereinafter referred to as DP (dynamic programming)) matching (for example, see Non-Patent Document 2). DP matching does not require a particularly complicated process and has a good recognition result. However, it is necessary to specify the start and end of the gesture for recognition.
Masato Aoba and Yoshiyasu Takefuji: “Motion feature extraction using second-order neural network and self-organizing map for gesture recognition”, Information Processing Society of Japan Research Report, 2004-MPS-52, pp.5-8, December 2004 Osaki, R., Shimada, M. and Uehara, K .: Extraction of primitivemotions by using clustering and segmentation of motion-captured data, JSAIVol.15, No.5, pp.878-886, (2000).

Specifically, in conventional DP matching, the start and end of a gesture are specified and matching is performed for the entire gesture. Therefore, it is necessary to consider the operation time of the gesture. That is, even with the same gesture, there are cases of a fast operation and a slow operation, and time changes depending on the operation speed. In order to recognize these operations as gestures by conventional DP matching, it is necessary to match them dynamically forcibly by matching the time.
In addition, since the entire gesture is matched, the recognition process starts from the end of the operation, and processing time is required.
Accordingly, an object of the present invention is to provide a gesture recognition device capable of recognizing a gesture in real time without extracting the beginning and end of the gesture by any method, an online operation system using the same, a gesture recognition method, and a computer readable To provide a medium.

The present invention for solving the above problems relates to the following items.
[1] Imaging means for continuously inputting A gestures as frame images;
B Frame image processing means comprising a vector acquisition unit that acquires the motion direction of the moving object as a vector from the frame image continuously input from the imaging unit, and a vector output unit that continuously outputs the acquired vector When,
C: normalizing the length of the vector based on the continuously output vector, storing state feature map data created from the vector feature amount weighted to the angle, and storing the continuous feature amount and state feature map Gesture element registration means for storing state weight map data created from
D: A state feature map and a state weight map stored in the gesture element registration unit are input, a state position initialization unit for initializing each state position, and a vector output by the frame image processing unit as a vector feature amount Converted, continuously input, and further input a state map stored in the gesture element registration means, and based on both, temporarily store and output as a comparison value of a predetermined state and the state before and after that A vector feature amount matching unit, a state transition determination unit that determines and outputs a transition of each state based on the initialized state position and the comparison value based on the comparison value, and a transition and a state weight in each state A gesture determination unit that compares the map and determines whether to establish a gesture recognition flag with a predetermined threshold;
Gesture recognition means comprising:
A gesture recognition device comprising:

The gesture element registration means includes:
A vector accumulating unit for accumulating the continuously output vectors;
A vector matrix obtaining unit for obtaining a vector matrix from the accumulated vectors;
A vector feature amount conversion unit that normalizes the length of the vector from the acquired vector matrix and converts it into a vector feature amount weighted to an angle;
A state feature map creating unit for creating and storing a state feature map from continuous feature quantities;
A state weight map creating unit for creating and storing a state weight map from the continuous feature quantity and the state feature map;
The gesture recognition device according to [1], comprising:
[3] The gesture recognition device according to [1] or [2], further comprising display means for continuously displaying the acquired vectors.

[4] A gesture recognition method for recognizing a gesture using the gesture recognition device according to any one of [1] to [3],
Inputting a state feature map and a state weight map acquired in advance (S101);
A step (S102) of initializing each state position;
Continuously inputting a gesture from the imaging means as a frame image (S103);
Obtaining a moving direction of a moving object as a vector from frame images continuously input from the imaging means (S104);
A step of continuously outputting the acquired vector (S105);
Converting the output vector into a vector feature (S106);
Each of the converted vector feature values is continuously input, and a state map prepared in advance is further input, and based on both, temporarily stored as a comparison value between a predetermined state and a state before and after that, and output A step (S107) of performing,
Based on the comparison value, the transition of each state is determined based on the initialized state position and the comparison value, the transition in each state is compared with the state weight map, and a gesture recognition flag is set based on a predetermined threshold. A step of determining whether or not to establish (S108);
Including
If the gesture recognition flag is established in the step S108, the gesture recognition is completed. If the gesture recognition flag is not established, the gesture recognition method moves to the step (S103).

[5] The initialization is performed by creating zero matrices for the number of states, and the transition determination is performed on a vector obtained by frame processing. Each comparison value is determined to proceed if the degree of coincidence of the next state is high, and the value of the advanced state is set to 1, if the degree of coincidence of the current state is high, the stay is set, and the value of the current state is +1 The gesture recognition method according to [4], wherein when the degree of coincidence of the previous state is high, the value of the current state is set to −1, and when it becomes 0, it is determined that the state is retreated.
In [5], it is also possible to determine the transition in the step (S108) based on a predetermined threshold.

[6] A gesture analysis method for analyzing an input gesture using the gesture recognition device according to any one of [1] to [3],
A gesture registration method for registering an input gesture using the gesture recognition device according to any one of claims 1 to 3.
Continuously inputting a gesture from the imaging means as a frame image (S201);
Obtaining a moving direction of a moving object as a vector from frame images continuously input from the imaging means (S202);
A step of continuously outputting the acquired vector (S203);
Storing the continuously output vectors (S204);
Obtaining a vector matrix from the accumulated vectors (S205);
Normalizing the length of the vector from the acquired vector matrix and converting it into a vector feature quantity weighted to an angle (S206);
C-4 creating and storing a state feature map from consecutive feature quantities (S207);
C-5 creating and storing a state weight map from the continuous feature quantity and the state feature map (S208);
The gesture registration method characterized by including.

  In the step of acquiring the vector, the resolution of the frame image continuously input from the imaging unit is converted into a low resolution image, and the converted low resolution image is converted into L * a * b * image and L *. a moving object is detected from the a * b * image converted image, the L * a * b * image converted image is smoothed, the smoothed image is binarized, and the binarized image is clustered. It is also possible to acquire the moving direction of the moving object as a vector from the frame images continuously input from the imaging means after the tracking processing after clustering processing, and it is also possible to detect the moving body by the frame difference method It is.

  [7] In the vector conversion step, the value of the angle in the same direction is set to 1, the value of the angle in the reverse direction is set to 0, and mapping is performed as a graph of x-axis 0 to 360 and y-axis 0 to 1. The gesture recognition method according to [6].

[8] A computer-readable medium storing a program for executing the process according to any one of [4] to [7].
In the computer readable medium of the present invention, the gesture recognition result may be associated with the operation of the computer application.
[9] An online operation system that operates a device that performs a predetermined operation online, and includes the gesture recognition device according to any one of claims 1 to 3, the gesture recognition device, and a public line. An on-line operation system comprising a device connected and performing a predetermined operation in accordance with a recognition result of the gesture recognition device.

According to the present invention, a gesture is set as a transition of “state”. That is, instead of recognizing a gesture as an entire motion, it recognizes an instantaneous movement and recognizes a gesture as a stack of the gestures, so that the gesture can be recognized in real time.
Furthermore, since the gesture is set as the “state” in this way, it can be recognized regardless of the operation speed. Furthermore, unlike the prior art, it is possible to recognize a gesture by a simple method without depending on the operation speed without specifying the start and end of the gesture.
By connecting this gesture recognition device to an operation target device that is desired to operate online, it is possible to efficiently recognize the operation target device online without being aware of the beginning and end of the gesture, and to operate more easily online.

  Hereinafter, the present invention will be described with reference to the accompanying drawings. In the present invention, the gesture recognition device refers to a sequential processing device (for example, a computer system) that sequentially captures images from a camera as an imaging means, and detects a moving object, typically a hand, by digital image processing. And a device for recognizing gestures.

The term “gesture” used in the present invention refers to an action such as “waving a hand” or “drawing a circle”. The present invention is based on a unique image processing algorithm and an apparatus embodying the algorithm in such a gesture recognition apparatus.
The present invention is characterized in that a gesture is set as a “state” and recognized based on the transition of the set state.

(Gesture recognition device)
First, the gesture recognition device of the present invention will be described with reference to FIGS.
FIG. 1 is a schematic diagram showing the entire gesture recognition apparatus of the present invention, and FIG. 2 is a schematic diagram showing the frame image processing means, gesture element registration means, vector feature amount matching means, and gesture recognition means shown in FIG. 3 is a diagram illustrating a state in which a gesture is vector-divided for each frame in the vector storage unit 31, FIG. 4 is a diagram illustrating an example of weighting, and FIG. 5 is an example of a state feature map. 6 is a diagram showing an example of a state weight map, FIG. 7 is a diagram showing an example of obtaining a comparison value in the vector feature amount matching unit 42, and FIG. FIG. 9 is a diagram illustrating an example of state transition, and FIG. 9 is a diagram illustrating stay values to be compared by the gesture determination unit.

  As shown in FIG. 1, the gesture recognition device 1 of the present embodiment includes an imaging unit 10 for continuously inputting a gesture as a frame image, a frame image processing unit 20, a gesture element registration unit 30, and a gesture recognition unit. 40. More generally, the gesture recognition apparatus 1 according to the present embodiment includes an imaging unit 10 such as a digital (video) camera and a computer system connected to the imaging unit 10.

That is, an interface for inputting continuous frame images from the imaging means 10, a memory that is a storage device that temporarily stores images input via the interface, and an image that is temporarily stored in the memory The gesture recognition apparatus 1 according to the present embodiment includes a CPU that is an operator for processing data, a processing program to be described later, storage means for storing various data, and a computer system including output means for outputting a gesture recognition result. Is generally constructed. Preferably, the output recognition result is continuously output to display means such as a display.
However, the present invention is not limited to the imaging means 10 and the computer system as long as the apparatus has an equivalent function, and is within the scope of the present invention if it has an equivalent function.

  The imaging means 10 in the gesture recognition device 1 of the present embodiment is not particularly limited as long as it can continuously input a moving body that is a gesture operation main body and output it to a subsequent computer system. Digital video cameras well known in the technical field can be used. Further, the resolution and format of frame images to be output continuously are not particularly limited as long as the present invention can be embodied, and the resolution is appropriately determined in view of ease of processing and image quality in the subsequent stage. It is possible. The image format applicable in the present invention is preferably image binary data in a general-purpose image compression format such as jpg, bmp, and png.

  The frame image processing means 20 in the present embodiment is means for extracting an operation target from the frame image captured and output by the imaging means 10 and outputting it as a continuous vector. As shown in FIG. The vector acquisition unit 21 acquires a motion direction of a moving object as a vector from frame images continuously input from the imaging unit, and the vector output unit 22 outputs the acquired vector continuously.

  The vector acquisition unit 21 is composed of, for example, a central operator in a computer system, a memory, a storage medium, and a program stored in the storage medium, and a continuous image from the imaging unit, for example, an RGB image image is obtained by a conventionally known technique. After capturing, converting resolution (lowering resolution), converting L * a * b * image, detecting the moving object that is the main subject of gesture, smoothing, binarizing, clustering and tracking, the present invention It is a component which acquires the motion direction of the moving object which is the feature of as a vector.

The vector output unit 22 has a function of continuously outputting the vector acquired by the vector acquisition unit 21 to the subsequent gesture element registration unit 30 and the gesture recognition unit 40.

The gesture element registration means 30 in the present embodiment is means for registering (learning) elements necessary for recognition by a gesture recognition means 40 described later. That is, the gesture element registration means 30 is means for previously registering elements necessary for recognizing a gesture by the gesture recognition means 40. Generally, each element of the state feature map data and the state weight map data Storage means for storing the.
These data may be stored in advance in the gesture element registration unit 30, but in the present embodiment, a case will be described in which a function for registering and storing a gesture is provided.
The gesture element registration means 30 can be provided, for example, in a ROM storing elements registered in advance (elements of state feature map data and state weight map data). However, when it is desired to register a gesture to be newly recognized. In this case, it is necessary to create and store the state feature map data and the state weight map data based on the vectors input by the imaging unit 10 and output from the frame image processing unit 22.
That is, the gesture element registration unit 30 in the present embodiment includes a vector accumulation unit 31 that accumulates the continuously output vectors from the vector output unit 22, and a vector matrix acquisition unit that obtains a vector matrix from the accumulated vectors. 32, a vector feature amount conversion unit 33 that normalizes the vector length from the acquired vector matrix and converts it into a vector feature amount weighted to an angle, and a state feature that creates and stores a state feature map from the continuous feature amount The map creation unit 34 includes a state weight map creation unit 35 that creates and stores a state weight map from the continuous feature amount and the state feature map.
The term “vector feature weighted by angle” used in the present invention is an amount indicating how much the direction of the acquired vector is relative to an angle of 0 to 360 °, for example, an angle value in the same direction. Means a feature amount of 0 to 360 on the x axis and 0 to 1 on the y axis where the reverse angle value is 0. In this way, in the present invention, the vector acquired by acquiring the vector feature amount weighted to the angle indicates one point from 0 ° to 360 ° as shown in the following figure, while the surrounding weight is set. By doing so, even slightly shifted vectors can be handled in the same direction.
Further, the term “state feature map” used in the present invention means a set of continuous vector feature amounts representing a gesture, and indicates that the direction is changed in a predetermined order.

These vector accumulation units 31 have a function of accumulating vectors in order to obtain a vector matrix of gestures, and the vector matrix acquisition unit 32 is temporally continuous for each gesture accumulated in the vector accumulation unit 31. It has a function to acquire a vector matrix from a vector.
Specifically, as shown in FIG. 3, the gesture is divided into vectors for each frame (V (t), V (t-1), V (t-2), V (t-3),. A set of vectors is a gesture vector matrix.

  The vector feature amount conversion unit 33 normalizes the length of the vector (that is, sets the vector length to 1) and weights the angle for the purpose of acquiring the feature amount with emphasis on the direction and angle of the vector that is not influenced by time. It has a function of converting into the feature amount. The weight of the obtained angle takes a large value if the difference from the detected angle is small, and takes a small value if the difference is large. As an example of the weight, a normal distribution is used as shown in FIG. Note that the angle may be quantized as desired.

The state feature map creating unit 34 has a function of creating a state feature map in order to represent a gesture by a transition of a representative vector feature amount (state).
That is, the state feature map has a function of collecting several representative vector feature amounts in the vector matrix converted into vector feature amounts.
Considering a typical vector feature as one state, initialize the number of states to the vector matrix and gradually reduce the number of states. At this time, each vector state amount in the vector matrix most closely matches which state. To find out. The state group when the matching sum of the entire state becomes smaller than the threshold is used as the state feature map.
When adding a gesture, the previous state feature map is compared with the state feature map subjected to the above processing, and the state is integrated, added, or deleted to create a new state feature map.

The matching sum of the representative state is created using the norm of the difference sum, the product sum norm, and the variance value.
The state feature map is, for example, a three-dimensional map in which vector feature amounts exist for each number of representative states as shown in FIG.

  The state weight map creation unit 35 learns the weight ratio map of each representative state and uses it for each of the states in the state feature map for the purpose of determining the gesture recognition. Has the function of counting the most matching states and creating a state weight map graphed by the weight ratio.

That is, a state weight map is created from the matrix of temporally continuous vector feature amounts from the vector matrix acquisition unit 32 and the state feature map from the state feature map creation unit 34.
The state weight map is a two-dimensional map in which the weight for each representative state as shown in FIG.

These vector storage unit 31, vector matrix acquisition unit 32, vector feature quantity conversion unit 33, and state feature map creation unit 34 are, for example, a central operator in a computer system, a memory, a storage medium, and a program stored in the storage medium. Composed.
The gesture element registration unit 30 normalizes the length of the vector based on the continuously output vectors, stores state feature map data created from vector feature amounts weighted to angles, and stores the continuous feature vectors. It is also possible to store the state weight map data created from the feature quantity and the state feature map and output the necessary data to the subsequent gesture recognition means as necessary.

  The gesture recognition unit 40 receives the state feature map and the state weight map stored in the gesture element registration unit 30 and outputs the state position initialization unit 41 for initializing each state position and the frame image processing unit 20. The converted vector is converted into a vector feature amount, and a state map stored in the gesture element registration unit 30 is input, and temporarily stored as a comparison value between a predetermined state and a state before and after the predetermined state based on both. , A vector feature value matching unit 42 to output, and a state transition determination to determine and output the transition of each state based on the initialized state position and the comparison value based on the comparison value in the vector feature value matching unit 42 The unit 43 compares the transition in each state and the state weight map, and establishes a gesture recognition flag with a predetermined threshold. Comprised determines the gesture determination unit 44 whether.

  The state position initialization unit 41 reads a motion feature map composed of a state feature map and a state weight map obtained by the gesture analysis process (or stored in advance in the gesture element registration unit), and initializes the state position. It has a function to set the state. That is, the state position initialization unit 41 reads the feature map of the gesture to be recognized, and sets the state position to the initial state in order to prepare for state transition recognition. With the function of the state position initialization unit 41, a motion feature map (state feature map + state weight map) exists in the memory, and the state position is set as the initial position.

The vector feature amount matching unit 42 includes a recognition vector feature amount conversion unit 42A that converts the vector output from the frame image processing unit 20 into a vector feature amount in the same manner as the vector matrix acquisition unit 32, and the gesture element registration unit. The matching unit 42B is configured to input the state map stored in 30, and temporarily store and output as a comparison value between a predetermined state and a state before and after the predetermined state based on both.
Specifically, the motion feature map (state feature map + state weight map) existing in the memory by the function of the state position initialization unit 41, and the vector feature amount output in real time from the recognition vector feature amount conversion unit 42A And the comparison value is continuously output in real time.
For example, as shown in FIG. 7, the comparison value is acquired using a product sum norm or a variance value in addition to the norm of the difference sum. In the preferred embodiment, three comparisons are made: the current state, the previous state, and the next state.

  The state transition determination unit 43 has a function of transitioning the state between advance, stay, and retreat according to a threshold value based on a comparison value obtained by vector feature matching in order to perform state transition in the state feature map. Each state has a function of providing a counter and storing it as a stay value.

  That is, the state transition determination unit 43 reads the state feature map stored in the gesture element registration unit 30, and further uses the vector feature matching comparison value determined by the state transition determination unit 43 to change the state transition in the state feature map. I do. Specifically, for example, when the degree of matching in the next state is high as shown in FIG. 8, the state is advanced. The stay value is 1 as the value of the transitioned state position. If the matching degree of the current state is high, the state is stayed. The stay value is incremented by 1 to the state position. If the above two are not satisfied, the stay value is decremented by 1 to the state position. When it reaches 0, the state is retracted. Based on these, the state transition is output as a stay graph and stored.

More specifically, a zero matrix corresponding to the number of states is first created (initialization of states [0 0 0 0 0...), And the current state and the previous and next states are compared with the vector obtained by frame processing. Each comparison value is obtained by comparing three. Proceeds when the degree of matching of the next state is high. Set the value of the advanced state to 1. If the degree of coincidence of the current state is high, the stay and the current state value are incremented by one. When the degree of coincidence with the previous state is high, the value of the current state is set to -1, and when it becomes 0, the state moves backward. In practice, a threshold is set for the degree of coincidence.
More specifically, it is conceivable that the state is recognized only by the transition of the feature state, but in this case, even if there is a slight error, the state may retreat, and correct gesture recognition cannot be performed. There is. Therefore, it was possible to absorb the error by correcting the state so that the state moves backward after a little weight.
Moreover, since the gesture is established at the moment when the final state is reached, there is a possibility of erroneous recognition. Therefore, the stay value and the state weight described above are introduced, and the establishment of the gesture is determined based on the stay ratio in each state.
As a result, it is resistant to instantaneous noise and can be handled in the same manner whether the operation speed is slow or fast depending on the stay ratio.

The gesture determination unit 44 compares the transition in each state with the state weight map to determine whether or not to establish a gesture, and determines whether or not to establish a gesture recognition flag based on a predetermined threshold.
That is, the gesture determination unit 44 reads the state feature map stored in the gesture element registration unit 30, and further calculates the stay value in each state from the state transition determination unit 43, for example, as shown in FIG. Alternatively, it is determined whether or not the gesture recognition flag is established by comparison using the product-sum norm or the variance value.

The gesture recognition apparatus of the present invention configured as described above easily recognizes a gesture without specifying the start and end of the gesture in any way, and regardless of the operation speed of the gesture, unlike the related art. It is possible.
The gesture recognition apparatus of the present invention has a small amount of data for recognition, and can easily register a gesture to be recognized.
Furthermore, in another embodiment of the present invention, a gesture recognition device capable of recognizing a gesture in real time with a simple configuration can be configured by previously registering elements to be recognized in a storage means such as a ROM or a hard disk. It becomes possible.

Furthermore, in a preferred embodiment of the present invention, it is possible to provide display means (not shown) for continuously displaying the acquired vectors. By continuously displaying the acquired vectors in this way, the user can visually confirm the result of the gesture. Further, by displaying the related gesture in advance by the display unit, the user can cope with the case by forgetting the gesture to be recognized by the display unit.
Furthermore, the gesture recognition apparatus of the present invention recognizes not only a single gesture but also a suitable gesture by, for example, analyzing each of a plurality of gestures, storing a state feature map, and processing the recognition means in parallel. It is also possible.

  In another embodiment of the present invention, an online operation system that operates a device that performs a predetermined operation online, is connected to the gesture recognition device of the present invention, the gesture recognition device via a public line, An online operation system is provided that includes an apparatus that performs a predetermined operation according to a recognition result of the gesture recognition apparatus.

  Note that a device that performs a predetermined operation online in the present invention is a device that operates when there is some input, and is not particularly limited as long as it can be connected to the gesture recognition device of the present invention online. Absent. Specifically, it covers the home security field, baths, air conditioners, elderly care systems, car navigation systems, televisions, audios, DVDs, videos, televisions, lighting equipment, games, etc.

(Operation: Gesture recognition method)
Next, the operation of the gesture recognition apparatus of the present invention, that is, the gesture recognition method will be described with reference to FIGS.
FIG. 10A is a flowchart showing a gesture registration method according to the present invention, and FIG. 10B is a flowchart showing a gesture recognition method according to the present invention. FIG. 11 is a graph for explaining step S206 of FIG. 10A.

  In the gesture registration method according to the present invention, as shown in FIG. 10A, first, a gesture from the imaging means is continuously input as a frame image (S201).

Next, the motion direction of the moving object is acquired as a vector from the frame images continuously input in step S201 (S202), and the acquired vector is continuously output (S203).
In S202 and S203, a continuous image from an imaging unit, for example, an RGB image image is captured by a conventionally known technique, the resolution is converted (lower resolution), the L * a * b * image is converted, and the gesture operation is performed. After detecting the moving body that is the subject, smoothing, binarizing, clustering, and tracking, the moving direction of the moving object can be acquired as a vector.
The moving object can be detected by a conventionally known method, for example, a frame difference method.

Next, in step S204, the continuously output vectors are accumulated, a vector matrix is obtained from the accumulated vectors (S205), the vector length is normalized from the obtained vector matrix, and the angle is weighted Conversion into vector feature amounts (S206), a state feature map is created and stored from the continuous feature amounts (S207), and a state weight map is created and stored from the continuous feature amounts and the state feature map (S208). .
In other words, steps S204 to S208 are steps in which a vector obtained by accumulating and matrixing each gesture is input, converted into a vector feature amount, and a motion feature map including a state feature map and a state weight map is generated and output.

In step S206, it is preferable that the value of the angle in the same direction is 1 and the value of the angle in the reverse direction is 0, and mapping is performed as a graph of x-axis 0 to 360 and y-axis 0 to 1.
That is, as shown in FIG. 11, the vector indicates the direction, and only one point from 0 to 360 degrees is obtained. On the other hand, by giving weights to surrounding angles, conversion to vector feature amounts is facilitated.

As described above, the gesture recognition elements can be easily registered in the gesture recognition device of the present invention by the steps shown in FIG. 10A. In other words, this gesture registration method can be said to be a gesture analysis method for the gesture recognition method of the present invention. That is, acquisition of these recognition elements can be said to be a preparation stage for use in the gesture recognition method described below. However, in a specific embodiment of the present invention, it is possible to register a gesture recognition method in advance.
Next, a method for recognizing a gesture in the gesture recognition device of the present invention will be described with reference to FIG. 10B.
First, in step S101, a state feature map and a state weight map are input, and each state position is initialized (S102).
Next, a frame image is input (S103), the moving direction of the moving body is acquired as a vector (S104), and the length of the vector is normalized from the acquired vector matrix in the same manner as in step S206, and the angle is weighted. Conversion into a vector feature amount (S105).
Then, this vector feature amount is compared with the transition in each state and the state weight map, and it is determined whether or not the gesture recognition flag is established based on a predetermined threshold (that is, the obtained vector feature amount is established as a gesture). (S106).
At this time, based on the input state map and the vector feature amount, it is temporarily stored and output as a comparison value between the predetermined state and the state before and after it (S107), and is initialized based on the comparison value. The transition of each state is determined based on the state position and the comparison value, the transition in each state is compared with the state weight map, and it is determined whether or not the gesture recognition flag is established based on a predetermined threshold.
When the gesture recognition flag is established (Yes), the gesture recognition is completed (end). On the other hand, if the gesture recognition flag is not established, the process proceeds to step 103 and a frame image is input.

The initialization in step S102 is performed by creating zero matrices corresponding to the number of states (see paragraph 0045), and the transition determination in step S108 is performed with respect to the vector obtained by the frame processing and Three of the states are compared, and each comparison value is determined to be advanced if the degree of coincidence of the next state is high, the value of the advanced state is set to 1, and if the degree of coincidence of the current state is high, it is regarded as stay Preferably, the current state value is set to +1, the current state value is set to -1 when the degree of coincidence of the previous state is high, and the state is determined to retreat when it becomes 0.
The reason for this determination is that, as described above, it is resistant to instantaneous noise and can perform recognition processing corresponding to various operating speeds.
Further, the transition determination in the step S108 can be performed based on a predetermined threshold value.

In the gesture recognition method of the present invention, the gesture registration (analysis) in FIG. 10A can be regarded as a preliminary gesture analysis for gesture recognition.
That is, a vector feature amount is created in advance, a state feature map and a state weight map are created, and if available, the registration process shown in FIG. 10A can be omitted and the recognition process shown in FIG. 10B can be directly executed. is there.
The gesture recognition method of the present invention configured as described above recognizes a gesture in real time in a simple manner without depending on the operation speed without specifying the start and end of the gesture, unlike the prior art. Is possible. Moreover, since the amount of information handled is very small compared to the conventional gesture recognition method, it is possible to perform gesture recognition without imposing a great load on the device (with a cheaper device).

  Further, by storing a program for executing each step of the gesture recognition method in a computer-readable medium, it is possible to operate the application program according to the gesture in association with an application program such as a presentation program. It is.

As mentioned above, although embodiment of this invention was described, this invention is not limited to these, It can apply widely.
For example, an image conversion unit that converts the resolution of continuously input frame images into a low-resolution image in the imaging unit of the present invention, and an image that has been reduced in resolution by the image conversion unit is converted into L ^* a ^* b. ^* and L ^* a ^* b ^* image converter for image conversion, and the L ^* a ^* b ^* moving body detection unit for detecting a moving object from the L ^* a ^* b ^* image converted image by the image conversion unit, L ^* a smoothing filter for smoothing the image obtained by a ^* b ^* image conversion, a binarization processing unit for binarizing the image smoothed by the smoothing filter, and a clustering processing unit for clustering the binarized image; It is possible to provide. When such an imaging unit is applied to the gesture recognition apparatus of the present invention, various processes in the subsequent stage are reduced. In addition to gesture recognition, it can be used for devices that detect moving objects, such as crime prevention and parking management devices. Since the data that can be output is binarized data that has been processed in advance, the load when processing the moving object is reduced.

BRIEF DESCRIPTION OF THE DRAWINGS Schematic which shows the whole gesture recognition apparatus of this invention. FIG. 2 is a schematic diagram illustrating a frame image processing unit, a gesture element registration unit, a vector feature amount matching unit, and a gesture recognition unit shown in FIG. 1. The figure which shows the state which divided the gesture into the vector for every frame in the vector storage part. Drawing which shows the example of weighting. The figure which shows an example of a state characteristic map. The figure which shows an example of a state weight map. The figure which shows the example which calculates | requires the comparison value in a vector feature-value matching part. The figure which shows an example of the transition of the state in a state feature map. The figure which shows the stay value compared by a gesture judgment part. The flowchart which shows the gesture registration method in this invention. The flowchart which shows the gesture recognition method in this invention. The graph for demonstrating process S206 of FIG. 10A.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Gesture recognition apparatus 10 Imaging means 20 Frame image processing means 30 Gesture element registration means 31 Vector accumulation | storage part 32 Vector matrix acquisition part 33 Vector feature-value conversion part 34 State feature map creation part 35 State weight map creation part 40 Gesture recognition means 41 State Position initialization unit 42 Vector feature value matching unit 42A Recognition vector feature value conversion unit 42B Matching unit 43 State transition determination unit 44 Gesture determination unit

Claims (9)

A imaging means for continuously inputting a gesture as a frame image;
B, a vector acquisition unit that acquires a motion direction of a moving object as a vector from frame images continuously input from the imaging unit;
A frame image processing means comprising a vector output unit for continuously outputting the acquired vector;
C: normalizing the length of the vector based on the continuously output vector, storing state feature map data created from the vector feature amount weighted to the angle, and storing the continuous feature amount and state feature map Gesture element registration means for storing state weight map data created from
D a state position initialization unit that inputs a state feature map and a state weight map stored in the gesture element registration unit and initializes each state position;
The vector output by the frame image processing means is converted into a vector feature amount, continuously input, and further a state map stored in the gesture element registration means is input. Based on both, a predetermined state and its state A vector feature amount matching unit that temporarily stores and outputs the comparison value of the previous and subsequent states;
Based on the comparison value, a state transition determination unit that determines and outputs a transition of each state based on the initialized state position and the comparison value;
A gesture determination unit that compares the transition in each state with the state weight map and determines whether or not to establish a gesture recognition flag based on a predetermined threshold;
Gesture recognition means comprising:
A gesture recognition device comprising: The gesture element registration means includes:
A vector accumulating unit for accumulating the continuously output vectors;
A vector matrix obtaining unit for obtaining a vector matrix from the accumulated vectors;
A vector feature amount conversion unit that normalizes the length of the vector from the acquired vector matrix and converts it into a vector feature amount weighted to an angle;
A state feature map creating unit for creating and storing a state feature map from continuous feature quantities;
A state weight map creating unit for creating and storing a state weight map from the continuous feature quantity and the state feature map;
The gesture recognition device according to claim 1, comprising:   The gesture recognition apparatus according to claim 1, further comprising display means for continuously displaying the acquired vectors. A gesture recognition method for recognizing a gesture using the gesture recognition device according to any one of claims 1 to 3.
Inputting a state feature map and a state weight map acquired in advance (S101);
A step (S102) of initializing each state position;
Continuously inputting a gesture from the imaging means as a frame image (S103);
Obtaining a moving direction of a moving object as a vector from frame images continuously input from the imaging means (S104);
A step of continuously outputting the acquired vector (S105);
Converting the output vector into a vector feature (S106);
Each of the converted vector feature values is continuously input, and a state map prepared in advance is further input, and based on both, temporarily stored as a comparison value between a predetermined state and a state before and after that, and output A step (S107) of performing,
Based on the comparison value, the transition of each state is determined based on the initialized state position and the comparison value, the transition in each state is compared with the state weight map, and a gesture recognition flag is set based on a predetermined threshold. A step of determining whether or not to establish (S108);
Including
If the gesture recognition flag is established in the step S108, the gesture recognition is completed. If the gesture recognition flag is not established, the gesture recognition method moves to the step (S103). The initialization is performed by creating a zero matrix for the number of states,
For the vector obtained by the frame processing for the transition determination, the current state and the previous and next states are compared, and each comparison value is determined to proceed when the degree of coincidence of the next state is high. The value of the completed state is 1, and if the degree of coincidence of the current state is high, it is stayed, the value of the current state is +1,
5. The gesture recognition method according to claim 4, wherein when the degree of coincidence of the previous state is high, the value of the current state is set to -1, and when it becomes 0, it is determined that the state is retreated. A gesture registration method for registering an input gesture using the gesture recognition device according to any one of claims 1 to 3.
Continuously inputting a gesture from the imaging means as a frame image (S201);
Obtaining a moving direction of a moving object as a vector from frame images continuously input from the imaging means (S202);
A step of continuously outputting the acquired vector (S203);
Storing the continuously output vectors (S204);
Obtaining a vector matrix from the accumulated vectors (S205);
Normalizing the length of the vector from the acquired vector matrix and converting it into a vector feature quantity weighted to an angle (S206);
Creating and storing a state feature map from continuous feature quantities (S207);
Creating and storing a state weight map from the continuous feature quantity and the state feature map (S208);
The gesture registration method characterized by including.   7. In the vector conversion step, the angle value in the same direction is set to 1, the angle value in the opposite direction is set to 0, and mapping is performed as a graph of x-axis 0 to 360 and y-axis 0 to 1. The gesture registration method described in.   A computer-readable medium in which a program for executing the process according to any one of claims 4 to 7 is stored. An online operation system for operating a device that performs a predetermined operation online, wherein the gesture recognition device according to any one of claims 1 to 3 is connected to the gesture recognition device via a public line. An online operation system comprising an apparatus that performs a predetermined operation in accordance with a recognition result of the gesture recognition apparatus.

Images