JP6349800B2 - Gesture recognition device and method for controlling gesture recognition device - Google Patents

Description

  The present invention relates to a gesture recognition device that recognizes an input operation by a gesture.

  Devices that can input with gestures to computers and electronic devices are showing signs of widespread use.

  For example, Patent Document 1 describes an input device that captures a gesture made by a user in space with a camera or the like and converts the gesture into an input command. The apparatus stores a specific gesture and a specific input command in association with each other, and has means for recognizing the gesture and means for converting the recognized gesture into an input command. As a result, the user can input an arbitrary command by directly performing a gesture operation in front of the device without directly operating the input device.

JP 2012-123608 A

  In an input device that recognizes a gesture, generally, by extracting a point representative of the position of the part (hereinafter referred to as a representative point) from the image of the part to be gestured, and tracking the movement of the extracted representative point, It is determined what kind of gesture has been made. For example, when a user performs a gesture of drawing a figure by opening the palm, the center point of the hand is detected and the locus of the center point is tracked to recognize the shape of the figure drawn by the gesture.

However, when performing a gesture using the hand, it is not always good to set the representative point at the center of the hand. For example, when performing a gesture with an index finger raised, a more natural recognition result can be obtained for the user by tracking the tip of the index finger. In such a case, if the representative point is set at the center of the hand, there is a possibility that the input may not be performed correctly when only the fingertip is moved without moving the hand greatly.
Thus, when performing a gesture using a part of the body, the conventional gesture recognition device cannot properly determine where the representative point is, and as a result, the trajectory expressed by the gesture is used. In some cases, it could not be recognized as intended.

  The present invention has been made in consideration of the above problems, and provides a technique for recognizing a gesture as intended by a user in a gesture recognition apparatus that recognizes a gesture performed by a user moving a target part. The purpose is to do.

  In order to solve the above-described problem, the gesture recognition device according to the present invention has a configuration in which the shape of a target part, which is a body part that performs a gesture, is determined, and the position of a representative point is determined in consideration of the shape. .

Specifically, the gesture recognition device according to the present invention includes:
A gesture recognition device that detects a gesture from an acquired image and generates an instruction for a target device corresponding to the gesture, and an image acquisition unit that acquires the image, and a target site where the gesture is performed from the acquired image Target part extraction means for extracting, target part shape specifying means for specifying the shape of the extracted target part, and representative points for setting a representative point that is a point representing the position of the target part for the target part Determining means; gesture recognition means for recognizing a gesture based on movement of the representative point; and command generation means for generating a command corresponding to the recognized gesture, wherein the representative point determination means The position of the representative point corresponding to the target part is determined using the shape of the target part.

The target part is a part where the user performs a gesture, and is typically a human hand, but may be the entire human body or an input marker held by the user. Also good. The input gesture can be recognized by tracking the movement of the representative point corresponding to the target part. However, in the gesture recognition device according to the present invention, the position of the representative point in the target part is determined by the shape of the target part. To determine.
The gesture recognition device according to the present invention recognizes a gesture input by moving the target part as intended by setting the position of the representative point in consideration of the shape of the target part in this way. can do.

  Further, the representative point determining means sets a representative point at the tip of the protruding portion when the shape of the target portion specified by the target portion shape recognizing means is a shape including the protruding portion. It is good.

  When a part of the target part has a protruding shape, there is a high possibility that the gesture input is performed by the movement of the protruding part. For example, there are cases where the user is standing with some fingers or holding a stick-shaped input marker. In such a case, it is preferable to set a representative point at the tip of the part.

  Further, the target part is a human hand, and the representative point determination unit determines whether the shape of the target part specified by the target part shape recognition unit is the first shape or the first shape. It is determined whether the second shape is different. When the shape of the target portion is the first shape, a representative point is set at a position corresponding to the fingertip, and the shape of the target portion is the second shape. In some cases, a representative point may be set at a position corresponding to the center of the hand.

When the target part is a human hand, there are two possible patterns: a case where a gesture is performed using a fingertip and a case where a gesture is performed using the entire hand. Therefore, the position of the representative point is preferably either the fingertip or the center of the hand.
If it can be estimated that the gesture is performed using the fingertip instead of the entire hand, a representative point is set at a position corresponding to the fingertip. Thus, even when a gesture is performed with a small fingertip movement, the gesture can be recognized as intended by the user.

  In addition, the representative point determination unit is the first shape when the shape of the hand that is the target site specified by the target site shape recognition unit is a shape in which some fingers are extended. It may be determined that the second shape is determined when all fingers are extended or all fingers are bent. Further, when the shape of the hand that is the target portion is a shape in which only one finger is extended, the shape may be determined as the first shape.

When a part of the five fingers is extended, it can be estimated that a gesture is performed using the fingertip. Further, when all fingers are extended or all fingers are bent, it can be estimated that a gesture is performed by moving the entire hand. Moreover, when only one extending finger is detected, it can be estimated that a gesture is performed using the fingertip. The state where some fingers are extended does not include the state where all fingers are opened.

  Further, the representative point determining means may use the center of gravity of the region corresponding to the hand extracted by the target part extracting means as the position corresponding to the center of the hand.

  As the position corresponding to the center of the hand, it is preferable to use a center of gravity that is easy to calculate.

  In addition, the gesture recognition device according to the present invention may further include notification means for notifying the user of the position of the set representative point.

  The notification means is, for example, a display device or an audio output device. For example, where the representative point is set may be displayed on the screen. With this configuration, the user can know where the gesture recognition device is tracking and can further improve usability.

  In addition, this invention can be specified as a gesture recognition apparatus containing at least one part of the said means. Further, the control method of the gesture recognition device, the program for operating the gesture recognition device, and the recording medium on which the program is recorded can be specified. The above processes and means can be freely combined and implemented as long as no technical contradiction occurs.

  ADVANTAGE OF THE INVENTION According to this invention, in the gesture recognition apparatus which recognizes the gesture which the user performed by moving a target part, a gesture can be recognized as a user intends.

It is a lineblock diagram of the gesture recognition system concerning a first embodiment. It is a figure explaining the example of gesture definition data. It is a figure explaining the example of extraction of an object part. It is a figure explaining the difference in the shape of an object part. It is a flowchart figure showing the process which a gesture recognition apparatus performs in 1st embodiment. It is a flowchart figure showing the process which a gesture recognition apparatus performs in 1st embodiment. It is an example of the notification screen in 2nd embodiment.

(First embodiment)
An outline of the gesture recognition system according to the first embodiment will be described with reference to FIG. 1 which is a system configuration diagram. The gesture recognition system according to the first embodiment is a system including the gesture recognition device 100 and the target device 200.

The gesture recognition device 100 is a device that recognizes a gesture performed by a user using a camera, generates a command corresponding to the gesture, and transmits the command to the target device 200.
The target device 200 is a device (control target device) that receives a command from the gesture recognition device 100, and is typically an electrical product such as a television, a video recorder, a computer, an air conditioner, or a video conference system. The target device 200 may be any device as long as it can receive a command from the gesture recognition device 100 by wire or wireless. In the present embodiment, the target device 200 is a television receiver, and the gesture recognition device 100 is a device built in the television.

  The gesture recognition device 100 will be described in detail. The gesture recognition device 100 includes an image acquisition unit 101, a gesture extraction unit 102, a gesture recognition unit 103, and a command generation unit 104.

The image acquisition unit 101 is a means for acquiring an image from the outside. In this embodiment, a user is imaged using a camera (not shown) attached to the upper front part of the television screen. The camera used by the image acquisition unit 101 may be a camera that acquires an RGB image, or may be a camera that acquires a grayscale image or an infrared image. The image is not necessarily acquired by the camera, and may be an image (distance image) representing a distribution of distances generated by the distance sensor, for example. Further, a combination of a distance sensor and a camera may be used.
The image acquired by the image acquisition unit 101 (hereinafter referred to as a camera image) may be any image as long as the movement of the gesture performed by the user and the shape of the body part where the gesture is performed can be acquired. Good. The angle of view of the camera image may be substantially the same as the viewing angle of the television.

The gesture extraction unit 102 is a unit that detects a body part (hereinafter, a target part) on which a gesture is performed from the camera image acquired by the image acquisition unit 101. In the case of this embodiment, a user shall perform a gesture using a hand. For example, the gesture extraction unit 102 detects an area representing a human hand from a camera image.
In addition, the gesture extraction unit 102 sets a representative point, which is a point representing the position of the target part, for the detected target part, and tracks the movement of the representative point. Thereby, the movement expressed by the gesture can be extracted.
The shape and representative points of the target part will be described in detail later.

  The gesture recognizing unit 103 is means for specifying the content of the gesture based on the movement of the representative point extracted by the gesture extracting unit 102. For example, as shown in FIG. 2, data (gesture definition data) that associates the movement of the representative point with the content of the gesture is stored, and the instruction expressed by the gesture is specified. As shown in FIG. 2, the instruction expressed by the gesture may be a series of gesture operations corresponding to a single instruction, or is displayed on the screen based on the moving amount and moving direction of the representative point. It may be an instruction to move the pointer.

  The command generation unit 104 is a unit that generates a command corresponding to the gesture specified by the gesture recognition unit 103. The command is a signal for controlling the target device 200, and may be an electric signal, a wirelessly modulated signal, a pulse-modulated infrared signal, or the like.

  The gesture recognition device 100 is a computer having a processor, a main storage device, and an auxiliary storage device, and a program stored in the auxiliary storage device is loaded into the main storage device and executed by the processor, whereby each of the above-described means is performed. Functions (a processor, a main storage device, and an auxiliary storage device are not shown).

Next, gesture recognition processing will be described in detail.
FIG. 3 is an example of an image acquired by the image acquisition unit 101. That is, it is a view of the user facing the television screen viewed from the screen side.
The gesture recognition process includes a process for detecting a target part to be gestured from an image, a process for setting a representative point corresponding to the detected target part, and a process for identifying the content of the gesture by tracking the movement of the representative point. And consist of

First, detection of a target part will be described.
The gesture extraction unit 102 detects an area (reference numeral 31) including a human hand from the acquired image. An area including a human hand may be determined by, for example, color or shape, or may be determined by detecting a feature point. Alternatively, the determination may be made by comparing with a model or template stored in advance. Since a known technique can be used for detecting the hand, a detailed description is omitted.

Next, representative points will be described.
The representative point is a point representing the position of the target part, and one is set for one target part. For example, when the target part is a human hand, a point (symbol 32) representing the center of the palm can be used as the representative point.
By setting a representative point for the target part and tracking the movement of the representative point, it is possible to acquire a gesture represented by the movement of the hand. For example, gestures such as “move the entire hand to indicate direction” and “draw a figure” can be acquired.

Here, the problems of the prior art will be described with reference to FIG.
FIG. 4 is an example of a camera image (illustration is omitted except for the target part). 4A shows a gesture for moving the fingertip of the index finger, and FIG. 4B shows a gesture for moving the palm in parallel.
Both of these gestures mean “move to the left”, but when making a gesture with a finger raised, often the fingertips are trying to make a gesture and the hand is opened. When performing a gesture, the gesture is often attempted by a large movement of the entire arm. Therefore, the recognition accuracy of the gesture changes depending on which part of the target part the representative point is set.
For example, in the case of FIG. 4A, when a representative point is set at the fingertip of the index finger, the moving distance is the distance indicated by reference numeral 41. However, when a representative point is set at the center of the palm, the moving distance is indicated by reference numeral 42. Distance. That is, when a different command is assigned depending on the movement distance, a command different from a desired command may be input. In addition, when it is desired to move the pointer according to the moving distance, there is a possibility that the moving amount desired by the user cannot be obtained.

  Therefore, in the gesture recognition apparatus according to the present embodiment, the gesture recognition accuracy is improved by determining the position of the representative point using the shape of the target part.

  In this embodiment, the shape of the target part is identified by the number of standing fingers. For example, in the case of FIG. 4A, the number of standing fingers is one, and in the case of FIG. 4B, it is determined that the number of standing fingers is five. In the following description, a state in which one finger is raised is referred to as “shape 1”, and a state in which the hand is opened and five fingers are raised is referred to as “shape 5”. Further, “shape n” means that there are n standing fingers.

  For example, the number of standing fingers may be determined by searching for a pointed shape (or protruding part) that seems to be a finger around the detected hand, or a template image or a skeleton model of the hand. It may be determined by performing matching using, for example. Since a known method can be used to determine the number of fingers, detailed description is omitted.

In the present embodiment, when the gesture extraction unit 102 determines the shape of the hand and determines that the gesture is performed with the fingertip, the representative point is set at a position corresponding to the fingertip. In other cases, a representative point is set at the center of gravity of the detected hand. The shape that can be determined that the fingertip is performing the gesture can be, for example, the case where the number of standing fingers is one (the shape of the target part is “shape 1”).

Next, the entire process performed by the gesture recognition apparatus 100 according to the present embodiment will be described with reference to FIGS. 5 and 6 which are process flowcharts.
The process illustrated in FIG. 5 is a process performed by the image acquisition unit 101 and the gesture extraction unit 102. The processing may be started when there is an operation indicating the start of input (for example, when a function that requires gesture input is activated on the target device side), or when the power of the target device is turned on. You may start.

First, the image acquisition unit 101 acquires a camera image (step S11). In this step, for example, an RGB color image is acquired using a camera provided in the upper front portion of the television screen.
Next, the gesture extraction unit 102 tries to detect a target part (hand) from the acquired camera image (step S12). As described above, the target part may be detected based on the color and shape, or may be performed by pattern matching or the like. If the target part is not detected, a new image is acquired after waiting for a predetermined time, and the same processing is repeated.

Next, the gesture extraction unit 102 determines the shape of the detected target part (step S13). In this example, it is determined whether the shape of the target part is the shape 1 or the other. If the shape of the target part is other than that defined in advance, the process may be interrupted and the process may return to step S11, or the process may be continued as “not applicable”.
Next, it is determined whether the shape of the detected target part has changed since the previous execution (step S14). When step S14 is executed for the first time, it is determined that there is “change”.
If it is determined that there is “change” in step S14, the process proceeds to step S15, and the gesture extraction unit 102 sets a representative point in the detected target part. Specifically, if the shape of the target part determined in step S13 is shape 1, a representative point is set at the tip of the extending finger, and in other cases, a representative point is set at the center of gravity of the hand. To do.
If it is determined in step S14 that there is no change, the process proceeds to step S16.

Next, the gesture extraction unit 102 acquires the coordinates of the representative points in the camera image and transmits them to the gesture recognition unit 103 (step S16).
The processes of steps S11 to S16 are repeatedly executed, and the coordinates of the representative points are sequentially transmitted to the gesture recognition unit 103.

The process illustrated in FIG. 6 is a process performed by the gesture recognition unit 103 and the command generation unit 104. This process is started simultaneously with the process shown in FIG.
Step S21 is processing in which the gesture recognition unit 103 recognizes a gesture based on the acquired coordinates of the representative point. For example, in the case where there is the gesture definition data shown in FIG. 2, when it is detected that the coordinates of the representative point have moved 100 pixels or more in the right direction, it is determined that a gesture indicating “volume up” has been made. To do.

Note that the information transmitted from the gesture extraction unit 102 to the gesture recognition unit 103 does not necessarily have to be coordinates in the camera image as long as it can represent the movement direction and movement amount of the representative point. For example, the coordinates of the representative point may be converted into a coordinate system with the user as the origin, or other data representing the movement direction or movement amount.
Moreover, you may make it transmit simultaneously the information showing the magnitude | size of the object site | part in a camera image. Since the movement amount of the target part acquired by the gesture recognition device changes according to the distance between the user and the apparatus, the movement amount can be corrected according to the size of the target part.

  Next, the command generation unit 104 generates a control signal corresponding to the recognized gesture and transmits it to the target device 200 (step S22). In the example described above, for example, a control signal (a signal for raising the volume by one level) associated with the command “volume up” is generated and transmitted to the target device 200.

  The processing shown in FIGS. 5 and 6 is periodically executed, and ends when there is an operation indicating the end of input (for example, when an operation using gesture input is ended on the target device side).

  As described above, the gesture recognition device according to the first embodiment sets the position of the representative point to a different position depending on the shape of the target part where the gesture is performed. Thereby, even if it is a case where a gesture is performed using a fingertip, or a case where it is performed using the whole hand, the said gesture can be recognized correctly.

In the description of the embodiment, the shape of the target part is determined in step S13. However, this step is executed only once after the target part is detected, and is skipped after the gesture is started. It may be. By doing so, the processing amount can be suppressed.
However, since the gesture may end and a different gesture may start subsequently, in such a case, the step may be executed again. For example, if the shape or size of the target part has changed significantly, or if the target part has been out of the image and then in again, it is determined that a different gesture has started, and step S13 is executed again. May be. Moreover, you may re-execute by explicit operation.

(Second embodiment)
The second embodiment is an embodiment in which means for notifying the user of the position of the representative point is added to the gesture recognition system according to the first embodiment. The configuration of the gesture recognition system according to the second embodiment is the same as that of the first embodiment except for the points described below.

The flowchart of the process performed by the gesture recognition device 100 according to the second embodiment is the same as that in FIGS. 5 and 6, but when the target region is set in step S <b> 15, the gesture extraction unit 102 determines that the target device 200 is the target device 200. It is different in that the user is notified of the position where the representative point is set through the screen (the television screen in this example).
FIG. 7 is an example of an image displayed on the television screen as the target device. For example, if the shape of the detected target part is shape 1, notify that the gesture can be performed by moving the fingertip, and if the shape is other than that, notify that the gesture can be performed by moving the entire hand. To do.
Note that the method for notifying the user may be other methods. For example, guidance may be displayed only with text, or a guidance window may be added to a normal operation screen to display graphics and text. Moreover, you may notify by an audio | voice etc.

  According to the second embodiment, since the user can recognize where the representative point to be tracked by the system is, a more intuitive gesture input can be performed.

(Modification)
The description of each embodiment is an exemplification for explaining the present invention, and the present invention can be implemented with appropriate modifications or combinations without departing from the spirit of the invention.
For example, in the description of the embodiment, the gesture recognition apparatus 100 is an apparatus incorporated in the target device 200, but the gesture recognition apparatus 100 may be an independent apparatus.
The gesture recognition device 100 may be implemented as a program that operates on the target device 200. When implemented as a program, the processor may be configured to execute a program stored in a memory, or may be configured to be executed by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like. May be.

In the description of the embodiment, an example of acquiring an image using a camera has been described. However, if a gesture can be acquired and the shape of a target part can be identified, for example, an image is received via a network. The image may be acquired by a method other than the exemplified method.
In addition, the target part is not necessarily a human hand. For example, it may be another body part or a gesture input marker. When using a marker, the gesture extraction unit 102 may detect the presence of the marker and set a representative point at the tip of the marker.

  In addition, the “shape of the target part” in the present invention means a shape recognized by the gesture recognition device through the image, and does not necessarily have to be a physical deformation of the target part.

In the description of the embodiment, two types of “shape 1” and “other shapes” are identified for the shape of the target part, but other shapes may be identified. The other shape may be, for example, a gripped hand or a state where two fingers are standing. Also, three or more types of shapes may be identified. In either case, the position of the representative point may be determined based on the identified shape, and processing may be performed by the method described above.
For example, the position where the representative point is set may be as follows.

When the shape is 0: The center of gravity of the fist is the representative point. When the shape is 1: The fingertip of the standing finger is the representative point. When the shape is 2: The midpoint of the two standing fingertips. In case of shape 3: Among the three standing fingers, the fingertip of the middle finger is the representative point. In case of shapes 4 and 5: The center of gravity of the palm is the representative point.

  In the description of each embodiment, after detecting the target part from the camera image in step S12, the shape of the target part is identified in step S13. However, these processes are performed simultaneously by template matching or the like. It may be. As long as the position where the target part exists and the shape of the target part can be acquired, the contents and order of processing are not particularly limited.

DESCRIPTION OF SYMBOLS 100 ... Gesture recognition apparatus 101 ... Image acquisition part 102 ... Gesture extraction part 103 ... Gesture recognition part 104 ... Command generation part 200 ... Target apparatus

Claims (7)

A gesture recognition device that detects a gesture from an acquired image and generates a command for a target device corresponding to the gesture,
Image acquisition means for acquiring images;
Target part extraction means for extracting a human hand , which is a target part to be gestured, from the acquired image;
Target part shape specifying means for specifying the shape of the extracted target part;
Representative point determining means for setting a representative point, which is a point representing the position of the target portion, with respect to the target portion;
Gesture recognition means for recognizing a gesture based on the movement of the representative point;
Command generating means for generating an instruction corresponding to the recognized gesture;
Have
In the representative point determining means, the shape of the hand, which is the target part specified by the target part shape specifying means, is a first shape in which some fingers are extended, or all fingers are extended. Or a second shape that is a shape in which all fingers are bent, and when the shape of the target portion is the first shape, a representative point at a position corresponding to the fingertip , And a representative point is set at a position corresponding to the center of the hand when the shape of the target part is the second shape . The representative point determining means sets a representative point at the tip of the protruding portion when the shape of the target portion specified by the target portion shape specifying means is a shape including the protruding portion. The gesture recognition device according to claim 1. The representative point determining means, as a position corresponding to the center of the hand, extracted by the target region extraction means, which comprises using a center of gravity of the area corresponding to the hand, either claim 1 or 2 The gesture recognition device according to any one of the above. Characterized by further comprising a notification means for notifying the location of the set representative point to the user, the gesture recognition apparatus according to any one of claims 1 to 3. A method for controlling a gesture recognition device that detects a gesture from an acquired image and generates a command for a target device corresponding to the gesture,
An image acquisition step of acquiring an image;
A target part extraction step of extracting a human hand that is a target part to be gestured from the acquired image;
A target part shape specifying step for specifying the shape of the extracted target part;
A representative point determination step for setting a representative point, which is a point representing the position of the target portion, with respect to the target portion;
A gesture recognition step for recognizing a gesture based on the movement of the representative point;
A command generation step of generating an instruction corresponding to the recognized gesture;
Including
In the representative point determining step, the shape of the hand that is the target portion specified in the target portion shape specifying step is a first shape that is a shape in which some fingers are extended, or all fingers are extended. Or a second shape that is a shape in which all fingers are bent, and when the shape of the target portion is the first shape, a representative point at a position corresponding to the fingertip And a representative point is set at a position corresponding to the center of the hand when the shape of the target portion is the second shape . In a gesture recognition device that detects a gesture from the acquired image and generates a command for the target device corresponding to the gesture,
An image acquisition step of acquiring an image;
A target part extraction step of extracting a human hand that is a target part to be gestured from the acquired image;
A target part shape specifying step for specifying the shape of the extracted target part;
A representative point determination step for setting a representative point, which is a point representing the position of the target portion, with respect to the target portion;
A gesture recognition step for recognizing a gesture based on the movement of the representative point;
A command generation step of generating an instruction corresponding to the recognized gesture;
And execute
In the representative point determining step, the shape of the hand that is the target portion specified in the target portion shape specifying step is a first shape that is a shape in which some fingers are extended, or all fingers are extended. Or a second shape that is a shape in which all fingers are bent, and when the shape of the target portion is the first shape, a representative point at a position corresponding to the fingertip , And a representative point is set at a position corresponding to the center of the hand when the shape of the target part is the second shape . A recording medium on which the gesture recognition program according to claim 6 is recorded.

Images