JP7163649B2 - GESTURE DETECTION DEVICE, GESTURE DETECTION METHOD, AND GESTURE DETECTION CONTROL PROGRAM - Google Patents

Description

The present invention relates to a gesture detection device, a gesture detection method, and a gesture detection control program for detecting gestures used for input operations.

As a conventional gesture detection device, for example, one described in Patent Document 1 is known. The gesture detection device (hand pattern switch device) disclosed in Patent Literature 1 reliably distinguishes whether an operator's gesture is made intentionally or carelessly, thereby suppressing the occurrence of erroneous recognition. is doing. Specifically, when the operator stops the position of the finger in the shape of the finger preset by the operator in the imaging region of the camera for a predetermined time, the detection device detects the operation by the operator's intentional gesture. has started. Then, after that, the detection device executes a gesture operation detection process based on the shape and movement of the fingers.

Japanese Patent Application Laid-Open No. 2004-145723

However, in Patent Document 1, in order to recognize the start of an operator's intentional gesture operation, the detection device needs to be stopped for a predetermined period of time in a preset finger shape, and the gesture operation is complicated. It is a (time-consuming) thing.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a gesture detection device, a gesture detection method, and a gesture that enable recognition of a gesture intended by an operator without increasing the time and effort required for gesture detection. An object of the present invention is to provide a detection control program.

The present invention adopts the following technical means in order to achieve the above objects.

In a first invention, in a gesture detection device for detecting a gesture when performing an input operation on an operation target based on a gesture of an operator,
Based on the feature amount data extracted from the imaging data of the operator imaged by the imaging device (10), the first gesture of a pair of gestures similar to each other is the gesture just before the start, and the latter is the end gesture. a gesture segment extraction unit (120) for extracting a segment of the gesture as the gesture immediately after;
a gesture identification unit (130) that identifies a gesture performed between the gesture just before the start and the gesture just after the end as an input gesture ;
The gesture segment extraction unit
a similarity calculation unit (122) that calculates the similarity for each of a plurality of sets of gestures that are similar to each other;
a start/end determination unit (123) that extracts a set of gestures having the highest similarity among the similarities calculated by the similarity calculation unit as a set of gestures, and determines a gesture immediately before the start and a gesture immediately after the end; It is characterized by having

Further, in the second invention, in a gesture detection method for detecting a gesture when performing an input operation on an operation target based on a gesture of an operator,
Based on the feature amount data extracted from the imaging data of the operator imaged by the imaging device (10), the first gesture of a pair of gestures similar to each other is the gesture just before the start, and the latter is the end gesture. Immediately after extracting the gesture interval as a gesture,
identifying gestures made between the just-beginning gesture and the just-ending gesture as input gestures ;
Calculating the degree of similarity for each of a plurality of sets of gestures that are similar to each other,
A gesture having the highest similarity among the calculated similarities is extracted as a set of gestures, and a gesture immediately before the start and a gesture immediately after the end are determined .

Further, in the third invention, in the gesture detection control program for detecting the gesture when performing the input operation on the operation target based on the operator's gesture,
the computer,
Based on the feature amount data extracted from the imaging data of the operator imaged by the imaging device (10), the first gesture of a pair of gestures similar to each other is the gesture just before the start, and the latter is the end gesture. a gesture segment extraction unit (120) for extracting a segment of the gesture as the gesture immediately after;
Functioning as a gesture identification unit (130) for identifying a gesture performed between the gesture immediately before the start and the gesture immediately after the end as an input gesture ,
In the gesture segment extraction unit,
a similarity calculation unit (122) that calculates the similarity for each of a plurality of sets of gestures that are similar to each other;
Functions as a start/end determination unit (123) that extracts a set of gestures with the highest similarity among the similarities calculated by the similarity calculation unit as a set of gestures, and determines the gesture just before the start and the gesture just after the end. It is characterized by

In general, the operator takes similar postures at the start and end of gestures. That is, when starting a predetermined gesture, the operator first takes an action of holding his/her hand in a predetermined finger shape (basic shape of the gesture), and after executing the predetermined gesture, performs the same predetermined finger shape as the beginning. to end the gesture.

Therefore, according to the present invention, by capturing a set of gestures that are similar to each other, it is possible to extract the gesture just before the start and the gesture just after the end, thereby extracting the gesture section. By identifying the gesture made between the gesture just before the start and the gesture just after the end as the gesture for input, for example, it can be ensured that it is a gesture intended by the operator, not a gesture made carelessly. can be recognized. Therefore, it is possible to recognize the gesture for input without increasing the trouble of operation for gesture detection as in the conventional technology, and furthermore, it is possible to suppress the occurrence of erroneous recognition.

It should be noted that the reference numerals in parentheses of the above means indicate the corresponding relationship with specific means described in the embodiments to be described later.

1 is a block diagram showing a gesture detection device; FIG. FIG. 10 is an explanatory diagram showing a procedure for extracting start/end candidates of a gesture; FIG. 11 is an explanatory diagram showing a procedure for calculating a similarity between gesture start/end candidates; FIG. 10 is an explanatory diagram showing a procedure for determining the start and end of a gesture;

A plurality of modes for carrying out the present invention will be described below with reference to the drawings. In each form, the same reference numerals may be given to the parts corresponding to the matters described in the preceding form, and overlapping explanations may be omitted. When only a part of the configuration is described in each form, the previously described other forms can be applied to other parts of the configuration. Not only combinations of parts that are explicitly stated that combinations are possible in each embodiment, but also partial combinations of embodiments even if they are not explicitly stated unless there is a particular problem with the combination. is also possible.

(First embodiment)
A gesture detection device 100 according to the first embodiment will be described with reference to FIGS. 1 to 4. FIG. A gesture detection device 100 according to the present embodiment is mounted on a vehicle, and detects gestures when performing input operations on various vehicle devices based on movements (gestures) of specific parts of the body of a driver (operator). It is a device.

Various types of vehicle equipment include, for example, an air conditioner that air-conditions the interior of the vehicle, a car navigation device that displays the current position of the vehicle or displays guidance to a destination (hereinafter referred to as a car navigation device), television broadcasting, radio broadcasting, CD / Audio device for playing back DVDs, etc. These various vehicle devices correspond to the operation targets of the present invention. Various types of vehicle equipment are not limited to those described above, but include a head-up display device, a room lamp device, a rear seat sunshade device, an electric seat device, a glove box opening and closing device, and the like.

As shown in FIG. 1 , the gesture detection device 100 detects gestures for input operations based on imaging data of a specific part of the driver's body imaged by the imaging device 10 . Various vehicle devices are operated based on the detected gesture, and the result of the operation (operating state, etc.) is displayed on the information device 20 .

The specific part of the driver's body can be, for example, the finger, palm, arm, or the like of the driver's hand. In this embodiment, as gestures for input operation, fingers (pointing gesture, circle gesture, wave gesture, flick gesture, etc.) are mainly used.

In the case of an air conditioner, the set temperature and the air volume of the air-conditioned air can be changed, for example, by gestures for input operation of various vehicle devices. On the audio device, the TV station and radio station are changed, music is selected, volume is changed, and so on.

The imaging device 10 continuously (as time elapses) captures motion of a specific part of the driver's body, and outputs the captured data to the gesture detection device 100 (hand detection unit 110).

As the imaging device 10, a camera that forms a luminance image of an object, a distance image sensor that forms a distance image, or a combination thereof can be used. Examples of cameras include a near-infrared camera that captures near-infrared light, a visible light camera that captures visible light, and the like.

In addition, as a range image sensor, for example, a stereo camera that measures information in the depth direction from parallax by photographing simultaneously with multiple cameras, or depth is measured by the time it takes for the light from the light source to be reflected by the target object. There are ToF (Time of Flight) cameras and the like for measurement. In this embodiment, a camera is used as the imaging device 10 .

The information device 20 is a display unit that displays the operation states of various vehicle devices, and includes an air conditioning display unit, a car navigation display unit, an audio display unit, and the like. The information equipment 20 is formed by, for example, a liquid crystal display or an organic EL display.

The gesture detection device 100 is, for example, a computer including a CPU, a ROM, a RAM, etc., and includes a hand detection unit 110, a gesture segment extraction unit 120, a gesture identification unit 130, and the like.

The hand detection unit 110 identifies an area where the driver's hand exists from the imaging data (image) captured by the imaging device 10, and outputs data of the identified area to the gesture segment extraction unit 120. ing. Techniques for detecting a hand are disclosed, for example, in the following documents.
・M. Kolsch and M. Turk.: Robust Hand Detection. FGR, 2004
・Wei Fan, Li Chen, Wei Liu, Yuan He, Jun Sun, Shutao Li: Monocular Vision Based Relative Depth Estimation for Hand Gesture Recognition. MVA 2013.

The gesture segment extraction unit 120 is a part that extracts a segment of a gesture intentionally made by the driver for an input operation, and includes a start/end candidate extraction unit 121, a similarity calculation unit 122, a start/end determination unit 123, and the like. (details will be described later).

The gesture identification unit 130 is a part that identifies a gesture in the section extracted by the gesture section extraction unit 120 as a gesture intentionally made by the driver for an input operation. The gesture identification unit 130 is a part that outputs the identified gestures to various vehicle devices and the information device 20 .

The gesture detection device 100 of the present embodiment is configured as described above, and the operation and effects will be described below with reference to FIGS. 2 to 4. FIG.

Imaged data imaged by the imaging device 10 is output to the hand detection unit 110 . Hand detection section 110 identifies a region in which a hand exists in the imaged data, and outputs data of the identified region to start/end candidate extraction section 121 of gesture segment extraction section 120 .

The start/end candidate extraction unit 121 extracts frame candidates for the start and end of the gesture from the frames of the imaging data output from the hand detection unit 110, as shown in FIG. Here, the frame at the start of the gesture is a frame that indicates the state at the start of the gesture by the driver. For example, if the gesture to be executed is a flick gesture, the start of the gesture first corresponds to a state in which the finger is held up. Also, when the gesture ends, when the flick gesture ends, it corresponds to a state in which the posture of the hand with the finger up is maintained in the same manner as when starting the flick gesture.

The start/end candidate extraction unit 121 uses three or more consecutive imaging frames (imaging data) to extract a static feature amount (feature amount data) based on one frame and dynamic features based on a plurality of frames. Quantities (feature quantity data) are extracted and combined to create one feature quantity. Preferably, HOG (Histogram of Oriented Gradient) can be used as the static feature amount, and HoOF (Histogram of Optical Flow) can be used as the dynamic feature amount.

The database of the start/end candidate extracting unit 121 stores in advance reference data obtained by learning classification criteria for feature amounts at the start and end of a gesture and feature amounts at other times. Then, the start/end candidate extracting unit 121 classifies the feature amount extracted from the actual captured frame according to the reference data, and extracts the feature amount that satisfies the reference as the feature amount at the start and end as candidates. A machine learning method can be used for learning and classification, and Random Forest can be preferably used. In FIG. 2 , the portions indicated by reference numerals 1, 2, 3, and 4 among the continuous captured frames are portions (frames) extracted as candidates for the start and end of the gesture.

The similarity calculation unit 122 forms all combinations of two of the plurality of candidates extracted by the start/end candidate extraction unit 121, and calculates static A feature amount (feature amount data) is extracted, and the degree of similarity for each pair is calculated using the extracted feature amount. FIG. 3 shows that all pairs of pairs are 1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4, and 3 and 4. ing. The similarity calculation unit 122 calculates the similarity of each of the two candidates for all these combinations.

The start/end determination unit 123 extracts a combination (one pair) having the highest similarity among the similarities calculated by the similarity calculation unit 122, The frame of the gesture is determined, and the later one (later) is determined as the frame of the gesture immediately after the end. Then, the start/end determination unit 123 determines a frame between the frame immediately before the start and the frame immediately after the end as the segment of the gesture performed by the driver for the input operation. FIG. 4 shows that the combination of codes 1 and 2 has the highest similarity among the combinations in FIG. 3, and the period between codes 1 and 2 is determined as the gesture section.

Then, the gesture identification unit 130 extracts static feature amounts and dynamic feature amounts for all captured frames in the gesture period determined by the start/end determination unit 123 . The database of the gesture identification unit 130 stores in advance gesture reference data indicating a reference for classifying the feature amount of each gesture motion. Based on the gesture reference data, the gesture identification unit 130 identifies (classifies) which gesture motion the feature amount of the captured frame in the gesture period belongs to.

Gesture identification unit 130 outputs the identified gesture to various vehicle equipment and each information equipment 20, so that various vehicle equipment performs an input operation corresponding to the gesture, and each information equipment 20 receives an input operation corresponding to the gesture. Operation status etc. are displayed.

In general, the driver assumes similar postures at the beginning and at the end of gesturing. That is, when starting a predetermined gesture, the driver first takes an action of holding his/her hand in a predetermined finger shape (basic shape of the gesture), and after executing the predetermined gesture, the driver performs the same predetermined finger shape as the beginning. to end the gesture.

Therefore, according to the present embodiment, the gesture segment extraction unit 120 captures a pair of gestures that are similar to each other, extracts the gesture just before the start and the gesture just after the end, and extracts the gesture segment. Become. Then, the gesture identification unit 130 identifies the gesture made between the gesture just before the start and the gesture just after the end as an input gesture, so that, for example, the gesture is not an inadvertent gesture, but the intention of the driver. can be reliably recognized as a gesture with Therefore, it is possible to recognize the gesture for input without increasing the trouble of operation for gesture detection as in the conventional technology, and furthermore, it is possible to suppress the occurrence of erroneous recognition.

Further, in the present embodiment, the start/end candidate extraction unit 121 is provided to extract a plurality of sets of mutually similar gesture candidates before the similarity calculation unit 122 performs processing. As a result, by extracting a plurality of sets of gesture candidates in advance, the similarity calculation unit 122 can sequentially calculate the similarity for each set of candidates, thereby reducing the calculation load. .

(Other embodiments)
In the first embodiment described above, the hand detection unit 110 is provided, but the hand detection unit 110 may be eliminated, and the imaging data captured by the imaging device 10 may be output to the gesture segment extraction unit 120. .

In the first embodiment described above, the start/end candidate extracting unit 121 is provided, but the start/end candidate extracting unit 121 may be eliminated if the calculation load is not considered important.

Also, the target operator is not limited to the driver, and may be a passenger. In this case, the passenger's gesture is also detected by the gesture detection device 100, and various vehicle devices can be operated.

The contents executed by the gesture detection device 100 described in the first embodiment correspond to the "gesture detection method" of the present invention. Further, in the above-described first embodiment, the gesture detection device 100 is used for gesture detection. By having them function as 110, 120, 130, gesture detection is possible.

10 imaging device 100 gesture detection device 120 gesture segment extraction unit 121 start/end candidate extraction unit 122 similarity calculation unit 123 start/end determination unit 130 gesture identification unit

Claims (4)

A gesture detection device for detecting a gesture when performing an input operation on an operation target based on a gesture of an operator,
Based on the feature data extracted from the imaging data of the operator captured by an imaging device (10), the first gesture of a pair of gestures similar to each other among the gestures is referred to as the immediately-before-start gesture, and the subsequent one. a gesture segment extraction unit (120) for extracting a segment of the gesture as a gesture immediately after the end of
a gesture identification unit (130) that identifies the gesture performed between the gesture immediately before the start and the gesture immediately after the end as an input gesture ;
The gesture segment extraction unit
a similarity calculation unit (122) that calculates the similarity for each of a plurality of sets of gestures that are similar to each other;
A start/end determination of extracting a set of gestures having the highest degree of similarity among the degrees of similarity calculated by the degree of similarity calculation unit as the set of gestures, and determining the gesture immediately before the start and the gesture immediately after the end. (123) . 2. The gesture detection device according to claim 1 , wherein the gesture segment extraction unit has a candidate extraction unit (121) for extracting in advance the plurality of sets of similar gesture candidates. A gesture detection method for detecting a gesture when performing an input operation on an operation target based on a gesture of an operator,
Based on the feature amount data extracted from the imaging data of the operator imaged by the imaging device (10), among the gestures, the first gesture of a pair of gestures similar to each other is the gesture just before the start, and the latter one. is extracted as a gesture immediately after the end of
Identifying the gesture made between the just-beginning gesture and the just-ending gesture as an input gesturewith
Calculating the degree of similarity for each of a plurality of sets of gestures that are similar to each other,
Among the calculated degrees of similarity, a set of gestures with the highest degree of similarity is extracted as the set of gestures, and the gesture just before the start and the gesture just after the end are determined.Gesture detection method. In a gesture detection control program for detecting the gesture when performing an input operation on an operation target based on the gesture of the operator,
the computer,
Based on the feature data extracted from the imaging data of the operator captured by an imaging device (10), the first gesture of a pair of gestures similar to each other among the gestures is referred to as the immediately-before-start gesture, and the subsequent one. a gesture segment extraction unit (120) for extracting a segment of the gesture as a gesture immediately after the end of
Functioning as a gesture identification unit (130) that identifies the gesture performed between the gesture immediately before the start and the gesture immediately after the end as an input gesture ,
In the gesture segment extraction unit,
a similarity calculation unit (122) that calculates the similarity for each of a plurality of sets of gestures that are similar to each other;
A start/end determination of extracting a set of gestures having the highest degree of similarity among the degrees of similarity calculated by the degree of similarity calculation unit as the set of gestures, and determining the gesture immediately before the start and the gesture immediately after the end. A gesture detection control program functioning as a part (123) .

Images