JP2017227966A - Gesture recognition device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a gesture recognition device with a reduced processing load.SOLUTION: As shown in the figures 1(a) and 1(b), a gesture recognition device comprises; a main image capturing unit 3 for capturing an image of a main imaging range 30; a sub image capturing unit 4 for capturing an image of a sub imaging range 40 that partially overlaps with the main imaging region 30; and a control unit 6 configured to use a sub image 41 captured by the sub image capturing unit 4 as well as a main image 31 captured by the main image capturing unit 3 to identify a gesture when the gesture cannot be identified from the main image 31.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a gesture determination device.

  As a conventional technique, a gesture for determining whether a gesture is push or pull as a function of the distance from the subject to be gestured to the camera and the feature of the movement trace of the subject in the imaging plane of the two cameras. Is known (see, for example, Patent Document 1).

Special table 2013-525909 gazette

  However, a conventional device for recognizing a gesture processes images captured by two cameras, and thus has a problem of a large processing load.

  Accordingly, an object of the present invention is to provide a gesture determination apparatus that can reduce the processing load.

  According to one aspect of the present invention, a main imaging unit that images a main imaging region, at least one slave imaging unit that images a slave imaging region that is partially in common with the main imaging region, and a main image captured by the main imaging unit When a gesture cannot be determined based on the above, a gesture determination device is provided that includes at least one sub-image captured by at least one sub-imaging unit and a determination unit that determines a gesture using the main image.

  According to the present invention, the processing load can be reduced.

1A is a schematic diagram of the inside of a vehicle in which an example of a gesture determination device according to an embodiment is arranged, and FIG. 1B is a block diagram illustrating an example of a gesture determination device. (C) is a block diagram showing an example of a vehicle communication system to which a gesture determination device is connected. FIG. 2 is a schematic diagram illustrating an example of a camera arrangement and an imaging region of the gesture determination device according to the embodiment. FIG. 3 is a flowchart illustrating an example of the operation of the gesture determination device according to the embodiment.

(Summary of embodiment)
The gesture determination apparatus according to the embodiment images a main imaging unit that images a main imaging region, at least one slave imaging unit that images a slave imaging region that is partly in common with the main imaging region, and the main imaging unit. When it is not possible to determine a gesture based on a main image, at least one sub image captured by at least one sub image capturing unit and a determination unit that determines a gesture using the main image are schematically configured.

  This gesture determination apparatus uses at least one sub-image captured by at least one sub-imaging unit different from the main imaging unit so that the main image is complemented only when the gesture cannot be determined based on the main image. Therefore, the processing load can be reduced as compared with the case where the determination is always performed based on the images captured by the plurality of imaging units.

[Embodiment]
(Outline of gesture determination device 1)
1A is a schematic diagram of the inside of a vehicle in which an example of a gesture determination device according to an embodiment is arranged, and FIG. 1B is a block diagram illustrating an example of a gesture determination device. (C) is a block diagram showing an example of a vehicle communication system to which a gesture determination device is connected. FIG. 2 is a schematic diagram illustrating an example of a camera arrangement and an imaging region of the gesture determination device according to the embodiment. Note that, in each drawing according to the embodiment described below, the ratio between figures may be different from the actual ratio. Moreover, in FIG.1 (b) and FIG.1 (c), the flow of main information is shown by the arrow.

  The gesture determination device 1 enables, for example, an operation of an electronic device mounted on the vehicle 8 based on a gesture made by a detection target. The operator can operate the function of the electronic device corresponding to the gesture by performing a predetermined gesture. This electronic device is, for example, the navigation device 84, the music and video playback device 85 shown in FIG. As an example, the gesture determination device 1 and the electronic device constitute a vehicle communication system 81 as shown in FIG. In the present embodiment, the operator's hand 9 is a detection target.

  As shown in FIG. 1A to FIG. 2, the gesture determination device 1 captures at least one main imaging unit 3 that images the main imaging region 30, and at least one sub-imaging region that is partially shared by the main imaging region 30. When a gesture cannot be determined based on one sub imaging unit and the main image 31 captured by the main imaging unit 3, the gesture is determined using at least one sub image captured by at least one sub imaging unit and the main image 31. And a control unit 6 as a determination unit.

  The control unit 6 is configured to detect the hand 9 performing a gesture based on the main image 31 and to track the movement of the hand 9. In addition, the control unit 6 has a precondition for the gesture related to the shape of the hand 9, and when the shape of the hand 9 does not match the precondition by the main image 31, the control unit 6 uses at least one sub-image together with the main image 31. It is configured to check whether or not the shape and the precondition match. This precondition is included in the control unit 6 as precondition information 61 described later.

  The at least one slave imaging unit of the present embodiment is the slave imaging unit 4 as an example. For example, as shown in FIG. 2, the slave imaging unit 4 captures a slave imaging region 40 and generates a slave image 41. The common area of the main imaging area 30 and the secondary imaging area 40 is, for example, the common area 5 shown in FIG. The common area 5 is an area where the main imaging area 30 and the secondary imaging area 40 intersect, and is formed on the front surface of the display device 87.

  When there are a plurality of slave imaging units, as an example, the slave imaging units may be arranged at least one each on the left and right sides of the main imaging unit 3, may be arranged at least one above and below, You may arrange | position at least 1 each on either side.

(Configuration of the main imaging unit 3 and the secondary imaging unit 4)
The main imaging unit 3 and the secondary imaging unit 4 are imaging apparatuses using semiconductor elements such as a CCD image sensor (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  As an example, the main imaging unit 3 and the secondary imaging unit 4 are arranged side by side on a display device 87 arranged on the center console 80 of the vehicle 8 as shown in FIG. The main imaging unit 3 and the secondary imaging unit 4 are configured to irradiate near infrared rays in the respective imaging region directions, for example.

  The main imaging unit 3 is arranged so that a gesture performed on the front surface of the display device 87 is within the main imaging region 30. The slave imaging unit 4 is arranged so that a gesture performed on the front surface of the display device 87 is within the slave imaging region 40.

The main imaging unit 3 periodically images the main imaging region 30, generates image information S ₁ as information of the captured main image 31, and outputs the image information S ₁ to the control unit 6. The slave imaging unit 4 captures the slave imaging region 40 when required by the control of the controller 6, generates image information S ₂ as information of the captured slave image 41, and outputs the image information S ₂ to the controller 6.

(Configuration of control unit 6)
The control unit 6 includes, for example, a CPU (Central Processing Unit) that performs operations and processes on acquired data according to a stored program, a RAM (Random Access Memory) that is a semiconductor memory, a ROM (Read Only Memory), and the like. Microcomputer. In this ROM, for example, a program for operating the control unit 6 is stored. For example, the RAM is used as a storage area for temporarily storing calculation results and the like.

  For example, as shown in FIG. 1B, the control unit 6 has hand shape information 60, precondition information 61, and gesture information 62 in a ROM or RAM. The control unit 6 is electrically connected to the main imaging unit 3 and the sub imaging unit 4.

  The hand shape information 60 is information regarding the shape of the hand. For example, the control unit 6 determines the shape of the hand based on the pattern matching method based on the main image 31 and the hand shape information 60.

  The precondition information 61 is information regarding preconditions for determining a gesture. The determination of the precondition is performed by, for example, a pattern matching method. For example, it is a precondition for the gesture of releasing the hand to detect the shape of the hand with the hand open and the thumb up. A gesture for drawing a circle with a hand is precondition that, for example, the shape of the hand with the index finger pointing toward the display device 87 is detected while the hand is held. The gesture for moving the hand up and down is, for example, a precondition that the shape of the hand with the palm of the hand facing down is detected as shown in FIG.

  The gesture information 62 is information related to a gesture, for example. The gesture performed by the hand 9 is, for example, a gesture for paying the hand 9, a gesture performed to draw a circle by the hand 9, a gesture for moving the palm of the hand 9 up and down as described above, and the like. . For example, the control unit 6 determines the gesture by comparing the movement of the hand 9 with the gesture information 62 based on the pattern matching method.

  Specifically, for example, the control unit 6 first confirms whether or not the hand 9 is detected in the main imaging region 30. Next, when the hand 9 is detected, the control unit 6 determines the shape of the hand 9 based on the hand shape information 60. Next, the control unit 6 determines whether the determined shape of the hand 9 matches the precondition based on the precondition information 61. Then, when the shape of the hand 9 matches the precondition, the control unit 6 tracks the movement of the hand 9 based on a plurality of main images 31 acquired periodically, and gesture information 62 determines whether the movement of the hand 9 is a gesture. Determine based on.

Control unit 6 generates and outputs the operation information S ₃ as the information about the determined gesture. Operation target electronic device, for example, performs an operation corresponding to the gesture based on the vehicle LAN (Local Area Network) 82 operation information S ₃ obtained through the described later in the vehicle communication system 81.

  Here, the control unit 6 includes a mode for determining a gesture with a single eye (monocular mode) and a mode for determining a gesture with a compound eye (compound eye mode).

  In the monocular mode in the present embodiment, the gesture is determined based on the main image 31 captured by the main imaging unit 3. In the monocular mode, imaging of the slave image 41 by the slave imaging unit 4 is stopped.

  In the compound eye mode in the present embodiment, the gesture is determined based on the main image 31 and the sub image 41 captured by the main image capturing unit 3 and the sub image capturing unit 4. In other words, the compound eye mode is a mode in which a gesture is also determined using the slave image 41 captured by the slave imaging unit 4 when the gesture cannot be determined based on the master image 31 captured by the master imaging unit 3.

  As shown in FIG. 2, when a gesture is performed with the palm of the hand 9 down, for example, a main image 31 obtained by imaging the hand 9 from an oblique direction is obtained. As shown in FIG. 2, the main image 31 is an image in which the main imaging range 35 is mainly imaged. Therefore, although the thumb, index finger, middle finger, and the like included in the main imaging range 35 are imaged, the ring finger and the little finger may not be imaged due to being hidden by other fingers. Thus, when a part of the finger of the hand 9 is not imaged, for example, it may be difficult to determine the shape of the hand 9 or to determine the precondition.

  Therefore, the control unit 6 shifts from the monocular mode to the compound eye mode, and the sub image 41 including the sub image capturing range 45 that complements the main image capturing range 35 and the main image 31 including the main image capturing range 35 complement each other. Gesture determination is performed using two images. With the main imaging range 35 and the secondary imaging range 45, all the images of the finger of the hand 9 are prepared, and the shape of the hand 9 and the precondition can be determined.

(Configuration of vehicle communication system 81)
The vehicle communication system 81 is a communication system using a standard such as CAN (Controller Area Network) as an example. In the vehicle communication system 81, for example, the gesture determination device 1 and the navigation device 84 communicate with each other via a vehicle LAN (Local Area Network) 82.

  As an example, the vehicle communication system 81 includes a gesture determination device 1, a vehicle control unit 83, a navigation device 84, a music and video playback device 85, an air conditioner 86, and a display device 87 as shown in FIG. It is configured to communicate with each other. The vehicle control unit 83 is, for example, a microcomputer that controls the vehicle communication system 81.

If navigation device 84 is to be operated, the operation information S ₃ output from the gesture determination apparatus 1 is input to the navigation device 84 via the vehicle LAN82. The navigation device 84 performs an operation based on the operation information S ₃ entered.

  Below, an example of the operation | movement regarding determination of the gesture of the gesture determination apparatus 1 of this Embodiment is demonstrated according to the flowchart of FIG.

(Operation)
Control unit 6 of the gesture determination apparatus 1, when the power supply of the vehicle 8 is turned on, controls the main image pickup unit 3 modes as monocular mode, acquires image information S ₁ of the captured primary imaging region 30 periodically (Step 1).

Control unit 6, or by hand 9 based on the image information S ₁ acquired is captured, that is to check whether the hand 9 is detected. When the hand 9 is detected (Step 2: Yes), the control unit 6 determines the shape of the hand 9 based on the main image 31 and the hand shape information 60 (Step 3).

  When the shape of the hand 9 is determined (Step 4: Yes), the control unit 6 determines whether or not the shape of the hand 9 matches the precondition of the gesture based on the precondition information 61. When the preconditions match (Step 5: Yes), the control unit 6 starts the gesture determination (Step 6) and sets the mode to the monocular mode (Step 7). Then, the control unit 6 determines a gesture based on the acquired main image 31 and the gesture information 62.

When the determination of the gesture is completed (Step 8: Yes), the control unit 6 performs the hand operation based on the acquired plurality of main images 31 until the gesture is completed or the hand 9 is not captured (the hand 9 is out of frame). Track 9 moves. Control unit 6, "Yes" is established in step 9, that is, if either hands 9 gesture is completed and frame-out (Step9: Yes), generates and outputs operation information S ₃ indicating the determination result of the gesture (Step 10) The process proceeds to Step 1 in order to detect the next gesture.

  Here, in Step 2, when the hand 9 is not detected (Step 2: No), the control unit 6 advances the process to Step 1.

  In step 4, if the shape of the hand 9 cannot be determined by the main image 31 (Step 4: No), the control unit 6 sets the mode to the compound eye mode (Step 11), and controls the sub imaging unit 4 to control the sub imaging region 40. To obtain the sub image 41. The control unit 6 determines the shape of the hand 9 based on the main image 31 and the sub image 41 (Step 12), and proceeds to step 5.

  Furthermore, in step 5, the control part 6 advances a process to step 1, when a precondition does not correspond (Step5: No). If the operation of the hand 9 is completed or the frame is out before the preconditions match, the process proceeds to step 1.

Furthermore, in step 8, when the gesture determination is not completed (Step 8: No), the control unit 6 confirms whether the hand 9 is out of frame. When the hand 9 is not out of frame (Step 13: No), the control unit 6 proceeds to Step 8 to determine a gesture. The control unit 6, when the hand 9 is out of frame (Step13: Yes), the operation information S ₃ to could not be determined _gesture, that outputs operation information S ₃ to no gesture as a determination result of the gesture ( Step 10).

(Effect of embodiment)
The gesture determination apparatus 1 according to the present embodiment can reduce the processing load. The gesture determination device 1 uses a sub image 41 captured by a sub image capturing unit 4 different from the main image capturing unit 3 so that the main image 31 is complemented only when a gesture cannot be determined based on the main image 31. Therefore, it is possible to suppress power consumption and reduce processing load as compared to a case where determination is always performed based on images captured by a plurality of imaging units.

  In the gesture determination apparatus 1, the image capturing unit 4 is arranged so as to compensate for a portion that cannot be captured by the main image capturing unit 3, so that the gesture determination accuracy is improved as compared with a case where this configuration is not adopted.

  Since the gesture determination apparatus 1 uses the monocular mode and the compound eye mode properly, the gesture determination speed is improved without always processing a plurality of images captured with the compound eye as compared with the case where the gesture is always determined with the compound eye.

  The gesture determination apparatus 1 according to the above-described embodiment and modification is, for example, a program executed by a computer, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, depending on applications. It may be realized by.

  As mentioned above, although some embodiment and modification of this invention were demonstrated, these embodiment and modification are only examples, and do not limit the invention based on a claim. These novel embodiments and modifications can be implemented in various other forms, and various omissions, replacements, changes, and the like can be made without departing from the scope of the present invention. In addition, not all combinations of features described in these embodiments and modifications are necessarily essential to the means for solving the problems of the invention. Further, these embodiments and modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 ... Gesture determination apparatus, 3 ... Main imaging part, 4 ... Sub imaging part, 5 ... Common area, 6 ... Control part, 8 ... Vehicle, 9 ... Hand, 30 ... Main imaging area, 31 ... Main image, 35 ... Main Imaging range, 40 ... subordinate imaging region, 41 ... subordinate image, 45 ... subordinate imaging range, 60 ... hand shape information, 61 ... precondition information, 62 ... gesture information, 80 ... center console, 81 ... vehicle communication system, 82 ... Vehicle LAN, 83 ... Vehicle control unit, 84 ... Navigation device, 85 ... Video playback device, 86 ... Air conditioning device, 87 ... Display device

Claims (3)

A main imaging unit that images the main imaging region;
At least one slave imaging unit that images a slave imaging region that is partly in common with the main imaging region;
A determination unit that determines the gesture using at least one sub-image captured by the at least one sub-imaging unit and the main image when a gesture cannot be determined based on the main image captured by the main imaging unit;
Gesture determination device comprising: The determination unit performs detection of a detection target that performs the gesture based on the main image, and tracks movement of the detection target.
The gesture determination apparatus according to claim 1. The determination unit has a precondition of the gesture related to the shape of the detection target, and when the shape of the detection target and the precondition are not matched by the main image, the at least one sub image is displayed together with the main image. Use to check if the shape of the detection target and the precondition match,
The gesture determination apparatus according to claim 2.

Images