WO2015136952A1

WO2015136952A1 - Gesture recognition device, head-mounted display, and portable terminal

Info

Publication number: WO2015136952A1
Application number: PCT/JP2015/050539
Authority: WO
Inventors: 義朗平原; 片桐　哲也
Original assignee: コニカミノルタ株式会社
Priority date: 2014-03-12
Filing date: 2015-01-09
Publication date: 2015-09-17
Also published as: JP2017083916A

Abstract

[Problem] To provide a gesture recognition device, a head-mounted display, and a portable terminal, which guide a user so as to correct the position of a gesture intuitively. [Solution] A gesture recognition device (1) which is used when worn or held by a user, said gesture recognition device being equipped with: an imaging unit (42) that continuously captures images; a control unit (43-46) that detects a change in the orientation of the imaging unit (42), tracks a prescribed object using the images captured by the imaging unit (42), determines whether the tracking operation is possible, calculates an amount of change for the detected orientation when the result of the determination is that tracking of the object is not possible, and identifies the cause when tracking is not possible; and a notification unit (47) that provides a notification based on the cause that has been identified.

Description

Gesture recognition device, head mounted display, and portable terminal

The present invention relates to a gesture recognition device, a head mounted display, and a mobile terminal.

In recent years, a head mounted display (HMD) has been developed as one of wearable computers. A head mounted display is a display device that is used by being mounted on a user's head. The user can view various contents such as image information displayed in front of the user with the head mounted display mounted on the head.

Some of such head mounted displays can capture a part of the user's body using a small camera and recognize the movement (gesture) of the body as an input operation to the head mounted display.

Although Patent Document 1 is not an example using a head-mounted display, a small camera is installed on the ceiling, and a user's gesture made under the small camera is captured and recognized, and input to a computer. Techniques for determining commands are described.

When a small camera is used as a fixed camera as in Patent Document 1, when the user's body is moved out of the field of view of the small camera, the user can be prompted to correct the position of the gesture.

JP 2013-196482 A

However, when using a head-mounted display, the small camera is mounted on the head-mounted display. Therefore, if the user's head is shaken, the small camera is also shaken. Therefore, when the user's body moves out of the field of view of the small camera, it cannot be distinguished whether the small camera has moved or the user's body has moved. In particular, in a situation where the user wears a head-mounted display and is gazing at the display, it tends to be difficult to distinguish whether the small camera has moved or the user's body has moved. Therefore, even if the user is prompted to correct the position of the gesture without showing a specific method for correcting the position of the gesture, the user cannot determine how to correct the position of the gesture.

The present invention has been made in view of the above circumstances, and an object thereof is to provide a gesture recognition device, a head-mounted display, and a mobile terminal that guide a user to intuitively correct the position of a gesture.

(1) A gesture recognition device that is worn or held by a user and that uses an image capturing unit that continuously captures images, changes in the orientation of the image capturing unit, and images captured by the image capturing unit Tracking the predetermined target, determining whether the tracking is possible, and if the target cannot be tracked based on the determination result, the amount of change in the detected direction is calculated, and the reason why the tracking is impossible A gesture recognizing apparatus comprising: a control unit that identifies a notification; and a notification unit that performs notification based on the identified cause.

(2) The gesture recognition device according to (1), wherein the predetermined target is a part of a user's body.

(3) The gesture recognition device according to (1) or (2), wherein the notification unit performs different notifications according to the cause specified by the specifying unit.

(4) The gesture recognition according to any one of (1) to (3), wherein the notification by the notification unit includes at least one of notification by display, notification by sound, and notification by vibration apparatus.

(5) The gesture recognition device according to any one of (1) to (4), wherein the tracking by the control unit is performed in a recognition region set within a field of view of the imaging unit.

(6) The control unit determines a command according to the tracked movement of the target, and the notification unit notifies when the target cannot be tracked when the command is determined by the control unit. The gesture recognition device according to any one of (1) to (5), wherein:

(7) A head-mounted display having a function as the gesture recognition device according to any one of (1) to (6) above.

(8) A mobile terminal having a function as the gesture recognition device according to any one of (1) to (6) above.

According to the present invention, when a predetermined target (part of the user's body) can no longer be tracked, the cause is identified and appropriate notification based on the identified cause is performed. This notification allows the user to easily determine in which direction the head should be moved and in which direction the hand should be moved so that the head-mounted display can correctly recognize the gesture. In this way, the user can intuitively correct the position of the gesture.

It is a figure which shows the example of an external appearance of the head mounted display (HMD) apparatus which concerns on this embodiment. It is a block diagram which shows the hardware structural example of an HMD apparatus. It is a block diagram which shows the function structural example of an HMD apparatus. It is a flowchart which shows the procedure of gesture recognition processing. It is a flowchart which shows the procedure of a warning process. (A) It is a figure which shows the example of a state when object moves to the right direction. (B) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the right side of a gesture recognition area. (A) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the left side of a gesture recognition area. (B) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the upper side of a gesture recognition area. (C) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the lower side of a gesture recognition area. (D) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the upper left side of a gesture recognition area. (A) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the upper right side of a gesture recognition area. (B) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the lower left side of a gesture recognition area. (C) It is a figure which shows the example of notification (notification A) when a target remove | deviates to the lower right side of a gesture recognition area. (A) It is a figure which shows the example of a state when a user's head turns to the left direction. (B) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the right side of a gesture recognition area. (A) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the left side of a gesture recognition area. (B) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the upper side of a gesture recognition area. (C) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the lower side of a gesture recognition area. (D) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the upper left side of a gesture recognition area. (A) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the upper right side of a gesture recognition area. (B) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the lower left side of a gesture recognition area. (C) It is a figure which shows the example of notification (notification B) when a target remove | deviates to the lower right side of a gesture recognition area. It is a figure which shows the modification of a gesture recognition area. It is a figure for demonstrating the example which applies this invention to a portable terminal.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios.

FIG. 1 is a diagram showing an example of the appearance of a head mounted display (HMD) device according to this embodiment. FIG. 2 is a block diagram illustrating a hardware configuration example of the HMD device.

Hereinafter, with reference to FIG. 1 and FIG. 2, a schematic configuration of the HMD device 1, particularly a hardware configuration will be described.

<HMD device 1 (hardware configuration)>
The HMD device 1 according to the present embodiment is an image display device that is used by being mounted on the head of a user (hereinafter also referred to as “observer”). For example, the HMD device 1 has a structure imitating eyeglasses for correcting vision. That is, the HMD device 1 includes a pair of left and right temples L1 and L2, a bridge B, and a pair of left and right transparent members G1 and G2.

Temples L1 and L2 are, for example, long bar-shaped members made of an elastic material. Each of the temples L1 and L2 is provided with an ear hook portion that is hung on the user's ear, and the transparent members G1 and G2 are fixed to the other end portion. In addition, a control unit U is mounted in the vicinity of the ear hook portion that is hung on one ear of the user.

The bridge B is a short bar-like member for connecting the pair of left and right transparent members G1 and G2 to each other. Transparent members G1 and G2 are fixed to both ends of the bridge B. The pair of left and right transparent members G1 and G2 are held at a predetermined interval.

The transparent members G1 and G2 are transparent materials (glass, plastic, film, etc.) that can transmit light from the outside world to the user's eyes so that the user wearing the HMD device 1 can observe the outside world It is formed by.

In the present embodiment, the imaging device 11 is provided above the transparent member G1 corresponding to the user's right eye, and the display device 12 is provided so as to overlap the transparent member G1.

The imaging device 11 images the outside of the camera lens in the optical axis direction (hereinafter also referred to as “imaging direction”). The imaging device 11 is fixedly held with respect to the HMD device 1 so that the optical axis of the camera lens substantially coincides with the user's line-of-sight direction. As a result, the imaging device 11 can take an image of the user's front visual field. For example, the imaging device 11 is a digital camera or a video camera including an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). For example, a captured image such as JPEG or MPEG2 / 4 (or a moving image captured at a predetermined frame rate) is generated by imaging by the imaging device 11 and transmitted to the control unit U connected by wire. In the present embodiment, a personal-use web camera (with an angle of view of about 72 degrees) is used as the imaging device 11.

The display device 12 displays various contents such as a predetermined image (video) on the transparent members G1 and G2. The display device 12 may display an image (AR image) that is additionally superimposed on light from the outside. For example, the display device 12 can display a pop-up image including information such as a name and work place as if there is a person near the person in the direction of the user's line of sight. The display device 12 may be a binocular type instead of the monocular type as shown in FIG.

Although not shown in FIG. 1, an inertial sensor 13 is provided in the vicinity of the imaging device 11. The inertial sensor 13 is a sensor that detects shaking of the HMD device 1 and provides data necessary for calculating the shaking amount of the imaging device 11 at that time. For example, the inertial sensor 13 includes an acceleration sensor, a gyro sensor, or the like. The inertial sensor 13 may have any form, name, or structure as long as it can provide data necessary for calculating the amount of shaking of the imaging device 11.

The control unit U has a configuration necessary for controlling the imaging device 11, the display device 12, and the inertial sensor 13 to function as a head mounted display as a whole. For example, the control unit U recognizes the user's gesture from the captured image (moving image) transmitted from the imaging device 11, executes a command corresponding to the recognized gesture, and displays the result on the display device 12. Perform the process. A specific configuration of the control unit U will be described later.

(Specific configuration of the control unit U)
Next, a specific configuration of the control unit U will be described.

As shown in FIG. 2, the control unit U includes a CPU (Central Processing Unit) 31, a memory 32, a storage 33, an input device 34, and a display controller 35.

The CPU 31 is a control circuit composed of a processor or the like that executes control of each unit and various arithmetic processes according to a program, and each function of the HMD device 1 is exhibited by the CPU 31 executing a corresponding program. The

The memory 32 is a high-speed accessible main storage device that temporarily stores programs and data as a work area. As the memory 32, for example, DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Access Memory), SRAM (Static Random Access Memory), or the like is adopted.

The storage 33 is a large-capacity auxiliary storage device that stores various programs including an operating system and various data. As the storage 33, for example, a flash memory or the like is employed.

The input device 34 is composed of physical keys and buttons for inputting instructions such as power-on of the HMD device 1. Moreover, you may utilize an external controller (for example, a smart phone, a remote controller, etc.) as the input device 34. FIG.

The display controller 35 causes the display device 12 to display various contents such as a predetermined image (video), an AR image, and the like. For example, the display controller 35 reads a predetermined image from the memory 32 every predetermined display cycle, converts it into a signal for input to the display device 12, and generates a pulse signal such as a horizontal synchronizing signal and a vertical synchronizing signal. To do.

The HMD device 1 having the hardware configuration as described above has the following functional configuration.

FIG. 3 is a block diagram illustrating a functional configuration example of the HMD device.

<HMD device 1 (functional configuration)>
As shown in FIG. 3, the HMD device 1 includes, as a functional configuration, a control unit 41, an imaging control unit 42, a shake detection unit 43, a tracking unit 44, a tracking availability determination unit 45, a cause identification unit 46, A notification unit 47, a command determination unit 48, a display control unit 49, and an application unit 50 are included.

Hereinafter, each functional unit 41 to 50 will be described in detail. The function units 41 to 48 and 50 are realized by the CPU 31 reading the program installed in the storage 33 into the memory 32 and executing it, and the display control unit 49 is realized by the display controller 35. The functional units 41 to 48 and 50 are not limited to this, and may be realized by hardware such as an ASIC.

The control unit 41 controls the entire HMD device 1.

The imaging control unit 42 controls the imaging device 11 to continuously image the outside world. Thereby, a captured image (moving image) is generated.

The shake detection unit 43 detects a change in the direction of the imaging device 11 (for example, the optical axis direction of the camera lens) when the user wearing the HMD device 1 shakes. For example, the shake detection unit 43 acquires the sensor value of the inertial sensor 13 and detects a change in the orientation of the imaging device 11.

The tracking unit 44 tracks a predetermined target (for example, the user's hand, fingertip, etc.) in order to recognize the user's gesture. For example, the tracking unit 44 sets an area for recognizing a user's gesture in advance (hereinafter referred to as “gesture recognition area”), and tracks a predetermined target in the gesture recognition area. In the present embodiment, the gesture recognition area is set within the field of view of the imaging device 11.

The tracking availability determination unit 45 determines whether tracking by the tracking unit 44 is possible.

The cause identification unit 46 calculates the amount of change in the orientation of the imaging device 11 detected by the shake detection unit 43 when the predetermined target cannot be tracked based on the determination result of the tracking enable / disable determination unit 45, and indicates that the tracking is impossible. Identify the cause. For example, when the amount of change in the orientation of the imaging device 11 is greater than or equal to the threshold value, the cause identifying unit 46 identifies that tracking is impossible due to the user's body shaking and the imaging device 11 moving. On the other hand, if the amount of change in the orientation of the imaging device 11 is less than the threshold value, it is determined that the predetermined target cannot be tracked due to the movement (gesture) of the predetermined target itself.

The notification unit 47 performs notification based on the cause specified by the cause specifying unit 46. For example, the notification unit 47 performs different notifications depending on the cause specified by the cause specifying unit 46. The notification performed by the notification unit 47 includes at least one of notification by display (display device 12), notification by voice (not shown speaker), and notification by vibration (small vibrator not shown). included.

The command determination unit 48 determines the user's instruction content, that is, the command to the HMD device 1 according to the movement (gesture) of the predetermined target tracked by the tracking unit 44.

The display control unit 49 causes the display device 12 to display various contents such as a predetermined image (video), an AR image, and the like.

The application unit 50 executes processing corresponding to the command determined by the command determination unit 48. For example, the application unit 50 performs processing such as display screen enlargement processing, reduction processing, content playback, pause, and stop.

The detailed operation of each functional unit 41 to 50 will be described in detail below.

<Operation of HMD Device 1>
FIG. 4 is a flowchart showing the procedure of the gesture recognition process. FIG. 5 is a flowchart showing the procedure of the warning process. FIG. 6A is a diagram illustrating a state example when the object moves in the right direction. FIGS. 6 (B), 7 (A) to (D), and FIGS. 8 (A) to (C) are respectively the right side, left side, upper side, lower side, upper left side, upper right side of the gesture recognition area, It is a figure which shows the example of notification (notification A) when it remove | deviates to the lower left side and the lower right side. FIG. 9A is a diagram illustrating an example of a state when the user's head is directed leftward. 9 (B), 10 (A) to (D), and 11 (A) to (C), the objects are the right side, left side, upper side, lower side, upper left side, upper right side of the gesture recognition area, respectively. It is a figure which shows the example of notification (notification B) when it remove | deviates to the lower left side and the lower right side.

Hereinafter, the gesture recognition process and the warning process will be described with reference to FIGS.

(Gesture recognition processing)
The HMD device 1 starts the gesture recognition process shown in FIG. 4 at the timing when the power is turned on. However, the timing of starting the gesture recognition process is not limited to this, and the gesture recognition process may be started when an operation for starting the gesture recognition process is performed on the input device 34. The gesture recognition process may be started when it is detected that the HMD device 1 is mounted on the user's head.

Note that immediately after the start of the gesture recognition process, the control unit 41 calibrates various parameters necessary for controlling the HMD device 1. For example, registration of the reference value of the inertial sensor 13 and setting of a gesture recognition area are performed. In the present embodiment, the control unit 41 acquires a sensor value from the inertial sensor 13 and registers it in the memory 32 as a reference value for the orientation of the imaging device 11. Further, the control unit 41 sets the entire field of view of the imaging device 11 (the entire region of the captured image) as a gesture recognition area.

[Step S101]
After calibration, the HMD device 1 functions as the imaging control unit 42 and continuously images the outside world at a predetermined frame rate (a predetermined number of frames). Specifically, the HMD device 1 generates a captured image (moving image) in the optical axis direction of the camera lens by controlling the imaging device 11 and starting imaging. Then, the HMD device 1 stores the generated captured image (moving image) in the memory 32 or the storage 33. In addition, while the imaging device 11 is imaging, the user can perform a gesture operation by moving a part of the body (for example, a hand) so that the HMD device 1 can recognize it. Examples of such gesture movements include pinch-out that spreads the thumb and index finger from the closed state, pinch-in that closes the thumb and index finger from the expanded state, slide of gesture that moves the hand or extended index finger, spread hand or Tap with a finger, double tap to tap it continuously, swipe to move your hand left and right or up and down.

[Step S102]
The HMD device 1 functions as the shake detection unit 43 and acquires a sensor value from the inertial sensor 13. For example, the HMD device 1 acquires acceleration in the triaxial direction from the inertial sensor 13. In step S102, the HMD device 1 acquires a sensor value every time a captured image for one frame is generated in step S101, and associates the captured image with the sensor value for each frame and stores them in the memory 32 or the like. .

[Step S103]
The HMD device 1 functions as a tracking unit 44 and tracks a predetermined target (for example, a user's hand) by a general pattern matching method. At this time, the HMD device 1 functions as the traceability determination unit 45 to determine whether a predetermined target is traceable or untraceable. Specifically, the HMD device 1 has features (colors and shapes) that are the same as or similar to the template image of “hand” prepared in advance from the captured images (each frame) generated in step S101. Find the area you have. Then, the HMD device 1 calculates the movement amount of the “hand” between all successive frames. The HMD device 1 determines that the “hand” can be tracked when the movement amount of the “hand” between all the consecutive frames is less than a predetermined amount. On the other hand, the HMD device 1 determines that the movement amount of the “hand” between at least one continuous frame is equal to or greater than a predetermined amount, or there is a frame in which no “hand” exists and If the amount of movement cannot be calculated, it is determined that the “hand” cannot be tracked.

When it is determined that the “hand” can be tracked, the HMD device 1 determines that the predetermined target can be tracked (step S103: YES), and the process proceeds to step S105. On the other hand, when it is determined that the “hand” cannot be tracked, the HMD device 1 determines that the predetermined target cannot be tracked (step S103: NO), and the process proceeds to step S104.

[Step S104]
The HMD device 1 executes a warning process for prompting the user to redo the gesture. Details of the warning process will be described later.

[Step S105]
The HMD device 1 functions as the shake detection unit 43 and detects a change in the orientation of the imaging device 11 (for example, the optical axis direction of the camera lens). For example, the shake detection unit 43 obtains the difference between the sensor values of the first and last frames among the sensor values of all the frames acquired in step S102, and if the difference is less than a predetermined amount, the imaging device 11 Judge that there is no change in the direction of the. On the other hand, the HMD device 1 determines that there is a change in the orientation of the imaging device 11 when the difference between the sensor values between the first and last frames is a predetermined amount or more.

When the HMD device 1 determines that there is no change in the orientation of the imaging device 11 (step S105: NO), the process of step S106 is omitted, and the process proceeds to step S107. On the other hand, when the HMD device 1 determines that there is a change in the orientation of the imaging device 11 (step S105: YES), the process proceeds to step S106.

[Step S106]
The HMD device 1 functions as the shake detection unit 43 and corrects the locus of movement of a predetermined target in consideration of the amount of change in the orientation of the imaging device 11. That is, the HMD device 1 converts the movement amount of the “hand” between all successive frames into a movement amount corresponding to the actual movement of the “hand”. Specifically, the HMD device 1 may synthesize the movement amount (vector) of the “hand” in the captured image and the change amount (vector) of the orientation of the imaging device 11 between consecutive frames.

[Step S107]
The HMD device 1 functions as the command determination unit 48 and recognizes the user's gesture from the movement amount of the “hand” calculated in step S103 or the movement amount of the “hand” corrected in step S106. Then, the HMD device 1 determines a user instruction content, that is, a command to the HMD device 1 according to the recognized gesture. Note that gestures and commands are associated with each other in a one-to-one relationship and registered in advance in a predetermined table, and the HMD device 1 may read commands associated with the recognized gesture from the table.

[Step S108]
The HMD device 1 functions as the command determination unit 48 and determines whether or not the command determined in step S107 is a regular command. Specifically, the HMD device 1 is not a legitimate command when there is no command corresponding to the gesture recognized in step S107, or when it is a command for instructing control restricted at that time. judge. On the other hand, if the command corresponding to the gesture recognized in step S107 is a command for instructing control that can be executed at that time, the HMD device 1 determines that the command is a regular command.

If the HMD device 1 determines that the command is a regular command (step S108: YES), the process proceeds to step S109. If the HMD device 1 determines that the command is not a regular command (step S108: NO), the process proceeds to step S110. Proceed to

[Step S109]
The HMD device 1 functions as the application unit 50 and executes processing corresponding to the command determined by the command determination unit 48. For example, if the command determined in step S107 is a command corresponding to a gesture such as pinch-out or pinch-in, the HMD device 1 executes enlargement processing, reduction processing, and the like. In addition, when the command determined in step S107 is a command corresponding to a gesture such as slide, tap, or double tap, the HMD device 1 executes processing such as content playback, pause, and stop.

[Step S110]
The HMD device 1 functions as the notification unit 47 and the display control unit 49 and notifies an error indicating that processing corresponding to the user's gesture cannot be executed. For example, the HMD device 1 causes the display device 12 to display a message such as “Please retry the gesture at the same position”.

[Step S111]
The HMD device 1 functions as the control unit 41 and determines whether or not an instruction to end the gesture recognition process has been given by the user. Specifically, the HMD device 1 may determine whether or not the input device 34 has been operated to end the gesture recognition process.

The HMD device 1 returns the process to step S101 when the instruction to end the gesture recognition process has not been given (step S111: NO), and continues the gesture recognition process. On the other hand, if an instruction to end the gesture recognition process is given (step S111: YES), the HMD device 1 ends the gesture recognition process.

(Warning process S104)
Next, details of the warning process (step S104) will be described.

When the process proceeds to step S104 in the gesture recognition process described above, the HMD device 1 starts a warning process.

[Step S201]
The HMD device 1 functions as the shake detection unit 43 and detects a change in the orientation of the imaging device 11 (for example, the optical axis direction of the camera lens). For example, the shake detection unit 43 obtains the difference between the sensor values of the first and last frames among the sensor values of all the frames acquired in step S102, and if the difference is less than a predetermined amount, the imaging device 11 Judge that there is no change in the direction of the. On the other hand, the HMD device 1 determines that there is a change in the orientation of the imaging device 11 when the difference between the sensor values between the first and last frames is a predetermined amount or more.

If the HMD device 1 determines that there is no change in the orientation of the imaging device 11 (step S201: NO), the process proceeds to step S202. On the other hand, when the HMD device 1 determines that there is a change in the orientation of the imaging device 11 (step S201: YES), the process proceeds to step S206.

[Step S202]
The HMD device 1 functions as the command determination unit 48 and recognizes the user's gesture from the movement amount of the “hand” calculated in step S103. Then, the HMD device 1 determines a user instruction content, that is, a command to the HMD device 1 according to the recognized gesture. However, if the movement amount of the “hand” cannot be calculated in step S103, there is a high possibility that the gesture includes a movement in which a predetermined target (for example, the user's hand) goes out of the gesture recognition area. . Therefore, the HMD device 1 recognizes the user's gesture from the amount of movement of the “hand” between the frames captured while the “hand” is moving in the gesture recognition area, and determines the corresponding command. To do.

[Step S203]
The HMD device 1 functions as the command determination unit 48 and determines whether or not the command determined in step S202 is a regular command. Specifically, the HMD device 1 is not a legitimate command when there is no command corresponding to the gesture recognized in step S202, or when it is a command for instructing control restricted at that time. judge. On the other hand, if the command corresponding to the gesture recognized in step S202 is a command for instructing control that can be executed at that time, the HMD device 1 determines that the command is a regular command.

If the HMD device 1 determines that the command is a regular command (step S203: YES), the process proceeds to step S204. If the HMD device 1 determines that the command is not a regular command (step S203: NO), the process proceeds to step S205. Proceed to

[Step S204]
The HMD device 1 functions as the application unit 50 and executes processing corresponding to the command determined by the command determination unit 48. For example, when the command determined in step S203 is a command corresponding to a gesture such as swipe, the HMD device 1 performs a process of turning a displayed page.

[Step S205]
The HMD device 1 functions as the notification unit 47 and the display control unit 49, and the imaging device 11 is not moving, but a predetermined target (for example, a user's hand) is out of the gesture recognition area (FIG. 6A). Notification A for informing the user of the state shown in FIG.

For example, as shown in FIG. 6A, when the target is not on the right side of the gesture recognition area as viewed from the user, the HMD device 1 controls the display device 12 and imitates the hand on the right side of the display screen. An image is displayed (FIG. 6B). Thus, the user can easily grasp that the hand has moved to the right side of the gesture recognition area, and intuitively move the gesture position to the left side.

Further, when the target is removed from the left side of the gesture recognition area as viewed from the user, the HMD device 1 controls the display device 12 to display an image imitating the hand on the left side of the display screen (FIG. 7A). )). Thus, the user can easily grasp that the hand has moved to the left side of the gesture recognition area, and intuitively move the gesture position to the right side.

Further, when the target is removed from the upper side of the gesture recognition area as viewed from the user, the HMD device 1 controls the display device 12 to display an image imitating a hand on the upper side of the display screen (FIG. 7B )). Thus, the user can easily grasp that the hand has moved out of the gesture recognition area, and intuitively move the gesture position downward.

Similarly, when the target is removed from the lower side, upper left side, upper right side, lower left side, lower right side of the gesture recognition area, the HMD device 1 displays the lower side, upper left side, upper right side of the display screen. Then, the image imitating the hand is displayed on the lower left side and the lower right side (FIG. 7C, FIG. 7D, FIG. 8A, FIG. 8B, FIG. 8C). As described above, the HMD device 1 displays an image for warning on the side where the target is excluded from the gesture recognition area, and allows the user to intuitively confirm the direction in which the operation is returned.

After the notification A as described above, the HMD device 1 functions as the control unit 41 and returns the process to the gesture recognition process.

[Step S206]
The HMD device 1 functions as the shake detection unit 43 and corrects the locus of movement of a predetermined target in consideration of the amount of change in the orientation of the imaging device 11. That is, the HMD device 1 converts the movement amount of the “hand” between all successive frames into a movement amount corresponding to the actual movement of the “hand”. Specifically, the HMD device 1 may synthesize the movement amount (vector) of the “hand” in the captured image and the change amount (vector) of the orientation of the imaging device 11 between consecutive frames.

However, if the movement amount of the “hand” cannot be calculated in step S103, there is a high possibility that the gesture includes a movement in which a predetermined target (for example, the user's hand) goes out of the gesture recognition area. . Therefore, the HMD device 1 moves only the movement amount of the “hand” between the frames captured while the “hand” is moving in the gesture recognition area as much as the “hand” is actually moved. Convert to quantity.

[Step S207]
The HMD device 1 functions as the command determination unit 48 and recognizes the user's gesture from the movement amount of the “hand” corrected in step S206. Then, the HMD device 1 determines a user instruction content, that is, a command to the HMD device 1 according to the recognized gesture.

[Step S208]
The HMD device 1 functions as the command determination unit 48 and determines whether or not the command determined in step S207 is a regular command. Specifically, the HMD device 1 is not a legitimate command when there is no command corresponding to the gesture recognized in step S207, or when it is a command instructing control restricted at that time. judge. On the other hand, if the command corresponding to the gesture recognized in step S207 is a command for instructing control that can be executed at that time, the HMD device 1 determines that the command is a regular command.

If the HMD device 1 determines that the command is a regular command (step S208: YES), the process proceeds to step S209. If the HMD device 1 determines that the command is not a regular command (step S208: NO), the process proceeds to step S210. Proceed to

[Step S209]
The HMD device 1 functions as the application unit 50 and executes processing corresponding to the command determined by the command determination unit 48. For example, if the command determined in step S207 is a command corresponding to a gesture such as swipe, the HMD device 1 performs a process of turning a displayed page.

[Step S210]
The HMD device 1 functions as the notification unit 47 and the display control unit 49, and the predetermined target (for example, the user's hand) has moved out of the gesture recognition area due to the movement of the imaging device 11 (FIG. 9A). Notification B for informing the user of the state shown in FIG.

For example, as illustrated in FIG. 9A, when the target is moved to the left as the user moves to the right side of the gesture recognition area, the HMD device 1 controls the display device 12 to display the display screen. An image simulating an arrow is displayed on the right side (FIG. 9B). Thereby, the user can easily grasp that the imaging device 11 is facing the left direction, and can intuitively point the head to the right direction.

In addition, when the user turns to the right and the target moves to the left side of the gesture recognition area, the HMD device 1 controls the display device 12 to display an image imitating an arrow on the left side of the display screen. (FIG. 10A). Thus, the user can easily grasp that the imaging device 11 is facing the right direction, and intuitively turn the head to the left.

In addition, when the user is directed downward and the target moves out of the gesture recognition area, the HMD device 1 controls the display device 12 to display an image imitating an arrow on the upper side of the display screen. (FIG. 10B). Thereby, the user can easily grasp that the imaging device 11 is facing downward, and understand that it is only necessary to intuitively point the head upward.

Similarly, when the target is out of the gesture recognition area due to the user facing upward, lower right direction, lower left direction, upper right direction, upper left direction, the HMD device 1 controls the display device 12, Images imitating arrows are displayed on the lower side, upper left side, upper right side, lower left side, and lower right side of the display screen (FIGS. 10C, 10D, 11A, and 11B). ), FIG. 11 (C)). As described above, when the target is away from the gesture recognition area due to the user facing in a certain direction, an image is displayed on the opposite side of the direction in which the user is facing. Therefore, the user can intuitively confirm the direction in which the head is directed.

After the notification B as described above, the HMD device 1 functions as the control unit 41 and returns the process to the gesture recognition process.

When the above-described gesture recognition process and warning process are executed in the HMD device 1 and a predetermined target (part of the user's body) cannot be tracked, the cause is identified and the identified cause Appropriate notifications A and B are made based on this. With these notifications A and B, the user can easily determine in which direction the head should be moved and in which direction the hand should be moved so that the HMD device 1 can correctly recognize the gesture. . In this way, the user can intuitively correct the position of the gesture.

Each processing unit in each flowchart described above is divided according to main processing contents in order to facilitate understanding of the HMD device 1. The invention of the present application is not limited by the method of classification of the processing steps and the names thereof. The processing performed in the HMD device 1 can be divided into more processing steps. One processing step may execute more processes.

<Modification>
In addition, said embodiment intends to illustrate the summary of this invention, and does not limit this invention. Many alternatives, modifications, and variations will be apparent to those skilled in the art.

For example, in the above embodiment, the entire field of view of the imaging device 11 (the entire area of the captured image) is set as the gesture recognition area. However, the present invention is not limited to this, and a partial region in the field of view of the imaging device 11 may be set as the gesture recognition area.

FIG. 12 is a diagram showing a modification of the gesture recognition area. In order to make it easy to recognize a gesture with the user's right hand, such as a region surrounded by a broken line in FIG. 12, a region on the right side of the visual field of the imaging device 11 may be set as a gesture recognition area. The gesture recognition area may or may not be displayed on the transparent members G1 and G2.

Further, the present invention is applicable to any device as long as it is a device worn or held by the user and has a function capable of recognizing the user's gesture. Therefore, the present invention is not limited to the example applied to the HMD device 1 of the above embodiment.

FIG. 13 is a diagram for explaining an example in which the present invention is applied to a mobile terminal. As shown in FIG. 13, the present invention can also be applied to a portable terminal 60 that is used while being held by a user's hand. Here, the mobile terminal includes a mobile information terminal device having a display screen capable of displaying various types of information and contents and an imaging device, such as a smartphone, a tablet PC, and a PDA (Personal Digital Assistant).

When the present invention is applied to the mobile terminal 60, the mobile terminal 60 has the same configuration as that of the above embodiment, and recognizes the movement of the user's face as a gesture. Then, when the mobile terminal 60 moves away from the gesture recognition area, the user's hand holding the mobile terminal 60 (that is, the imaging device 11) has moved or the user's face (gesture) has moved. The user is notified so that it can be distinguished.

Further, the notification unit 47 may output a different warning sound according to the cause specified by the cause specifying unit 46. Further, different vibration patterns may be generated by a motor or the like depending on the cause specified by the notification unit 47 and the cause specifying unit 46.

In the embodiment and the modification, the HMD device 1 is a see-through image display device. However, the present invention is not limited to this, and the HMD device 1 may be a non-transmissive image display device.

The above-described configuration of the HMD device 1 is not limited to the above-described configuration because the main configuration has been described in describing the features of the above-described embodiments and modifications. In addition, the configuration of the general HMD device 1 is not excluded.

In addition, each functional configuration of the HMD device 1 described above is classified according to main processing contents in order to facilitate understanding of each functional configuration. The present invention is not limited by the way of classification and names of the constituent elements. Each functional configuration can be classified into more components according to the processing content. Moreover, it can also classify | categorize so that one component may perform more processes.

The processing of each functional configuration of the HMD device 1 described above can also be realized by a dedicated hardware circuit. In this case, it may be executed by one hardware or a plurality of hardware.

The program for operating the HMD device 1 may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, a CD-ROM, or may be provided online via a network such as the Internet. . In this case, the program recorded on the computer-readable recording medium is usually transferred and stored in the memory 32, the storage 33, or the like. Further, this program may be provided as, for example, a single application software, or may be incorporated in the software of each device as one function of the HMD device 1.

1 HMD device,
11 imaging device,
12 display device,
13 Inertial sensor,
31 CPU,
32 memory,
33 storage,
34 input devices,
35 Display controller,
41 control unit,
42 imaging control unit,
43 Shake detection unit,
44 Tracker,
45 Tracking availability determination unit,
46 Cause identification part,
47 Notification section,
48 Command decision part,
49 Display control unit,
50 application section,
60 mobile devices,
U Control unit.

Claims

A gesture recognition device worn or used by a user,
An imaging unit that continuously captures images;
When a change in the orientation of the imaging unit is detected, a predetermined target is tracked using an image captured by the imaging unit, whether the tracking is possible, and the tracking of the target is impossible based on the determination result A control unit for calculating the detected change amount of the orientation and identifying the cause of the tracking failure;
A gesture recognition device comprising: a notification unit that performs notification based on the identified cause.
The gesture recognition device according to claim 1, wherein the predetermined target is a part of a user's body.
3. The gesture recognition apparatus according to claim 1, wherein the notification unit performs different notifications according to the cause specified by the specifying unit.
The gesture recognition apparatus according to any one of claims 1 to 3, wherein the notification by the notification unit includes at least one of notification by display, notification by sound, and notification by vibration.
The gesture recognition apparatus according to any one of claims 1 to 4, wherein the tracking by the control unit is performed in a recognition region set in a field of view of the imaging unit.
The controller determines a command according to the tracked movement of the object;
The notification unit performs notification when tracking of the target becomes impossible when a command is determined by the control unit,
The gesture recognition device according to any one of claims 1 to 5.
A head-mounted display having a function as the gesture recognition device according to any one of claims 1 to 6.
A mobile terminal having a function as the gesture recognition device according to any one of claims 1 to 6.