JP5865615B2 - Electronic apparatus and control method - Google Patents

Description

  Embodiments described herein relate generally to an electronic device and a control method for controlling the device.

  In recent years, in an electronic device such as a personal computer (PC), a new input interface for controlling the electronic device has been developed without using an operation device such as a keyboard, a mouse, or a remote control unit.

  One such new input interface is a hand gesture interface. The hand gesture interface recognizes the shape and movement of the user's hand (hand gesture) using an image taken by the camera, and controls the electronic device based on the recognition result.

  However, in general, the types of hand gestures that can be recognized are limited. On the other hand, with the diversification and complexity of functions of electronic devices, the types of operations for controlling electronic devices tend to increase.

  For this reason, recently, hand gesture interfaces having several operation modes corresponding to the types of operations have begun to be developed. As a method of switching the operation mode to be used, a method of displaying an operation mode menu such as a top menu and selecting an item corresponding to the target operation mode from a number of items on the operation mode menu can be considered.

JP 2009-230460 A

  An item may be selected using a hand adjuster. However, in this case, it may be necessary to recognize several hand gestures such as a hand gesture for moving the cursor to a target item on the operation mode menu and a hand gesture for inputting an enter key. Absent. Therefore, in the above-described method using the operation mode menu, it takes time until the target operation mode becomes available, which may reduce the operability of the hand gesture interface.

  An object of the present invention is to provide an electronic device and a control method that can easily select an operation mode used in hand gesture recognition.

According to the embodiment, the electronic device includes a face detection unit, a hand shape detection unit, an operation mode selection unit, and a hand gesture recognition unit. The face detection unit recognizes a face in an image taken by a camera and detects the position of the face in the image. The hand shape detecting means recognizes a first shape hand in the image and detects a position of the first shape hand in the image. The operation mode selection unit calculates a first value indicating a relative position of the detection position of the hand of the first shape with respect to the detection position of the face, and selects a plurality of operation modes based on the first value. The first operation mode is selected. The hand gesture recognition means recognizes the shape and movement of a hand related to the hand gesture to be used in the selected first operation mode using the video, and the electronic device according to the recognition result. Control the operation of the equipment. The operation mode selection means calculates the degree of coincidence between the first value and each of a plurality of reference values corresponding to the plurality of operation modes, and selects an operation mode having a degree of coincidence higher than a predetermined threshold. The first operation mode is selected.

FIG. 2 is a perspective view illustrating an appearance of the electronic apparatus according to the embodiment. 2 is an exemplary block diagram showing an example of the system configuration of the electronic apparatus of the embodiment. FIG. 4 is an exemplary block diagram illustrating a configuration example of a hand gesture recognition program executed by the electronic apparatus of the embodiment. FIG. FIG. 6 is an exemplary view showing an arrangement example of operation modes used in the electronic apparatus of the embodiment. 6 is an exemplary view showing an execution example of a select operation mode which is one of operation modes used in the electronic apparatus of the embodiment. FIG. FIG. 6 is an exemplary view showing an execution example of an AV control menu mode which is one of operation modes used in the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing an execution example of a volume operation mode which is one of operation modes used in the electronic apparatus of the embodiment. FIG. 6 is a view showing another arrangement example of operation modes used in the electronic apparatus of the embodiment. 6 is a diagram for explaining selection of an operation mode used in the electronic apparatus of the embodiment. FIG. The figure which shows the operation mode selection screen used with the electronic device of the embodiment. FIG. 6 is a view for explaining a reference value of each operation mode used in the electronic apparatus of the embodiment. The figure explaining the coincidence used when selecting the operation mode in the electronic device of the embodiment. 6 is an exemplary flowchart illustrating an example of a procedure of an operation mode switching process which is executed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing an example of a coordinate position of an operation mode when an operation mode is selected which is executed by the electronic apparatus of the embodiment. FIG. 6 is an exemplary view showing a display screen displayed when using a hand gesture recognition program executed by the electronic apparatus of the embodiment.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 is a perspective view illustrating an external appearance of an electronic apparatus according to an embodiment. This electronic apparatus is realized as a notebook type personal computer 10, for example. In addition, the electronic device may be a digital television, for example. As shown in FIG. 1, the computer 10 includes a computer main body 11 and a display unit 12. The display unit 12 incorporates an LCD (liquid crystal display) 17 and a video camera 20. The display unit 12 is attached to the computer main body 11 so as to be rotatable between an open position where the upper surface of the computer main body 11 is exposed and a closed position covering the upper surface of the computer main body 11.

  The video camera 20 is provided at a substantially central position of the upper end portion of the display unit 12. The video camera 20 is a camera for taking an image of the user. An image (video) captured by the camera 20 is used for hand gesture recognition that recognizes the shape and movement of the user's hand. Note that the position where the video camera 20 is provided may not be the upper end of the display unit 12, for example, the lower end of the display unit 12. Further, the video camera 20 may be an external video camera having a USB interface, for example.

  The computer main body 11 has a thin box-shaped casing. On the upper surface of the computer main body 11, a keyboard 13, a power button 14 for powering on / off the computer 10, an input operation panel 15, a touch pad 16, Speakers 18A, 18B, etc. are arranged. Various operation buttons are provided on the input operation panel 15. A USB connector 19 for connecting a USB cable or a USB device is provided on the right side surface of the computer main body 11.

FIG. 2 is a diagram showing a system configuration of the computer 10.
As shown in FIG. 2, the computer 10 includes a central processing unit (CPU) 101, a north bridge 102, a main memory 103, a south bridge 104, a graphics processing unit (GPU) 105, a video memory (VRAM) 105A. , Sound controller 106, BIOS-ROM 107, LAN controller 108, hard disk drive (HDD) 109, optical disk drive (ODD) 110, video camera 20, USB controller 111A, card controller 111B, wireless LAN controller 112, embedded controller / keyboard controller ( EC / KBC) 113, EEPROM 114, and the like.

  The CPU 101 is a processor that controls the operation of each unit in the computer 10. The CPU 101 executes various application programs such as an operating system (OS) 201 and a hand gesture recognition program 202 that are loaded from the HDD 109 to the main memory 103. The hand gesture recognition program 202 recognizes the shape and movement of the user's hand (hand gesture) using, for example, each image frame (that is, video) acquired by the video camera 20, and performs the main gesture based on the recognition result. It is a program for controlling the operation of the computer 10. More specifically, several application programs executed by the computer 10 are controlled based on the recognition result of the hand gesture. In addition, the hand gesture recognition program 202 detects and recognizes a face (face image) included in the video.

  As an application program (hand gesture compatible application) controlled by the hand gesture recognition program 202, for example, a web browser, a media content playback application, a file for viewing a large number of folders and a large number of files managed by a file system This is a TV application for reproducing contents such as an operation program and a TV broadcast program.

  The CPU 101 also executes a BIOS (basic input / output system) stored in the BIOS-ROM 107. The BIOS is a program for hardware control.

  The north bridge 102 is a bridge device that connects the local bus of the CPU 101 and the south bridge 104. The north bridge 102 also includes a memory controller that controls access to the main memory 103. The north bridge 102 also has a function of executing communication with the GPU 105 via, for example, a PCI EXPRESS serial bus.

  The GPU 105 is a display controller that controls the LCD 17 used as a display monitor of the computer 10. A display signal generated by the GPU 105 is sent to the LCD 17.

  The south bridge 104 controls each device on a peripheral component interconnect (PCI) bus and each device on a low pin count (LPC) bus. Further, the south bridge 104 includes an IDE (Integrated Drive Electronics) controller for controlling the HDD 109 and the ODD 110. Further, the south bridge 104 has a function of executing communication with the sound controller 106.

  The sound controller 106 is a sound source device and outputs audio data to be reproduced to the speakers 18A and 18B. The LAN controller 108 is a wired communication device that executes, for example, IEEE 802.3 standard wired communication, while the wireless LAN controller 112 is a wireless communication device that executes, for example, IEEE 802.11 standard wireless communication. The USB controller 111A executes communication with an external device. The card controller 111B executes data writing and reading with respect to a memory card such as an SD card inserted into a card slot provided in the computer main body 11.

  The EC / KBC 113 is a one-chip microcomputer in which an embedded controller for power management and a keyboard controller for controlling the keyboard 13 and the touch pad 16 are integrated. The EC / KBC 113 has a function of turning on / off the computer 10 in accordance with the operation of the power button 14 by the user.

Next, the operation of the hand gesture recognition program 202 will be described with reference to FIG.
FIG. 3 is an example of a block diagram illustrating a system configuration of functions and operations of the hand gesture recognition program 202 of the present embodiment.
The hand gesture recognition program 202 includes a hand gesture recognition unit 300, a face detection unit 304, and a control unit 305. The hand gesture recognition unit 300 includes a hand shape detection unit 301, a hand movement detection unit 302, and a function execution unit 303. The control unit 305 includes an operation mode selection unit 306, a graphical user interface (GUI) display unit 307, a position storage unit 308, and a position setting unit 309.

  Before describing specific functions of each block, an outline of an operation mode selection method in an interface (hereinafter also referred to as a hand gesture interface) using the hand gesture recognition program 202 implemented in the present embodiment will be described.

  The hand gesture recognition program 202 has a plurality of operation modes respectively corresponding to a plurality of operation types for operating the application. Examples of operation types (operation modes) include, for example, AV control (play, pause, fast forward, etc.), select operation (arrow keys, enter, return), mouse cursor operation, launcher menu selection operation, volume operation, etc. Can be mentioned.

  When a user uses a hand gesture-compatible application, the user may selectively use a plurality of operation modes as necessary to operate the application. It is difficult to simultaneously support all the above-described types of operations by the hand gesture interface because of the limited types of hand gestures that can be recognized. Therefore, it is necessary to switch the operation mode. In addition, it is inconvenient for the user if a plurality of operation modes cannot be smoothly switched when the application is used. Therefore, in the present embodiment, a user's face and a specific hand shape (for example, a palm) serving as an operation trigger are detected using a video (image frame group) captured by the video camera 20, and a specific position relative to the position of the face is detected. One of a plurality of operation modes is selected based on the relative position of the hand-shaped position. For a plurality of operation modes, the positions (relative positions) of these operation modes with respect to the face are determined in advance. The user selects the target operation mode by placing his / her hand (eg, palm) at a position corresponding to the target operation mode among a plurality of operation mode positions set in advance with respect to the face position. Can do. When switching the operation mode, the user can lower the hand once and put the hand (for example, a palm) again at a position corresponding to another operation mode to switch to the other operation mode. .

Hereinafter, functions and the like of each block in FIG. 3 will be described.
An image captured by the video camera 20 is sent to the hand gesture recognition unit 300 and the face detection unit 304. The face detection unit 304 recognizes a face (face image) in the video using the video sent from the video camera 20. The face detection unit 304 detects the position of the face in the video. The position of the face in the video is, for example, the position of the face (face image) in the image frame. Further, the face detection unit 304 detects a face tilt and a face orientation in addition to the face position. For example, when the face in the video is tilted somewhat to the left with respect to a predetermined axis, the face detection unit 304 recognizes that the face in the video is tilted somewhat to the left. Furthermore, the face detection unit 304 also detects the size of the recognized face (face image). Further, the face detection unit 304 sends detection results regarding these faces to the operation mode selection unit 306. The face detection unit 304 operates in accordance with a control command from the control unit 305. For example, when a plurality of faces are included in an image, the control command causes a face to be detected to be selected from a plurality of faces with a face having the largest size among the plurality of recognizable faces as a recognition target face. It may be a control command.

  The hand gesture recognition unit 300 recognizes or interprets a user's hand gesture (hand shape and movement) using an image sent from the video camera 20 (hereinafter referred to as recognition). The hand gesture recognition unit 300 has the above-described plurality of operation modes, and executes hand gesture recognition corresponding to the operation mode specified by the control unit 305. Each operation mode may define one or more hand gestures to be recognized and a function to be executed when the one or more hand gestures are recognized. The hand gesture recognition unit 300 operates in the operation mode selected by the control unit 305. That is, the hand gesture recognition unit 300 recognizes the shape and movement of the hand related to the hand gesture to be used in the selected operation mode, and transmits a command to the hand gesture corresponding application 311 according to the recognition result. Thereby, the operation of the computer 10 is controlled.

  The hand shape detector 301 recognizes the hand shape (or hand shape) in the video sent from the video camera 20 and detects the position of the hand shape in the video. The position of the hand shape in the video is the position of the hand shape (hand shape image) in the image frame. The hand shape detected by the hand shape detection unit 301 includes a hand grip (fist), a hand shape (thumb up) with a specific finger (thumb) raised, a palm (palm), and the like. The hand shape detection unit 301 sends the hand shape detection result to the function execution unit 303 and the operation mode selection unit 306.

  The hand movement detection unit 302 detects a hand movement in the video transmitted from the video camera 20. The movement of the hand is, for example, a movement in which the detected hand moves up and down or left and right, and a back and forth movement with respect to the video camera 20. The detection result of the hand movement detected by the hand movement detection unit 302 is sent to the function execution unit 303.

  The function execution unit 303 is based on the hand gesture recognition result, that is, based on the hand shape detection result from the hand shape detection unit 301 and the hand motion detection result from the hand movement detection unit 302. Send a command to 311. For example, if the movement of the hand that instructs video playback is detected, the function execution unit 303 sends a command that instructs video playback to the hand gesture compatible application 311. In addition, the function execution unit 303 displays the operation screen (graphical user interface) according to the hand gesture recognition result so that the operation screen (graphical user interface) on the LCD 17 changes according to the hand gesture recognition result. To control. Here, the case where the function execution unit 303 is realized as one module in the hand gesture recognition unit 300 is illustrated, but the function execution unit 303 may be realized as one module in the control unit 305. Good.

  The hand gesture compatible application 311 is an application corresponding to a hand gesture controlled by the hand gesture recognition program 202 or the like. The hand gesture compatible application 311 executes the function according to the command from the function execution unit 303.

  The GPU 105 displays an operation mode screen (graphical user interface) or the like on the LCD 17 in accordance with a command or the like from the graphical user interface display unit 307 function execution unit 303 or the graphical user interface display unit 307.

  The control unit 305 controls the hand gesture recognition program 202. For example, the control unit 305 may instruct the hand gesture recognition unit 300 of the hand shape to be recognized. Further, the control unit 305 may instruct the face detection unit 304 to start face recognition processing.

Next, the position setting unit 309 included in the control unit 305 will be described.
The position setting unit 309 sets the position of each of the plurality of operation modes. Although details will be described later, the position of each operation mode can be set to a position (relative position) with respect to the face position. The position setting unit 309 sets the relative position for each operation mode. This relative position is used as a reference position for each operation mode.

  The position storage unit 308 stores data relating to the position of each operation mode set by the position setting unit 309.

  The operation mode selection unit 306 is based on the face detection result by the face detection unit 304, the hand shape detection result by the hand shape detection unit 301, and the data related to the position stored in the position storage unit 308. To select one of the operation modes. Although a specific description of this selection method will be described later, for example, a value indicating the relative position of the detection position of a specific hand shape with respect to the detection position of the face is calculated, and one of a plurality of operation modes is determined based on the value. Selected. For example, when selecting one of the operation modes, the operation mode selection unit 306 calculates a value indicating the relative position of the hand detection position with respect to the face detection position, and corresponds to the calculated value and each operation mode. The degree of coincidence between each reference value is calculated. The operation mode selection unit 306 compares the calculated degrees of coincidence and determines the degree of coincidence having the highest degree of coincidence. Further, the operation mode selection unit 306 compares the degree of coincidence with a predetermined threshold value. The operation mode selection unit 306 selects an operation mode having a matching degree higher than a predetermined threshold. The operation mode selection unit 306 notifies the selected operation mode to the graphical user interface display unit 307 and the hand gesture recognition unit 300.

  The graphical user interface display unit 307 displays an operation screen (graphical user interface) corresponding to the operation mode notified by the operation mode selection unit 306 on the display screen of the LCD 17 via the GPU 105.

Next, the relationship between the position of the face and the position of each operation mode will be described with reference to FIG.
FIG. 4 is a diagram illustrating an example of the arrangement of operation modes used by the hand gesture recognition program of the present embodiment.
The position of each operation mode is determined in advance based on the position of the face in the image taken by the video camera 20. A face 406 represents the face of the user who uses the computer 10. For example, three operation modes are assigned to the position of the face 406. The three operation modes are a select operation 400, an AV control menu 402, and a volume operation 404. As shown in FIG. 4, the select operation 400, the AV control menu 402, and the volume operation 404 are arranged on the left side, the right side, and the upper side of the face 406 as viewed from the user. When the user places his / her hand on the left side of his / her face when viewed from the user, the select operation 400 is selected. When the user places his / her hand on the right side of his / her face when viewed from the user, the AV control menu 402 is selected. When the user places his / her hand above the face of the user as viewed from the user, the volume operation 404 is selected.

  Each of the three operation modes has a fixed square area as shown by a dotted line in the figure, but may be a circular or triangular area other than the square. Furthermore, although the operation mode area is shown as a two-dimensional area in the figure, for example, an operation mode area in a three-dimensional space may be arranged around the face 406. The select operation 400 is an operation mode for selecting a direction in the application, and is used to send commands such as “arrow key”, “decision”, and “return” to the application. The AV control menu 402 is one of operation modes used in video playback devices and the like, and is used to send commands such as “play”, “pause”, and “fast forward” to the application. The volume operation 404 is an operation mode in which operations such as volume adjustment of sound output from the speakers 18A, 18B, etc. are performed when the application is used.

Next, the three operation modes shown in FIG. 4 will be described with reference to FIGS.
FIG. 5 is a diagram illustrating an example of a select operation mode that is one of the operation modes of the present embodiment.
The left diagram in FIG. 5 assumes a case where the user places his / her hand on the left side with respect to the position of the face 406. When the user puts his hand to the left side with respect to the position of the face 406, an operation screen 500 as shown on the right side of FIG. The operation screen 500 is a graphical user interface corresponding to the select operation mode. As shown in the figure, for example, select keys 501, 502, 503, and 504 are displayed on the operation screen 500. The select keys 501, 502, 503, and 504 are indicated by arrow marks indicating up, right, down, and left, respectively. Note that the directions indicated by the select keys 501, 502, 503, and 504 are not limited to up, right, down, and left, but may be, for example, right up or down left. Further, the shape of the select keys 501, 502, 503, and 504 may not be the arrow mark in the direction that the user wants to select. For example, all of the select keys 501, 502, 503, and 504 are indicated by circular marks. May be. Alternatively, arbitrary marks indicating the characters “up”, “right”, “down”, and “left” may be used for the select keys 501, 502, 503, and 504, respectively. The hand mark 505 is a mark corresponding to “Enter”. The select keys 501, 502, 503, and 504 may indicate effect effects such as changing the color on the screen when the user moves the hand in the direction that the user wants to select.

  An example of a user's hand movement (gesture) in the select operation mode will be described. When the user wants to select the select key 502 in the right direction, the user selects the select key 502 by waving his hand to the right, and a command (right arrow key command) indicating the right selection is sent to the application. Note that the gesture when the right select key 502 is selected may be moved to the right without changing the shape of the hand, for example, in addition to waving to the right. A command (enter key command) indicating “OK” is sent to the application by a gesture that pushes out a hand (for example, a palm). The gesture for inputting the enter key command to the application may be, for example, a gesture for changing the hand into a specific shape in addition to pushing out the hand.

FIG. 6 is a diagram illustrating an example of an AV control menu mode that is one of the operation modes of the present embodiment.
The left diagram in FIG. 6 assumes a case where the user places his / her hand on the right side with respect to the position of the face 406. When the user puts his / her hand to the right side with respect to the position of the face 406, an AV control menu screen 600 as shown on the right side of FIG. The AV control menu screen 600 is a graphical user interface corresponding to the AV control menu mode. The AV control menu screen 600 includes icons 601 to 607. The icon 601 is an icon indicating a function for playing back or pausing a video played back by an application. Icons 602 to 605 are icons indicating functions such as fast-forwarding, rewinding, moving chapters, and the like of images reproduced by an application. An icon 606 is an icon indicating a function for terminating the application. The icon 607 is an icon for executing another specific function. In addition to these, an icon indicating a function necessary for controlling an application for reproducing a video or the like may be displayed.

  An example of the user's gesture in the AV control menu mode will be described. When the user shakes his / her hand to the left or right, the menu selection focus moves between the icons 601 to 607. Instead of moving the focus between the icons 601 to 607, each of the icons 601 to 607 displayed on the screen 600 may be moved by the user shaking his / her hand to the left or right. For example, when the icon is arranged as shown on the right side of FIG. 6, the icon 601 currently positioned at the foremost position moves to the position of the icon 603 when the user shakes his hand once to the right. At the same time, the icon 602 moves to the frontmost position. Further, when the user shakes his / her hand three times to the right, the icon 601 currently located at the foremost position is moved to the position of the icon 607 and the icon 606 is moved to the position of the position closest to the front. In this way, the icon indicating the function that the user wants to use is moved to the frontmost position, that is, the position of the icon 601 in the figure by shaking the hand a plurality of times. Thereafter, the user can send a command for executing the function indicated by the icon located in the foremost position to the application by performing a gesture such as pushing the hand (for example, palm) forward. Note that as a gesture other than waving, for example, a gesture for moving a hand in a direction in which the icon is desired to be moved without changing the shape of the hand may be used.

FIG. 7 is a diagram illustrating an example of a volume operation mode that is one of the operation modes of the present embodiment.
The diagram on the left side of FIG. 7 assumes a case where the user places his / her hand upward with respect to the position of the face 406. When the user puts his hand upward with respect to the position of the face 406, an operation screen 700 as shown on the right side of FIG. The operation screen 700 is a graphical user interface corresponding to the volume operation mode. This operation screen 700 shows an arrow 701 for drawing a circle. When the user wants to adjust the volume, the user can move his hand to draw a circle as shown by an arrow 701. For example, if the user wants to increase the volume, the user can move his / her hand in the direction indicated by the arrow 701 (clockwise). On the other hand, if the volume is to be lowered, the hand may be moved in the direction opposite to the direction indicated by the arrow 701 (counterclockwise). In addition, when the user moves his hand to draw a circle, the display of the arrow 701 may be changed. For example, when the user moves his / her hand clockwise to increase the volume, the shape of the arrow 701 may be changed in correspondence with the shape of the arrow 701 and the current volume level.

  Various display forms can be used for the graphical user interface in each operation mode as described above. Various correspondences between each operation and hand gesture in each operation menu mode can be used.

  Thus, based on the relative position of the hand position with respect to the position of the face 406, the operation mode corresponding to the position is selected. Thereafter, in the selected operation mode, the user can perform various functions in the operation mode by performing a hand gesture.

Next, an example of another arrangement of operation modes different from the arrangement of operation modes as shown in FIG. 4 will be described with reference to FIG.
FIG. 8 is a diagram illustrating another example of the operation mode arrangement in the present embodiment.
In FIG. 8, four operation menu modes are arranged with respect to the position of the face 406. The four operation menus are a select operation 400, an AV control menu 402, a volume operation 404, and a mouse cursor operation 800. As shown in the figure, the four operation menus may be arranged, for example, radially at equal intervals with respect to the position of the face 406. Note that the size and shape of each operation menu mode area and the area indicated by the dotted line in the figure may be other than those shown in the figure.

In the four operation menus, descriptions of functions similar to those in FIG. 4 are omitted.
As for the mouse cursor operation 800, the mouse cursor operation 800 is selected when the user moves the hand position to the lower right with respect to the position of the face 406, in the same manner as the selection method of other operation menus described above. The mouse cursor operation 800 is one of operation modes in which a function equivalent to an operation performed by operating a pointing device such as the touch pad 16 is performed by a hand gesture. For example, the mouse cursor displayed on the LCD 17 is moved based on the relative movement distance of the hand. Further, a desired place displayed on the LCD 17 can be clicked by changing the shape of the hand.

  As described above, the operation menu mode is selected based on the relative positional relationship between the position of the face 406 and the hand, and operations in the selected operation menu mode can be performed according to various hand gestures. That is, when a user operates an application corresponding to a hand gesture that can be operated in a plurality of operation modes with a hand gesture, the user does not perform a specific hand gesture such as changing the shape of the hand in the plurality of operation modes. The operation mode can be selected smoothly based on the relative position between the position and the position of the hand put out by the user. Note that the graphical user interface is not necessarily displayed on the LCD 17. That is, the user may select an operation mode according to the position of the hand that is raised with respect to the position of the face 406, and may perform a hand gesture in the selected operation mode in the same manner as when the graphical user interface is displayed. . In the AV control menu mode, different hand shapes may be assigned to “play”, “pause”, and “fast forward”.

FIG. 9 is a diagram for explaining a process until an operation mode is selected.
FIG. 9 assumes a case where there are three operation modes for the position of the face 406, as in FIG. In FIG. 9, P1 indicates the relative position of the area (center position of the area) of the select operation mode 400 with respect to the position of the face 406, and P2 indicates the area (center position of the area) of the AV control menu mode 402 with respect to the position of the face 406 The relative position is indicated, and P3 indicates the relative position of the volume operation mode area (the center position of the area) with respect to the position of the face 406. P4 indicates the relative position of the position (palm center position) of the hand shape (for example, palm) of the operation start trigger with respect to the position of the face 406. The position of the face 406 may be the center position of the face 406. All of P1 to P4 are determined based on the position of the face 406. The hand gesture recognition program 202 calculates the distance (matching degree or corresponding degree) between each position P1 to P3 of the region of each operation mode and the hand position P4. The operation mode having the highest matching degree among the calculated matching degrees is selected. The high degree of coincidence means that in FIG. 9, for example, the distance between P2 and P4 is shorter than the distance between P1 and P4 or the distance between P3 and P4. Note that P1 to P3 may be other than the center position of each operation mode, and may be, for example, the position closest to the face 406 or the farthest position in each operation mode region.

  Further, when the degree of coincidence exceeds a predetermined threshold, an operation mode having a degree of coincidence higher than the threshold may be selected. That is, in FIG. 9, the distance between P2 and P4, which is the degree of coincidence corresponding to the AV control menu 402, is shorter than the other coincidence indicated by the distance between P1 or P3 and P4. For example, if the degree of coincidence corresponding to the AV control menu 402 does not exceed a predetermined threshold, the AV control menu 402 is not yet selected at that time. Thereafter, the position of the hand P4 or the face 406 moves, the degree of coincidence corresponding to the AV control menu 402 increases, and when the degree of coincidence exceeds a predetermined threshold, the AV control menu 402 is selected.

Next, processing executed when the degree of coincidence calculated according to the position P4 of the hand put out by the user does not exceed a predetermined threshold will be described with reference to FIGS.
FIG. 10 is a diagram showing an example of the operation mode menu screen of the present embodiment.
FIG. 10 shows an operation screen 800 that displays an AV control menu GUI 1001, a select operation GUI 1002, and a volume operation GUI 1003. These three GUIs correspond to the positions of the three operation modes for the face 406 as shown in FIG. For example, a volume operation GUI 1003 corresponding to the volume operation mode is displayed on the upper side of the screen 800. The screen 800 is displayed on the LCD 17, for example.

  When the degree of coincidence does not exceed the threshold value, GUIs of all operation modes that can be selected by the user are displayed as shown in FIG. However, even when the degree of coincidence has already exceeded the threshold at the time when the user takes a hand, the screen 800 as shown in FIG. 10 is displayed, and then a predetermined time (for example, as short as 0.5 seconds). The screen 800 may fade out in time, and an operation mode GUI corresponding to the degree of coincidence exceeding the threshold may be displayed. In this way, by displaying the GUI of all selectable operation modes, even if the user does not know the types of operation modes that can be selected and the positions of the operation modes, the user is informed of them. Can do. Furthermore, since the process and state from when the user puts out the hand until the desired operation mode is selected are shown by animation, the user can easily understand the process and procedure until the desired operation mode is selected.

  If the degree of coincidence does not exceed a predetermined threshold, the size of each of the three operation mode GUIs may be changed and displayed on the screen 800 as shown in FIG. Specifically, as shown in FIG. 9, three matching degrees corresponding to each of the three operation modes are calculated, and the calculated matching degrees are associated with the GUI size of each operation mode. For example, the GUI of the operation mode corresponding to the highest matching degree among the three calculated matching degrees may be displayed on the screen 800 the largest. By doing so, the size of the GUI in each operation mode also changes in accordance with the change in the degree of coincidence calculated by the change in the hand position P4, so which operation mode is visually selected by the user. Easy to understand.

Next, the setting of the center positions P1 to P3 of the operation mode area used when calculating the degree of coincidence as described above will be described with reference to FIG.
FIG. 11 is a diagram for explaining the setting of each operation mode position according to the present embodiment.
P1 to P3 are used to determine the position of each operation mode with respect to the position of the face 406, in addition to being used when calculating the degree of coincidence. That is, P1 to P3 are also positions serving as reference positions for the respective operation modes.

  More specifically, first, as shown in FIG. 11, in the XY plane coordinate system, the center position of the face 406 detected at the origin P0 is set. In addition to the center of the face, P0 may be the midpoint of the eye position of the face 406 or the nose position. Next, the size of the face 406 is measured. In the drawing, the size of the face 406 is an area surrounded by a rectangular dotted line with x in the horizontal direction and y in the vertical direction. Based on the measured size of the face 406, the coordinates of P1 to P3 are determined. For example, the parameters a and b are set so that the coordinates of P1 and P2 are (-ax, -by) and (ax, -by) as shown in the figure. Similarly, the parameter c is set so that the coordinates of P3 are (0, cy). Regarding the parameters a, b, and c, for example, the parameter a is set so that the distance between P0 and P1, the distance between P0 and P2, and the distance between P0 and P3 are equal to each other. , B, and c may be set, but any other parameters can be set.

A method different from the method for setting the positions P1 to P3 serving as the reference for the position of each operation mode as shown in FIG. 11 will be described.
Although not shown, for example, the positions P1 to P3 may be determined by the method described above, and then the user can freely set an arbitrary operation mode at the positions P1 to P3. For example, when the four operation modes such as the select operation 400, the AV control menu 402, the volume operation 404, and the mouse cursor operation 800 as described above are usable operation modes, the user operates these four operations at the position P1. Any one operation mode selected from the modes can be set. The same can be set for P2 and P3.

  As another method for setting the position of the operation mode, for example, instead of setting the positions P1 to P3 by the method shown in FIG. 11, a setting method that allows the user to freely set the positions P1 to P3 may be used. . For example, when there are four usable operation modes as described above, the position candidates of the four operation modes are displayed on the operation screen, and the user adjusts the displayed position candidates by an operation such as calibration. Thus, the position of each operation mode can be set freely.

Next, the relationship between the positions P1 to P3 described in FIG. 11 and the hand position P4 will be described with reference to FIG.
FIG. 12 is a diagram for explaining the calculation of the degree of coincidence in the process of selecting the operation mode according to this embodiment.
As described with reference to FIG. 11, it is assumed that after the operation mode reference points P <b> 1 to P <b> 3 are set, the user places a hand-shaped hand as indicated by the hand position P <b> 4 in FIG. 12. State. When the user puts his / her hand at a coordinate position as shown in the figure, that is, at the upper left position with respect to the face 406 position, the hand gesture recognition program 202 causes the relative position of the hand position to the face position and three operations. The degree of coincidence between the reference values of each mode, that is, the distance between the hand position P4 and the reference value P1, the distance between the hand position P4 and the reference value P2, the hand position P4 and the reference value P3 The distance between is calculated. In the figure, since the distance between P3 and P4 is the shortest, the degree of coincidence between the operation modes associated with the reference value P3 is the highest. Note that when the threshold value of the matching degree as described above is not considered, the operation mode associated with the reference value P3 having the highest matching degree is selected.

With reference to FIG. 13, the procedure of the operation mode switching process executed by the hand gesture recognition program 202 as described above will be described.
At block 1300, the hand gesture recognition program 202 is executed. The hand gesture recognition program 202 may be automatically executed at the same time as an application corresponding to the hand gesture recognition program is executed.

  In block 1302, the hand gesture recognition program 202 determines whether the user's face 406 is detected by the face detection unit 304. If the face 406 is not detected, the hand gesture recognition program 202 waits until the face 406 is detected. If face 406 is detected, proceed to block 1304.

  In block 1304, the hand gesture recognition program 202 determines whether the hand of the user is detected by the hand shape detection unit 301. The hand to be detected here is a hand having a specific hand shape (for example, a palm). Of course, the present invention is not limited to the palm, and a hand shape such as a gripping hand may be detected.

  If no hand is detected, the process returns to block 1302 and the hand gesture recognition program waits until both the face 406 and a hand with a specific hand shape are detected. If a hand is detected at block 1304, proceed to block 1306. Note that block 1304 may be processed before block 1302. That is, the processing procedure may be such that a hand is detected before the face 406 is detected, and then the process proceeds to block 1306 when the face 406 is detected.

  In block 1306, the operation mode selection unit 306 of the hand gesture recognition program 202 determines the relative position of the hand to the position of the face 406 based on the position of the face 406 detected in block 1302 and the position of the hand detected in block 1304. A value indicating the position is calculated, and a degree of coincidence between the value and a reference corresponding to each of the plurality of operation modes is calculated.

  In block 1308, the operation mode selection unit 306 determines whether the relative position of the hand with respect to the face 406 corresponds to the relative position of the select operation mode with respect to the face 406 based on the degree of coincidence calculated in block 1306. To correspond to the select operation mode, for example, the highest match degree among the match degrees corresponding to each of the plurality of operation modes calculated in block 1306 corresponds to the select operation mode, and the match degree has a threshold value. It means that it is over. If it corresponds to the select operation mode, the process proceeds to block 1318. If the select operation mode is not applicable, the process proceeds to block 1310.

  In block 1310, the operation mode selection unit 306 determines whether the relative position of the hand with respect to the face 406 corresponds to the relative position of the AV control menu mode with respect to the face 406 based on the degree of coincidence calculated in block 1306. The determination of whether this is the case is the same as described in the description of block 1308. If the AV control menu mode is applicable, the process proceeds to block 1324. If the AV control menu mode is not applicable, the process proceeds to block 1312.

  In block 1312, the operation mode selection unit 306 determines whether the relative position of the hand with respect to the face 406 corresponds to the relative position of the volume operation mode with respect to the face 406 based on the degree of coincidence calculated in block 1306. The determination of whether this is the case is the same as described in the description of block 1308. If it corresponds to the volume operation mode, the process proceeds to block 1330, and if it does not correspond to the volume operation mode, the process proceeds to block 1314.

  In block 1314, when none of the operation modes selectable in block 1308, block 1310, and block 1312 corresponds, GUIs of all displayable operation modes are displayed as shown in operation screen 800 of FIG. . Also, as described above with reference to FIG. 10, the graphical user interface display unit 307 changes, changes, or adjusts the size of the GUI in each operation mode in accordance with the plurality of degrees of coincidence calculated in block 1306. It is displayed on the screen 500. That is, the GUI for each operation mode is displayed in a size based on the degree of matching corresponding to the operation mode.

  In block 1316, the operation mode selection unit 306 determines whether the hand position or the face position has changed based on the detection results received from the hand motion detection unit 302 and the face detection unit 304, respectively. When a change in the position of the hand or face is detected, the process proceeds to block 1306, and the operation mode selection unit 306 calculates the degree of coincidence again.

Next, a case where it is determined in block 1308 that the relative position of the hand with respect to the face 406 corresponds to the select operation mode will be described.
In this case, the process proceeds to block 1318, and a select operation mode GUI such as the operation screen 500 is displayed in accordance with an instruction from the function execution unit 303. In block 1320, the hand gesture recognition unit 300 recognizes the select operation hand gesture, that is, one or more hand shapes and movements associated with the hand gesture to be used in the select operation mode. Then, the hand gesture recognition unit 300 issues a command (arrow key, determination, etc.) defined in the select operation mode based on the recognition result.

  In block 1322, the hand gesture recognition unit 300 determines whether a movement of lowering the hand has been detected. If no movement of lowering the hand is detected, the process returns to block 1320 to recognize the gesture in the select operation mode again. If the hand gesture recognition unit 300 detects a movement of lowering the hand, the process returns to block 1302. That is, when the user performs a gesture of lowering his / her hand, the state in which the select operation mode is selected is canceled, that is, the operation in the select operation mode is ended. Note that the gesture of lowering the hand is a gesture of moving the hand out of a shooting area that can be shot by the video camera 20, for example. Of course, the gesture of lowering the hand may be a gesture that is not included in the gesture in the select operation mode recognized by the hand gesture recognition unit 300 in block 1320, and may be a gesture that indicates a change in a specific hand shape.

  Thus, the user can easily switch the operation mode by ending the operation mode by a simple hand movement or gesture of lowering the hand, and then returning the hand to a predetermined position as described above. .

  If it is determined in block 1310 that the relative position of the hand with respect to the face 406 corresponds to the AV control menu mode, the process proceeds to block 1324. In block 1324, an AV control menu GUI such as the operation screen 600 is displayed in accordance with an instruction from the function execution unit 303. At block 1326, the hand gesture recognizer 300 recognizes an AV control menu hand gesture, ie, one or more hand shapes and movements associated with the hand gesture to be used in the AV control menu mode. Then, the hand gesture recognition unit 300 issues a command (playback, pause, fast forward, etc.) defined in the AV control menu mode based on the recognition result.

At block 1328, it is determined whether the hand gesture recognition unit 300 has detected a movement to lower the hand. If the movement of lowering the hand is detected, the process returns to block 1302, and the AV control menu mode operation is completed.

  Similarly, if it is determined in block 1312 that the relative position of the hand to the face 406 corresponds to the volume operation mode, the process proceeds to block 1330. In block 1330, a volume operation GUI such as operation screen 700 is displayed in accordance with an instruction from function execution unit 303. In block 1332, the hand gesture recognition unit 300 recognizes a gesture in the volume operation mode. At block 1334, it is determined whether the hand gesture recognition unit 300 has detected a movement to lower the hand. When the movement of lowering the hand is detected, the process returns to block 1302 and the operation in the volume operation mode is completed.

  Next, a process for selecting an operation mode corresponding to the relative position of the hand with respect to the face 406 will be described with reference to FIG.

  When the orientation of the face 406 is tilted to the left as shown in FIG. 14, the hand gesture recognition program 202 detects the orientation of the face 406. The position of each operation mode is determined in accordance with the orientation of the face 406. For example, even if the coordinates shown in FIGS. 11 and 12 are rotated in accordance with the orientation of the detected face 406 and the inclination thereof, for example. Good. That is, P1 to P1 when the face 406 is tilted based on coordinates obtained by converting the coordinates P1 to P3 set based on the coordinates as shown in FIGS. 11 and 12 using a rotation matrix or the like. You may set the coordinate of P3. Note that the GUI of each operation mode displayed on the screen 500 when the face 406 is tilted may be the same display as in FIG. That is, even if the face 406 is inclined, the UI display direction of each operation mode displayed on the screen 500 may not change.

Finally, with reference to FIG. 15, a display screen when using an application corresponding to the hand gesture recognition program of the present embodiment will be described.
FIG. 15 is a view showing an example of a display screen displayed when the hand gesture recognition program 202 is used.
On the LCD 17, an application window 1500 corresponding to the hand gesture recognition program, a sub window 1501, and an operation mode GUI 1502 are displayed. The window 1500 is, for example, a web browser window or a video playback application window. The sub window 1501 may display, for example, an image captured by the video camera 20, or display what can be visually recognized by the user that the hand shape or the face is analyzed using the captured image. Also good. The GUI 1502 in the operation mode displays the GUI in the select operation mode in the drawing, but may display the GUI in the AV control menu mode as described above. Note that the position and size of the sub window 1501 on the LCD 17 can be arbitrarily changed, and a second operation mode GUI similar to the operation mode GUI 1502 may be additionally displayed on the LCD 17.

  As described above, according to the present embodiment, a value indicating the relative position of the detection position of a hand with respect to the detection position of the face is calculated, and based on the calculated value, the corresponding operation is performed from a plurality of operation modes. The operation mode to be selected is automatically selected. Therefore, since an operation mode corresponding to the position where the hand is put out can be selected, the time until the target operation mode can be used can be shortened. Therefore, the operation mode used for hand gesture recognition can be easily selected. For example, when the user uses an interface using a hand gesture recognition program, the user can instantaneously select an operation mode according to the relative position of the hand to the position of the face. That is, when the user starts the operation, the target operation can be performed smoothly. Furthermore, when the operation mode is switched, the operation mode can be switched smoothly by simply lowering the hand and moving the hand to a certain relative position with respect to the face position.

  It is also possible to switch the operation mode using a specific operation mode switching gesture (for example, holding the palm and swinging it sideways). For example, when a hand shape serving as an operation start trigger is issued, the select operation mode is set first. When starting, in order to perform an operation such as playback / pause, it is necessary to select and determine a playback / pause menu after performing an operation mode switching gesture. In the present embodiment, as described above, the operation mode can be selected according to the position where the hand shape serving as the operation start trigger is issued, so that the time until the target operation mode can be used can be greatly shortened. .

  In the present embodiment, the operation mode is selected based on the relative position of the hand position of the specific hand shape with respect to the face position, but this is to shorten the time required for selecting the operation mode as much as possible. It is.

  The specific hand shape may be a specific hand shape used as a trigger for starting the hand gesture recognition process of the hand gesture recognition program 202. When the user wants to start the hand gesture recognition process, the user puts out a hand having a specific hand shape. At this time, the user can start the hand gesture recognition process immediately in the target operation mode by placing his hand at a position corresponding to the target operation mode corresponding to the operation he wants to use.

  Also, the processing procedure of the hand gesture interface of this embodiment can be executed entirely by software. For this reason, the same effect as this embodiment can be easily realized only by installing and executing this program on a normal computer through a computer-readable storage medium storing the program for executing this processing procedure. .

  3 may be realized by hardware such as a dedicated LSI or DSP.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 20 ... Video camera, 202 ... Hand gesture recognition program, 300 ... Hand gesture recognition part, 301 ... Hand shape detection part, 302 ... Hand motion detection part, 303 ... Function execution part, 304 ... Face detection part, 306 ... Operation mode Selection unit, 307 ... operation menu display unit, 308 ... position storage unit, 309 ... position setting unit, 311 ... hand gesture compatible application.

Claims (7)

Electronic equipment,
Face detection means for recognizing a face in an image taken by a camera and detecting the position of the face in the image;
Hand shape detecting means for recognizing a first shape hand in the image and detecting a position of the first shape hand in the image;
A first value indicating a relative position of the detected position of the first shape hand with respect to the detected position of the face is calculated, and a first operation mode is selected from a plurality of operation modes based on the first value Operation mode selection means to
A hand that recognizes the shape and movement of the hand related to the hand gesture to be used in the selected first operation mode using the video and controls the operation of the electronic device according to the recognition result Gesture recognition means,
The operation mode selection means calculates the degree of coincidence between the first value and each of a plurality of reference values corresponding to the plurality of operation modes, and selects an operation mode having a degree of coincidence higher than a predetermined threshold. An electronic device configured to be selected as the first operation mode.   The electronic device according to claim 1, wherein the operation mode selection unit displays a graphical user interface corresponding to the selected first operation mode on a display. The operation mode selection unit selects a plurality of graphical users corresponding to the plurality of operation modes without selecting any of the plurality of operation modes when there is no operation mode having a matching degree higher than the predetermined threshold. electronics interface, according to claim 1, characterized in that is configured to display the size Dede Isupurei based on the calculated degree of matching.   The electronic device according to claim 3, wherein the operation mode selection unit recalculates the degree of coincidence when the position of the face or the position of the first shape hand in the video changes. A setting unit configured to set a plurality of reference values indicating relative positions of the plurality of operation modes with respect to the position of the face according to a user operation;
The operation mode selection unit calculates a degree of coincidence between the first value and each of the plurality of reference values set by the setting unit, and selects an operation mode having a degree of coincidence higher than a predetermined threshold. The electronic device according to claim 1, wherein the electronic device is selected as the first operation mode. Recognizing a face in an image taken by a camera, detecting the position of the face in the image,
Recognizing a first shape hand in the image, detecting a position of the first shape hand in the image;
A first value indicating a relative position of the detected position of the first shape hand with respect to the detected position of the face is calculated, and a first operation mode is selected from a plurality of operation modes based on the first value And
Recognizing a shape and movement of a hand related to a hand gesture to be used in the selected first operation mode using the video, and controlling an operation of the electronic device according to the recognition result. And
Selecting the operation mode means calculating an agreement between the first value and each of a plurality of reference values corresponding to the plurality of operation modes, and having an agreement higher than a predetermined threshold. A control method comprising selecting a mode as the first operation mode. On the computer,
Recognizing a face in an image taken by a camera and detecting the position of the face in the image;
Recognizing a first shape hand in the image and detecting a position of the first shape hand in the image;
A first value indicating a relative position of the detected position of the first shape hand with respect to the detected position of the face is calculated, and a first operation mode is selected from a plurality of operation modes based on the first value And the steps to
Recognizing the shape and movement of a hand associated with a hand gesture to be used in the selected first operation mode using the video, and controlling the operation of the computer according to the recognition result; And execute
The procedure for selecting the operation mode includes calculating an agreement between the first value and each of a plurality of reference values corresponding to the plurality of operation modes, and having an agreement higher than a predetermined threshold. A program comprising selecting a mode as the first operation mode.

Images