JP5733056B2 - Data input method with virtual mouse - Google Patents

Description

  The present invention operates a virtual mouse that does not physically exist in a communication device or an information device, detects the movement of an operator's hand and the movement of a fingertip, and moves or clicks a displayed pointer. The present invention relates to a method for realizing data input equivalent to normal mouse input by executing the screen operation.

  2. Description of the Related Art In recent years, with the advancement of semiconductor technology, a portable personal computer for mobile use (hereinafter abbreviated as a mobile personal computer) having a small size, light weight and high performance has been spread. Further, screen operations and data input means in application software installed in a mobile personal computer are generally operations based on a graphical user interface (GUI) such as click and drag, etc., using a mouse in addition to a keyboard.

  For this reason, for example, when using a mobile PC, a normal mouse is wired or wirelessly connected at home or at home, and a GUI is operated. It is customary to operate the GUI using a device. This is because it avoids carrying a normal mouse as a separate unit in addition to the mobile personal computer to the destination, and it seems that it is originally more comfortable to operate with a normal mouse.

  As a method for improving this, there is a technique in which, for example, a hand movement direction is analyzed from an image captured by a camera, and command input processing is executed based on the analysis information (for example, Patent Document 1). This technology makes it possible to identify four types, for example, from bottom to top, from left to right, from top to bottom, and from right to left as directions in which a hand or finger as a recognition target object crosses the recognition area (hotspot). When three regions are set as regions, 64 (= 4 × 4 × 4) ways of identification are possible, and these operations are associated with 64 GUI operation commands. Thus, the user can perform a predetermined command to operate the screen without using a physical mouse or the like by performing a predetermined gesture operation on the camera.

  However, although the technique described in Patent Document 1 seems to be suitable for remote control of the volume and channel of a TV being viewed without a remote control device, for example, the hand or finger is recognized. This is a gesture input that requires a large amount of movement to identify the direction across the area, which is very different from the general sense of operating a mouse. Further, the operator needs to learn many predetermined gesture operations, which is insufficient as a technique for replacing a physical mouse.

JP 2007-34525 A

  Therefore, the problem of the present invention is to operate a virtual mouse that does not physically exist in a communication device or information device such as a mobile personal computer, and to detect the movement of its hand and fingertips. Another object of the present invention is to provide a method for realizing data input equivalent to normal mouse input by executing screen operations such as a moving operation and a click operation of a displayed pointer.

In order to solve the above problems, a first aspect of the present invention analyzes the image data from a predetermined video camera for photographing the hand and fingers of the operator,
A data input method of inputting data to determine the operation content for the virtual mouse no entity according to the movement and click operation of a pointer displayed in the predetermined image display apparatus, wherein by analyzing the image data a movement detection step for detecting a movement of the operator's hand operating the virtual mouse, a fingertip detection step of detecting the position and operation of one or more fingertip for operating the virtual mouse by analyzing the video data, the And when the movement detection step detects a movement of the hand in a state where the fingertip detection step detects a predetermined specific shape by the one or more fingertips, a display is being performed according to the movement of the hand pointer outputs movement commands to move to the predetermined information processing step, detecting a specific shape where the fingertip detection step reaches a predetermined by the one or more fingertips When in it is state detects the reciprocation of one of the one or more fingertips, a scroll command to scroll the screen in the display output to the information processing step, the fingertip detection step wherein one or more A click command for notifying that a click operation has been performed is detected to the information processing step when a pressing operation by any one of the one or more fingertips is detected in a state where a predetermined specific shape is detected by the fingertip. It is characterized by outputting.

  According to the present invention, since a virtual mouse that does not physically exist is operated as if a normal mouse was operated, it is not necessary to memorize a special gesture command. Therefore, even when the mobile personal computer is used at the destination and the normal mouse is not carried, there is an effect that the comfortable operability equivalent to that of the normal mouse can be provided.

Overall explanatory diagram of the method of the present invention Block configuration diagram of an apparatus to which the method of the present invention is applied Operation flowchart of apparatus to which method of the present invention is applied

  Hereinafter, as an embodiment of the present invention, a case where the present invention is applied to a mouse input unit of an information device or a communication device will be described as an example with reference to the drawings.

  FIG. 1 is an overall explanatory view of the method of the present invention, in which 1 is an information device or communication device (hereinafter abbreviated as this device) to which the method of the present invention is applied, 2 is a video camera incorporated in the device 1, 30 is a display unit incorporated in the apparatus 1, 31 is a pointer displayed on the display unit 30, 32 and 33 are click areas displayed on the display unit 30, and 40 is an operator operating the virtual mouse. The hand, 41 is the fingertip of the index finger of the hand 40, 42 is the fingertip of the middle finger of the hand 40, 50 is a virtual mouse, 51 is a left click switch of the virtual mouse, and 52 is a right click switch of the virtual mouse. Although not shown, there is a scroll key between the left click switch 51 and the right click switch 52 of the virtual mouse 50 as in a normal mouse.

  50 to 52 indicated by broken lines are not physically present. While looking at the pointer 31 displayed on the display unit 30, the operator imagines that the virtual mouse 50 that is not physically present is being grasped with the hand 40, moves the virtual mouse 50 on the desk with the hand 40, and moves the fingertip 41 and 42 are used to perform a click operation and a screen scroll operation. The hand 40 and fingertips 41 and 42 of the operator who operates the mouse are photographed by the video camera 2, and the video data from the video camera 2 is processed and analyzed to analyze the movement of the hand 40 and the fingertip. Left-click or right-click operations by 41 and 42 are detected. The pointer 31 moves according to the detection results.

  FIG. 1 shows an example in which the pointer 31 displayed on the display unit 30 is moved to the click area 32 of YES and clicked. Note that the position where the pointer 31 is first displayed is appropriately set by processing of application software such as the center of the display unit 30, the top and bottom, left and right edges, and the center of a predetermined click area.

  FIG. 2 is a block diagram of the apparatus 1, wherein 11 is a fingertip detection unit, 12 is a specific shape detection unit, 13 is a movement detection unit, 14 is a mouse setting unit, 15 is a click determination unit, and 16 is a mouse interface. , 17 are information processing units.

  The fingertip detection unit 11 analyzes the video data of each fingertip that operates the virtual mouse 50 taken by the video camera 2, detects the coordinates where the hand or each fingertip is located, and the specific shape detection unit 12, movement detection The detected data is notified to the unit 13 and the click determination unit 15.

  The specific shape detection unit 12 analyzes the data from the fingertip detection unit 11 to determine the shape of each finger, and when detecting a specific shape registered in advance (for example, Janken's face) Notify the mouse interface 16.

  The movement detection unit 13 analyzes the data from the fingertip detection unit 11, detects the movement of the hand 40 that operates the virtual mouse 50, and notifies the mouse interface 16 of the movement direction and movement amount.

  The mouse setting unit 14 operates the virtual mouse 50 such as the function of the left click switch 51 and the right click switch 52 of the virtual mouse 50 and the movement ratio of the pointer 31 with respect to the movement amount of the hand 40 (that is, the movement amount of the virtual mouse 50). It is means for setting such data. The data to be set in the mouse setting unit 14 may be input by an input device other than the virtual mouse 50 such as a keyboard (not shown), or a predetermined shape by each fingertip via the specific shape detection unit 12. It is also possible to input predetermined data corresponding to the above.

  The click determination unit 15 analyzes the data from the fingertip detection unit 11 to monitor the pressing operation of the fingertip 41 or the fingertip 42. When the pressing operation is detected, the left click switch 51 or the right click switch 52 of the virtual mouse 50 is detected. It is determined which of the buttons is clicked, and the determination result is notified to the mouse interface 16. For example, when the pressing operation by the fingertip 41 is detected, it is determined that the left click switch 51 is clicked. When the pressing operation by the fingertip 42 is detected, it is determined that the right click switch 52 is clicked.

  The mouse interface 16 outputs the movement data of the pointer 31, the click command, and the scroll command to the information processing unit 17 based on the data input from the specific shape detection unit 12, the movement detection unit 13, and the click determination unit 15.

  For example, in the shape of Janken, the virtual mouse 50 is gripped by the hand 40 and the fingertip 41 of the right index finger and the fingertip 42 of the right hand ring finger are set at the positions of the left click switch 51 and the right click switch 52 of the virtual mouse 50. This movement is defined as movement or dragging of the virtual mouse 50 to which the pointer 31 should be moved. In this case, movement in a shape other than choki (for example, par or goo) moves the virtual mouse 50 without moving the pointer 31 (corresponding to an operation of lifting the mouse back to the original position in normal mouse operation). It becomes. As a result, the operator only has to move each fingertip with the feeling of operating a normal mouse, so the feeling of strangeness is small.

  The information processing unit 17 is information processing means using application software in general that involves input of movement data, a click command, and a scroll command using a mouse.

  The movement data, click command, and scroll command input to the information processing unit 17 are the same as data from a normal physical mouse, and the mouse corresponding unit (not shown) of the information processing unit 17 is a mouse interface. The pointer 31 being displayed is moved according to the movement data and the click command input from 16, and the shape of the pointer 31 being displayed is transformed into a shape corresponding to a click operation or a drag operation. Further, the displayed image is scrolled according to the scroll command input from the mouse interface 16.

  FIG. 3 is an operation flowchart of the apparatus 1. Hereinafter, the operation flow of the present apparatus 1 will be described with reference to FIGS. This flow starts when the power of the apparatus 1 is turned on and the information processing unit 17 waits for data input from the mouse (S300). In this state, the pointer 31 is displayed at a home position or a specific position designated by the application software running the information processing unit 17.

  When the hand 40 of the operator who is operating the apparatus 1 and video data of the fingertip (captured by the video camera 2) are input to the fingertip detection unit 11 (S310), the fingertip detection unit 11 analyzes the video data. The position of the hand 40 and each fingertip is detected (S311).

  The specific shape detection unit 12 monitors whether or not a predetermined specific shape (for example, choki) is detected from the coordinate data at which each fingertip is detected detected by the fingertip detection unit 11, and detects the specific shape (S320, YES). ), It is determined that the operator is holding the virtual mouse 50, and the process proceeds to S330. If the specific shape is not detected (S320, NO), the process returns to S310.

  In S330, the movement detection unit 13 monitors the movement of the hand 40 from the history of coordinate data where the hand 40 is notified, which is notified from the fingertip detection unit 11, and detects the movement of the hand 40 (S330, YES). The movement data including the movement direction and the movement amount of 40 is output to the information processing unit 17 via the mouse interface 16 (S331), and the process proceeds to S340. Note that the value of the movement amount included in the movement data is a value obtained by multiplying the movement amount of the hand 40 by a parameter relating to movement set in the mouse setting unit 14. The information processing unit 17 relatively moves the displayed pointer 31 according to the input movement data. If NO in S330, the process skips S331 and proceeds to S340.

  In S340, the click determination unit 15 monitors the pressing operation of each fingertip from the history of the coordinate data where the fingertip of the index finger 41 and the middle finger 42 are notified from the fingertip detecting unit 11, and detects the pressing operation (YES in S340). ) A click command corresponding to right click or left click is output to the information processing unit 17 via the mouse interface 16 (S341), and the process proceeds to S350.

  Note that the click operation performed by the operator includes a single click operation, a double click operation, and a drag operation with the click switch held down, but these are identified by the information processing unit 17 (not shown). Further, the right click and the left click may be identified based on whether the fingertip that has been pressed is the fingertip 41 or the fingertip 42. If NO in S340, the process skips S341 and proceeds to S350.

  In S350, when the mouse interface 16 detects a reciprocating motion in the vertical or front-rear direction corresponding to the operation of the scroll key (not shown) of the virtual mouse 50 by the fingertip 41 or the fingertip 42 (S350, YES), the mouse interface 16 Referring to the mouse setting unit 14, the command related to the screen scroll is output to the information processing unit 17 (S351), and the process returns to S310.

  Here, the scroll direction of the screen may be defined as, for example, upward scrolling of the screen when the reciprocating motion of the fingertip 41 is detected, and downward scrolling of the screen when the reciprocating motion of the fingertip 42 is detected. If NO in S350, S351 is skipped, the process returns to S310, and S310 and subsequent steps are repeated.

  As described above, the video data from the video camera that shot the hand and fingertip of the operator who operates the virtual mouse that does not exist is analyzed by image processing technology, and the same movement data, click command, The method for outputting the scroll command to the information processing unit 17 has been described.

  In this method, as a specific method for detecting the position of each fingertip of the operator, for example, by using image recognition processing technology, a nail part may be extracted and the center position determined as the fingertip position. . Moreover, what is necessary is just to let the position of a hand be the position of the center of each detected said fingertip, or a gravity center, for example.

  Furthermore, as a method for determining the position of the fingertip by a simple process, an object having a feature point such as nail art is attached to each nail of each fingertip, a picture or mark having a feature point is drawn, or there is a feature point A glove or finger sack may be worn, and the feature points may be detected by image processing technology.

  By the way, in the embodiment of the present invention, as a method for detecting the position and movement of each fingertip, a method for analyzing and detecting video data from the video camera 2 has been described as an example. It is not limited to this. Although a detailed description is omitted, for example, instead of the video camera 2, a method of detecting the operation content on the virtual mouse 50 by using data from a sensor that directly detects the position of each fingertip can be applied. For example, a magnetic material or a wireless tag may be attached to the nail or glove of each fingertip, and their position and operation may be detected by a predetermined sensor (for example, a hall element or a wireless tag reader).

  It is also possible to freely define a specific shape identified by the specific shape detector 12 (not shown). For example, in addition to goo, choki, and par with each fingertip of the hand 40, the home position of the pointer 31 can be achieved by various shapes or operations such as an OK sign with the thumb and forefinger, and a shape in which a predetermined number of fingers are extended from the shape of goo. It is possible to define various screen operation commands such as moving to, displaying / stopping the pointer 31, and enlarging / reducing the image displayed on the display unit 30.

DESCRIPTION OF SYMBOLS 1 ... Apparatus which applied method of this invention 2 ... Video camera 11 ... Fingertip detection part 12 ... Specific shape detection part 13 ... Movement detection part 14 ... Mouse setting part 15 ... Click determination unit 16 ... mouse interface 17 ... information processing unit 30 ... display unit 31 ... pointer 32 ... click area (YES)
33 ... Click area (NO)
40 ... Hand for operating virtual mouse 41 ... Fingertip 42 for index finger operating virtual mouse ... Fingertip 50 for middle finger operating virtual mouse ... Virtual mouse 51 ... Left click switch of virtual mouse 52 ... Virtual mouse right click switch

Claims (1)

Hands and fingers of the operator analyzes the image data from a predetermined video camera for capturing, moving and click operation of a pointer to determine the operation content for the virtual mouse no entity is displayed in the predetermined image display device A data input method for inputting data related to
A movement detection step for detecting a movement of the operator's hand operating the virtual mouse by analyzing the video data, the position and operation of one or more fingertip for operating the virtual mouse by analyzing the video data A fingertip detection step for detecting
When the movement detection step detects the movement of the hand in a state where the fingertip detection step detects a predetermined specific shape by the one or more fingertips, a pointer being displayed according to the movement of the hand A movement command to be moved is output to a predetermined information processing step, and the one or more fingertips are reciprocated while the fingertip detection step detects a predetermined specific shape by the one or more fingertips. If detected, a scroll command for scrolling the displayed screen is output to the information processing step, and the fingertip detection step detects the predetermined specific shape by the one or more fingertips. When a pressing operation by one of one or more fingertips is detected, a click command notifying that a click operation has been performed is sent to the information processing step. Data input method by the virtual mouse and outputting to.

Images