KR20130096073A - Virtual mouse driving method using hand motion recognition - Google Patents

Abstract

PURPOSE: A virtual mouse driving method using a hand movement recognition is provided to analyze the moving trajectory of whole hands for recognizing the movement, and to implement a virtual mouse in low resolution. CONSTITUTION: A device analyzes a photographed hand movement, and activates a virtual mouse. The device extracts a portion of the hand performing the predetermined movement. The device analyzes the trajectory of the hand by analyzing the location change of the extracted portion from photographed images (S601-S609). The device classifies the trajectory into a click action, a drag and drop action, and a pointer movement. [Reference numerals] (A1) Start; (B1) Analyze the moving trajectory of hands; (B2,C2,C3,C6,D2,E2,E6,E9,E7) No; (B3,B4,C1,C5,D1,E1,E4,E5,E8) Yes; (C4) Is a dragging initiated?; (E3) Is a dragging not initiated?; (F1) Perform a click action; (F2) Perform a drag initiating movement; (F3) Successive dragging; (F4) Perform a drop action; (G1) Release a mouse control reserving; (G2) Move a pointer; (S601) Is a trajectory in a stopping movement?; (S602) Did a stopping movement occur in the near past?; (S603) Did the trajectory change the direction to 180°?; (S604) Is it a regular pointer movement?; (S605) Is the mouse control in a reserving state?; (S606) Set the mount control reserving state; (S607) Is the trajectory moved downwards?; (S608) Set the mount control reserving state; (S609) Is the trajectory moved to one direction

Description

Virtual mouse driving method using hand motion recognition

The present invention relates to a method of driving a virtual mouse of a display device, which recognizes the motion of a hand in an image obtained from a camera, tracks a specific area of the hand, extracts information about the position and motion of the hand, and controls the display device. It is about how to make it available.

As display devices have evolved into smart systems, interaction with display devices has become important. Smart display devices should be able to input commands according to their position on the display device screen, similar to a computer. The input device most commonly used as a means for inputting such a command is a mouse. In addition, in the case of smart phones that are in the spotlight recently, a command input according to a position on a screen may be performed using a touch screen.

The input method using a conventional touch screen for giving a location-based command transmits a command through contact with a display device, causing various limitations. That is, it can be used only when the display device is within a reachable distance by hand. Also, the mouse is not a smart input tool because it has a physical size and shape.

Recently, input devices capable of transmitting a command to a display device in a contactless manner have emerged. In particular, in the game field, a method for transmitting a command through motion recognition using a 3D camera has been developed. The method of using the 3D camera has an advantage of easily separating the object image and the background image from which the gesture is performed from the input image, but the input device is complicated and expensive. In addition, since the resolution is low, a user must input a command using a large operation, which causes a great inconvenience for the user.

In order to construct an inexpensive input device, an attempt has been made to control a smart TV or a set-top box by recognizing a gesture of a hand in a 2D camera image as shown in Patent Publication 10-2009-0124172.

In addition, the motion recognition function implemented in the recently launched smart TV raises the arm vertically upward toward the camera and then spreads the hand to shake the left and right until the camera recognizes the hand to display the pointer on the screen. Position the mouse on the icon you want to click on so that all five fingers are folded and extended.

However, such a click method can increase the hand fatigue by moving the finger muscles a lot by repeating the action of folding and releasing the finger when a lot of clicking operations are required such as keyboard input or a game. Therefore, it is necessary to implement the click function more conveniently.

In addition, another method of implementing a click may use an operation of folding and unfolding the index finger. However, when implementing a motion recognition at a long distance such as a TV, a high resolution camera is required to recognize one finger, which increases the cost.

In implementing a virtual mouse using hand gesture recognition for controlling a display device, it is a problem to be solved in the present invention to reduce the user fatigue and to increase convenience and lower cost.

In the method of driving a virtual mouse according to the present invention, in the method of driving a virtual mouse using hand movement, by analyzing the movement of the photographed hand, the virtual mouse is activated when a predetermined hand movement as a virtual mouse activation operation is made. And extracting a partial region of a hand that has performed the movement of the predetermined hand, analyzing a change in the position of the extracted region in the captured image, and analyzing a trajectory of the hand; And dividing the image into a click operation, a drag and drop operation, and a pointer movement operation according to the analyzed hand trajectory.

According to the invention, the hand is characterized in that moving the entire finger in the open state.

In addition, according to the present invention, the click motion is defined as an operation of stopping the moving speed of the hand at a position to be clicked and then stopping, then moving the hand in one direction and moving in the opposite direction to the one direction again. desirable.

Further, according to the present invention, the click operation is a predetermined error at the position to be clicked by moving the hand in a direction opposite to the predetermined direction after moving the hand by a predetermined distance from a position to be clicked. It is preferably defined as an operation returning to a position within a distance.

In addition, according to the present invention, it is preferable that the click operation be performed within a predetermined time.

According to the present invention, it is preferable that the position of the pointer on the screen is fixed while the click operation is performed.

In addition, according to the present invention, the drag and drop operation includes a drag operation and a drop operation. The drag operation includes a drag start operation for initiating a drag and a dragging operation after the drag is started. The drag initiation operation is performed by decelerating and stopping the moving speed of the hand at a position to start dragging, and then moving the hand within a predetermined time by a predetermined distance in the downward direction and stopping for a predetermined time. The continuous motion is preferably defined as an motion of moving a hand to a target position after the start of dragging.

In addition, according to the present invention, the drop operation is preferably defined as an operation of stopping a predetermined time after moving a hand within a predetermined time by a predetermined distance in a predetermined direction after the drag operation is recognized.

According to the present invention having the above-described configuration, the user can conveniently perform a click operation, a drag and drop operation, etc. with little fatigue, and analyze the trajectory of the entire hand to recognize the motion, thereby realizing the low resolution. Have

1 is a schematic block diagram of an apparatus for implementing a virtual mouse driving method according to an embodiment of the present invention.
2 is an embodiment of a click operation.
3 is a mouse pointer trajectory during the click operation of FIG. 2.
4 is an embodiment of a drag and drop operation.
5 is a mouse pointer trajectory during the drag and drop operation of FIG. 4.
FIG. 6 is a flowchart for distinguishing a hand movement trajectory from a normal pointer movement and a click, drag initiation, and drop operation. The flowchart is repeated in real time whenever an image is captured by a camera while tracking a hand trajectory. .

Hereinafter, a method of driving a virtual mouse according to a preferred embodiment of the present invention will be described with reference to the accompanying drawings.

1 is a schematic configuration diagram of an apparatus for implementing a virtual mouse driving method according to an embodiment of the present invention, FIG. 2 is an embodiment of a click operation, and FIG. 3 is a mouse pointer trajectory during the click operation of FIG. 4 is an embodiment of a drag and drop operation, FIG. 5 is a mouse pointer trajectory during the drag and drop operation of FIG. 4, and FIG. 6 is a general pointer movement and click, drag start, and drop operation of a hand movement trajectory. This flowchart is repeated in real time whenever an image is taken by the camera in the process of tracking the hand trajectory.

1 to 6, first, a virtual mouse driving method according to the present embodiment is implemented in a virtual mouse system. The virtual mouse system 100 includes a camera 10, an image input unit 20, and a hand gesture recognition unit. 30, and the command transmission unit 40.

The camera 10 captures and outputs an image input from a lens through an imaging device such as a CCD or a CMOS, and may be implemented as, for example, a digital camera. The camera 10 captures an image of a user's hand and transmits the image to a video input unit. . In particular, in the present embodiment, a two-dimensional camera is used for the camera. As described above, the use of the two-dimensional camera is more economically advantageous than the three-dimensional camera, and the resolution is relatively high.

The image input unit 20 receives an image captured by the camera in real time.

The hand gesture recognition unit 30 extracts a region that may be a feature of the hand from a plurality of consecutive input images by using a known image processing technique, and sets position coordinates of the hand region. In particular, in the present embodiment, the user moves the hand in a state where all the fingers are open, and the hand gesture recognition unit extracts a feature region from the hand in an extended state of the finger to set the position coordinates of the hand. For example, the position coordinate of the hand region may be a coordinate of the center of gravity of the palm.

Recognizing the movement of the hand in a state where the finger is opened in this way is that the recognition of the distance from the camera is far (eg The virtual mouse is used in the TV), and since the outer area of the hand is distinguished from other objects around when the finger is open, it is advantageous for recognition.

The hand gesture recognition unit analyzes the position coordinates of the hand region in a plurality of consecutive input images, analyzes the hand trajectory, and activates a virtual mouse activation, click action, drag and drop operation, and pointer according to the analyzed hand trajectory. It is divided into movement operation.

First, a virtual mouse activation operation means an operation for activating a virtual mouse, where activation of a virtual mouse means synchronization of a pointer (mouse pointer) to be displayed on a display device (for example, a monitor or a TV screen). do. The virtual mouse activation operation may be defined as an arbitrary operation, for example, an operation of drawing a circle by hand, and when the defined activation operation is performed, the hand gesture recognition unit analyzes it to activate the virtual mouse. When the virtual mouse is activated, a pointer is displayed on the display device, and the pointer is moved on the screen of the display device in response to the change of the position coordinate of the hand.

Hereinafter, a virtual mouse activation operation, a click operation, a drag and drop operation, and a pointer movement operation will be described.

The click operation refers to an operation of clicking a menu icon (ie, an icon for recognizing the click operation) displayed on the screen of the display device, for example, a folder, a shortcut icon, a hypertext, and the like. In the same way, an action is defined.

First, as shown in FIG. 2, assuming that the user stops the movement of the hand while the palm is facing in front, that is, the position to click Pa, the user moves the predetermined distance downward as shown in FIG. 2 to click. Afterwards (ie, moving to Pb in FIG. 2), the hand is moved upwards to return to the original position Pc (where Pc means the same position as Pa or a position within Pa and a predetermined error distance). Stop

Herein, the analysis procedure at time t2 in FIG. 2 will be described with reference to FIG. 6. At this time, the predetermined time limit for the click operation is set to C _tp , and the threshold for the amount of movement in one direction of the hand during the click operation is set as C _dp . If the stop motion is recognized at time t2 in Fig. 2 (Fig. 6.601), then the position of the hand is called Pc. Then, it is analyzed whether the stop operation has occurred between adjacent past times (t2-C _tp ) to (t2) (Fig. 6.602) and finds the past stop time t1. If the trajectory of the hand between t1 and t2 is changed to 180 ° as shown in FIG. 3, the trajectory descends downward and ascends upward (FIG. 6.603), it is recognized as a click operation. At this time, the click time t2-t1 must be smaller than the predetermined time limit C _tp , and the one-way movement distance Pb-Pa of the hand must satisfy a condition smaller than the threshold C _dp . The position Pc at which the click operation is stopped is recognized as a correct click operation only when it is within a preset error distance based on the click start position Pa.

The drag and drop operation refers to an operation of changing a position of an icon or the like on the screen of the display device. The drag and drop operation is composed of a drag operation of determining an icon and moving a position, and a drop operation of placing an icon at the moved position.

The drag operation includes a drag start operation for initiating a drag and a continuous operation for dragging (ie, moving a position) after the start of the drag. As shown in FIG. 4, the drag initiation operation is performed by decelerating and stopping the movement speed of the hand at a position (for example, an icon) to start dragging, and then moving the hand within a predetermined time by a predetermined distance in one direction (down direction). It is defined as an operation that stops for a certain time after moving, and it is defined as a continuous operation of dragging when a hand is moved after this operation.

The analysis procedure at time t2 at which the hand of FIG. 4 is stopped will be described with reference to FIG. 6 as follows. In this case, it is assumed that the threshold for the time required for the drag start operation is D _tp , and the threshold (preset distance) for the amount of downward movement of the hand for the drag start operation is D _dp . By analyzing the position Pa, Pb and the stop time t1, t2 of the hand through step 601 of FIG. 6 and step 602 of FIG. 6, the trajectory of the hand between t2 and t1 is downward as shown in FIG. If the trajectory is moved to 180 ° but the direction is not changed 180 °, it is recognized as a drag start operation as in step 607 of FIG. At this time, the drag start time t2-t1 must be smaller than the threshold D _tp , and the one-way movement distance Pb-Pa of the hand must satisfy a condition smaller than the threshold D _dp . Then, the movement of the hand is regarded as a continuous operation of dragging to move the predetermined icon to the desired position.

On the other hand, in the start of drag operation, the direction of hand movement is similar to the action of pushing a button of a conventional mouse, and the hand is moved downwards in the direction of gravity to lift the hand so that the energy of the hand is less. To relieve fatigue.

After the drag operation is recognized, as shown in FIG. 4, the drop operation stops by decelerating the moving speed of the hand at the position Pc where the drop operation is to be started, and then, within the preset time by a predetermined distance in the preset direction. After moving (the position of Pd), it is defined as an operation to stop for a predetermined time. An analysis procedure at time t4 at which the hand of FIG. 4 is stopped will be described with reference to FIG. 6. In this case, it is assumed that the threshold for the time required for the drop operation is D _tp and the threshold for the one-way movement amount of the hand for the drop is D _dp . The position Pc, Pd of the hand, and the stop times t3 and t4 are obtained through step 601 of FIG. 6, step 602 of FIG. 6 and step 603 of FIG. If it has moved in one direction, as in step 609 of FIG. 6 (at this time, dragging is started), it is recognized as a drop operation. At this time, the drop operation time t4-t3 must be smaller than the threshold D _tp , and the one-way movement distance Pd-Pc of the hand must satisfy a condition smaller than the threshold D _dp .

On the other hand, if the pointer of the mouse is unconditionally moved according to the movement of the hand, the position of the mouse pointer on the display device is changed even during the click operation, the drag start operation, and the drop operation, making it difficult to accurately control the position. Therefore, in the present invention, the user's position is fixed on the screen during the click, drag start, and drop operation by distinguishing whether the hand movement is performing a click, drag start, and drop operation or a general mouse pointer movement. Control to perform the operation at a desired position.

To this end, if it is assumed that the stop operation does not occur during the adjacent past time t1-C _tp of t1 at the time t1 of FIG. 2, the pointer is not moved by retaining the mouse control state as in step 606 of FIG. 6. After analyzing the hand trajectory, if the mouse control is in a reserved state (steps 601 and 605 of FIG. 6), it is determined whether the hand trajectory is a general pointer movement (step 604 of FIG. 6). The control reservation is released (step 608 of FIG. 6) to move the pointer, and if the click, drag start, and drop operations are performed instead of the normal pointer movement, the mouse control reservation is maintained. At this time, it is determined whether the trajectory moves in one direction defined in advance, whether the movement distance is within the thresholds C _dp and D _dp , or within the operation time thresholds C _tp and D _tp .

Meanwhile, the moving distance, the required time (ie, the threshold time), the moving direction, and the stopping time described above may be determined in advance. However, in this embodiment, the moving distance threshold is set within a range (100mm) where the camera does not require much energy to move the hand from the distance (20mm) from which the camera can analyze the trajectory, and the operation time threshold is set by the user. As a result of analyzing the experiment for the general public by setting the time (0.5 seconds ~ 2 seconds) to the extent that the operation can be performed without difficulty, the moving distance threshold is less than 40mm, the operating time threshold is It was found that setting the condition within 1 second was a condition that the subjects could easily learn and use the operation.

In addition, the command transfer unit 40 converts a command related to the position, click, drag and drop, and pointer movement of the virtual mouse into a protocol that can communicate with the display device and transmits the converted command.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the above-described specific preferred embodiments, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Various modifications can be made by those skilled in the art, and such changes are within the scope of the claims.

100 ... Virtual Mouse System (100)
10 ... camera (10) 20 ... video input
30 ... Hand gesture recognition part 40 ... Command delivery part

Claims (9)

In the method of driving a virtual mouse using the movement of the hand,
Analyzing the movement of the photographed hand and activating the virtual mouse when a predetermined hand movement as a virtual mouse activation is performed;
Extracting a partial region of a hand on which the preset hand movement is performed;
Analyzing the position of the extracted region in the captured image to analyze the trajectory of the hand;
And classifying the mouse into a click operation, a drag and drop operation, and a pointer movement operation according to the analyzed hand trajectory. The method of claim 1,
The hand is a method of driving a virtual mouse, characterized in that for moving the entire finger in the open state. The method of claim 2,
The click operation is defined as an operation of declining the moving speed of the hand at a position to be clicked and then stopping and then moving the hand in one direction and then moving in the opposite direction to the one direction to stop the virtual mouse. Driving method. The method of claim 2,
The click operation moves a hand by a predetermined distance in a predetermined direction from a position to be clicked, and then moves the hand in a direction opposite to the predetermined direction to return to a position within a predetermined error distance from the position to be clicked. Method of driving a virtual mouse, characterized in that the coming operation. The method according to claim 3 or 4,
And wherein the click operation should be performed within a preset time. The method of claim 5,
The position of the pointer on the screen is fixed while the click operation is performed. The method of claim 2,
The drag and drop operation is composed of a drag operation and a drop operation,
The drag operation may include a drag start operation of initiating a drag and a continuous operation of performing dragging after the drag is started.
The drag initiation operation is defined as an operation of stopping a hand for a predetermined time after moving the hand within a predetermined time by a predetermined distance in a downward direction after decelerating and stopping the movement speed of the hand at a position to start dragging.
The continuous operation is a virtual mouse driving method, characterized in that the movement of moving the hand to the target position after the start of the drag. The method of claim 7, wherein
The drop operation may be defined as an operation of stopping a hand for a predetermined time after moving a hand within a predetermined time in a predetermined distance after the drag operation is recognized. 9. The method according to claim 7 or 8,
The method of driving a virtual mouse, wherein the position of the pointer on the screen is fixed during the drag start operation and the drop operation.

Images