KR20070030398A - Mobile device controlling mouse pointer as gesture of hand and implementing method thereof - Google Patents

Abstract

The present invention relates to a mobile communication terminal and a method for controlling the movement of a mouse pointer according to a user's hand gesture. More particularly, the present invention relates to recognizing and responding to the motion of a hand using an image photographing apparatus provided in the mobile communication terminal. The present invention relates to a mobile communication terminal and a method in which a mouse pointer moves. The mobile communication terminal according to the present invention includes an image capturing unit for capturing an image related to a user's hand gesture, a gesture information generating unit generating hand gesture information based on the captured image signal, and mouse coordinates according to the hand gesture information. And a tracking operator for generating information and direction information, a mouse pointer controller for moving the mouse pointer corresponding to the mouse coordinate information and the mouse direction information, and a display unit for displaying the moved mouse pointer. According to the present invention, the convenience of the user can be increased by adjusting the mouse pointer on the mobile communication terminal with a simple hand operation.

Gesture, Tracking, Human Mouse, Hand, Mobile Communication Terminal

Description

MOBILE DEVICE CONTROLLING MOUSE POINTER AS GESTURE OF HAND AND IMPLEMENTING METHOD THEREOF}

1 is a view for explaining a method of selecting a menu in a graphical user interface (GUI) of a conventional mobile communication terminal.

2 is a block diagram illustrating a mobile communication terminal providing a mouse pointer operated by a hand gesture according to an embodiment of the present invention.

3 is a diagram for explaining an example in which mouse coordinates change according to a hand motion according to another exemplary embodiment of the present invention.

4 is a view for explaining an example in which the position of the mouse pointer on the display changes in response to the position of the index finger of FIG. 3.

5 is a view for explaining a method of distinguishing the color of the hand according to another embodiment of the present invention.

6 is a diagram for describing a method of executing a click command according to a hand gesture according to another exemplary embodiment of the present invention.

7 is a flowchart illustrating a method of operating a mouse pointer by a hand gesture in a mobile communication terminal according to the present invention.

8 is a flowchart illustrating a method of setting a photographing area of a hand gesture according to the present invention.

9 is a flowchart illustrating a method of changing the position of a midpoint as a reference for tracking coordinates in a photographing area according to the present invention.

<Explanation of symbols for the main parts of the drawings>

200: mobile communication terminal

210: video recording unit

220: gesture information generation unit

225: color detection unit

230: tracking calculation unit

240: mouse pointer control

250: display unit

The present invention relates to a mobile communication terminal and a method for controlling the movement of a mouse pointer according to a user's hand gesture. More particularly, the present invention relates to recognizing and responding to the motion of a hand using an image photographing apparatus provided in the mobile communication terminal. The present invention relates to a mobile communication terminal and a method in which a mouse pointer moves.

In modern society, mobile communication terminals are widely used as a necessity for modern people as a means of convenient communication without regard to place. Recently, as the performance of the processor is improved and the resources of the memory are increased, the mobile communication terminal provides various functions as well as the conventional mobile communication service.

As described above, although the method of using a mobile communication terminal is diversified and a high level user interface is used, a conventional mobile communication terminal uses a keypad or a touch screen as a user interface for executing various functional items presented on a display. When using the keypad, a preset menu is provided on the display unit, and the user can input a desired command by pressing a numeric key or a direction key of the keypad. An example of a command input method of a conventional mobile communication terminal is shown in FIG.

1 is a view for explaining a method of selecting a menu in a graphical user interface (GUI) of a conventional mobile communication terminal. The user can select how to press the down arrow key four times or the numeric key "4" button to select an item to execute ("4. Specify update notification").

However, there is a problem in that a command input method using a keypad cannot be freely and quickly selected due to the inconvenience of having to select a specific menu item displayed on the display unit one by one or pressing a number button while checking a number corresponding to each item. .

Accordingly, recently, as an improved user interface, a terminal such as a PDA adopts a method of directly inputting a command by simply clicking on a sensor covering the display unit with a menu item or a command icon displayed on the display unit using a touch screen. However, this method also has to be equipped with hardware for implementing a touch screen in addition to the resources provided by the conventional mobile communication terminal causes a cost increase of the mobile communication terminal, there is a problem that also increases the volume or weight.

As another example, a small mouse can be used by connecting to a mobile communication terminal, but it is cumbersome to carry the mouse together, and a separate port for installing a mouse must be additionally provided to the mobile communication terminal to increase the cost It may be difficult to commercialize.

Therefore, there is an urgent need for the emergence of a new technology that can maximize user convenience by allowing users to freely select and execute various menu items without increasing costs or volume by utilizing resources provided on the mobile communication terminal. have.

The present invention has been made to solve the problems of the prior art as described above, the object is to increase the convenience of the user by simply moving the mouse pointer appearing on the display unit provided with the mobile communication terminal by hand motion. do.

Another object of the present invention is to reduce the production cost of the terminal because it is not necessary to have a separate mouse by moving the mouse pointer by utilizing the resources of the mobile communication terminal.

Another object of the present invention is to be able to conveniently use it to control or play a multimedia program such as mobile game content requiring a human computer interface (HCI).

Still another object of the present invention is to sense a hand gesture in consideration of the illuminance around the hand input through the image photographing apparatus of the mobile communication terminal.

Another object of the present invention is to detect not only the movement of the mouse pointer but also various hand gestures to control the mouse pointer's click operation.

It is still another object of the present invention to continuously generate a mouse pointer corresponding to a hand gesture to control the natural movement of the mouse pointer.

It is another object of the present invention to track a fast moving hand gesture by limiting the area for photographing the hand gesture according to the size or the moving distance of the hand gesture.

Still another object of the present invention is to effectively divide the hand gesture and the surrounding noise by converting the image signal photographing the hand gesture into various color models.

In order to achieve the above object and to solve the above-mentioned problems of the prior art, the mobile communication terminal according to the present invention, an image capturing unit for capturing an image related to the user's hand gesture, based on the captured image signal A gesture information generator for generating hand gesture information, a tracking operator for generating mouse coordinate information and direction information according to the hand gesture information, a mouse pointer for moving a mouse pointer in response to the mouse coordinate information and the mouse direction information And a display unit for displaying the moved mouse pointer.

For reference, the term "mobile communication terminal" herein refers to a PDC (Personal Digital Cellular) phone, PCS (Personal Communication Service) phone, PHS (Personal Handyphone System) phone, CDMA-2000 (1X, 3X) phone, WCDMA ( Wideband CDMA phone, Dual Band / Dual Mode phone, Global Standard for Mobile (GSM) phone, Mobile Broadband System (MBS) phone, Digital Multimedia Broadcasting (DMB) phone, Smart phone, Handheld devices that may include communication functions such as mobile phones, personal digital assistants (PDAs), hand-held PCs, notebook computers, laptop computers, WiBro terminals, MP3 players, MD players, etc. And a portable electric and electronic device for all kinds of handheld-based wireless communication devices including an international roaming service and an international mobile telecommunication-2000 (IMT-2000) terminal that provides an extended mobile communication service. , Code Divis (CDMA) Predetermined devices such as wireless communication devices with GPS chips to enable location tracking via ion multiplexing access modules, Bluetooth modules, infrared data associations, wired and wireless LAN cards, and global positioning systems (GPS). It can be provided with a communication module of, and is equipped with a microprocessor capable of performing a multimedia playback function is interpreted as a concept that collectively refers to a terminal capable of performing a certain operation.

The hand color according to the present invention may include one or more of various colors such as white, flesh color, pink, brown, and black. In addition, it is obvious to those skilled in the art that the gesture information is not limited to a specific hand gesture, and the technical spirit of the present invention may be applied to other body parts or instruments (artificial arm, etc.) as it is. Will do.

Hereinafter, exemplary embodiments of a mobile communication terminal providing a mouse pointer operated by a hand gesture according to the present invention will be described with reference to the accompanying drawings.

2 is a block diagram illustrating a mobile communication terminal providing a mouse pointer operated by a hand gesture according to an embodiment of the present invention.

As shown in FIG. 2, the mobile communication terminal 200 according to the present invention includes an image capturing unit 210, a gesture information generating unit 220, a color detecting unit 225, a tracking calculating unit 230, and a mouse pointer control unit. 240, the display unit 250 may be included. Hereinafter, the function of each component will be described.

The image capturing unit 210 captures an image related to a gesture of a hand such as a hand shape and an operation of the terminal user. The camera module installed in the mobile communication terminal may be used as the image capturing unit 210. As the camera module, various sensors capable of detecting any device or object capable of capturing images, such as a digital camera, a CMOS camera, a Charged Coupled Device (CCD) camera, etc. The device can be used.

In addition, the image capturing unit 210 may be configured to automatically switch to a specific mode when the hand movement is stopped for a predetermined time or an image (“hand gesture image”) related to the hand movement is not input. For example, if the hand gesture image is not detected by the image capturing unit 210 for a predetermined time or the stopped hand gesture is maintained for a long time, the mobile communication terminal determines that the user is not operating the mouse pointer and saves power or saves the screen saver. You can control to switch to the mode or return to the initial screen. Therefore, power consumption can be minimized in the camera shooting mode which consumes a lot of battery capacity, and user convenience can be increased.

The gesture information generation unit 220 generates hand gesture information based on the hand gesture image signal captured by the image capturing unit 210. As an example of the generated hand gesture information, a hand gesture indicating a direction with an index finger or a hand gesture like pressing a button of a mouse with an index finger and a middle finger may be input from the image capturing unit 210 and used as the hand gesture information. The pointing operation of unfolding only the black of the human hand movement is the most commonly used operation for indicating a direction, and the present specification will be described taking the pointing operation of the index finger as an example.

At this time, the hand gesture and the background image irrelevant to the hand gesture are separated and only the hand gesture image is extracted as the hand gesture information so that the input hand gesture has a meaning and which mouse pointer operation corresponds to. As one method for distinguishing a hand gesture from a background other than the hand gesture, a method of detecting color may be used.

As an example of a detection method using color, the gesture information generator 220 includes a color detector 225 for distinguishing a color of the user's hand and a background color other than the hand (that is, two colors). Only hand gesture information needed to create an action can be extracted. An example of a figure in which the image signal of the hand gesture is converted by the color detector 225 is illustrated in FIG. 4 and will be described later.

The tracking operator 230 generates mouse coordinate information and direction information corresponding to the hand gesture information generated by the gesture information generator 220. In order to detect in which direction the position of the mouse pointer moves on the mobile communication terminal 200, a method of calculating a moving distance between a start point and an end point based on a coordinate axis may be used. For example, the moving distance and the direction can be calculated by calculating the end of the index finger as the point where the mouse pointer is located and how much the point moves in which direction.

For example, in the hand gesture image in which the index finger is pointing in a specific direction, the direction of the mouse can be detected by measuring the angle of the fist of the hand with the center of the coordinate axis and the index finger with respect to the X or Y axis of the XY coordinate system. Can be. Also, in the hand gesture image, an end portion (or end portion) of the index finger may be designated as the coordinates of the mouse pointer. An example of the tracking calculation method is shown in FIG. 3 and will be described later.

The mouse pointer controller 240 moves the mouse pointer corresponding to the mouse coordinate information and the mouse direction information. As mentioned above, a coordinate value may be assigned to a point at which the tip of the index finger is positioned around the coordinate axis. The mouse pointer may be positioned on the display unit corresponding to the coordinate values.

The display unit 250 visually displays the moved mouse pointer so that the user can recognize it. By displaying the mouse pointer continuously in real time, the user can view the mouse pointer as naturally moving from one point to another by optical illusion. For example, suppose that the mouse pointer corresponding to the coordinate value of the index finger reaches a point (5, 5) by linear movement at a 45 degree angle from the point (1, 1) of the two-dimensional coordinate axis. By sequentially displaying (2, 2), (3, 3), (4, 4) before reaching (5, 5), it is possible to feel the movement of the continuous mouse pointer.

Hereinafter, an embodiment in which each component described above will operate will be described with reference to the accompanying drawings.

3 is a diagram for explaining an example in which mouse coordinates change according to a hand motion according to another exemplary embodiment of the present invention.

As shown in FIG. 3, an image of a hand may be input through the image capturing unit 210. The tip of the index finger can be set as the position of the mouse pointer based on the two-dimensional coordinate axis. In FIG. 3A, the tip of the index finger points to the coordinate values (X11, Y11). When the position of the index finger is moved to another position, the index finger may have a value changed to (X12, Y12).

In addition, the tracking operator 230 may determine a change in the mouse coordinate information and the mouse direction information according to a change in the probability distribution of the hand color based on a predetermined midpoint in the hand gesture information. That is, the gesture information generator 220 may generate a color probability distribution of the hand gesture by referring to a look-up table.

In this case, the lookup table used is a table having a function similar to that of a reference table formed by considering hue and saturation at the same time, and may be used when generating a color probability distribution for interpreting hand gesture information. The elements constituting the lookup table may not only use artificially created colors but also various natural colors (eg, multiple users, skin color information of various races, and skin color information changed according to various lighting environments) photographed through the image capturing unit 210. Information) may be used.

As an example, the tracking operator 230 may generate mouse coordinate information and mouse direction information based on the 2D coordinate axis using the color probability distribution. Through a process of newly inputting a color into the two-dimensional lookup table ("learning process") by a method such as image capturing as described above, the subject in the photographing area is a color representing the unique color of the subject and the unique Saturation representing the saturation of the color of may be normalized to a value between 0 and 250, respectively. Thus, a histogram corresponding to this value can be generated. In this case, the lookup table may be configured with a probability value obtained through Equation (1) based on a histogram.

L (h, s) = H (h, s) / max H (i, j)-(1)

L (h, s): Position value at (h, s) in lookup table

H (h, s): The value of (h, s) in the histogram

max H (i, j): The maximum value of H (i, j), but max range is i, j = 0 to 255

h: index of Hue

s: index of saturation

The secondary lookup table consists of two axes, Hue and Saturation, and L (h, s) represents the value of the (h, s) position of the lookup table. In the process of dividing the hand gesture image from the background image, a probability distribution image indicating a probability that a pixel of an image signal input through the image capturing unit 210 corresponds to a hand gesture using gray scale or the like is obtained using the lookup table value. Can be.

4 is a view for explaining an example in which the position of the mouse pointer on the display changes in response to the position of the index finger of FIG. 3.

As shown in FIG. 4, the tip of the index finger in the hand gesture image input through the image capturing unit 220 may point to the point (-3, 4) when the center of the coordinate axis is set to (0, 0). (FIG. 4A). In the display unit 250, the mouse pointer is positioned corresponding to the hand gesture image, so that the mouse pointer may be checked at a point corresponding to (-3, 4) on the display unit 250 (FIG. 5B). .

Further, according to another embodiment of the present invention, the moving distance of the first mouse pointer 410 is the distance difference between the start coordinates and the end point coordinates in the end coordinates of the inter-frame index finger, and the resolution of the input image in the image capturing unit 210. , And may be calculated considering at least one of the difference in resolution of the display unit 250.

For example, the resolution of an image input to the image capturing unit 210 may be 1024 * 768, and an image signal generated at the resolution may be displayed at 640 * 480 resolution of the display unit 250. At this time, since the distance between the start coordinate of the mouse pointer and the end coordinate of the mouse pointer is different, in the process of converting the hand gesture image according to the resolution, the number of pixels is added or subtracted to match the resolution or the distance difference between the start point and the end point of the coordinate. You can control the mouse pointer to move in response to hand gestures by adding operations to extend or reduce values to match its resolution. In addition, a method of defining the motion equation of the mouse pointer moving along the coordinates of the index finger end and determining the position of the mouse pointer may be used.

In addition, according to another embodiment of the present invention, the display unit 250 continuously displays the mouse pointer to check the natural movement of the mouse pointer. The first mouse pointer 410 may have an effect similar to the mouse movement of the computer by allowing the mouse pointer to be displayed on the path where the hand gesture is moved while being moved to the position of the second mouse pointer 420.

5 is a view for explaining a method of distinguishing the color of the hand according to another embodiment of the present invention.

The image signal input to the image capturing unit 210 may be largely divided into parts other than the hand gesture and the hand gesture. At this time, in order to extract the gesture information of only the hand part, each part may be divided according to the color difference. As shown in FIG. 5, in the image signal input through the image capturing unit 210, the hand gesture image 510 is designated as the light and dark color, and the background image 520 other than the hand is designated as the dark contrast color. Can be.

In addition, since the hand gesture image 510 may be affected by the illumination of the environment around the hand, an image signal input to the image capturing unit 210 may be incorrectly determined. That is, if the color of the hand is detected only by considering the color Hue, different hand gesture information may be extracted according to the surrounding environment of the user. That is, if the user's hand is photographed against a red background having a color similar to that of the skin, it may be difficult to detect only the user's hand gesture area.

Accordingly, the gesture information generator 220 may divide the image signal photographed by the image capturing unit 210 into a hand gesture region and a background region without being affected by a noise color (ie, a color other than a hand gesture). The conversion to a specific color model method can be performed.

As an example, the image signal may be more accurately detected by converting the image signal into an HSV (Hue, Saturation, Value) color model to generate the hand gesture information. The HSV (Hue, Saturation, Value) refers to a color model most similar to a human visual model based on human intuitive vision. The model uses three attributes: Hue, Saturation, and Value or Brightness. Humans can distinguish between 128 (H) × 130 (S) × 23 (B) = 382720 colors.

In addition, other color models that may be used by the gesture information generation unit 220 may be an RGB (red, green, blue) color model, a CMY (Cyan, Magenta, Yellow) color model, or the like. By converting to any one of the color models described above, the hand gesture information may be generated in consideration of the purpose, performance, and the like of the mobile communication terminal 200.

RGB (Red, Green, Blue) refers to a system that represents colors to be used in computer displays. Red, green, and blue can be mixed in various proportions to achieve any color within the visual spectrum. As an example, the levels of R, G, and B may vary from 0 to 100%, respectively. Each level is represented by a total of 256 levels ranging from 0 to 255 decimal, which means 00000000 to 11111111 in binary and 00 to FF in hexadecimal. Thus, the number of possible colors can be 16,777,216, which is the result of the calculation of 256x256x256.

CMY (Cyan, Magenta, Yellow) is a model frequently used in computer graphics or digital printing. It is a color model used for printing. The primary colors of the CMY color model are cyan (primary blue, cyan), magenta (magenta, red, magenta), and yellow (yellow, original, yellow).

The mutual conversion between the RGB color model, the CMY color model and the HSV color model may be arbitrarily defined by a user using a predefined color using a color look-up table (CLUT). have.

The "hand gesture information" used in the present invention may be converted into any one of a HSV (Hue, Saturation, Value) color model, an RGB (Red, Green, Blue) color model, and a CMY (Cyan, Magenta, Yellow) color model. The present invention is not limited to a specific color model, and may apply all color models to the mouse coordinates and may be used according to the purpose or performance of the mobile communication terminal.

In addition, according to another embodiment of the present invention, the gesture information generation unit 220 includes a color gesture unit 225 for distinguishing a hand gesture from a portion other than the hand gesture according to the color of the hand gesture image 510. The hand gesture information may be generated by using, and the tracking operator 230 may generate mouse coordinate information and direction information by using the hand gesture information. In addition, in order to analyze and track the hand gesture, a basic algorithm for accurate segmentation of the hand region may be provided.

For example, a database of skin colors by race or individual characteristics for various skin colors may be built to have all optimized color distribution data for determining hand gestures. In another method, color distribution data optimized for each user may be provided in consideration of changes in lighting due to changes in user's environment and hand motion in real time.

In addition, the color detecting unit 225 may consider a dark portion of the hand gesture as a part of the hand to detect the changed color of the skin region due to the change in the hand motion according to the illumination of the hand. By comparing the color of the background and the background color other than the hand may be determined by hand only when the difference between each color exceeds a predetermined threshold. For example, a user with a brown skin color may include yellow or maroon similar to this color to be recognized as an area of hand gestures. In addition, in addition to color, saturation and contrast may also be included in the area of the hand gesture.

6 is a diagram for describing a method of executing a click command according to a hand gesture according to another exemplary embodiment of the present invention.

In the case of capturing an image related to a hand gesture through the camera module attached to the mobile communication terminal 200 as the image capturing unit 210, a command other than the movement of the mouse pointer is input to the mobile communication terminal 200 by modifying the hand gesture. You can do that. In FIG. 6, as an example, gesture information different from a straight index finger (FIG. 3) can be input by inputting a curved index finger as an image signal. For example, the straight index finger may simply be a gesture of changing the coordinate axis to move the mouse pointer. In another embodiment, the bent index finger can be designated as a hand gesture that is performing a click action of a mouse button. Therefore, the gesture information generation unit 220 may generate the click operation information of the mouse pointer based on the image signal of the bent index finger.

7 is a flowchart illustrating a method of operating a mouse pointer by a hand gesture in a mobile communication terminal according to the present invention.

In operation S710, the mobile communication terminal captures an image of a user's hand gesture. The image input to the mobile communication terminal is not limited to a specific hand gesture, and various hand shapes and hand gestures may be input.

In operation S720, the mobile communication terminal generates hand gesture information based on the captured video signal. As one of the methods for generating the hand gesture information by the mobile communication terminal, only the hand gesture information may be generated by color-dividing the hand gesture and the parts other than the hand gesture.

In operation S730, the mobile communication terminal generates mouse coordinate information and direction information according to the hand gesture information. As an example, the end of the index finger can be designated as the coordinate axis of the mouse pointer and mouse coordinate information and direction information corresponding to the end can be generated. In addition to generating the mouse coordinate information and the direction information by changing the gesture of the hand, a mouse click command may be specified.

In step S740, the mobile communication terminal moves the mouse pointer corresponding to the mouse coordinate information and the mouse direction information. That is, the mouse pointer may be controlled to move to a corresponding position on the display of the mobile communication terminal in response to a user's hand gesture.

In step S750, the moved mouse pointer is displayed. Thus, the user can move his or her hand as if holding a virtual mouse, and reproduce the operation of opening a folder or executing a command as if the mouse is controlled by the PC.

Hereinafter, in the performing of the present invention described with reference to FIG. 7, a method of defining a photographing area for capturing a more optimized hand gesture area and a method of changing the position of a center point as a reference for tracking coordinates in the photographing area are changed. It will be described in detail.

8 is a flowchart illustrating a method of setting a photographing area of a hand gesture according to the present invention.

In step S810, the mobile communication terminal sets the size of the photographing area of the hand gesture. It is possible to efficiently photograph the hand gesture area by minimizing the size of the photographing area by capturing only the hand gesture area out of the entire area photographed through the mobile communication terminal.

In operation S820, a color probability distribution is generated by referring to a predetermined look-up table in the photographing area. A color probability distribution may be generated to extract the hand gesture region, and the color probability distribution may include color and saturation information of a 2D lookup table, and the color probability distribution may be represented by a histogram.

In this case, the photographing area may be determined in consideration of the two-dimensional moving distance and the size change of the hand gesture. If there is a background similar to the hand in addition to the user's hand, the mobile communication terminal may enlarge the photographing area to lose information of the hand gesture area being tracked. In addition, when the change in the size of the hand gesture becomes small, it is not necessary to photograph all the areas unnecessarily, so that the photographing area can be reduced to efficiently photograph.

In step S830, the size and direction of the hand gesture are calculated from the generated color probability distribution, and the end point coordinates of the hand are calculated using the calculated size and direction (step S840). The size and direction may be calculated in a similar manner as described above, and thus will be omitted.

Hereinafter, the process of moving the position of the center point in a method for tracking the coordinates of the end of the index finger corresponding to the mouse pointer in the photographing area will be described.

9 is a flowchart illustrating a method of changing the position of a midpoint as a reference for tracking coordinates in a photographing area according to the present invention.

In step S910, the position of the center point is searched for in the photographing area, and then the photographing area is moved around the found center point (step S920). The photographing area may be limited to a photographing area considering only a change in the size of the hand gesture. In order to effectively limit the shooting area of the hand gesture, it is necessary to be able to track the fast movement of the hand and to change the size of the shooting area of the hand gesture (adjust the horizontal and vertical ratio) to track the coordinate points of the end of the hand. have.

By repeating steps S910 and S920 until the movement of the midpoint becomes less than a predetermined threshold in step S930, the photographing area is more efficiently photographed by limiting the gradually moving hand motion. Can be specified. The threshold value can be set arbitrarily by a user. In addition, since the position of the photographing area moves along the midpoint of the user's hand gesture area, it is difficult to track the motion of the hand when there is noise around the hand gesture area. Therefore, the more accurately the two-dimensional lookup table is prepared, the easier tracking is. In this case, a pixel having a small probability less than or equal to a threshold in the probability distribution of the hand gesture region obtained by dividing the hand gesture region from the surrounding background is not regarded as the hand gesture region, thereby affecting the surrounding background (or surrounding noise). Can be minimized.

So far, the method of controlling the mouse pointer of the mobile communication terminal according to the present invention has been described, and the above-mentioned details of the embodiments of FIGS. Will be omitted.

The method for controlling a mouse pointer of a mobile communication terminal according to the present invention can be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. The medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, the present invention has been described by specific embodiments such as specific components and the like, but the embodiments and the drawings are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. However, various modifications and variations are possible to those skilled in the art to which the present invention pertains.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and all the things that are equivalent to or equivalent to the claims as well as the following claims will belong to the scope of the present invention. .

According to the present invention, the user's convenience can be increased by simply moving the mouse pointer displayed on the display unit provided in the mobile communication terminal by a hand motion.

In addition, according to the present invention, it is possible to expect the effect of reducing the production cost of the terminal by moving the mouse pointer utilizing the resources of the mobile communication terminal.

In addition, according to the present invention, there is an effect that can be conveniently used to freely control or play a multimedia program such as mobile game content that requires a human computer interface (HCI).

In addition, according to the present invention, it is possible to detect a hand gesture in consideration of the illuminance around the hand input through the image photographing apparatus of the mobile communication terminal.

In addition, according to the present invention, it is possible to detect not only the movement of the mouse pointer but also various hand gestures to perform a click operation of the mouse pointer.

In addition, according to the present invention, by continuously generating a mouse pointer corresponding to the hand gesture, there is an effect that it is possible to represent the movement of the natural mouse pointer.

In addition, according to the present invention, in addition to the distance difference between the start point and the end point of the coordinates of the end of the hand, it is possible to track the mouse pointer more precisely by considering the difference in the resolution of the input image and the resolution of the display.

In addition, according to the present invention, when the hand gesture is not input for a predetermined time or more, it is possible to reduce the electricity consumption of the mobile communication terminal by switching to a specific mode.

In addition, according to the present invention, it is possible to track the fast moving hand gesture by limiting the area for photographing the hand gesture according to the size or the moving distance of the hand gesture.

In addition, according to the present invention, by converting the image signal photographed by the hand gesture to a variety of color models, there is an effect that can effectively divide the hand gesture and the surrounding noise.

Claims (18)

In the mobile communication terminal, An image capturing unit configured to capture an image of a user's hand gesture; A gesture information generator configured to generate hand gesture information based on the captured image signal; A tracking calculator configured to generate mouse coordinate information and mouse direction information according to the hand gesture information; A mouse pointer controller for moving a mouse pointer corresponding to the mouse coordinate information and the mouse direction information; And A display unit displaying the moved mouse pointer Mobile communication terminal is installed, characterized in that it comprises a mouse pointer control device. The method of claim 1, The gesture information generation unit may convert the photographed image signal into any one of an HSV (Hue, Saturation, Value) color model, an RGB (Red, Green, Blue) color model, and a CMY (Cyan, Magenta, Yellow) color model. Generating the hand gesture information by converting Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 1, The gesture information generator includes a color detector, The color detecting unit is a mobile communication terminal with a mouse pointer control device, characterized in that for distinguishing the color of the user's hand and the background color other than the hand. The method of claim 1, The gesture information generator includes a color detector, The color detecting unit generates the hand gesture information in consideration of the changed color of the skin region due to the change of the hand motion according to the illuminance around the hand. Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 3, The color detecting unit compares the color of the hand with the background color other than the hand and judges the hand gesture information when the difference exceeds a predetermined threshold. Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 1, The gesture information generator generates a probability distribution of the hand gesture by referring to a look-up table. The tracking operation unit generates mouse coordinate information and mouse direction information from the probability distribution. Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 6, The lookup table is updated with new color information input through the mobile communication terminal. Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 1, The gesture information generation unit generates click operation information of the mouse pointer based on the image signal. Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 1, The tracking calculator calculates a moving distance of the mouse pointer in consideration of at least one of a distance difference between the start coordinates and the end point coordinates of the mouse coordinate information, the resolution of the image capturing unit, and the resolution of the display unit. Mobile communication terminal is installed, characterized in that the mouse pointer control device. The method of claim 1, And the display unit continuously displays the mouse pointer in real time. The method of claim 1, And the image capturing unit is a camera module. The method of claim 1, The image capturing unit switches to a specific mode when the hand movement is stopped for a predetermined time or an image related to the hand movement is not input. Mobile communication terminal is installed, characterized in that the mouse pointer control device. In the method of controlling the mouse pointer of the mobile communication terminal, Photographing an image relating to a user's hand gesture; Generating hand gesture information based on the captured image signal; Generating mouse coordinate information and direction information according to the hand gesture information; Moving a mouse pointer corresponding to the mouse coordinate information and the mouse direction information; And Displaying the moved mouse pointer Method of controlling a mouse pointer of a mobile communication terminal comprising a. The method of claim 13, Generating the hand gesture information Setting a shooting area of the hand gesture; Generating a color probability distribution by referring to a predetermined two-dimensional look-up table for the photographing area; And Generating the hand gesture information from the generated color probability distribution Method of controlling a mouse pointer of a mobile communication terminal comprising a. The method of claim 14, And the photographing area is determined in consideration of a two-dimensional moving distance and a change in the size of a hand gesture. The method of claim 14, Setting the shooting area of the hand gesture, (a) searching for a position of a center point in the photographing area; And (b) moving the photographing area around the found center point Including, And repeating steps (a) and (b) until the moving distance of the midpoint becomes equal to or less than a predetermined threshold. The method of claim 16, When the moving distance of the midpoint is less than or equal to the threshold, determining the photographing area in consideration of a change in the size of the hand gesture; Method of controlling a mouse pointer of a mobile communication terminal, characterized in that. A computer-readable recording medium having recorded thereon a program for executing the method of claim 12.

Images