KR101559424B1 - A virtual keyboard based on hand recognition and implementing method thereof - Google Patents

Abstract

The present invention relates to a virtual keyboard based on hand recognition and an implementation method thereof. The method for implementing a virtual keyboard by using at least one processor includes the steps of: recognizing a user′s hand by using a sensor; combining two or more keys by assigning the key combinations to the respective fingers of the hand recognized in the previous step; presenting one of the key combinations assigned to the respective fingers as a candidate key group based on the slope of the extracted hand against a specific reference; and selecting at least one key in the candidate keys present for the respective fingers as an input value according to a selection gesture using the motion of the user′s fingers.

Description

Technical Field [0001] The present invention relates to a virtual keyboard based on hand recognition,

In particular, the present invention relates to a keyboard technology that is implemented not in a physical physical keyboard but in particular a keyboard technique using a sensing sensor to recognize a user's hand and input a key corresponding to the movement of a user's hand or finger An apparatus and a method for implementing a virtual keyboard as far as possible, and a recording medium on which the method is recorded.

A classic and common method for controlling a computer or mobile terminal device is to enter letters or numbers using an input interface such as a keyboard / keypad or a mouse. For example, in a mobile communication terminal, it is very natural for us to enter letters or telephone numbers using a keypad and to control various application programs using a keyboard.

Since the input device such as a keyboard or a keypad is composed of a plurality of keys, it is inconvenient to occupy a relatively large space as compared with other interfaces. In the case of a computer, a keyboard is a useful device for performing various commands and operations, but it is inconvenient because it is the largest input device. In addition, in the case of a mobile terminal device such as a PDA, a mobile phone, and the like, the size of the mobile terminal device is becoming smaller. Therefore, the existing keypad interface is not suitable for miniaturization of the terminal.

New interfaces such as touch screen, virtual keyboard, pen, wheel, etc. have been suggested as an alternative to the conventional inconvenient interface, but they have a disadvantage in that a separate additional device is required or only a limited environment can be used. Therefore, it has been required to develop an input means for detecting and tracking the position and movement of the user's hand in real time so as to enable key input corresponding to the user's hand.

In the prior art cited below, a virtual input device using a three-dimensional hand model generation technique is proposed based on the recognition of such a problem. However, only a conventional keypad has been brought using a virtualization technology, There is a distance from the input device that can help.

Korean Patent No. 10-2010-0041464, Korean University Industry-University Collaboration Foundation, Apr. 22, 2010

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to solve the inconvenience of input difficulty and visual disruption due to the fixed input interface or continuous change according to the movement of a finger, In case of applying to augmented reality, we try to solve the problem of spatial sacrifice due to display of keyboard by utilizing the form of keyboard.

According to an aspect of the present invention, there is provided a method of implementing a virtual keyboard using at least one processor, the method comprising: recognizing a user's hand using a sensing sensor; Assigning at least two keys to each finger of the recognized hand; Presenting, as candidate keys, any one of a combination of keys for each finger based on a slope of a hand extracted in comparison with a predetermined criterion; And selecting at least one key among the candidate keys presented for each finger according to the selection gesture using the movement of the user's finger.

In the method of implementing a virtual keyboard according to an embodiment, the step of assigning the two or more keys in combination includes dividing a plane parallel to the user's hand into slope sections according to the inclination of the hand, One key is allocated as a candidate key. Meanwhile, the number of divided sections may be equal to the number of combinations of the keys allocated for each finger.

In the method of implementing the virtual keyboard according to an embodiment, the keys may be respectively assigned to a user's finger according to a key arrangement of a QWERTY keyboard.

In the method of implementing a virtual keyboard according to an embodiment, a combination of keys assigned to the respective fingers may be displayed through a display unit by fixing a virtual keyboard with reference to the user's hand or wrist.

In the method of implementing a virtual keyboard according to an embodiment of the present invention, the step of extracting a hand inclination with respect to the predetermined reference may include: detecting a wrist for distinguishing a forearm region from a recognized hand region; And setting the forearm axis and the forearm axis respectively from the backlight region and the forearm region and extracting the inclination of the hand based on the correlation between the detected wrist and the forearm axis or forearm axis.

According to an embodiment of the present invention, when the gesture using the wrist rotation is detected, the method may further include changing a character combination assigned to each finger and assigning a new finger to each finger. In addition, the character combination may include at least two of a language-specific character set, a number set, and a special character set. Further, in the step of detecting the gesture using the wrist rotation, a virtual circle is formed in the hand area extracted from the recognized hand, and a coefficient indicating the degree of the circle shape close to the ellipse is calculated and compared with the threshold value , The wrist rotation can be determined.

According to another aspect of the present invention, there is provided a method of implementing a virtual keyboard using at least one processor, comprising: recognizing a user's hand using a sensing sensor; Receiving a selection gesture using a tilt of a hand and a movement of a finger with respect to a plane parallel to the back of the hand using the sensing sensor; And selecting any one of at least two keys assigned to the finger of the hand based on the inputted inclination of the hand and the selection gesture as an input value.

According to another embodiment of the present invention, when the slope of the hand is inputted in advance of the selection gesture, the step of selecting any one of the keys as the input value may include: The method according to claim 1, wherein the at least one key is a combination of at least two keys, and the at least one of the at least two keys is assigned a combination of the at least two keys. And selects at least one key among the candidate keys presented for each finger as an input value.

According to another embodiment of the present invention, when the selection gesture is input prior to the inclination of the hand, the step of selecting the one of the keys as the input value may include: At least two keys are combined as a candidate key, and one of the candidate keys assigned to each finger is selected as an input value based on a slope of the extracted hand in comparison with a predetermined reference.

Furthermore, the present invention provides a computer-readable recording medium having recorded thereon a program for causing a computer to implement the above-described virtual keyboard implementation method.

According to an aspect of the present invention, there is provided an apparatus for implementing a virtual keyboard, comprising: a sensor for recognizing movement of a user's hand and a finger; A memory for storing a program for implementing a virtual keyboard; And at least one processor for operating a program implementing the virtual keyboard, wherein the program stored in the memory is configured to assign a combination of at least two keys to each finger of the recognized hand, The method of any one of the preceding claims, wherein each of the plurality of keys comprises at least one of a plurality of keys, Includes commands to select.

In the apparatus for implementing a virtual keyboard according to an embodiment, a program stored in the memory divides a plane parallel to the user's hand into a slope-based interval according to a slope of the hand, As a candidate key.

In an embodiment of the virtual keyboard according to an embodiment, the keys may be respectively assigned to a user's finger according to a key arrangement of a QWERTY keyboard.

The virtual keyboard according to an embodiment of the present invention may further include a display unit displaying a combination of keys allocated to the respective fingers, wherein a combination of keys allocated to the respective fingers is independent of movement of the fingers A virtual keyboard is fixed at a predetermined reference position and can be displayed through the display unit.

The apparatus for implementing a virtual keyboard according to an embodiment may further include a display unit for displaying a combination of keys assigned to the respective fingers, wherein the display unit allows the user to visually recognize Through display means for displaying a combination of keys assigned on the transmitted hand while overlapping the displayed hands.

In the apparatus for implementing a virtual keyboard according to an embodiment of the present invention, when a gesture using a wrist rotation is detected through the detection sensor, a program stored in the memory changes a character combination assigned to each finger, And may further include an instruction to assign. In addition, the character combination may include at least two of a language-specific character set, a number set, and a special character set.

Embodiments of the present invention provide a virtual keyboard in a non-fixed / adaptive form while at the same time fixing the virtual keyboard to a less active area, allowing for easier key entry, and using a tilt of the user's wrist, By implementing a one-to-many relationship that maps key combinations to one finger each, minimizing the sacrifice of space on the keyboard display, and utilizing the rotation of the wrist to change the character set assigned to the finger, Can be input.

1 is a diagram for explaining the principle of implementation of a virtual keyboard based on hand recognition adopted by embodiments of the present invention.
Figs. 2A to 2C are diagrams illustrating a candidate key assigned to each finger according to the inclination of the hand based on the implementation principle of Fig.
3 is a flowchart illustrating a method for implementing a virtual keyboard based on hand recognition according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a technique for extracting a hand slope in a method of implementing a virtual keyboard according to embodiments of the present invention.
5 is a diagram illustrating a technique for extracting a back and forth forearm axis using a circular model in a method of implementing a virtual keyboard according to embodiments of the present invention.
FIG. 6 is a diagram for explaining a method of changing a character combination assigned to a finger using the rotation of the wrist in the method of implementing a virtual keyboard according to the embodiments of the present invention.
FIG. 7 is a diagram for explaining a technique for separating a hand region and a finger region using a circular model in a method of implementing a virtual keyboard according to embodiments of the present invention.
FIG. 8 is a diagram for explaining a method of detecting the rotation of the wrist using the original model in the method of implementing a virtual keyboard according to the embodiments of the present invention.
FIG. 9 is a flowchart illustrating a method of implementing a virtual keyboard based on hand recognition according to another embodiment of the present invention.
10 is a block diagram illustrating an apparatus for implementing a virtual keyboard based on hand recognition according to an embodiment of the present invention.

Prior to describing the embodiments of the present invention, after studying the characteristics and problems in using the virtual keyboard technology in the Augmented Reality field, in order to solve such problems, the embodiments of the present invention Provide an overview of the technical means.

As a virtual keyboard interface method for smart glasses, two methods can be suggested. One is a fixed virtual keyboard method using a real keyboard interface, and the other is a non-fixed type / adaptive virtual keyboard type which is specialized in a hand .

First, when a fixed three-dimensional interface of a real keyboard is augmented on a two-dimensional screen, it is necessary to visually concentrate a virtual keyboard and a finger position in the screen, and to discriminate an interaction with a finger corresponding to a specific button There is a drawback that it is difficult. For example, when a finger touches a button area and enters a different button area when a finger presses a gesture, the recognition and processing of the finger movement becomes unclear.

Second, the hand - specific non - fixed / adaptive virtual keyboard method uses a non - fixed / adaptive interface to recognize the hand and enhance the button on the finger to improve the fixed virtual interface. However, the non-fixed type / adaptive type interface has a problem that the position where the virtual button is augmented according to the movement of the finger is continuously changed, which can give visual confusion to the user and can not provide a stable interface.

Furthermore, since both of the above methods are used to implement a virtual keyboard in the form of a keyboard, it is inevitable that a virtual keyboard occupies a large area in the screen. This characteristic can be pointed out as a disadvantage that it is prominent in a mobile device or a head-up display (HUD) device having a relatively small display area.

To solve this, embodiments of the present invention perform user hand recognition based on a three-dimensional hand model using a sensing sensor (e.g., can be a three-dimensional depth sensor) (For example, it may be a hand with little movement depending on the batter, unlike a finger), and the three-dimensional virtual keyboard is augmented, and the three-dimensional hand inclination / angle change, finger movement, And a key input unit for inputting the key input.

In particular, in embodiments of the present invention, a keyboard is fixed and visualized based on a fixed area (e.g., a hand), a combination of virtual keys that can be input is augmented by dividing each finger, You can select a letter. For this purpose, the key of the character to be selected can be inputted as a 'click' gesture of the corresponding finger. On the other hand, the character combination of the visualized augmented reality keyboard can be focused on the experience of using the keyboard or keypad currently used by the majority of users. For example, a two-handed keyboard may be used in addition to a QWERTY keyboard or a conventional double-edged keyboard for a two-handed hitter, or a keypad of a portable terminal may be used for a one-handed hitter. Furthermore, the character combination of the augmented reality keyboard can be changed / selected by using the rotation of the wrist.

Hereinafter, embodiments of the present invention for solving the above-mentioned technical problems will be described in detail with reference to the drawings. In the following description and the accompanying drawings, detailed description of well-known functions or constructions that may obscure the subject matter of the present invention will be omitted. It is to be noted that the same components are denoted by the same names and reference numerals as possible throughout the drawings.

FIG. 1 is a diagram for explaining a principle of implementation of a virtual keyboard based on hand recognition adopted by embodiments of the present invention. For convenience of explanation, a QWERTY keyboard for a two-handed hitter is assumed.

Referring to FIG. 1, embodiments of the present invention propose a fixed / adaptive interface that provides a keyboard interface fixed around the back of a hand using a hand recognition technology as a virtual keyboard interface. This virtual keyboard interface is divided into five input areas (L1, L2, L3, L4, and L5 areas of the left hand and R1, R2, R3, R4, and R5 areas of the right hand) based on the starting joint position of each finger Can be visualized, and each area can visualize the set of input characters assigned to that area. In order to solve the finger-button relation of one-to-many (1: N, N is a positive integer) relationship, as an interaction method, a combination of inputable buttons is selected by adjusting the angle / tilt of the hand, We propose a method of inputting assigned characters. In addition, the keyboard may play an additional role, such as changing or editing the input mode using the front-to-rear axis and the horizontal axis rotation motion during the three-dimensional rotational movement of the hand.

1, a combination of letter keys corresponding to the left hand is assigned in the L5 area and a combination of letter keys "2wsx" in the L4 area corresponding to the left finger ring is assigned , A combination of character keys "3edc" corresponding to the left-hand stop is assigned to the L3 area, a combination "4rfv" of the character keys corresponding to the left-hand detection is assigned to the L2 area, "5tgb" A combination of character keys "6yhn" corresponding to the left hand thumb is assigned to the R1 area, a combination key "7ujm" corresponding to the left hand is assigned to the R2 area, Quot ;, and a combination of character keys "9ol." Corresponding to the left hand fingerprint is assigned to the R4 area and a combination of character keys "0p /" .

As illustrated in FIG. 1, the interaction technique adopted by the embodiments of the present invention uses a key button input, a key button selection, and a keyboard combination selection as independent interactions, , And provides an interaction method that can be easily input if one gets used without paying attention. In order to realize this, 3D rotation recognition technology of the back of the hand is required, and especially, a recognition rate which is not affected by the movement of the finger is required. Therefore, the main supporting structure of the present invention includes a finger-free three-dimensional wrist tilt and handwheel rotation recognition technique.

2A to 2C are diagrams for explaining a candidate key assigned to each finger according to the inclination of the hand based on the implementation principle of FIG. 1. For convenience of explanation, only the English characters of the QWERTY keyboard are assigned to respective fingers . That is, since the keyboard corresponding to the alphabetic character is a total of 3 lines, three slope distinctions are required to select three characters in each combination. Therefore, when three virtual areas are formed in a fan shape around the user's wrist, one character can be selected in the key combination according to the left-right inclination of the hand.

Referring to FIG. 2A, the wrist is inclined to the left with respect to a plane parallel to the back of the user with respect to the left hand of the user. The first row of the alphabetical letters of the QWERTY keyboard may be assigned to each finger . That is, the characters 'q', 'w', 'e', 'r', and 't' can be presented as candidate keys in order from left hand to thumb. After the candidate key is selected in this way, the user can select the key as an input value by taking a gesture for selecting at least one of the presented candidate keys.

Referring to FIG. 2B, there is illustrated a case where the wrist is not shifted to the left or the right with respect to the plane parallel to the back of the user with respect to the left hand of the user, and the second row of the alphabetical letters of the QWERTY keyboard is assigned to each finger . That is, the characters 'a', 's', 'd', 'f', and 'g' can be presented as candidate keys in the order from left hand to thumb. After the candidate key is selected in this way, the user can select the key as an input value by taking a gesture for selecting at least one of the presented candidate keys.

Referring to FIG. 2C, a wrist is inclined to the right with respect to a plane parallel to the back of the user with respect to the left hand of the user, and a third row among the alphabetical rows of the QWERTY keyboard may be assigned to each finger . That is, the letters 'z', 'x', 'c', 'v', and 'b' can be presented as candidate keys in the order from left hand to thumb. After the candidate key is selected in this way, the user can select the key as an input value by taking a gesture for selecting at least one of the presented candidate keys.

FIG. 3 is a flowchart illustrating a method of implementing a virtual keyboard based on hand recognition according to an embodiment of the present invention, and may be implemented in a form of a computer program driven using at least one processor to be provided in a virtual keyboard device .

In step S310, the virtual keyboard device having at least one processor recognizes the user's hand using the detection sensor. Such a sensing sensor may be a variety of shape / motion recognition means such as a depth sensor or a three-dimensional camera, and a product such as a Kinect sensor or a Leap Motion may be adopted.

In step S320, the virtual keyboard apparatus combines and assigns at least two keys to each finger of the hand recognized in step S310. In particular, in step S320, a plane parallel to the user ' s hand is divided into slope sections according to the slope of the hand, and one key may be assigned as a candidate key for each section. In this case, the number of divided intervals may be generally the same as the number of combinations of the keys allocated for each finger.

As described above with reference to FIG. 1, the key of the key mapping proposed by the embodiments of the present invention is a one-to-many relationship in which at least two keys are assigned to one finger.

In addition, the keys may be assigned to the user's fingers according to the key arrangement of the QWERTY keyboard for the two-handed hitter, and a portion of the equity keyboard or a keypad may be assigned for the one-handed hitter.

Preferably, the combination of the keys allocated to the respective fingers is displayed through the display unit by fixing the virtual keyboard with reference to the user's hand or wrist. As mentioned above, when the key is strengthened according to the position of the finger, since the position of the user's finger changes rapidly, the image of the enhanced key moves accordingly, thereby causing visual confusion to the user. Therefore, in the embodiments of the present invention, a virtual keyboard is augmented and displayed on the basis of a body part, i.e., a back or a wrist, which is relatively less influenced by movement of a finger, in order to eliminate such inconvenience.

In step S330, the virtual keyboard device presents any one of the key combinations for each finger as a candidate key based on the inclination of the hand extracted in comparison with the predetermined reference. Herein, the predetermined reference is a reference to be able to compare with the inclination of the hand, and the virtual area is divided according to the inclination of the hand according to the inclination of the hand, and it is determined whether the user's hand is included in the specific area among the divided inclination areas do. That is, the slope determination is necessary to select any one of the one-to-many key combinations assigned through step S320. A specific method of setting a certain criterion will be described later with reference to FIGS. 4 to 5. FIG.

In step S340, the virtual keyboard device selects at least one key among the candidate keys presented for each finger according to the selection gesture using the movement of the user's finger. As illustrated in FIG. 2, when a user's selection gesture is generated under a situation where one candidate key among the key combinations assigned to each finger is presented according to the inclination of the user's hand, .

Hereinafter, a method of extracting the inclination of the hand will be described with reference to FIGS. 4 and 5. FIG. FIG. 4 is a diagram illustrating a technique of extracting the inclination of the hand in the method of implementing a virtual keyboard according to the embodiments of the present invention. FIG. circle model and a forearm axis are extracted using a circle model. For convenience of description, embodiments of the present invention use a depth sensor, with the arm pointing to the bottom of the screen, the hand pointing to the top of the screen, and the distance between the hand and the depth sensor .

To recognize the hand regardless of the finger, the wrist is first recognized in the whole arm area and then the hand model is fitted. In order to accurately recognize the wrist portion, PCA (Principal Component Analysis) is first performed on all the pixels corresponding to the arm region, and the principal axes of the pixels are found and rotation normalization is performed. Then, a distance transform is used to find the center point of the hand. This is based on the principle that the pixel corresponding to the center of the hand has the highest value when the contour line of the neighboring points of the whole image of the arm is drawn. 4 (a) of Fig.

Now, if we raise the radius of the lower part of the circle corresponding to the arm centered on the center of the hand, it will meet with the outline of the hand, and the radius corresponding to the model of the back of the hand can be obtained. The wrist region can be defined as a line orthogonal to the direction of the hand axis and in contact with the model of the back of the hand, and is illustrated in FIG. 5, the central axis (Hand Center, P _H) the center circle model and wrist (Wrist Line, L _W), the back of the hand shaft (Palm Axis, L _P) determined therefrom having a radius r _H a in the back of the hand and It can be seen that Forearm Axis (L _F ) can be extracted.

As described above, if the wrist region is defined, the main axis can be found for the hand and the arm based on the distance transform local maxima of each region, This process has been sequentially illustrated through (b) and (c) of FIG. Thus, the user can recognize the tilt of the hand (vertical axis rotation of the hand) using the two pieces of main axis information and select a specific key button according to the user's selection gesture.

In summary, in the embodiments of the present invention, the process of extracting the inclination of the hand is performed by detecting a wrist for distinguishing the forearm region from the recognized hand region, Axis and the forearm axis, and extracting the inclination of the hand based on the correlation between the detected wrist and the forearm axis or forearm axis.

The embodiments of the present invention can provide a function of changing the character combination of the keyboard by using the rotation of the wrist as well as selecting the key of the keyboard using only the tilt of the hand. To this end, embodiments of the present invention can change a character combination allocated to each finger and assign it to each finger newly when a gesture using the wrist rotation is detected. Here, the character combination may include at least two of a language-specific character set, a number set, and a special character set.

6 is a diagram for explaining a method of changing a character combination assigned to a finger using rotation of the wrist in a method of implementing a virtual keyboard according to embodiments of the present invention, .

6 (a) shows a situation immediately before the user rotates the wrist. Assume that the keyboard / keypad assigned to the finger is a first keyboard (e.g., English keyboard). 6 (b), when the user rotates his / her wrist to turn the handwheel toward the front, the virtual keyboard device according to the embodiments of the present invention recognizes a character set change command. Then, when the user rotates the wrist again to return to the original position in FIG. 6C, the virtual keyboard device can be a second keyboard (for example, a Korean keyboard) in the first keyboard assigned previously. The character set will be changed. Likewise, by repeating this process, it is possible to change from the second keyboard to the first keyboard or sequentially change to a third keyboard (for example, a numeric keyboard or a special character keyboard).

Hereinafter, the process of recognizing the user's wrist rotation will be described in more detail with reference to FIGS. 7 and 8. FIG.

FIG. 7 is a diagram for explaining a technique for separating a hand region and a finger region using a circular model in a method of implementing a virtual keyboard according to an embodiment of the present invention. Referring to FIG. 7, .

As described above with reference to FIG. 6, the virtual keyboard technology according to embodiments of the present invention uses the horizontal axis rotation of the back of the hand for changing / selecting the keyboard combination. When the back of the hand is estimated to be close to the circle, the back of the hand is not rotated to the horizontal axis and the keyboard can be input. When the hand lamp is estimated to be close to the ellipse, the hand lamp is rotated on the horizontal axis and the keyboard combination selection command is executed. Therefore, in order to estimate the shape of the back of the hand regardless of the change of the finger, information of each finger region as well as the wrist region is required. The wrist region was estimated while increasing the radius of the circle, and information on the finger region was obtained at the same time as the wrist region information was obtained. These results are illustrated in FIG. The conic coefficient can be obtained using the position information of the pixel corresponding to the wrist region, the finger region, and the outer angle of the hand. The back of the hand corresponds to three types of conic: hyperbola, parabola, and ellipse. Since the short axis of the ellipse can not be longer than the long axis, the length of the short axis can be normalized to a value between '0' and '1' by dividing the length of the short axis by the length of the long axis.

8 is a view for explaining a method of detecting the rotation of the wrist using the original model in the method of implementing a virtual keyboard according to the embodiments of the present invention. .

Referring to FIG. 8, when the value is close to 1, the back of the hand is close to the circle, indicating that it is not rotated on the horizontal axis. On the other hand, if the value is close to 0, the hand lamp is recognized as being rotated to the horizontal axis near the ellipse, and the keyboard combination selection command can be executed. In order to know the degree of rotation of the 3D wrist, three-dimensional rotation information can be obtained by fitting the 3D depth information (3D points) of the back region selected by the ellipse to the 3D hand model using ICP (Iterative Closest Point) have.

Now, the embodiments of the present invention perform key button input only when the hand light representing a situation in which key button input is possible is close to a circle. The key button input can track the movement of each fingertip and determine the input according to its trajectory. For the binarized image corresponding to the hand region, the fingertip can be recognized by selecting five outermost border regions using a convexity defect.

In summary, the process of detecting a gesture using a wrist rotation in a virtual keyboard according to embodiments of the present invention includes forming a virtual circle in a hand area extracted from a recognized hand, And determines whether or not to rotate the wrist by comparing the threshold with a threshold value (which will be a reference value allowing key button input).

FIG. 9 is a flowchart illustrating a method of implementing a virtual keyboard based on hand recognition according to another embodiment of the present invention, and a series of processing steps is described focusing on the relationship between a hand tilt input and a selection gesture input. As described above with reference to FIG. 3, the method of implementing the virtual keyboard of FIG. 9 may also be implemented in the form of a computer program that is executed using at least one processor, and may be provided in a virtual keyboard device.

In step S910, the virtual keyboard device having at least one processor recognizes the user's hand using the detection sensor.

In step S920, the virtual keyboard device receives the selection gesture using the inclination of the hand and the movement of the finger with respect to the plane parallel to the back of the hand using the sensing sensor.

In step S930, the virtual keyboard device may select any one of at least two key combinations assigned to the finger of the hand based on the inclination of the hand and the selection gesture input through step S920 Select. At this time, there may be some differences in the key selection process depending on the relationship between the input of the hand slope and the input of the selection gesture.

First, when the inclination of the hand is inputted before the selection gesture, in step S930 of selecting any one of the keys as an input value, at least two or more keys are assigned to each finger of the recognized hand, The method comprising: presenting, as a candidate key, any one of a combination of keys for each finger based on a slope of a hand extracted in comparison with a predetermined criterion; and displaying at least one candidate key among the candidate keys presented for each finger according to a selection gesture using the movement of the user's finger You can select the key as the input value. This process is as shown in FIG. 3.

If the selection gesture is input prior to the inclination of the hand, the selection of the one of the keys as the input value is performed by combining at least two keys for each of the fingers inputted in the selection gesture, And a candidate key assigned to each finger based on a slope of the extracted hand in comparison with a predetermined criterion can be selected as an input value. The feature of this second case is that it can display the assigned combination key only for the selected finger when the user's key selection gesture is preceded and indicates that the input value can be determined using the inclination of the hand for this combination key .

FIG. 10 is a block diagram illustrating a virtual keyboard system 100 based on hand recognition according to an embodiment of the present invention. The operation of each configuration corresponds to each of the processes of FIGS. 3 and 9 described above. To avoid redundancy, we should only outline the outline, focusing on device characteristics.

The detection sensor 10 recognizes the movement of the user's hand and the finger.

The memory 30 stores a program 40 that implements a virtual keyboard.

At least one processor 20 may be included and reads a program 40 implementing the virtual keyboard from the memory 30 to drive a series of commands defined in the program.

Here, the program (40) stored in the memory (30) combines and assigns at least two keys to each finger of the recognized hand, and assigns a key to each of the keys based on the inclination of the extracted hand, And a command for selecting at least one key among the candidate keys presented for each finger according to the selection gesture using the movement of the user's finger.

The program 40 stored in the memory 30 divides a plane parallel to the user's hand into slope sections according to the inclination of the hand and assigns one key as a candidate key for each section . In addition, the keys may be assigned to the user's fingers according to the key arrangement of the QWERTY keyboard for the two-handed hitter, and a portion of the equity keyboard or a keypad may be assigned for the one-handed hitter.

The virtual keyboard system 100 according to an embodiment of the present invention may further include a display unit 50 displaying a combination of keys assigned to the respective fingers, It is preferable that the combination is displayed through the display unit 50 by fixing the virtual keyboard at a predetermined reference position so as to be independent of the movement of the finger.

In addition, the display unit 50 includes a see-through unit for overlapping and displaying a combination of keys allocated on the transparent hand, while guiding the user to visually recognize by transmitting the actual hand of the user. Display means. In this case, the display unit 50 may be utilized as a smart glass in the form of head-up to head-mounted.

In addition, when the gesture using the wrist rotation is sensed through the sensing sensor 10, the program 40 stored in the memory 30 may change the character combination assigned to each finger and newly assign the finger to each finger It is desirable to further include an instruction. At this time, the character combination may include at least two of a language-specific character set, a number set, and a special character set.

According to the embodiments of the present invention described above, it is possible to provide a virtual keyboard in a non-fixed / adaptive form while at the same time fixing a virtual keyboard to an area with less movement, By implementing a one-to-many relationship in which a plurality of key combinations are mapped to a single finger using the keyboard, the sacrifice of space according to the display of the keyboard is minimized, and the rotation of the wrist is used to change the character set assigned to the finger, You can use or enter a set.

Accordingly, the interaction between the conventional fixed virtual keyboard and the non-fixed type / adaptive virtual keyboard, and the fixed / adaptive virtual keyboard proposed by the embodiments of the present invention are as follows. In both cases, the key button input of the virtual keyboard is caused by the movement of the fingertip. However, in the conventional virtual keyboard techniques, since the key button selection method is not designed as an independent interaction with the key button input, there is a disadvantage that the user is required to pay attention to the omnipotent input because the probability of occurrence of the omitting input is very high. In addition, in the case of keyboard combination selection, the conventional virtual keyboard is dependent on the above-mentioned problem of the false input (that is, both the key button selection and the key button input) because it is designed as an interaction for pressing a specific key button. On the contrary, the virtual keyboard proposed by the embodiments of the present invention shows that these problems are solved.

The virtual keyboard proposed by embodiments of the present invention can be implemented using a hand movement in a QWERTY keyboard, which is the most widely used input method used in computers and the like, so that it can be easily and intuitively used by using the existing paradigm. In addition, since it is possible to input while observing the movement of the hand directly through the augmented reality in the display of the smart glasses, it is possible to input at a speed as high as the conventional keyboard input speed according to the accuracy of the hand recognition technology. Furthermore, since it can replace the basic input device of the smart glasses and the HMD or the fixed virtual keyboard, it can be utilized as a base technology for inputting smart glasses and HMD in the future.

Meanwhile, the embodiments of the present invention can be embodied as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device and the like, and also a carrier wave (for example, transmission via the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily deduced by programmers skilled in the art to which the present invention belongs.

The present invention has been described above with reference to various embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Implementation of virtual keyboard
10: detection sensor 20: processor
30: memory 40: program
50:

Claims (20)

A method for implementing a virtual keyboard using at least one processor,
Recognizing a user's hand using a detection sensor;
Assigning at least two keys to each finger of the recognized hand;
Presenting, as candidate keys, any one of a combination of keys for each finger based on a slope of a hand extracted in comparison with a predetermined criterion; And
And selecting at least one key among the candidate keys presented for each finger according to a selection gesture using the movement of the user's finger. The method according to claim 1,
Wherein the step of assigning the two or more keys in combination comprises:
Wherein a plane parallel to the user's hand is divided into slopes according to a slope of the hand, and one key is assigned as a candidate key for each slice. 3. The method of claim 2,
Wherein the number of divided intervals is equal to the number of combinations of keys assigned to the respective fingers. The method according to claim 1,
Wherein the keys are assigned to a user's finger according to a key arrangement of a QWERTY keyboard. The method according to claim 1,
Wherein a combination of keys assigned to the respective fingers is displayed on the display unit by fixing the virtual keyboard with reference to the user's hand or wrist. The method according to claim 1,
The step of extracting the inclination of the hand with respect to the predetermined reference,
Detecting a wrist separating a hand region and a forearm region from the recognized hand and setting a hand axis and a forearm axis respectively from the hand region and the forearm region and detecting the position of the hand and the forearm axis And extracting the slope of the hand based on the correlation. The method according to claim 1,
And changing a character combination assigned to each finger when the gesture using the wrist rotation is detected, and assigning a new finger to each finger. 8. The method of claim 7,
Wherein the character combination includes at least two of a language-specific character set, a number set, and a special character set. 8. The method of claim 7,
Wherein the step of detecting the gesture using the wrist rotation comprises:
Wherein a virtual circle is formed in the hand area extracted from the recognized hand, and a coefficient indicating a degree of the circle shape close to an ellipse is calculated and compared with a threshold value to decide whether to rotate the wrist Implementation method. A method for implementing a virtual keyboard using at least one processor,
Recognizing a user's hand using a detection sensor;
Receiving a selection gesture using a tilt of a hand and a movement of a finger with respect to a plane parallel to the back of the hand using the sensing sensor; And
Selecting one of the at least two keys assigned to the finger of the hand based on the input hand slope and the selection gesture as an input value. 11. The method of claim 10,
Wherein when the inclination of the hand is inputted before the selection gesture, the step of selecting any one of the keys as the input value comprises:
At least two or more keys are assigned to each finger of the recognized hand and a combination of the keys is presented as a candidate key on the basis of the slope of the extracted hand in comparison with a predetermined reference, Wherein at least one key among the candidate keys presented for each finger is selected as an input value according to a selection gesture using motion of the finger. 11. The method of claim 10,
Wherein when the selection gesture is input prior to the inclination of the hand, the step of selecting any one of the keys as the input value comprises:
Wherein each of the candidate gestures is assigned as a candidate key by combining at least two keys with respect to each of the input fingers, and a key selected among the candidate keys assigned to each finger based on a slope of the extracted hand in comparison with a predetermined reference, And selecting the virtual keyboard. A computer-readable recording medium storing a program for causing a computer to execute the method according to any one of claims 1 to 12. A sensing sensor for recognizing movement of a user's hand and a finger;
A memory for storing a program for implementing a virtual keyboard; And
And at least one processor for driving a program implementing the virtual keyboard,
A program stored in the memory,
At least two or more keys are assigned to each finger of the recognized hand and a combination of the keys is presented as a candidate key on the basis of the slope of the extracted hand in comparison with a predetermined reference, And selecting at least one key among the candidate keys presented for each finger according to the selection gesture using the movement of the finger. 15. The method of claim 14,
A program stored in the memory,
And dividing a plane parallel to the user's hand into slope sections according to a slope of the hand, and assigning one key as a candidate key for each section. 15. The method of claim 14,
Wherein the keys are respectively assigned to a user's finger according to a key arrangement of a QWERTY keyboard. 15. The method of claim 14,
And a display unit displaying a combination of keys assigned to the respective fingers,
Wherein a combination of keys assigned to the respective fingers is fixed to a predetermined reference position so as to be independent of the movement of the finger, and is displayed through the display unit. 15. The method of claim 14,
And a display unit displaying a combination of keys assigned to the respective fingers,
The display unit is a see-through display unit for displaying a combination of the keys allocated on the transmitted hand, while inducing the user to visually recognize by transmitting the actual hand of the user. Implementation of a virtual keyboard. 15. The method of claim 14,
A program stored in the memory,
Further comprising instructions for changing a character combination assigned to each of the fingers and allocating a new finger to each of the fingers when the gesture using the wrist rotation is detected through the detection sensor. 20. The method of claim 19,
Wherein the character combination includes at least two of a language-specific character set, a number set, and a special character set.

Images