KR101559502B1 - Method and recording medium for contactless input interface with real-time hand pose recognition - Google Patents

Abstract

The present invention relates to a non-contact type input interface method capable of recognizing a hand pose in real time only through a camera, further aligning a virtual keyboard on a user's hand and clicking a virtual keyboard with a finger to input desired information, .
A non-contact type input interface method for real-time hand pose recognition according to the present invention includes: inputting an image including a hand from a camera; Detecting the outline information of the hand through the depth value differential in the current frame; Calculating rotation angle information of the hand by holding arbitrary two points in the current frame; Extracting center point information of the hand using a distance map in the current frame; Extracting a plurality of point component information corresponding to the fingers of the hand and spaced apart from each other using a circle extending in the current frame with respect to the center point; And recognizing a hand pose of the current frame by performing matching with hand information obtained in a previous frame using hand information including at least the plurality of point component information obtained in the current frame, do.

Description

TECHNICAL FIELD [0001] The present invention relates to a contactless input interface method and a recording medium for real-

The present invention relates to a hand pose recognition method, and more particularly, to a hand pose recognition method which can recognize a hand pose of a user more accurately and quickly in real time, An interface method and a recording medium.

With the development of various sensors, computing power and interface technology, smart interaction between human and digital devices is becoming possible. In particular, the development of wearable computing environment in the glass environment has led to the demand and attention to the next generation interface that will replace the touch screen suitable for the smart phone's wearable terminal environment and gradually researching from the input through the device to the intuitive form recognition . It is necessary to develop a user-friendly interface that can transmit information without constraining user's behavior and audiovisual according to this flow.

There are multitouch interfaces, motion recognition interfaces using sensors, vision-based contactless interfaces, and the like as prior arts for intuitive, user-friendly interfaces that do not require learning. However, when a multi-touch interface or a sensor is used, a separate device is required, which makes it difficult to apply the method to the wearable terminal environment described above. On the other hand, vision - based contactless interface can be applied to various applications such as education, game, and presentation by recognizing the pose or motion of the hand in a non - contact manner out of the hardware such as a sensor.

The prior art using vision-based hand recognition is as follows. Liu Yun et al. Proposed a method of recognizing hand pose using a number of feature information (Multi-Feature) and template matching method (Template Matching). The method comprises the steps of storing the four features (Hu Moment, angle count, skin color angle, and non-skin color angle) information for each hand pose image by the preprocessing, detecting the skin region in the input image, And recognizes the hand pose through template matching and pre-stored feature information. This is advantageous in that it is robust against change of illumination and rotation of the hand, but there is a problem that the more the number of poses to be recognized increases, the longer the matching speed becomes, and the real time recognition becomes difficult.

As another conventional technique, Chris Harrison obtains an image of a divided hand through a differential image of a depth camera, obtains a rough hand position through template matching, and then converts a pixel distance to an actual distance to find a finger Respectively. The above method can be regarded as an input interface suitable for a bar-type terminal in which the position of the end point of a finger is corrected through a Kalman filter. However, it has a limit in that it can not recognize various hand pauses by supporting only an input interface through a finger end point.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above problems, and it is an object of the present invention to provide a handy pose recognition apparatus and a handy pose recognition method which can recognize a hand pose in real time through only a glass and a camera, The present invention also provides a non-contact type input interface method and a recording medium in which a virtual keyboard is matched on a user's hand and a virtual keyboard is clicked with a finger to input desired information.

Yet another object of the present invention is to provide a non-contact type input interface method and a recording medium which can simplify a complicated algorithm compared to the conventional hand pose recognition technology and greatly improve the processing speed thereof,

According to another aspect of the present invention, there is provided a non-contact type input interface for real-time hand pose recognition, comprising: inputting an image including a hand from a camera; Detecting the outline information of the hand through the depth value differential in the current frame; Calculating rotation angle information of the hand by holding arbitrary two points in the current frame; Extracting center point information of the hand using a distance map in the current frame; Extracting a plurality of point component information corresponding to the fingers of the hand and spaced apart from each other using a circle extending in the current frame with respect to the center point; And recognizing a hand pose of the current frame by performing matching with hand information obtained in a previous frame using hand information including at least the plurality of point component information obtained in the current frame, do.

Further, according to an embodiment of the present invention, the method further includes the step of distinguishing the right hand and the left hand of the hand of the input image using the outline information and the Otsu Algorithm. In this case, the plurality of point components may include at least one joint point information corresponding to the joint of the finger of the hand and end point information corresponding to the end of the finger of the hand, Mapping the virtual keyboard to the right or left hand of the hand by using the information and detecting the click using the end point information when the left or right hand is clicked on the virtual keyboard mapped to the right hand or the left hand .

According to the non-contact type input interface method and recording medium according to the present invention, the hand recognition is performed by the hand model-based matching method using the finger joints and the end points. Unlike the conventional art, There is a remarkable effect that it is possible to recognize a variety of hand pose even when there is movement such as a hand movement, a need for initial data learning, a relatively simple algorithm, a faster processing speed, and an improved accuracy (hand pose recognition rate).

Specifically, the average hand pose recognition rate was 95% or more on average, and the accuracy of virtual keyboard input was more than 80%. Also, the speed is about 32fps and it can be processed in real time.

As a result, it is possible to input not only a powerful real-time hand-pose recognition performance but also a virtual keyboard matching without a separate device in a glass environment, so that it can be utilized in various application programs.

1 is a block flow diagram illustrating a process flow of a non-contact input interface method through real-time hand pose recognition in accordance with the present invention.
2 is an example showing a hand outline detected by the image segmentation unit of the present invention.
FIG. 3 shows an example in which the right hand and the left hand are divided according to the present invention.
4 is an example showing line approximation according to the present invention.
5 is an example showing the hand rotation angle calculated according to the present invention.
6 is an example showing the hand center point extracted according to the present invention.
7 is a block flow diagram illustrating a detailed flow of a point component extraction method according to the present invention.
Figs. 8 (a) to 8 (e) are examples showing the processing results of the respective steps in Fig.
9 is a block flow diagram illustrating a detailed flow of a hand pose recognition method in accordance with the present invention.
10 is an example of recognizing a hand pose through matching with a hand model of the present invention.
11 is an example of a virtual keyboard of the present invention mapped on a user's hand.
12 is a diagram for explaining user click detection through the FCD technique of the present invention.
13 is an example of a user interface definition according to various hand pose.
14 is an example of a screen manipulation using a non-contact type input interface method using real-time hand pose recognition of the present invention.
15 is a comparison result of hand pose recognition rate data of the conventional method and the present invention.
16 is a graph of virtual keyboard input success rate experiment data according to the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Furthermore, although the terms first, second, etc. may be used herein to describe various components, the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

In the following, preferred embodiments, advantages and features of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block flow diagram illustrating a process flow of a non-contact input interface method through real-time hand pose recognition in accordance with the present invention.

Referring to FIG. 1, a non-contact type input interface method according to the present invention includes a camera image input step (step 1), a step of detecting an outline information of a hand through a depth value differential (step 2) (Step 4) of extracting the center point information of the hand, a step of extracting the point component information of the finger (step 5), a step of calculating the rotation angle information of the hand And a pose recognition step (step 6).

The image of step 1 includes a hand. The hand image may include both right and left hands. In this case, a step of distinguishing the right hand and the left hand of the hand between steps 2 and 3 (step 2 -1). Further, when the step 2-1 is further included, steps 3 to 6-1 are configured to be performed on at least one of the right hand and the left hand of the hand (particularly, the right hand) or both.

According to the extended embodiment of the present invention, a virtual keyboard is mapped on the basis of the hand information obtained in the above-described steps 2 to 5 (step 7). When the finger is clicked on the virtual keyboard, (Step < RTI ID = 0.0 > 8). ≪ / RTI >

Hereinafter, preferred embodiments according to the respective steps will be described in more detail.

≪ Step 1 (S10)

Step 1 of the present invention is a step in which a camera (preferably a depth camera) captures an image of a subject including a user's hand and inputs the captured image. Therefore, the input image is displayed with the background as well as the hand to be recognized. Further, the hand appearing in the input image may be either one of the hands (left hand or right hand), or both hands.

≪ Step 2 (S20)

In the image input through the camera of step 1, the hand and the background are expressed together as described above. In order to accurately recognize the pose of the hand, it is necessary to separate the hand area and the background.

Step 2 of the present invention is a step of separating the hand region and the background mixed in the input image, and detects the outline information of the hand displayed on the current frame through the depth value differential of the current frame.

More specifically, a depth value is obtained using a Kinect, and a boundary of the hand, that is, an outline 10, is detected through the first derivative using the depth value information of each pixel. 2 is an example showing the hand outline 10 detected by the image segmentation unit of the present invention. As shown in FIG. 2, the image dividing unit can extract only the hand region 10 except for the background by following Step 2 described above.

≪ Step 2-1 (S25) >

Step 2-1 of the present invention distinguishes between the right hand and the left hand of the hand using the outline information obtained in step 1 and the Otsu algorithm.

FIG. 3 shows an example in which the right hand and the left hand are divided according to the present invention. Referring to FIG. 3, the outline information of the hand obtained in step 1 is histogramged, and the inflection point is obtained through the Otsu algorithm. As a result, the hand is divided into the left hand 10a and the right hand 10b ).

Here, the Otsu algorithm is a thresholding algorithm that obtains a threshold value for dividing the two peaks when the frequency distribution of the outline information histogram is distributed in a bimodal form. This is an algorithm that is generally used in image processing, so a detailed description will be omitted.

≪ Step 3 (S30) >

Step 3 of the present invention is a step of calculating rotation angle information of a hand by capturing arbitrary two points in order to obtain hand information.

FIG. 4 is an example showing line approximation according to the present invention, and FIG. 5 is an example showing hand rotation angle calculated according to the present invention.

Referring to FIGS. 4 and 5, step 3 of the present invention performs a random access consensus (RANSAC) algorithm (Algorithm) at two arbitrary points in the current frame of the input image to obtain hand rotation angle Line approximation (line fitting, 20) is performed.

The RANSAC algorithm is a technique that predicts straight lines in an image through repetitive operations of mathematical model parameters from a series of data sets. It estimates model parameters by extracting arbitrary sample data for N times in the input image. the hypothesis process and the original data are well matched to the predictive model, and if it is correct, the validation process is repeated to obtain a new model from the set.

Assuming that n samples of arbitrary sample data are 'P g ' and 'P fail ', the probability that the data will have a straight line observation form is 'P g ' and the probability that the L model prediction attempts will fail is 'P fail ' , The maximum number of iterations that affects the performance is expressed by Equation 1 below.

4, the line approximation can be performed on various hand-pose images, and a plurality of straight lines 20 approximated through the random algorithm can be used as a basis (i.e., It is possible to calculate the rotational angle? 1 of the hand as shown in FIG.

≪ Step 4 (S40)

Step 4 of the present invention is a step of extracting center point information of a hand using a distance map (distance conversion, Distance Map) to obtain hand information.

6 is an example showing the hand center point extracted according to the present invention. Referring to FIG. 6, in step 4, first, an eight-way search is performed on each pixel in the hand region to generate a distance map storing the shortest distance value at the corresponding pixel position. Then, a pixel having the largest value among the values stored in the distance map is extracted as the central point 30 of the hand.

≪ Step 5 (S50)

Step 5 of the present invention is a step of extracting a plurality of point components corresponding to specific points of a finger of a hand using an expanding circle, and is performed by a hand information extracting unit.

FIG. 7 is a block flow chart showing a detailed flow of the point component extraction method according to the present invention, and FIGS. 8 (a) to 8 (e) are examples showing results of processing in each step of FIG.

7 and 8, a point component extraction method according to the present invention includes a step of extending a circle at a hand center point 30 (step 5a; S51), a hand boundary intersection point search and intermediate points detection step A finger joint point and an end point extraction step (step S5e; S55) corresponding to the finger, a midpoint extraction step (step S5d) corresponding to the finger, .

Step 5a (S51) is finally a preparatory step for extracting the finger joint point and end point information. More specifically, the large circle 40 is gradually expanded on the basis of the central point 30 of the hand extracted in step 4 Lt; / RTI > The plurality of circles 40 extended at this time are formed into a plurality of concentric circles based on the central point 30 of the hand as shown in FIG. 8A, and it is preferable that the circles 40 are formed so as to form a constant interval from each other Do.

Step 5b (S52) finds the intersections of the black pixels corresponding to the white and the out-of-hand regions corresponding to the hands while drawing the concentric circle 40 as in step 5a. Then, the coordinates of the intersecting points are determined as the boundary candidates of the finger, and they are connected to each other, and the intermediate points (50) of the connecting lines are obtained.

Here, the intersecting points to be interconnected are two pairs of intersecting points, and two pairs of intersecting points are located on the same circle among a plurality of concentric circles, and two closest points to each other can be interconnected.

8 (b) shows an example of the intermediate point 50 of the intersection connecting line detected in step 5b. As seen in FIG. 8 (b), the intermediate points 50 obtained by FIG. 5 (b) can be generated in the wrist or palm area in addition to the fingers. Therefore, in order to extract the point component 60 corresponding to the finger, it is necessary to remove the middle point of the wrist or palm area, which may occur in the step 5b.

Step 5c (S53) is a step of removing the midpoints 53 and 52 of the aforementioned wrist or palm region. According to a preferred embodiment, step 5c calculates the angle between each intermediate point 50 and the center point 30 of the hand on the basis of the rotation angle? 1 of the hand calculated in step 3, The point 53 is removed. 8 (c) shows a state in which the middle point 53 of the wrist area is removed by the above-described method.

Step 5d (S54) is a step of removing the non-removed finger region (e.g., palm area) in step 5c through clustering for classification of the intermediate point 51 corresponding to the finger area. According to a preferred embodiment, step 5d is a step 5c in which the intermediate point of the wrist area is removed through step 5c, and the angle between the respective intermediate points 51, 52 and the hand center point is within the respective finger critical range angles (i.e., Middle point) clustering among them. At this time, it is judged that the portion having a relatively short length is not a finger, and is removed.

After the steps 5c and 5d described above, the intermediate points 53 and 52 corresponding to the wrist or the palm area are finally removed as shown in Fig. 8 (d), and only the intermediate point 51 corresponding to the finger area is left .

In the above state, the intermediate points corresponding to the ends of the fingers are classified as the end points 61, and the joint points 62 corresponding to the joints are calculated using the remaining intermediate points (step S 5). In this case, the normal user has three joints per finger, and the end points and the respective joints are arranged at substantially the same spacing ratio, so that each joint point 62 can be extracted using the above fact. FIG. 8 (e) shows one end point 61 and three joint points 62 finally extracted by the above-described method, and the extracted end points and joint points correspond to the plurality of point components 60 ).

≪ Step 6 (S60)

Step 6 of the present invention recognizes the hand pose of the current frame by performing matching with the hand information obtained in the previous frame using the hand information obtained in the current frame.

Here, the hand information obtained in the current frame includes information on the outline 10 of the hand, the rotation angle? 1 of the hand, and information on a plurality of point components 60 obtained in steps 2 to 5, The plurality of point component (60) information includes joint point (62) and end point (61) information extracted in step (5).

FIG. 9 is a block flow diagram illustrating a detailed flow of a hand pose recognition method according to the present invention. Referring to FIG. 9, step 6 for recognizing hand pose of the present invention includes a hand model normalization step (S15 in FIG. 1), a hand model rotation step (step 6a; S61) (Step S6), a hand pose recognition step (step S6), and a hand model updating step (step S6e, step S65) of matching the hand model and the current frame hand information .

The non-contact type input interface method of real-time hand pose recognition according to the present invention is characterized in that hand displayed in the first frame of the input image is first hand model information (hereinafter referred to as 'initial hand model') for hand pose recognition Is used.

The term " hand model " used in the present invention refers to a matching object (i.e., matching frame) used to recognize a hand pose of a current frame, and preferably uses a frame immediately before the current frame.

Therefore, the first frame is subjected to a preprocessing operation for a subsequent frame. Specifically, the user follows the method of steps 2 to 5 described above in a state in which all five fingers are straightened, so that information on the outline 10 of the hand, The rotation angle? 1 information and the plurality of point component 60 information. The hand information for the first frame is used as the initial hand model (the hand model normalization step S15)

Thus, the hand model normalization step may be performed prior to performing each step for hand pose recognition (or virtual keyboard input) for the frame after the first frame, i.e., step 2 to step 6 (or step 2 to step 8) 2 to 5 are followed first.

When the initial hand model information is generated through the above process, the hand model generated in the previous frame and the hand information obtained in the current frame (i.e., joint point information, end point information, hand center point information, The distance between the two hands, and the rotation angle information of the hand), thereby recognizing the hand pose.

Accordingly, the first frame corresponds to the previous frame (i.e., the hand model) for the second frame, the previous frame (i.e., the hand model) for the third frame corresponds to the second frame, and N (N-1) th frame corresponds to the previous frame (i.e., the hand model) of the (n-1) th frame.

On the other hand, the size or rotation angle of the divided hand in the input image can be changed flexibly for each frame before matching with the hand model. In order to solve such a problem, the present invention normalizes the hand model and then rotates the hand model so as to coincide with the rotation angle of the hand of the current frame, and then transforms the hand model to fit the hand size of the current frame. Hand model size conversion step S62)

The matching is performed by comparing the angular difference between the hand model of the converted previous frame and each finger joint point and the end point of the hand obtained from the current frame and the center point of the hand (matching step S63 of the hand model and the current frame hand information) )

Then, when the hand model matching operation and the hand pose recognition according to the above are completed (hand pose recognition step S64), the used hand model is newly updated. That is, it is configured to update the hand information obtained in the current frame with new hand model information for hand pose recognition (i.e., matching with the hand model) for the next frame (hand model update step S65).

10 is an example of recognizing a hand pose through matching with a hand model of the present invention. As can be seen from FIG. 10, it is difficult to know exactly which finger the finger is when it is hidden when the finger is hidden or when it is not a thumb or small finger. By using the method according to the present invention, Recognition becomes possible.

≪ Step 7 (S70)

Step 7 of the present invention is a step of mapping a virtual keyboard based on the hand information obtained in the above-described steps 2 to 5. [

11 is an example of a virtual keyboard of the present invention mapped on a user's hand. Referring to FIG. 11, in order to increase the accuracy and immersion of the touch, the user can use the information of the point component 60 (in particular, the joint point 62) of the finger obtained by using the above- And the rotation angle of the palm is calculated in real time using the rotation angle of the hand (? 1) obtained through the above-described method, the virtual keyboard 70 is accurately mapped to the hand 1 as shown in FIG. 11 .

≪ Step 8 (S80)

Step 8 of the present invention is a step of detecting the click when the finger touches or closes (hereinafter, referred to as "click") the virtual keyboard 70 mapped to the user's hand 1 by step 7.

According to a preferred embodiment, step 8 is a technique for detecting when the fingertip of one hand of the user (e.g., the right hand) hits the palm of another hand (e.g., the left hand) Click Detection (FCD).

12 is a diagram for explaining user click detection through the FCD technique of the present invention. Referring to FIG. 12, the FCD method for user click detection of the present invention is configured to use the depth value 'Dp' of the finger endpoints 'p (61)' and 'p (61)' as shown in FIG.

More specifically, the depth value Di of the sixteen pixels (i1 to i16) around the fingertip) is compared with a value obtained by adding or subtracting the threshold value t to Dp to determine the number of True values as n , And when it satisfies 'n> = 12', it is configured to detect by clicking, otherwise it is not detected by click. This can be expressed by the following equation (2).

Meanwhile, when the present invention further includes steps 7 and 8 as described above, it is possible to detect a click of the virtual keyboard 70 based on recognizing the hand pose of the user in real time. In this case, Invoking a virtual keyboard 70 of a certain type for input.

The virtual keyboard calling step may be configured to be performed before any one of the first to sixth steps, and preferably may be set to be performed after step 3.

Further, the virtual keyboard 70 may be configured to be automatically called after completion of the set specific step or before the start, or may be configured to be called by the user's command (e.g., button operation, etc.).

Hereinafter, a specific embodiment of a non-contact type input method through real-time hand pose recognition according to the present invention will be described.

Example 1 was performed in an Intel Core I5 2500 processor, 4GB RAM, VS 2005 environment, OpenCV and OpenNI were used, and the camera used MS Kinect. The input image size of the camera used is 640 x 480.

In order to experiment with the method proposed in the present invention, an image viewer program command control through a hand pose and a key input experiment through a virtual keyboard input interface were performed, and the input image processing was performed in the respective steps described above.

First, to determine the accuracy and usability of hand pose recognition, we defined a hand-based user interface for controlling the image viewer program by hand motion defined as a series of hand pauses as shown in Fig.

14 is a result of applying the hand-based user interface defined above by implementing the multimedia viewer program. The user can freely control functions such as left / right mouse click, zooming, and image flipping using both hands.

Particularly, since the method proposed in the present invention can distinguish the fingers of each hand, it can be determined that the pose of the other hand is different even if the number of fingers of the hand pose is the same, there was.

In order to prove the remarkability of the effect of the present invention, it is compared with the conventional hand pose recognition method. The conventional hand pose recognition method, which is a comparison method, stores information of four features (Hu Moment, angle count, skin color angle, and non-skin color angle) for each hand pose image and then recognizes hand pose do.

As a result of three randomly performing pauses defined in FIG. 15 each time by the pauses defined in FIG. 15, there is a matching error between similar pauses in the conventional hand pose recognition method, whereas the hand pose recognition method according to the first embodiment of the present invention, It is found that the recognition accuracy is high because the hand model is newly updated through the acquired feature hand information and used for matching in the next frame.

On the other hand, in the case of the hand pose recognition speed, the conventional hand pose recognition method requires a preprocessing time of about 15 minutes and an average of 15 fps, whereas the hand pose recognition method according to the present invention requires no prior learning , And the average number of feature points required for hand pose recognition was small, resulting in an average of 32 fps. As a result, it was found that the processing time for hand pose recognition can be significantly reduced compared with the conventional art.

In order to test the accuracy and usability of the virtual keyboard input interface, we used a virtual keyboard with one hand and randomly clicked numbers 1 through 9 on the other hand. The test results are shown in FIG. As shown in FIG. 16, according to the virtual keyboard input method of the present invention, a high input success rate of 80% or more can be realized.

The non-contact type input interface method through real-time hand pose recognition of the present invention described and illustrated above can be realized in the form of a program and realized through an electronic device. Here, the electronic device is generically referred to as an electric / electronic information terminal including a central processing unit and a memory such as a general personal computer, a wearable computer, and a smart terminal.

In the above case, the recording medium of the present invention includes a function of inputting an image including an image of a hand picked up by a camera to an electronic apparatus; Detecting information of an outline (10) of the hand through a depth value differential in a current frame; A function of calculating rotation angle information (? 1) of the hand by holding arbitrary two points in the current frame; Extracting a center point (30) of the hand using a distance map of the hand in the current frame; Extracting a plurality of point component (60) information corresponding to the fingers of the hand and spaced apart from each other by using a circle (40) extending in the current frame with respect to the center point (30); And recognizing a hand pose of the current frame by performing matching with hand information obtained in a previous frame using hand information including at least the plurality of point components (60) information obtained in the current frame And a program for recording the program.

Preferably, the recording medium of the present invention is a medium that can be read by an electronic device that records a program for further realizing a function of distinguishing the right hand and the left hand of an input image by using outline information and an Otsu Algorithm .

According to the extended embodiment of the present invention, the recording medium of the present invention is characterized in that the plurality of point components include at least one joint point (62) information corresponding to each joint of the hand, And a program for realizing a function of mapping a virtual keyboard to the right hand (or left hand) of the hand using information.

In the recording medium of the present invention, the plurality of dot components include information on the end point 61 corresponding to the end of the finger of the hand, and the virtual keyboard 70 mapped to the right hand (or left hand) Or a right-handed user), when the user clicks the end point 61, detects the click or not by using the end point 61 information.

The recording medium of the present invention compares a value obtained by adding a threshold value to a depth value of a plurality of pixels around the finger end point 61 clicked on the virtual keyboard 70 and a depth value of the finger end point And may be a medium that can be read by an electronic device in which a program for realizing the function of detecting click or not is recorded.

The recording medium of the present invention may be a medium readable by an electronic device that records a program for further realizing a function of calling a virtual keyboard to the outside when a user wishes to input a keyboard.

While the preferred embodiments of the present invention have been described and illustrated above using specific terms, such terms are used only for the purpose of clarifying the invention, and it is to be understood that the embodiment It will be obvious that various changes and modifications can be made without departing from the spirit and scope of the invention. Such modified embodiments should not be understood individually from the spirit and scope of the present invention, but should be regarded as being within the scope of the claims of the present invention.

10: hand outline 20: approximated straight line
30: center of hand 40:
51: midpoint of the finger region 52: midpoint of the palm region
53: middle point of wrist area 61: finger end point
62: finger joint point 70: virtual keyboard

Claims (20)

delete delete delete delete A first step of receiving a hand image including a left hand or a right hand from a camera;
A second step of detecting the outline information of the hand through the depth value differential in the current frame;
A step 2-1 of discriminating whether the hand is a right hand or a left hand using the outline information and the Otsu algorithm;
A third step of calculating rotation angle information of the hand by holding arbitrary two points in the current frame;
A fourth step of extracting center point information of the hand using the distance map of the hand in the current frame;
Extracting a plurality of point component information corresponding to the fingers of the hand and spaced apart from each other using a circle extending in the current frame with respect to the center point, A fifth step of extracting at least one corresponding joint point information and the plurality of point components so as to include end point information corresponding to an end of a finger of the hand;
A sixth step of recognizing a hand pose of the current frame by performing matching with hand information obtained in a previous frame using hand information including at least the plurality of point component information obtained in the current frame;
A seventh step of mapping a virtual keyboard to a right hand (or a left hand) of the hand using the hand information including the joint point information; And
And an eighth step of detecting whether the virtual keyboard is mapped to the right hand (or left hand) by using the end point information when the left hand (or right hand) is clicked.
In the eighth step,
Wherein the real time hand pose recognition is performed by comparing a value obtained by adding a threshold value to a depth value of a plurality of pixels around a finger end point clicked on the virtual keyboard and a depth value of the finger end point, ≪ / RTI >
delete 6. The method of claim 5,
Further comprising the step of calling a virtual keyboard prior to any one of the first to sixth steps. ≪ RTI ID = 0.0 > 11. < / RTI >
6. The method of claim 5,
Wherein the step 2-1 is configured to distinguish the right hand and the left hand of the hand by obtaining an inflection point through the Otsu algorithm after histogramizing the outline information. Contactless input interface method.
6. The method of claim 5,
Wherein the third step is configured to calculate rotation angle information of the hand using line fitting through a RANSAC algorithm. ≪ RTI ID = 0.0 > 11. < / RTI >
6. The method of claim 5,
And the fourth step is configured to extract a pixel having the largest value among the values stored in the distance map as the center point of the hand.
6. The method of claim 5,
In the fifth step,
5a) extending the circle at the center point of the hand;
(B) detecting midpoints by searching a hand boundary crossing point using the expanding circle;
A fifth step (c) of removing an intermediate point corresponding to the wrist of the hand;
A fifth step of extracting an intermediate point corresponding to the finger of the hand; And
And a fifth step of extracting the plurality of point component information from the intermediate point extracted in the step (5d)
The plurality of point component information extracted in the step 5e includes at least one joint point information corresponding to the joint of the finger of the hand and one end point information corresponding to the end of the finger of the hand respectively A non - contact input interface method through real - time hand pose recognition.
6. The method of claim 5,
In the sixth step,
Rotating the hand of the previous frame so that the hand of the previous frame coincides with the hand of the current frame;
Transforming the size of the hand of the previous frame so that the hand of the previous frame coincides with the hand of the current frame;
Comparing the hand information of the hand of the previous frame converted through the steps (6a) and (6b) with the hand information obtained in the current frame, and performing matching; And
And recognizing a hand pose of the current frame through matching in the step (6c). ≪ Desc / Clms Page number 19 >
6. The method of claim 5,
The first frame of the input image in the first step is used as the first hand information for hand pose recognition of the second frame by performing the second step to the fifth step in a state in which all five fingers of the hand are spread, A non-contact type input interface method through real time hand pose recognition.
6. The method of claim 5,
Wherein the hand information obtained in the current frame is transferred to the hand information acquired in the previous frame upon hand pose recognition in the next frame.
delete delete delete delete In electronic devices,
A function of receiving an image including a hand picked up from a camera;
Detecting the outline information of the hand through the depth value differential in the current frame;
A function of distinguishing the right hand and the left hand of the hand using the outline information and the Otsu algorithm;
A function of calculating rotation angle information of the hand by holding arbitrary two points in the current frame;
A function of extracting the center point information of the hand using the distance map of the hand in the current frame;
Extracting a plurality of point component information corresponding to the fingers of the hand and spaced apart from each other using a circle extending in the current frame with respect to the center point, Extracting a plurality of point component information such that at least one corresponding joint point information and at least one end point information corresponding to an end of a finger of the hand are included in the plurality of point components;
Recognizing a hand pose of the current frame by performing matching with hand information obtained in a previous frame using hand information including at least the plurality of point component information obtained in the current frame;
A function of mapping a virtual keyboard to the right hand (or left hand) of the hand using the hand information including the joint point information;
A function of detecting whether the virtual keyboard is clicked using the joint point information or the end point information when a left hand (or right hand) is clicked on the virtual keyboard mapped to the right hand (or left hand); And
An electronic device in which a program for realizing a function of detecting the click or not by comparing a value obtained by adding a threshold value to a depth value of a plurality of pixels around a finger end point clicked on the virtual keyboard and a depth value of the finger end point Readable medium.
20. The method of claim 19,
And a program for realizing the function of calling the virtual keyboard.

Images