KR20150040580A - virtual multi-touch interaction apparatus and method - Google Patents

Abstract

The suggested are a virtual space multi-touch interaction device capable of recognizing a moment when the finger of a user touches a virtual object visualized through a wearable display which can provide cubic images and providing touch feedback as well as visual interactions for the user and a method thereof. The suggested device includes: a visualization unit which outputs a virtual image; a virtual space multi-touch recognition unit which recognizes multi-touches through detection of finger tips and movements of the user for the virtual image visualized by the visualization unit; and a touch feedback providing unit which provides touch feedback corresponding multi-touches of the user recognized by the virtual space multi-touch recognition unit.

Description

[0001] The present invention relates to a virtual multi-touch interaction apparatus and method,

The present invention relates to a virtual space multi-touch interaction apparatus and method, and more particularly, to a virtual space multi-touch interaction apparatus and method that enable realistic multi-touch interaction with a wearable display visualization object.

Input devices such as conventional remote controls, keyboards, and mice require additional devices or are flat and non-intuitive for the user to control the menu.

The existing gesture-based interface provides an intuitive control operation to solve this problem, but the user can not feel precisely whether or not the contact with a virtual object occurs.

In addition, the user interacts with the content through visual feedback based on whether or not the user touches the touch screen.

Relevant prior art teaches a three-dimensional virtual touch HMI system and method of operation that includes a three-dimensional TOF sensor, a three-dimensional non-eyeglass display, and a computer coupled to the sensor and the display are described in U. S. Patent Publication No. 2012-0306795 METHOD AND SYSTEM FOR THREE-DIMENSIONAL VIRTUAL-TOUCH INTERFACE).

In US Patent Publication No. 2012-0306795, a three-dimensional TOF sensor recognizes a user object in a three-dimensional sensor space, a three-dimensional non-eyeglass display visualizes an image in a three-dimensional display space, Map the location.

The above-described invention of U. S. Patent Publication No. < RTI ID = 0.0 > 2012-0306795 < / RTI > does not provide tactile feedback to a user's virtual touch.

In another prior art, a three-dimensional coordinate image input from a three-dimensional scanner is used to recognize a body part, and then a virtual touch screen can be generated through a user's gesture recognition. In a virtual touch screen, Korean Patent No. 1242848 (a device for generating and controlling a virtual touch screen) has been disclosed in which a monitor screen can be easily controlled by the movement of a cursor displayed on a monitor screen under the control of the controller.

The invention disclosed in Korean Patent No. 1242848 is based on the fact that the gesture recognition error due to the operation of the gesture unclear in the conventional gesture recognition and the unnatural activity due to the control of the gesture recognition in a free place where a large number of users such as the living room are present And it is possible to more easily control the monitor screen using the gesture.

However, in the above-described invention disclosed in Korean Patent No. 1242848, there is no description of the fingertip detection method, and there is no description of tactile feedback for the virtual touch of the user.

As another prior art, a driving operation of an electronic device using a virtual touch through a user's movement (gesture) at a remote place is applied to all electronic devices driven by a conventional remote controller, regardless of the presence or absence of a display device, The contents that can be made are disclosed in Korean Patent No. 1235432 (remote control apparatus and method using virtual touch of three-dimensionally modeled electronic device).

The above-mentioned Korean Patent No. 1235432 does not provide tactile feedback for the virtual touch of a user.

Another prior art is to detect and track the bare hands of a user without special equipment based on the color information and the depth information and to recognize the number of fingers by detecting the end point of the finger, Korean Patent Publication No. 2012-0134488 (Gesture recognition-based user interaction method and device therefor) discloses a technique for enabling a user to recognize a hand gesture and building a natural user interface.

Although the above-described Korean Patent Laid-Open Publication No. 2012-0134488 allows finger end point detection and gesture recognition, it is impossible to recognize whether the virtual space is touched or not, and does not provide tactile feedback on the virtual touch of the user.

Disclosure of Invention Technical Problem [8] Accordingly, the present invention has been made to solve the above-mentioned problems occurring in the prior art, and it is an object of the present invention to recognize a moment when a virtual object and a user's finger are touched through a wearable display capable of providing a stereoscopic image, The present invention provides a virtual space multi-touch interaction apparatus and method capable of providing feedback together.

According to an aspect of the present invention, there is provided a virtual space multi-touch interaction apparatus including: a visualization unit for outputting virtual images; A virtual space multi-touch recognition unit for recognizing a multi-touch through a user's fingertip detection and motion detection of a virtual image visualized through the visualization unit; And a touch feedback providing unit for providing touch feedback corresponding to the multi-touch of the user recognized by the virtual space multi-touch recognizing unit.

Preferably, the virtual space multi-touch recognition unit includes: a color image / depth information acquisition unit for acquiring a color image and depth information as the user photographs; A body joint tracker for tracking the rotation and movement of the user's body joint; A hand region detecting unit for detecting a hand region of the user from the color image and the depth information; A finger end point information detecting unit for detecting a finger end point from a calculated curvature value of a pixel existing on a hand outline of a hand region detected by the hand region detecting unit; A virtual touch recognition unit for recognizing whether the user has touched the virtual image based on information from the finger end point information detection unit; And a multi-touch gesture recognition unit for recognizing a gesture of the user corresponding to the touch point on which the user has touched the virtual image based on the result of the body joint tracking unit and the result of the virtual touch recognition unit .

Preferably, the hand region detection unit detects a hand region based on the tracking result value acquired by the body joint tracking unit with respect to the color image and the depth information, It is possible to collect the pixels having the depth value smaller than the threshold value among the pixels within the size range to detect the hand area.

Preferably, the hand region detection unit detects a hand region based on a tracking result value obtained by the body joint tracking unit with respect to the color image and depth information, and detects a face in the color image, Based on the information, it can be detected using the prior knowledge about the three-dimensional position of the face and the hand.

Preferably, the fingertip information detecting unit calculates a direction vector between each pixel separated in both directions in each pixel existing along the hand contour of the detected hand region, and when the angle of the two direction vectors becomes equal to or more than the threshold value, And sets a region of interest around the coordinates of the recognized finger end point. The depth value average of the hand region excluding the background in the region of interest is obtained from the depth map obtained by the color image / depth information obtaining unit It can correspond to the depth information of the finger end point.

Preferably, the multi-touch gesture recognition unit may use a touch duration, a motion tolerance, a motion moment vector, a distance between touch points, and a midpoint slope between touch points as recognition parameters.

Preferably, the virtual space multi-touch recognizing unit may further include a plurality of user processing units for selecting a user located at the forefront among the plurality of users as a main user based on the color image and the depth information.

Preferably, the visualization unit may be configured as a face wearable stereoscopic display.

Preferably, the touch feedback providing unit may provide tactile feedback to the user through wireless communication.

Meanwhile, in the virtual space multi-touch interaction method according to the preferred embodiment of the present invention, the virtual space multi-touch recognition unit recognizes the multi-touch through the finger end point detection and motion detection of the virtual image visualized through the visualization unit ; And providing the touch feedback providing unit with touch feedback corresponding to the multi-touch of the user recognized by recognizing the multi-touch of the user.

According to the present invention having such a configuration, it is possible to perform a touch while viewing a virtual object of a non-2D stereoscopic image displayed through a stereoscopic display with a bare hand without having an additional device for user input.

In addition, by receiving tactile feedback along with a touch through a tactile feedback device, it is possible to freely and freely touch a space much wider than touching the screen of the 2D display.

1 is a view illustrating an example of a wearable display visualization apparatus implementing a virtual space multi-touch interaction apparatus according to an embodiment of the present invention.
2 is a diagram illustrating a concept of a virtual space touch according to an embodiment of the present invention.
3 is a configuration diagram of a virtual space multi-touch interaction apparatus according to an embodiment of the present invention.
FIG. 4 is a diagram for explaining an operation of the touch pad bag providing unit shown in FIG.
FIG. 5 is an internal configuration diagram of the virtual space multi-touch recognition unit shown in FIG.
FIG. 6 is a view illustrating joint positions applied to the body joint tracking in the body joint tracker shown in FIG. 5. FIG.
7 is a diagram for explaining the operation of the fingertip information detecting unit shown in Fig.
FIG. 8 is a flowchart illustrating a virtual space multi-touch interaction method according to an embodiment of the present invention.
FIG. 9 is a flowchart detailing the virtual space multi-touch recognition process shown in FIG.
FIG. 10 is a flowchart detailing the multi-touch gesture recognition process shown in FIG.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

FIG. 1 is a view showing an example of a wearable display visualization apparatus implementing a virtual space multi-touch interaction apparatus according to an embodiment of the present invention, and FIG. 2 is a view showing a concept of a virtual space touch according to an embodiment of the present invention .

In order to realize the virtual space multi-touch interaction proposed by the present invention, it is necessary for a user to synthesize and present virtual contents in a situation where his / her body is visible. For this purpose, the wearable display visualization device proposes a wearable display of an optical see-through, which has an intuitive sense of distance and a small delay effect due to additional image processing among visualization techniques that output mixed reality images and virtual images. .

That is, when the stereoscopic image content is visualized in the user's visual space through the optical system as shown in FIG. 1, the user naturally touches the virtual content (for example, a three-dimensional menu) Interaction becomes possible. In the wearable display visualization apparatus of FIG. 1, the display units 1a and 1b may be components of the wearable display visualization unit 10 of the virtual space multi-touch interaction apparatus according to the embodiment of the present invention. In FIG. 1, reference numeral 2 denotes a camera for tracking the user's head and hands, and reference numeral 3 denotes a sensor marker for camera tracking.

When the virtual space multi-touch interaction is implemented by applying the wearable display, the position of the virtual contents arranged around the user can be always arranged in a fixed three-dimensional space based on the user area, or independently of the user It may be fixed to an absolute position in the background space. In the present invention, the latter method is adopted. In order to implement the latter method, the user's gaze movement must be tracked in real time and reflected in the control of the virtual content. In order to track eye movement in real time, it is possible to apply head tracking technology additionally. Typically, we use a camera-based image sensor and a gyro / accelerometer / electronic compass to track 6 degrees of freedom (6 degrees of freedom, position x / y / z, attitude pitch / yaw / roll) The immersion feeling can be improved.

The wearable display visualization device illustrated in FIG. 1 is an embodiment in which the present invention is implemented. The wearable display visualization device (i.e., eyeglass-type display (EGD 2.0) supporting optical vision shown in FIG. 1) utilizes an OLED panel of high resolution (SXGA class, 1280x1024) In a binocular manner. In addition, the wearable display visualization device illustrated in FIG. 1 supports a high-resolution video output module (not shown) and a viewing angle of about 41 degrees in a wearing portion of about 150 g, and supports two USB cameras 2, And a tracking sensor (not shown).

When the user wears the wearable display visualizing device illustrated in FIG. 1 on his / her face, the user can experience a stereoscopic image that is visible in the surrounding space as shown in FIG.

FIG. 3 is a block diagram of a virtual space multi-touch interaction apparatus according to an embodiment of the present invention, and FIG. 4 is a diagram for explaining an operation of the touch pad bag providing unit shown in FIG.

The virtual space multi-touch interaction apparatus according to an embodiment of the present invention includes a wearable display visualization unit 10, a virtual space multi-touch recognition unit 20, and a touch feedback unit 30. The wearable display visualization unit 10, the virtual space multi-touch recognition unit 20, and the touch feedback unit 30 may be installed in the wearable display visualization apparatus illustrated in FIG.

The wearable display visualization unit 10 can mix the real image and the virtual image and output the mixed image. The wearable display visualization unit 10 includes display units 1a and 1b, a video output module (not shown), a USB camera 2, and a head tracking sensor (not shown) of the wearable display visualization apparatus of FIG. . The wearable display visualization unit 10 may be an example of the visualization unit described in the claims of the present invention.

The virtual space multi-touch recognition unit 20 recognizes a user's multi-touch on a virtual image that is visualized through the wearable display visualization unit 10. [ The virtual space multi-touch recognition unit 20 controls and interacts with the virtual images visualized through the wearable display visualization unit 10. [ The internal configuration of the virtual space multi-touch recognition unit 20 will be described later.

The touch feedback providing unit 30 provides touch feedback corresponding to the multi-touch of the user recognized by the virtual space multi-touch recognizing unit 20 to the user. In other words, in order to provide touch feedback according to the recognition result finally obtained by the virtual space multi-touch recognition unit 20, the touch feedback providing unit 30 causes the finger to touch the virtual image (more specifically, The touch feedback device provides a feeling of touching the actual plane. One embodiment of the touch feedback device may be a device 30a (see FIG. 4) such as a smart device-based watch worn on the user's wrist. That is, the touch feedback providing unit 30 uses the device 30a to provide touch feedback through an effect such as vibration.

More specifically, the touch feedback providing unit 30 may use the following method of providing a touch feedback. That is, when the touch feedback providing unit 30 receives the result of the virtual plane and the finger end point intersection from the virtual space multi-touch recognizing unit 20, the touch feedback providing unit 30 transmits the touch widget to the touch widget through the wireless transmission protocol such as Bluetooth, A right hand, a finger penetration depth into a virtual plane, and a multi-touch gesture type.

In the touch widget running on the smart device watch, the received left and right hand information are separately distinguished and fed back to the user. The intensity of the vibration is proportional to the depth of penetration, so that the user can feel the touch of the virtual plane with the intensity of the touch by the degree of intensity. Also, depending on the type of the multi-touch gesture and each finger type, it is possible to provide a variety of touch effects by feeding different vibration patterns such as frequency.

FIG. 5 is an internal configuration view of the virtual space multi-touch recognition unit 20 shown in FIG. 3, and FIG. 6 is a view showing an example of a joint position applied when tracking a body joint in the body joint tracking unit 22 shown in FIG. And FIG. 7 is a diagram used for explaining the operation of the finger endpoint information detecting unit 25 shown in FIG.

The virtual space multi-touch recognition unit 20 includes a color image / depth information acquisition unit 21, a body joint tracking unit 22, a multiple user processing unit 23, a hand area detection unit 24, a finger end point information detection unit 25, A virtual touch recognition unit 26, and a multi-touch gesture recognition unit 27. [

The color image / depth information obtaining unit 21 obtains the color image and depth information for the user according to the photographing of the user. In other words, the color / depth information acquiring unit 21 utilizing the color image / depth information acquiring apparatus (not shown) includes a color image / depth information acquiring apparatus attached at an arbitrary position where the user's upper body can be sufficiently obtained The color image and the depth information of the user can be obtained. In one embodiment of the present invention, Kinect, a Microsoft company, was used. By using this equipment, not only the depth information but also corresponding color images can be acquired at the same time. At this time, a calibrated value of color image and depth information is provided in real time through a fixed relative positional relationship between a CMOS sensor (not shown) for acquiring a color image and a depth information acquiring module (not shown). Here, although not shown, the CMOS sensor and the depth information acquisition module are understood to be installed in a Kinect.

The body joint tracking unit 22 tracks the rotation and movement of the user's body joint based on the color image and the depth information from the color image / depth information obtaining unit 21. [ In other words, the body joint tracker 22 can simultaneously acquire the color image and the depth information of the user in front of the keynote using the Kinect and the Kinect SDK or the OpenNI library. In addition, the body joint tracking unit 22 can receive rotation and position motion information of the user's whole body joint in real time. For example, as illustrated in FIG. 6, the body joint tracker 22 can obtain the positions and the rotation values of 15 joints such as head, neck, shoulder, elbow, and knee at a speed of 30 fps.

The multi-user processing unit 23 selects the most forward user among the plurality of users as the main user based on the color image and the depth information. In other words, the multi-user processing unit 23 selects the most forward user among the plurality of users detected based on the user detection information (for example, color image and depth information) provided by the Kinect SDK or OpenNI as a main user, Or interfere with the gesture can be blocked in advance. The multiple user processing unit 23 obtains the torso value of the body joint tracking result in the region detected by the user and selects the user having the smallest depth value as the main user. If there are two or more users who input a gesture, the gesture of the frontmost user is selected to perform the multi-touch interaction.

The hand region detection unit 24 detects the hand region of the user based on the color image and the depth information. In other words, the hand region detection unit 24 detects the position of the user's hand from the color image and depth information every frame and tracks both hands of the user. In an embodiment of the present invention, a hand region is detected on the basis of a user's whole body tracking result value acquired by the body joint tracking unit 22. That is, the hand region detecting unit 24 collects pixels having a depth value smaller than the threshold value among the pixels within a predetermined size region centering on the hand coordinates tracked in the depth image (i.e., the image including the depth information) Area.

In another embodiment, the hand region detection unit 24 detects a face in a color image, and then detects the face using the prior knowledge of the three-dimensional position of the face and the hand based on the three-dimensional position information of the face have. That is, when the hand is detected after detecting the face, the distance between the hand and the face is set to the second threshold value when the hand reaches the first threshold value from the face forward to the maximum, and the object positioned in this area is selected as the hand candidate . Connected data is connected using the connected component method based on this candidate group. Finally, the two largest components in the region are assumed to be hands. At this time, in order to minimize the arm region in the detected region, only the region below the threshold value is detected as the hand region from the depth of the hand closest to the camera.

The finger end point information detection unit 25 detects a finger end point from the calculated curvature of a pixel existing on the hand outline of the hand region detected by the hand region detection unit 24. [ In other words, the finger end-point information detecting unit 25 detects the finger end point from the curvature calculation value of the pixel existing on the hand contour by utilizing the morphological prior knowledge of the finger. In one embodiment, after detecting a hand region from a detected hand image and converting the region into a binary image, the contour (i.e., the hand contour including the finger contour) is detected, and the detected contour And calculates the direction vector between each pixel separated in both directions in each pixel, and if the angle of the two direction vectors becomes equal to or more than the threshold, it is recognized as a candidate of the finger end point. Equation 1 below is a formula for obtaining the angles of two vectors forming a hand contour.

와

가 이루는 각도를 의미한다. P (i = 1, 2) denotes a point on the contour of the finger to which the curvature is to be calculated, and P (i = 1, 2) Means two points on the outline. θ is

Wow

.

If the finger endpoint candidate appears consecutively along the pixel of the finger contour, the pixel located in the middle of the consecutive pixels is set as the finger endpoint.

The detected depth values of the fingertip points can be obtained in correspondence with the corresponding coordinates in the depth map obtained by the color image / depth information obtaining section 21, but the depth values of the outline portions can be obtained due to the discontinuity of the depth values occurring in the outline It is not reliable enough. Further, the color image / depth information acquiring device, which is an embodiment of the present invention, is not so precise because the outline of the depth map is not precise. In order to solve such a problem, in the embodiment of the present invention, as shown in FIG. 7, the interest area 21a having a certain size is set around the two-dimensional coordinate of the finger end point, and the color image / depth information obtaining unit 21 The depth value average of the hand region excluding the background in the ROI 21a is obtained from the obtained depth map and is corresponded to the depth information of the finger end point 21b. 7A is an example of a color image photographed through a camera. FIG. 7B is a diagram illustrating a portion set as the region of interest 21a in FIG. 7A. Fig. 7C is a diagram illustrating the finger end point 21b in Fig. 7B.

The virtual touch recognition unit 26 recognizes whether the user has touched the virtual image based on the information from the finger end point information detection unit 25. [ In other words, the virtual-touch recognition unit 26 recognizes whether or not the virtual image (more specifically, the virtual plane) set by the user is touched with a finger. First, a parameter to set the virtual plane is presented. The parameters are presented to allow the user to freely set how much the plane deviates about the X axis from the Kinect in the Z axis direction. Only one virtual plane may be present, and in some cases it may be represented by a plurality of planes surrounding the user. In order to represent a plurality of planes, it is possible to divide an area corresponding to a desired space in the X, Y, and Z directions around the user or the key knot, and set the plane through the virtual plane setting parameter for each area. When a user extends a finger and touches the virtual plane, the three-dimensional coordinates of the finger end point are passed from one space to another space divided by the virtual plane. If the user goes beyond the space, the virtual plane is touched .

The multi-touch gesture recognition unit 27 recognizes the user's gesture with respect to the touch point on which the user has touched the virtual image based on the result of the body joint tracking unit 22 and the result of the virtual touch recognition unit 26 do.

Next, a virtual space multi-touch interaction method according to an embodiment of the present invention will be described with reference to the flowchart of Fig.

First, in S10, the wearable display visualization unit 10 visualizes the stereoscopic image content in the user's visual space.

Then, in S20, the user inputs a predetermined gesture by touching a virtual content (e.g., a three-dimensional menu) existing in the vicinity of the user with a finger.

Accordingly, in S30, the virtual space multi-touch recognition unit 20 recognizes the user's multi-touch on the virtual image visualized through the wearable display visualization unit 10 and outputs the result to the touch feedback providing unit 30, .

In step S40, the touch feedback providing unit 30 provides touch feedback according to the recognition result finally obtained by the virtual space multi-touch recognizing unit 20. [ For example, interactive feedback can be provided by providing touch feedback through effects such as vibration using a device such as a smart device-based watch that is worn on the wrist.

FIG. 9 is a flowchart detailing the virtual space multi-touch recognition process (S30) shown in FIG.

As the user gesture is input S20, the color image / depth information obtaining unit 21 in S31 utilizes the color image / depth information obtaining apparatus attached at an arbitrary position where the upper half of the user can be sufficiently obtained, And acquires the captured color image and depth information.

At S32, the body joint tracker 22 acquires the color image and depth information of the user in front of the keynote using the Kinect and the Kinect SDK or the OpenNI library, and rotates and rotates the user ' Location motion information is provided in real time.

Then, in step S33, the multi-user processing unit 23 determines whether or not the user detection information (for example, the color image and the depth information) provided by the Kinect SDK or OpenNI to block the gesture or obstructing gesture occurring at the same time And selects the most forward user among the detected multiple users as the main user.

In S34, the hand region detecting section 24 detects the hand region of the corresponding main user from each color image and depth information of the main user every frame.

Then, in S35, the finger endpoint information detection unit 25 detects a finger end point from the curvature calculation value of the pixel existing on the hand contour by utilizing the morphological prior knowledge of the finger.

In step S36, the virtual touch recognition unit 26 recognizes whether the virtual plane set by the user (i.e., the main user) has been touched with a finger, and sends the result to the multi-touch gesture recognition unit 27. [

If it is recognized that a touch has occurred, the multi-touch gesture recognition unit 27 determines in step S37 whether the gesture is intended by the user based on the plurality of touch points.

Finally, in step S38, the multi-touch gesture recognition unit 27 sends the recognition result to the touch feedback providing unit 30 so that the touch feedback corresponding to the recognition result can be provided to the user.

FIG. 10 is a flowchart detailing the multi-touch gesture recognition process (S37) shown in FIG.

First, the multi-touch gesture recognition unit 27 determines whether there is one touch point or two or more touch points (S50). The recognition parameters used in the multi-touch gesture recognition unit 27 include a touch duration, a motion tolerance, a motion momentum vector, a distance between touch points, and a midpoint slope between touch points.

If there is one touch point, it is determined whether or not the touch gesture and the press gesture are pressed and held based on the predetermined touch duration (th1, th2) and the touch discrimination area (touch plane area) . Here, if the touch duration time th1 is set to, for example, 1 second, the touch duration time th2 may be set to, for example, 3 seconds. The touch duration (th1, th2) can be adjusted as needed. Accordingly, the touch is performed during the touch duration time th1 and the touch duration time th2, and if the touch point exists within the touch discrimination area of the object of interest (S51 to S53), the gesture of the user is determined to be a click gesture S54). On the other hand, if the touch point exists in the touch discrimination area of the object of interest, but the touch is performed longer than the touch duration time th2, it is determined that the touch point is pressed and held (S55).

If the corresponding touch is made shorter than the touch duration t1 and the motion tolerance for the touch or touch point is greater than or equal to a specific value th3 and the touch point is moving (S51, S56, S57) (S58), it is determined as a flick gesture based on the motion momentum vector parameter (S59). Here, since a flick can be regarded as a short and fast linear movement, a flick gesture can be said to mean a movement in which a user performs a short and fast linear movement using a finger. If the touch is made shorter than the touch duration t1 and the motion tolerance for the touch or touch point is greater than or equal to a specific value th3 and the touch point is moving (S51, S56, S57) (S58), the drag gesture is determined (S60). For more stable drag and flick gesture discrimination, there is also a way to reflect the penetration depth of the virtual plane and fingertip.

On the other hand, if there are two or more touch points (for example, two) and the motion tolerance for the corresponding touches or touch points is less than the specific value th3 ("No" in S61) (S62).

If there are two or more touch points and the motion tolerance for the corresponding touches or touch points is equal to or greater than the specific value th3, it is determined by zooming in or zooming out according to the distance between the touch points. In other words, when the movement tolerance for the corresponding touches or touch points is equal to or greater than the specific value th3 ("Yes" at S61) and the distance between the touch points becomes wider ("Yes" at S63) (S64). If the movement tolerance for the corresponding touches or touch points is equal to or greater than the specific value th3 ("Yes" in S61) and the distance between the touch points becomes shorter ("No" in S63), it is determined as a zoom-out gesture (S65).

In addition, when there are two or more touch points and the motion tolerance for the corresponding touches or touch points is equal to or greater than the specific value th3, a midpoint tilt between touch points is measured for each frame (S66) (S67).

As described above, an optimal embodiment has been disclosed in the drawings and specification. Although specific terms have been employed herein, they are used for purposes of illustration only and are not intended to limit the scope of the invention as defined in the claims or the claims. Therefore, those skilled in the art will appreciate that various modifications and equivalent embodiments are possible without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

10: Wearable display visualization part
20: virtual space multi-touch recognition unit
21: color image / depth information acquisition unit 22: body joint tracking unit
23: multiple user processing unit 24: hand area detecting unit
25: finger end point information detection unit 26: virtual touch recognition unit
27: Multi-touch gesture recognition unit
30:

Claims (17)

A visualization unit for outputting a virtual image;
A virtual space multi-touch recognition unit for recognizing a multi-touch through a user's fingertip detection and motion detection of a virtual image visualized through the visualization unit; And
And a touch feedback unit for providing touch feedback corresponding to the multi-touch of the user recognized by the virtual space multi-touch recognition unit. The method according to claim 1,
Wherein the virtual space multi-
A color image / depth information acquisition unit for acquiring color image and depth information as the user photographs;
A body joint tracker for tracking the rotation and movement of the user's body joint;
A hand region detecting unit for detecting a hand region of the user from the color image and the depth information;
A finger end point information detecting unit for detecting a finger end point from a calculated curvature value of a pixel existing on a hand outline of a hand region detected by the hand region detecting unit;
A virtual touch recognition unit for recognizing whether the user has touched the virtual image based on information from the finger end point information detection unit; And
And a multi-touch gesture recognition unit for recognizing a gesture of the user corresponding to the touch point on which the user has touched the virtual image based on the result of the body joint tracking unit and the result of the virtual touch recognition unit Virtual space multi-touch interaction device. The method of claim 2,
Wherein the hand region detection unit detects a hand region on the basis of the tracking result value acquired by the body joint tracking unit with respect to the color image and the depth information, Detecting a hand region by collecting pixels having a depth value smaller than a threshold value among the pixels in the virtual space. The method of claim 2,
Wherein the hand region detection unit detects a hand region based on a tracking result value obtained by the body joint tracker with respect to the color image and depth information, detects a face from the color image, Dimensional space between the face and the hand by using a prior knowledge about the three-dimensional position of the face and the hand. The method of claim 2,
The fingertip information detecting unit calculates a direction vector between each pixel separated in both directions in each pixel existing along the hand contour of the detected hand region, and when the angle of the two direction vectors is equal to or more than the threshold value, Sets a region of interest centered on the coordinates of the recognized finger endpoint, obtains an average depth value of a hand region excluding a background in the region of interest from a depth map obtained by the color image / depth information obtaining unit, With the depth information of the virtual space. The method of claim 2,
Wherein the multi-touch gesture recognition unit uses a touch duration, a motion tolerance, a motion moment vector, a distance between touch points, and a midpoint slope between touch points as recognition parameters. The method of claim 2,
Wherein the virtual space multi-
Further comprising a multi-user processing unit for selecting a user positioned at the forefront among a plurality of users as a main user based on the color image and the depth information. The method according to claim 1,
Wherein the visualization unit comprises a face wearable stereoscopic display. The method according to claim 1,
Wherein the touch feedback providing unit provides tactile feedback to the user through wireless communication. A virtual space multi-touch recognition unit recognizing a multi-touch through a finger end point detection and motion detection of a virtual image visualized through a visualization unit; And
And providing the touch feedback providing unit with touch feedback corresponding to the multi-touch of the user recognized by the step of recognizing the multi-touch of the user. The method of claim 10,
The step of recognizing the multi-
Obtaining color image and depth information as the user photographs;
Tracking rotation and movement of the user's whole body joint;
Detecting a hand region of the user from the color image and the depth information;
Detecting a fingertip from a calculated curvature of a pixel on a hand contour line of the hand region;
Recognizing whether the user has touched the virtual image based on the information by the step of detecting the fingertip; And
Recognizing a user's gesture with respect to a touch point on which the user has touched the virtual image based on a result of the step of tracking the rotation and movement and a result of recognizing whether the virtual image is touched; Wherein the virtual space is a virtual space. The method of claim 11,
The step of detecting the hand region may include detecting a hand region based on a tracking result value acquired in the step of tracking rotation and movement of the user's body joint with respect to the color image and depth information, Detecting a hand region by collecting pixels having a depth value smaller than a threshold value among pixels within a predetermined size region centered on the hand coordinates traced by the user. The method of claim 11,
Wherein the step of detecting the hand region comprises detecting a hand region based on a tracking result value obtained in the step of tracking rotation and movement of the user's body joint with respect to the color image and depth information, And detecting the position of the virtual space using the prior knowledge of the three-dimensional position of the face and the hand based on the three-dimensional position information of the face. The method of claim 11,
The step of detecting the fingertip point may include calculating a direction vector between each pixel separated in both directions in each pixel existing along the hand contour of the detected hand region, and if the angle of the two directional vectors is greater than or equal to the threshold value, And sets the ROI around the coordinates of the recognized finger end point and obtains the depth value average of the hand region excluding the background in the ROI from the ROI obtained in the step of obtaining the ROI and the ROI And associating it with depth information of a finger end point. The method of claim 11,
Wherein the step of recognizing the user's gesture uses a touch duration, a motion tolerance, a motion momentum vector, a distance between touch points, and a midpoint slope between touch points as recognition parameters. The method of claim 11,
Wherein the step of recognizing the multi-touch further comprises the step of selecting a user positioned at the forefront among a plurality of users as a main user based on the color image and the depth information. The method of claim 10,
Wherein the step of providing the touch feedback provides tactile feedback to a user via wireless communication.

Images