KR20050060606A - Device and method for interaction between human and computer - Google Patents

Abstract

The present invention relates to a human computer interaction apparatus and method, and more particularly to a three-dimensional image using one image sensor and two non-contact optical markers capable of acquiring images of the entire palm and independent of wrist movement. A human computer interaction apparatus and method capable of inputting a hand shape and a spatial displacement of a hand into a computer.

The human computer interaction apparatus according to the present invention comprises an image sensor for acquiring a hand image at any movement or movement, and a non-contact optical mark on the palm of the hand so that the distance from the fingertip to the image sensor can be estimated. And a support for supporting the light source to project and the image sensor on the palm of the hand so as not to be affected by the movement of the wrist.

In addition, the human computer interaction method according to the present invention comprises a hand shape estimation step of estimating a hand shape using hand image data, and a hand shape restoration for restoring a hand shape on a computer using the data of the hand shape estimation step. And a hand space displacement estimating step of estimating a space displacement of a hand using background image data among the hand image data.

Description

Device and method for interaction between human and computer

According to the present invention, an image of the entire palm can be obtained and a three-dimensional hand shape and a spatial displacement of the hand can be input to a computer by using an image sensor and two non-contact optical markers independent of wrist movement. The present invention relates to a human computer interaction apparatus and method.

Despite the quantum leap in computer and communication technology, human computer interaction (HCI) methods are still inconvenient to use, and traditional HCI methods represented by a mouse and keyboard are intuitive and rich in human expression. Has reached its limit in effectively communicating There are many limitations, especially in mobile environments. Currently, many new input devices are being developed, and the most common input method is to use hand shape and displacement as input values.

The most common method of HCI using hand shape and displacement as input value is to attach variable resistor to each finger joint to detect the change of resistance value to estimate joint movement and to attach optical fiber in the glove to the finger Then, by comparing the amount of light input and the amount of light obtained from the output, the method of estimating the movement of the finger joint by reflectance and refraction index, and by attaching an accelerometer to the tip of the finger or the middle of the finger to detect the movement of the finger and the whole hand, etc. Are globe-based methods.

However, this method has a problem that it is cumbersome because the glove has to be worn, and the sensor is in close contact with the joint, thereby preventing the user's natural hand input and being considerably uncomfortable when used for a long time. In addition, there is a problem in that the equipment price is high due to the need for additional equipment such as a controller due to the use of sensors such as acceleration sensors and optical fibers.

In order to improve the problems of the globe-based methods, vision-based non-contact HCI technologies have been actively developed to acquire the image of the hand using an image sensor, find the feature values of the hand, and estimate the hand movement.

However, such a vision-based method has a problem in that wearing and moving are impossible because a fixed camera is used in space.

There has been another attempt to solve this problem. In consideration of the problems of the vision-based method described above, a method of inputting seven codes by detecting a change of three fingers through a camera attached to a wrist has been developed.

However, this is only a few encodings can not input the shape of the hand, wearing a camera on the wrist, there is a problem that can not get the result unintentional because you have to get a result dependent on the movement of the wrist.

SUMMARY OF THE INVENTION An object of the present invention is to provide a human computer interaction apparatus which greatly reduces the discomfort of wearing by using an image sensor and a non-contact optical marker.

Still another object of the present invention is to provide a human computer interaction apparatus capable of estimating a three-dimensional movement of a hand using only one image sensor.

It is still another object of the present invention to provide a human computer interaction apparatus which is simple in structure, practical and economical by using an image sensor and a non-contact optical marker.

It is still another object of the present invention to provide a human computer interaction device that is easy to carry and wear in a mobile environment.

It is still another object of the present invention to provide a human computer interaction method capable of restoring any hand shape and displacement on a computer.

The human computer interaction apparatus according to the present invention comprises an image sensor for acquiring a hand image at any movement or movement, and a non-contact optical mark on the palm of the hand so that the distance from the fingertip to the image sensor can be estimated. And a support to support the light source to project and the image sensor on the palm of the wrist so as not to be affected by the movement of the wrist.

In addition, the human computer interaction method using the shape and the displacement of the hand according to the present invention as input values, the hand shape estimation step of estimating the hand shape using the hand image data, and using the data of the hand shape estimation step And a hand restoring step of reproducing a hand shape on a computer, and a hand space displacement estimating step of estimating a spatial displacement of a hand using background image data among the hand image data.

In addition, the human computer interaction method according to the present invention includes a finger height information obtaining step of obtaining one-dimensional finger height information obtained from a two-dimensional hand image in which the hand shape estimation step is a color image segmentation, and two optical marks. And finger depth information obtaining step of obtaining finger depth information from the hand and predicting a hand shape by combining the finger height information and the depth information.

In addition, in the human computer interaction method according to the present invention, the hand shape feature values extracted in the hand shape estimating step may include parameters of the 3D hand model in which the hand restoring step is configured by various constraints. Mapping and reproducing the shape of the hand (reproduce) on the computer.

In addition, in the human computer interaction method according to the present invention, the motion space of the background image is estimated after the hand space displacement estimation step estimates the motion vector of the background image used in the hand shape propagation step by a block matching method. It is characterized by estimating the spatial displacement of the hand by calculating the reverse vector of.

Hereinafter, with reference to the accompanying drawings, the features of the present invention described above will be described in detail.

1 is a block diagram of a human computer interaction apparatus according to the present invention.

As shown in FIG. 1, the human computer interaction apparatus according to the present invention has one image sensor capable of acquiring a hand and a background image, and has two planar light beams that cross each other in a straight line at oblique angles at a predetermined distance. It consists of two non-contact markers and a support that is fixed on the palm of the hand so that the image sensor can capture the image without being affected by wrist movement.

As described above, since the human computer interaction apparatus according to the present invention uses only one image sensor, the number of parts is reduced compared to the case of using two. In addition, the human computer interaction apparatus according to the present invention is easier to use than the contact marker because it uses a non-contact marker. In addition, since the human computer interaction apparatus according to the present invention fixes the image sensor to the support on the palm of the hand, it is not affected by the movement of the wrist so that the change in the shape of the hand and the displacement can be restored more accurately than in the conventional case.

Human computer interaction method according to the present invention is largely divided into three steps.

The first step is to estimate hand shape by obtaining one-dimensional finger height information from color image segmented two-dimensional image and combining the one-dimensional finger height information and depth information from two optical marks. Step.

The second step maps the parameters of the three-dimensional hand model, which is composed of various constraints, with the hand-shaped feature values extracted in the hand estimation step.

It is a hand restoration step of reproducing a hand shape on a computer by apping.

The third step estimates the spatial displacement of the hand by estimating the motion vector of the background image among the hand images used in the hand shape estimation step by block matching method and then calculating the reverse vector of the motion vector of the background image. It is a hand space displacement estimation step.

Each step will be described in detail below.

<1. Hand estimation step>

1-1) Finger height information acquisition step (image segmentation)

The segmentation of the hand image divides the human skin color into Y-value intervals in the YCbCr color space, which is robust to brightness changes, and uses a chromatic map that defines regions of skin color on the Cb and Cr coordinate systems of each interval. In addition, when the Y value is low, in order to compensate for the case where the hand image is not extracted well, when the Y value is less than a specific value, the color of the area where the hand image always exists is used for the image segmentation process.

This is possible because an image sensor is attached to the hand so that part of the hand image always appears in a specific part of the entire image. Finally, the two light marks projected on the hand are very characteristic and thus simply extracted through thresholding in RGB space.

1-2) Finger height information acquisition step (1D histogram conversion of 2D image information)

2 is a diagram illustrating a process of extracting feature values of a hand.

As shown in FIG. 2, when the angle of the inclined finger is found in the image and the hand image is corrected vertically in the image, two-dimensional image information as shown in FIG. Can be converted to histogram information. In this case, it is assumed that the finger images obtained from the image sensor are parallel to each other and only the information of the height changes in the movement of bending the finger. In order to determine the degree of inclination of the hand, an edge is first extracted from the image of the divided hand. Next, the gradient information is obtained in a specific area that is predicted to have a finger image in advance by using the Hough transform. Finally, after the image is rotated by the obtained angle, the histogram of the one-dimensional finger height information is obtained through the x-axis projection.

1-3) Acquisition of Finger Depth Information and Prediction of Hand Shape (Estimate of Fingertip Position)

From the one-dimensional histogram obtained in the above step 1-2), a graph of the change of the fingertip position as shown in FIG. 2 (c) is obtained according to the change of the angle ( MCP θ) of the heavy ground node having a range of 0 ° ≤ MCP θ ≤ 90 °. I can make it. However, this information alone cannot determine whether the finger is close to or away from the camera based on the position of the highest point in the graph on FIG. 2 (c), max_ height _ angle . That is, it is not known whether the finger is folded or extended.

Therefore, in the human computer interaction method according to the present invention, two light sources on each finger are projected onto the finger by projecting a light source on the finger, which generates two surfaces with inclination to determine the distance information between the finger and the camera. Produce parallel straight lines. Here, the distance information of the two straight lines has the same result as that of the graph of FIG. 2 (e) according to MCP θ. In fact, the location information of the fingertip can be estimated only by the distance information generated by the two straight lines, but as shown in FIG. 2 (e), a change section of irregular information named dead zone appears in a certain area.

Therefore, the distance information of the two straight lines plays the role of creating the depth _ flag , which is a criterion for judging the perspective of the camera and the finger based on the max_ height _ angle , which is the MCP θ when the finger height histogram has the highest value. One-dimensional histogram information is used as main information. The height histogram graph of the finger according to the change of MCP θ in FIG. 2 (c) can be approximately formulated and curren which is the height value of the fingertip of the histogram.

relationship t_height and MCP θ is depth_ having 0 and 1 values, based on the height max_ angle _

Defined according to the flag value.

<2. Hand Restoration Steps>

** ** restore three-dimensional hand model

3 is a schematic side view of the finger in consideration of the joint angle.

Human computer interaction method according to the present invention is J.J. Based on Kuch's three-dimensional hand model. However, considering only the movement of the finger on the y-axis, the angular relationship between the finger joints may be briefly illustrated as shown in FIG. 3.

Therefore, it is possible to estimate the MCP θ know the height of the fingertip, which is represented by the one-dimensional histogram can be obtained and the PIP θ, θ DIP value, as shown in Figure 3 by the MCP θ. Eventually, the movement of the finger can be restored on the computer using the height information of the fingertip and the distance information between the finger and the camera.

The thumb has a different direction of motion than the four fingers. If you look on the screen, you can see that the thumb has a lateral movement. With this in mind, you can estimate the movement of your thumb by making a square area between your index finger and your thumb while holding your hands open and grasping the number of skin color pixels that appear here.

<3. Hand Space Displacement Estimation Step>

** Estimation of hand space displacement **

Since the human computer interaction apparatus according to the present invention is a hand input apparatus, the accurate displacement acquisition is not important when considering feedback, and thus, a 2D image block matching method is used to estimate the hand movement.

When the fist is clenched, the motion vector of the background image is extracted and conversely, the motion of the hand is estimated. This is possible because the camera is attached to the palm of the hand so that the image sensor is also dependent on the movement of the hand. Motion prediction is fast and uses a three step search (TSS) block matching method that produces good results for low quality video.

However, the result obtained by the block matching is a vector value of one pixel unit, so it is very sensitive to hand movement and is difficult to use by the input device. Therefore, the size of the image is increased to make the search range larger. Finally, the most representative of the vectors obtained from the blocks of the background image can be taken in the reverse direction to estimate the motion of the hand.

The overall operation of the human computer interaction method according to the present invention described above will be described using a flowchart.

4 is a flowchart illustrating a human computer interaction method according to the present invention. As shown in FIG. 4, the flow of the human computer interaction method according to the present invention can be summarized as follows.

(reset)

Hold your hands early in the system's operation to store the minimum and maximum height of your fingertips.

* Mode is divided into 0 and 1 by distinguishing whether or not the hand is held.

Mode 0 (the process of estimating the spatial displacement of the hand): the hand is clenched

1. Preprocess the image to gray image.

2. Reduce the resolution of the image so that no result is too sensitive to hand movement.

3. A representative vector of motion vectors of an image obtained by block matching is calculated for a predetermined background image part.

4. By calculating the inverse vectors of the motion vectors obtained through block matching, the motion vectors of the hands are obtained.

-Mode 1 (the process of estimating the shape of the hand): State other than Mode 0

1. Segment a hand image from the entire image.

2. Hough transform the tilted hand image to get the tilted angle and rotate it so that it is perpendicular to the image.

3. In the segmented hand image, the height of the fingertip is calculated as a relative value by referring to the minimum and maximum height of the fingertip obtained from the initialization.

4. Estimate the distance between the fingertip and the image sensor from the distance value between the two light lines projected on the finger.

5. Estimate the position of the fingertip by combining the height value of the fingertip obtained in step 3 and the distance value between the fingertip and the image sensor estimated in 4.

6. From the position of the fingertip estimated in step 5, calculate the metacarpophalangeal angle of each finger by referring to the relationship between the position of the fingertip and the metacarpophalangeal angle shown in FIG.

7. The shape of the hand is restored by applying the angle of the metastasis obtained in step 6 to the three-dimensional hand model.

Since the human computer interaction apparatus according to the present invention uses a non-contact image sensor and a marker, a vision-based method using an image sensor fixed in space cannot be used at the same time as it is more convenient to wear than a conventional globe method. It has the effect of overcoming the disadvantages.

In addition, the human computer interaction apparatus according to the present invention has the effect of estimating the three-dimensional movement of the hand using only one image sensor.

In addition, the human computer interaction apparatus according to the present invention may be applied to a hand input device such as a portable device when the hand input device is used for a long time, the input of a portable terminal device, the input of a sign language interpreter, a 3D CAD system, a rehabilitation medical device, In addition, it can be applied to fields such as DTV input, which is convenient to wear, portable, and enables the natural and intuitive input of the hand.

In addition, the human computer interaction apparatus according to the present invention has a simpler and more economical effect than the conventional method.

In addition, the human computer interaction method according to the present invention can restore the shape and displacement of any hand on the computer.

1 is a block diagram of a human computer interaction apparatus according to the present invention.

2 is a diagram illustrating a process of extracting feature values of a hand;

Figure 3 is a schematic side view of the finger considering the joint angle.

4 is a flow chart of a human computer interaction method in accordance with the present invention.

Claims (2)

In the human computer interaction device using the shape and displacement of the hand as an input value, The human computer interaction device An image sensor for acquiring a hand image in an arbitrary movement or movement, A light source for projecting a non-contact light mark on the palm of the hand so as to estimate the distance from the fingertip to the image sensor; Support for supporting the image sensor on the palm of the hand so as not to be affected by the movement of the wrist Human computer interaction apparatus comprising a. In the human computer interaction method using the shape and displacement of the hand as an input value, The human computer interaction method A hand estimation step of estimating a hand shape using hand image data; A hand restoration step of restoring a hand shape on a computer by using the data of the hand shape estimation step; Hand space displacement estimation step of estimating the spatial displacement of the hand using the background image data among the hand image data Human computer interaction method comprising a.