KR101883866B1 - Ground contact type finger input device and method - Google Patents

Abstract

FIELD OF THE INVENTION The present invention relates to a ground contact type finger input device and method. More particularly, the present invention relates to a method of generating an image by photographing a ground and a finger of a user, extracting a ground area and a finger area from the generated projection, determining whether or not to use the information at each of the overlapping points of the ground area and the finger area, And generates a command signal using the determined contact / non-contact information. Accordingly, it is possible to implement a variety of action functions and to provide a new concept of a mouse interface device that allows a virtual touch interface in a space free from constraints.

Description

[0001] The present invention relates to a ground contact type finger input device and method,

FIELD OF THE INVENTION The present invention relates to a ground contact type finger input device and method. More particularly, the present invention relates to a technique that can utilize finger touch and movement more variously than existing mice by using information that a finger tip touches the ground.

Recently, intuitive interface technologies (Natural Interfaces) have emerged between human and computer, which are natural and interactive. Research on user's intention and behavior recognition for interaction between human and computer has been actively carried out. Typical user interface (UI) technology such as a keyboard and a mouse is also referred to as a pointing device Device responsible for computer control and information exchange. In the case of a mouse, it is one of the most representative hardware. However, the existing mouse has a definite gesture setting. Once the user's body part is aligned to a predetermined position, a gesture is detected. Therefore, a predetermined function is performed in the computer device. That is, in the present method, once each gesture is detected, a predetermined action for a specific gesture is executed by a computer. The current method typically includes an initial set up step. In the set up step, a gesture of definite setting is defined, and the computer performs a function for a specific gesture. This means that the user must use the mouse only at a predetermined position, and the action according to the user's gesture is limited. This is not free to implement various actions according to various gestures.

As the projector and camera have been miniaturized due to the recent development of technology, they are being mounted on mobile devices, and these compact projectors and cameras are being manufactured in a wearable form to provide various services In addition to conceptual presentation (MIT Sixth Sense), a portable system (System) (Microsoft's Mobile Surface, Light Blue Optics' Light Touch) is being developed. Of course, by using such a system, digital information can be expressed on a real object (object) rather than on a screen on a digital terminal, and a new service can be created .

In such a system, an interaction method using a bare hand, which does not have a separate identification device such as a hand or a finger, should be provided for a user's convenient use, and the system should be provided with an interface The system should be provided with a load of the system and a method of manufacturing it at low cost.

Conventional image recognition technology is mainly based on recognition of face or gesture using a complex algorithm using a color camera. These technologies require a very fast computer and are not suitable for small-scale applications such as embedded systems. In particular, tracking a hand or recognizing a gesture in a projected image is different from simply recognizing an image using a camera in a place where there is no image of the projector. In other words, in a method of recognizing a specific color such as skin color, which is widely used in image recognition, in the projector environment, since the output image is projected on an object such as a hand, the recognition rate is remarkably decreased, The recognition itself can become impossible because the change is severe.

Meanwhile, as an example of interactions with projection images, a projector is mounted on a ceiling, a user's motion is recognized using a camera, and based on the recognition, a user interacts with a projection image There is a system. However, this not only requires a lot of system resources for image processing but also can not be applied to precise recognition such as finger movement and touch.

Korean Patent Registration No. 1446902 Korean Patent Publication No. 2012-0020045 Japanese Patent Publication No. 5926184

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide a mouse interface based on information that a fingertip touches the ground.

It is also intended to implement a mouse interface which can be used regardless of the place, since a separate initial setup is not required.

Also, it is aimed to be able to distinguish each finger that touches the ground through image processing.

In addition, since each finger can be distinguished, a user can implement various action functions according to a free gesture (gesture).

It is also intended to provide a device capable of a virtual touch interface in a space free from constraints.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a ground contact type finger input device including an image generation unit for generating an image by photographing a ground and a user's finger, a ground region and a finger region from the generated image, And a command signal generation unit for generating a command signal by using the determined contact information, and a command signal generation unit for generating a command signal by using the determined contact information.

In this case, the contact determination unit may include a ground boundary extracting unit for extracting a ground boundary from the generated image, a contact region extracting unit for extracting a contact region where the extracted ground boundary and the finger region overlap with each other, A finger end point extracting unit for extracting an end point of each finger from the image in which the ground area is removed, and a contact recognizing unit for determining that the end point of each extracted finger overlaps the contact area, do. The contact determination unit may further include an image conversion unit that converts the brightness value information of the generated image into a background area having a predetermined brightness value or less.

The ground boundary extracting unit is characterized by using a mask technique for extracting a ground boundary line using brightness value information of an image. In this case, the surface boundary line of the mask technique is a line connecting an excess point where the brightness value exceeds 0 when the brightness value is 0, and a line connecting the smallest brightness value point when the brightness value is over 0 .

The image converting unit may determine that a region having a brightness value of 75 or less is a background region, and converts the region into a brightness value of 0.

The command signal of the command signal generating unit controls the computer as any one of a cursor movement, a click, an up and down scroll, a left and right scroll, and a hot key.

According to an aspect of the present invention, there is provided a ground contact type finger input method, a method of generating an image by photographing a ground and a user's finger, a step of extracting a ground area and a finger area from the generated image, A contact determination step of determining whether or not to make contact with each other using the information of each point overlapping the ground area and the finger area, and a command signal generating step of generating a command signal using the determined contact presence / absence information.

In this case, the step of determining whether or not the contact is made includes a step of extracting a paper boundary line from the generated image, a step of extracting a contact area extracting a contact area in which the extracted paper boundary line and the finger area overlap each other, A finger end point extraction step of extracting an end point of each finger from the image in which the ground area is removed, and a contact recognition step of determining that the end point of each extracted finger is in contact with the contact area, . In addition, the contact determination step may further include an image conversion step of converting the brightness value information of the generated image into a background area less than a predetermined brightness value.

The ground boundary extracting step is characterized by using a mask technique for extracting a ground boundary line using brightness value information of an image. In this case, the surface boundary line of the mask technique is a line connecting an excess point where the brightness value exceeds 0 when the brightness value is 0, and a line connecting the smallest brightness value point when the brightness value is over 0 .

The image converting step determines that a region having a brightness value of 75 or less is a background region, and converts the region into a brightness value of 0.

The command signal in the command signal generating step controls the computer as any one of a cursor movement, a click, a scroll, a left and right scroll, and a hot key.

According to the ground-contact finger input device and method as described above,

First, there is an effect that a mouse interface technology can be implemented based on information that the fingertip touches the ground.

Second, it is not necessary to set up a separate initial setup, and it is possible to implement an available mouse interface technology regardless of the place.

Second, it has the effect of distinguishing the fingers in contact with the ground through image processing.

Third, since the respective fingers can be distinguished, the user can implement various action functions according to a free gesture (Gesture).

Fourth, there is an effect that a device capable of a virtual touch interface can be provided in an unrestricted space.

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a diagram of a ground-contacting finger input system in accordance with an embodiment of the present invention.
2 is a configuration diagram showing a ground-contact type finger input device.
FIG. 3 is a block diagram showing a contact determination unit of the ground contact type finger input device. FIG.
4 is a flowchart showing a ground contact type finger input method.
FIG. 5A is a view illustrating an image in which a finger and a finger of a user are photographed according to an embodiment of the present invention. FIG.
FIG. 5B is a diagram illustrating an image obtained by converting an area of a predetermined brightness value or less to a brightness value of 0 according to an exemplary embodiment of the present invention; FIG.
FIG. 5C is a view showing an image obtained by extracting a paper boundary line, a finger region, and a contact region according to an embodiment of the present invention; FIG.
5D is a diagram illustrating an image in which a ground area is removed according to an embodiment of the present invention.
FIG. 5E is a view showing an image of an end point of each finger extracted according to an embodiment of the present invention; FIG.
FIG. 5F is a view showing an image in which a contact area and each finger end point are matched one-to-one according to an embodiment of the present invention. FIG.
FIG. 6A is a view showing a mask for extracting a paper boundary line in a non-contact area in which the paper and the finger are not in contact with each other according to the embodiment of the present invention; FIG.
FIG. 6B is a view showing a mask for extracting a paper boundary line in a contact area where a finger is in contact with the paper according to an embodiment of the present invention; FIG.
FIG. 7A illustrates a finger image that implements a cursor motion instruction in accordance with an embodiment of the present invention. FIG.
7B illustrates a finger image implementing a Click command according to an embodiment of the present invention.
7C illustrates a finger image that implements a scroll command according to an embodiment of the present invention.
7D illustrates a finger image that implements a left and right scroll command according to one embodiment of the present invention.
7E illustrates a finger image that implements a Hotkey command in accordance with an embodiment of the present invention.

A ground contact type finger input apparatus and method according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. The present invention is capable of various modifications and various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. It should be understood, however, that the present invention is not intended to be limited to the particular forms disclosed, but includes all images falling within the spirit and scope of the present invention. Like reference numerals are used for like elements in describing each drawing. In the accompanying drawings, the dimensions of the structures are enlarged to illustrate the present invention, and are actually shown in a smaller scale than the actual dimensions in order to understand the schematic structure.

Also, the terms first and second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. On the other hand, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Referring to FIG. 1, there is shown a ground contact finger input system according to an embodiment of the present invention.

First, when the IR diffuse sensor 23 is illuminated to the finger 13, the light is reflected and enters the infrared pass filter camera 25. At this time, the infrared pass filter camera (IR Pass Filter Camera) 25 is a mono camera. When the IR diffuse sensor 23 is illuminated to the finger 13, the light of the finger 13 nearer to the finger 13 is reflected more. That is, the finger 13, which is relatively close to the input device 20, appears brighter when viewed by the image. The light reflected by the infrared pass filter camera 25 is input to the input device 20 and the image information is transmitted to the mouse controller 27. The mouse controller 27 can control the computer 30 according to the transmitted image information. The image information transmitted at this time is image information when the finger 13 comes in contact with the paper surface G, which generates various command signals that can be implemented with five fingers. If you use the left hand and the right hand together, you can create a variety of command signals that can be implemented with a total of 10 fingers. As shown in Fig. 1, the object to which the finger 13 is contacted is the ground G, but it should be interpreted that it can be any object to which the finger 13 can touch, such as a desk, a wall, and a book.

2 and 3, the ground contact type finger input device includes an image generation unit 100, a contact determination unit 200, and a command signal generation unit 300.

The image generating unit 100 photographs the ground and the user's finger to generate an image.

The contact determination unit 200 extracts the ground area and the finger area from the generated image, and determines whether or not to make contact using the information of each of the overlapping points of the ground area and the finger area.

At this time, the contact determination unit 200 includes a ground boundary extracting unit 211, a contact area extracting unit 213, a ground area removing unit 215, a finger end point extracting unit 217, and a contact recognizing unit 219 .

The ground boundary extracting unit 211 extracts a ground boundary from the generated image. The ground boundary extracting unit 211 uses a mask technique for extracting a ground boundary using brightness information of an image. In this case, the surface boundary line of the mask technique is a line connecting an excess point where the brightness value exceeds 0 when the brightness value is 0, and a line connecting the smallest brightness value point when the brightness value is over 0 .

The contact region extracting unit 213 extracts a contact region where the extracted ground boundary line and the finger region overlap.

The ground area removing unit 215 removes the ground area using the extracted ground boundary information.

The finger end point extraction unit 217 extracts the end points of the respective fingers from the image in which the ground area is removed.

The contact recognizing unit 219 judges that the end point of each of the extracted fingers is in contact with the contact area.

The contact determination unit 200 further includes an image conversion unit 220 that converts the brightness value information of the generated image into a background area having a predetermined brightness value or less. The image converting unit 220 determines that the area having a brightness value of 75 or less is a background area and converts the area into a brightness value of 0.

The command signal generator 300 generates a command signal using the determined contact information. At this time, the command signal controls the computer as any one of a cursor movement, a click, a scroll, a left and right scroll, and a hot key. The command signal is an instruction signal according to an embodiment of the present invention, and is not limited by the terms. The number of all cases that can be implemented with a maximum of 10 fingers can be implemented as a command signal, and the command signal should be interpreted as being capable of various operations according to the user's setting.

Referring to FIG. 4, the ground contact type finger input method includes an image generation step S100, a ground boundary extraction step S211, a contact area extraction step S213, a ground area removal step S215, a finger end point extraction step S217 ), A contact recognition step (S219), and a command signal generation step (S300).

In the image generation step S100, the image of the ground and the user's finger is photographed.

In the ground boundary extracting step S211, the brightness value information of the generated image is used to convert into a background area less than a certain brightness value. An area having a brightness value of 75 or less is determined as a background area, and the brightness value is converted into a brightness value of 0. In the step S211, a boundary line is extracted from the generated image using a mask technique for extracting a boundary line of the image using the brightness value information of the image. In this case, the surface boundary line of the mask technique is a line connecting an excess point where the brightness value exceeds 0 when the brightness value is 0, and a line connecting the smallest brightness value point when the brightness value is over 0 .

The contact area extraction step (S213) extracts a contact area where the extracted ground boundary line and the finger area overlap.

The ground area removing step (S215) removes the ground area using the extracted ground boundary information.

The finger end point extracting step (S217) extracts the end points of the respective fingers from the image in which the ground area is removed.

In the contact recognition step S219, when the extracted end point of each finger overlaps the contact area, it is judged as contact.

The command signal generation step (S300) generates a command signal using the determined contact / non-contact information. At this time, the command signal controls the computer as any one of a cursor movement, a click, a scroll, a left and right scroll, and a hot key. The command signal is an instruction signal according to an embodiment of the present invention, and is not limited by the terms. The number of all cases that can be implemented with a maximum of 10 fingers can be implemented as a command signal, and the command signal should be interpreted as being capable of various operations according to the user's setting.

Referring to FIGS. 5A and 6B, a ground contact finger input system according to an exemplary embodiment of the present invention is shown.

1) The image of the ground G and the finger of the user is photographed (Fig. 5A).

2) The generated image (FIG. 5A) is changed to the HSV color model (Hue Saturation Value Color Model) and converted into a background area less than a predetermined brightness value by using the brightness value information of the changed image. In this case, it is determined that a region having a brightness value of 75 or less is a background region, and converts the region into a brightness value of 0.

3) Extract the ground (G) area and the finger area (5) from the converted image (Fig. 5B). In addition, the paper boundary line 3 is extracted from the converted image (Fig. 5B). The extraction of the ground boundary line (3) uses a mask technique for extracting the ground boundary line (3) using the brightness value information. At this time, the mask plays a role of extracting the ground boundary line 3 from the converted image (FIG. 5B). A method of extracting the paper boundary line 3 by the mask technique of the present invention can be described with reference to FIGS. 6A and 6B. In the case of FIG. 6A, as a mask for extracting the ground boundary line 3 from the noncontact area where the finger G is not in contact with the ground G, a mask G is formed between the ground G having a brightness value of more than 0 and the background having a brightness value of 0 Is a border line (3). In the case of FIG. 6B, as a mask for extracting the ground boundary line 3 from the contact area 7 in which the ground G and the finger are in contact, a point having the smallest brightness value among the values exceeding 0 is referred to as a ground boundary line (3). The mask according to an embodiment of the present invention starts at any one of the left, right, upper, and lower sides of the image. However, the mask may be masked depending on the shape of the object to be extracted from the image using a mask It is not limited to one starting point. The mask according to an embodiment of the present invention is characterized in that a mask is extracted from the left side of the image and then moved to the right side of the image while the mask is located at the boundary line 3 . When the mask reaches the right side of the image, it shows that the ground boundary line (3) from the left to the right side of the image has been extracted. The mask technique used in the present invention is applicable to an uneven ground G. It is possible to extract the ground boundary 3 more precisely at an uneven ground G than an embodiment of the present invention . As shown in FIG. 5C, although the mask technique of the present invention uses the mask A and the mask B largely, it is understood that various masks can be used in addition to the FIGS. 6A and 6B according to the brightness value of the image. shall.

4) The ground (G) area is removed from the extracted image (FIG. 5C).

5) Each finger end point 9 is extracted from the image (FIG. 5D) from which the ground (G) area is removed. At this time, each finger end point 9 judges that the point having the largest y-axis value is each finger end point 9. If there are a plurality of points having the largest y-axis value in one finger, the x-axis values are extracted from each y-axis value, and the middle point of these x-axis values is determined as the end point of the corresponding finger. For example, referring to the enlarged view of FIG. 5D, if (0.9, 3), (0.6, 3), and (0.3, 3) are a plurality of points having the largest y- The average x-axis value is 0.6 ((0.9 + 0.6 + 0.3) / 3). Therefore, the coordinate point (0.6, 3) is determined as the end point of the finger. When there are a plurality of points having the largest y-axis value in one finger, it can be various cases that can occur depending on the contact method of the user, such as a situation in which the finger presses the ground G.

6) When the contact area 7 of FIG. 5C overlaps the respective finger end points 9 of FIG. 5E, it is judged as a contact and they are matched to each other on a one-to-one basis. The resulting image that is matched is shown in Figure 5f. Unlike the embodiment of the present invention, when the user touches the ground G with only five pieces of the finger 13, the point having the largest y-axis value from the finger 13 not in contact with the ground G It should be interpreted that the finger end point 9 can be extracted but the contact area 7 where the finger G contacts the ground G can not be extracted.

7A to 7E, a finger image for implementing a command signal according to an embodiment of the present invention is shown.

FIG. 7A is a diagram illustrating a finger image that implements a cursor movement instruction according to an exemplary embodiment of the present invention, and is the same as the conventional mouse model. In this case, the cursor movement according to an embodiment of the present invention is an indicator for accurately indicating a path through which the user moves the pointing means, and the shape and color of the mouse cursor can be variously modified. More than one mouse cursor can be created depending on the number of fingers or setting, and can be moved in a random direction.

7B is a view showing a finger image implementing a click instruction according to an embodiment of the present invention and recognizing an area 5 in which the ground G and the fingertip 15 contact each other, Lt; / RTI > The finger image shown in FIG. 7B implements a click instruction according to an embodiment of the present invention, but the finger image can be implemented as another instruction according to user setting.

FIG. 7C is a diagram illustrating a finger image implementing a scroll command according to an embodiment of the present invention. The scroll command is implemented by changing the brightness value of a finger. Although the finger image shown in FIG. 7C implements a scroll command according to an exemplary embodiment of the present invention, the finger image can be implemented as another instruction according to user settings.

FIG. 7D implements right and left scrolling commands by left and right movement of the contact area 5 recognized as a drawing of a finger image implementing a left and right scroll instruction according to an embodiment of the present invention. The finger image shown in FIG. 7D implements a right and left scroll command according to an embodiment of the present invention, but the finger image can be implemented by other instructions according to user setting.

FIG. 7E shows a finger image implementing a Hotkey command according to an embodiment of the present invention, and implements a Hotkey command in a combination of the contact areas 5. FIG. The finger image shown in FIG. 7E implements a hotkey command according to an embodiment of the present invention, but the finger image can be implemented as another command according to user setting. The number of combinations of the contact area (5) can be up to 32 on the basis of one hand, and up to 1,024 on the basis of both hands. In particular, the present invention should be construed as being applicable to a user who can not use a finger naturally or a user who can not use a finger as an embodiment.

In the above description, the ground-contacting type finger input device is mounted on the apparatus while the ground-contacting type finger input device is grounded on the floor. However, the ground-contacting type finger input device may be variously modified by a user such as a portable wearable It should be interpreted as being applicable.

Further, the ground contact type finger input device of the present invention should be construed as being applicable not only to a mouse but also to various control devices other than a computer. One embodiment of the control device may be a keyboard, a mobile phone, a car, a remote control, or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. It is clear that the present invention can be suitably modified and applied in the same manner. Therefore, the above description does not limit the scope of the present invention, which is defined by the limitations of the following claims.

G: Floor 3: Floor boundary
5: finger area 7: contact area
10: Hand 13: Finger
15: fingertip 20: input device
23: IR Diffuse Sensor
25: Infrared pass filter camera (Filter Camera)
27: Mouse Controller
30: Computer
100: image generating unit 200: contact area determining unit
211: paper boundary line extracting unit 213: contact area extracting unit
215: ground region removing unit 217: finger end point extracting unit
219: contact recognition unit 220: image conversion unit
300: command signal generating unit

Claims (14)

A ground contact type finger input device comprising:
An image generating unit for generating an image by photographing a ground and a user's finger;
A contact presence / absence determination unit for extracting a ground area and a finger area from the generated image, and determining whether or not to make contact using information of each of overlapping points of the ground area and the finger area; And
And a command signal generator for generating a command signal using the determined contact / non-contact information,
The contact-
A ground boundary extracting unit for extracting a ground boundary from the generated image;
A contact area extracting unit for extracting a contact area where the extracted ground boundary line and the finger area overlap;
A ground area removing unit for removing the ground area using the extracted ground boundary information;
A finger end point extraction unit for extracting an end point of each finger from the image in which the ground area is removed; And
And a contact recognition unit for determining that the extracted point is a contact when the end point of the extracted finger overlaps the contact area. delete The method according to claim 1,
The paper-
Wherein a mask (Mask) technique for extracting a paper boundary line using brightness value information of an image is used. The method of claim 3,
The ground boundary line of the mask technique,
If the brightness value is 0, the line connecting the excess point where the brightness value exceeds 0,
And when the brightness value is over 0, the smallest brightness value point is connected. The method according to claim 1,
The contact-
And an image converting unit for converting the brightness value of the generated image into a background area having a predetermined brightness value or less using the brightness value information of the generated image. 6. The method of claim 5,
The image converter may include:
Wherein the region having the brightness value of 75 or less is determined to be the background region, and the region is converted into the brightness value of 0. The method according to claim 1,
The command signal includes:
Wherein the controller controls the computer as any one of a cursor movement, a click, an up / down scroll, a left / right scroll, and a hot key. A ground contact finger input method comprising:
An image generation step of generating an image by photographing a ground and a finger of a user;
Extracting a ground area and a finger area from the generated image, and determining whether or not to make contact using the information of each of the overlapping points of the ground area and the finger area; And
And a command signal generating step of generating a command signal using the determined contact / non-contact information,
The contact determination step may include:
A ground boundary extracting step of extracting a ground boundary from the generated image;
A contact area extracting step of extracting a contact area in which the extracted ground boundary line and the finger area overlap;
A ground area removing step of removing the ground area using the extracted ground boundary information;
Extracting an end point of each finger from the image in which the ground area is removed; And
And a contact recognition step of determining that the extracted point is a contact when an end point of the extracted finger overlaps with the contact area. delete 9. The method of claim 8,
Wherein the step of extracting the paper-
Wherein the method comprises using a mask technique for extracting a ground boundary line using brightness value information of an image. 11. The method of claim 10,
The ground boundary line of the mask technique,
If the brightness value is 0, the line connecting the excess point where the brightness value exceeds 0,
And when the brightness value is over 0, the smallest brightness value point is connected. 9. The method of claim 8,
The contact determination step may include:
And converting the generated brightness value information into a background area less than a predetermined brightness value using the brightness value information of the generated image. 13. The method of claim 12,
Wherein the image conversion step comprises:
Wherein a region having a brightness value of 75 or less is determined to be a background region, and the region is converted into a brightness value of 0. 9. The method of claim 8,
The command signal includes:
Wherein the computer is controlled as any one of a cursor movement, a click, an up / down scroll, a left / right scroll, and a hot key.

Images