KR20160023417A - Non-contact multi touch recognition method and system using color image analysis - Google Patents

Abstract

The present invention relates to a method and system for recognizing a non-contact multi-touch by using a color image analysis. The method comprises the following steps: obtaining a color image by photographing a non-contact multi-touch of a user; extracting a skin region and an outer line of the skin region through a color analysis by considering an illuminance change in a color image, extracting a hand region by using complexity of an outer line, and extracting a feature point of the hand region by making an outer line in the extracted hand region to be approximate; generating a hand model with individually modeled fingers by dividing a color image so that each of feature points is included in different regions based on the feature point of the hand region; and recognizing a non-contact multi-touch by using the hand model.

Description

[0001] Non-contact multi-touch recognition method and system using color image analysis [

The present invention relates to a method and system for recognizing non-contact multi-touch of a user through color image analysis. More specifically, the present invention extracts a hand region by color analysis of a color image, and applies Voronoi tessellation And a noncontact multi-touch recognition system using the hand model, a recording medium thereof, and a contactless multi-touch recognition system.

Recently, with the rapid development and generalization of computer information devices (PC, smart phone, etc.), it has become necessary to introduce a more natural input device to replace a mouse and a keyboard. Generally, an input device of a conventional information device includes a keyboard, a mouse, and a touch pen. These input devices have been widely spread as the spread of computer has spread rapidly, and they are the most commonly used computer input devices nowadays. However, recently, due to the miniaturization of the computer and the appearance of the smart phone, the use environment is extremely limited and it is inconvenient to carry. To solve these problems, high performance computer information devices such as smart TV and smart phone are using NUI (Natural User Interface). NUI is a concept for users to get closer to reality in a computer environment. It refers to a next generation input device that uses human hands, face, voice, etc. to replace existing input devices using only a part of the body. As one of the next-generation input devices, human hands are focused on, and various researches are being conducted on the non-contact multi-touch field used as an input device by recognizing hands.

A method of recognizing non-contact multi-touch of a person includes a method using a sensor and a method using a vision. One of the most typical methods of using the sensor is a method using a data glove and a method using a motion recognition sensor. The method using the data globe is easy to measure, and when used in combination with the magnetic sensor, the direction of the hand and the approximate three-dimensional position can be known. Therefore, the data glove is attracting attention as an input device of an information system such as a virtual reality system. Has been actively studied in the field of three-dimensional motion recognition in the way that it can be used as an input device by understanding the motion of the hand as it is. However, the method of using the data glove and the method of using the motion recognition sensor have the inconvenience that the sensor should be attached to the user's body or the surrounding environment. Next, methods of using vision include 3D modeling, skeleton modeling, and extended neural network modeling. The 3D modeling method can model the entire shape of the hand in three dimensions to represent the shape and the motion of the hand in detail, and the skeleton modeling can construct the human hand skeleton model and analyze the hand movements from a geometrical perspective. Finally, the extended neural network modeling method recognizes the hand region by extending the neural network from arbitrary two points in the hand region.

However, the method of using the vision has a problem that the amount of computation increases sharply as the modeling of the hand becomes more specific and is not suitable for real-time processing. On the other hand, a device for recognizing noncontact multi-touch suitable for real-time processing has a problem that the recognition rate is generally lowered.

 B. Loureiro and R. Rodrigues. "Multi-touch as a natural user interface for elders: A survey." Information Systems and Technologies (CISTI), 2011 6th Iberian Conference on. IEEE, pp. 1-6, 2011.

SUMMARY OF THE INVENTION An object of the present invention is to solve the problem of an increase in the amount of calculation in proportion to the embodyment of hand modeling in a method for recognizing contactless multi-touch using a conventional vision, A multi-touch non-contact recognition method and system capable of real-time processing and maintaining the accuracy of a recognition rate in terms of achieving a goal.

According to an aspect of the present invention, there is provided a non-contact multi-touch recognition method comprising: acquiring a color image of a non-contact multi-touch of a user; Extracting a skin region and an outline of the skin region through color analysis in consideration of illumination change in the color image, and extracting a hand region using the complexity of the outline; Extracting a hand feature point of the hand region through approximation of an outline in a hand region of the color image; Generating a hand model individually modeling the fingers by dividing the color image so that each feature point is included in different areas based on the feature points of the hand region; And recognizing the non-contact multi-touch using the hand model.

In the non-contact multi-touch recognition method according to an exemplary embodiment, the extracting of the hand region may include classifying the skin color using a Red-Green-Blue (RGB) color model and a Hue-Saturation- Can be extracted.

In the non-contact multi-touch recognition method according to an exemplary embodiment, the extracting of the hand region may extract an outline of a skin region using a Canny Edge Detector.

In the non-contact multi-touch recognition method according to an exemplary embodiment, the extracting of the hand region may extract only the hand region according to the complexity of the outline analyzed using the spatial frequency technique for the extracted outline.

In the non-contact multi-touch recognition method according to an embodiment, the step of extracting feature points of the hand region may be performed by using a Douglas Peucker algorithm to approximate the hand outline within a predetermined threshold range, Can be extracted.

In the non-contact multi-touch recognition method according to an exemplary embodiment, the step of generating the hand model may divide the color image by applying Voronoi tessellation to the minutiae points of the hand region.

In the above-described embodiment, in the hand model, the feature point may be a Voronoi cell, and the divided region may be a Voronoi polygon.

In the above-described embodiment, all the Voronoi cells do not interfere with each other, and the Voronoi polygon has a maximum area as long as the Voronoi cells do not interfere with each other.

In the above-described embodiment, the step of recognizing the non-contact multi-touch may recognize the non-contact multi-touch using a change in the internal angle of the Voronoi polygon including the minutiae corresponding to the end points of the finger among the minutiae.

According to another aspect of the present invention, there is provided a contactless multi-touch recognition system including: an input unit receiving a color image of a non-contact multi-touch of a user; Extracting a skin region and an outline of the skin region through color analysis in consideration of illumination change in the color image, extracting a hand region using the complexity of the outline, extracting a feature point of the hand region by approximating the outline in the hand region, a feature point extracting unit for extracting a feature point; A hand model generating unit for generating a hand model in which fingers are individually modeled by dividing the color image so that respective feature points are included in different regions based on the feature points of the hand region; And a determination unit for recognizing the non-contact multi-touch using the hand model.

In the non-contact multi-touch recognition system according to another embodiment, the analyzing unit extracts a skin region by classifying skin color using Red-Green-Blue (RGB) color model and HSI (Hue-Saturation-Intensity) Extracts an outline of a skin region using a detector (Canny Edge Detector), and extracts only the hand region according to the complexity of the outline analyzed using a spatial frequency technique for the extracted outline.

In the non-contact multi-touch recognition system according to another embodiment, the analyzing unit can extract feature points of the hand region by approximating the hand outline within a predetermined threshold range using a Douglas Peucker algorithm.

In the non-contact multi-touch recognition system according to another embodiment, the hand model generation unit may divide the color image by applying Voronoi tessellation to the minutiae points of the hand region.

In the above-described embodiment, the feature point is a Voronoi cell, the divided region is a Voronoi polygon, all the Voronoi cells do not interfere with each other, and the Voronoi polygon is a Voronoi polygon, And has a maximum area as long as it does not interfere with each other.

In the above-described embodiment, the determination unit may recognize the non-contact multi-touch using a change in the internal angle of the Voronoi polygon including the feature point corresponding to the end point of the finger among the minutiae.

The present invention also provides a computer-readable recording medium storing a program for causing a computer to execute the non-contact multi-touch recognition methods described above.

According to embodiments of the present invention, the non-contact multi-touch of the user can be recognized so that the user can naturally interact with the displayed object through the analysis of the color image. In addition, by applying the RGB / HSI color model to continuous input images obtained from the camera, it solves the drawbacks that are sensitive to illumination, and by applying the spatial frequency technique, the hand region can be separated regardless of the size of the face region, Contact non-contact multi-touch recognition system using a hand model generated by applying the non-contact multi-touch recognition system.

1 is a flowchart of a non-contact multi-touch recognition method according to an embodiment of the present invention.
Fig. 2 is a diagram showing the spectra of the RGB color model and the HSI color model.
FIG. 3 is a color image (a) and an extracted skin region image (b) input according to the non-contact multi-touch recognition method according to an embodiment of the present invention.
FIG. 4 is a flowchart illustrating the detailed steps of step S120 in FIG.
Fig. 5 is an outline image (c) of the skin region extracted from the extracted skin region image (b) of Fig.
FIG. 6 is a diagram illustrating a process of extracting a hand region using a spatial frequency in a non-contact multi-touch recognition method according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating an example of a process of approximating the hand outline using the Douglas Parker algorithm in the non-contact multi-touch recognition method according to an embodiment of the present invention.
FIG. 8 is a conceptual illustration of a hand region image (a), an outline approximation image (b), and a feature point extraction result (c) extracted according to the noncontact multi-touch recognition method according to an embodiment of the present invention.
FIG. 9 is a diagram showing the minutiae extracted according to the non-contact multi-touch recognition method according to an embodiment of the present invention in the input color image, and showing the respective components of the Voronoi tessellation.
10 is a diagram illustrating three kinds of hand motions recognized according to the non-contact multi-touch recognition method according to an embodiment of the present invention.
11A is an experimental example showing an analysis process according to a non-contact multi-touch recognition method according to an embodiment of the present invention with respect to a color image in which a hand is larger than a face.
FIG. 11B is a screen view showing a result of recognizing noncontact multi-touch in frame order according to an embodiment of the present invention for a color image continuously photographed when a hand is larger than a face.
12A is an example of an analysis process of a non-contact multi-touch recognition method according to an embodiment of the present invention, with respect to a color image in which a hand is smaller than a face.
FIG. 12B is a screen view showing a result of recognition of the noncontact multi-touch according to an embodiment of the present invention in a frame order with respect to a color image continuously photographed when the hand is smaller than the face.
13 is a conceptual diagram illustrating a contactless multi-touch recognition system according to another embodiment of the present invention.

Prior to the description of the concrete contents of the present invention, for the sake of understanding, the outline of the solution of the problem to be solved by the present invention or the core of the technical idea is first given.

A person can control the device by using the contactless multi-touch. The non-contact multi-touch is an interface that enables a device to be operated only by a movement of a finger such as a user's touch on an arbitrary space (for example, air) without touching the device directly, . It can be defined as a visual touch by focusing on touching the device visually. In the embodiment of the present invention, the non-contact multi-touch will be described as a visual touch.

There are two methods of recognizing a non-contact multi-touch (visual touch) of a person include a method using a sensor and a method using a vision. There are 3D modeling, skeleton modeling, and extended neural network modeling in connection with the present invention. These methods have a problem in that the amount of computation increases sharply as the modeling of the hand becomes more concrete and is not suitable for real-time processing. On the other hand, A suitable visual touch system generally has a problem that the recognition rate is lowered. In other words, more specific modeling is required for accurate recognition. As the modeling becomes concrete, the amount of computation increases and a trade-off occurs in which real-time processing becomes impossible.

Accordingly, the present invention proposes a method of recognizing noncontact multi-touch (visual touch) by analyzing a color image using Voronoi tessellation in order to solve such a problem. The non-contact multi-touch recognition method according to the present invention can recognize non-contact multi-touch more stably than existing methods while real-time processing is possible.

1 is a flowchart of a non-contact multi-touch recognition method according to an embodiment of the present invention. Each step will be described in detail below.

In step S110, a color image of the non-contact multi-touch (visual touch) of the user is acquired. It is not a movement that a user directly touches a device but a color image that captures a movement of touching a device by using a hand in an arbitrary space. It is possible to obtain a color image obtained by photographing noncontact multi-touch through various kinds of cameras capable of shooting color images. In the case of acquiring a plurality of color images with a continuous color image, embodiments of the present invention recognize non-contact multi-touch (visual touch) for each color image in real time.

The skin region and the skin region are extracted through the color analysis in consideration of the illumination change in the color image input in operation S120, and the hand region is extracted using the complexity of the skin line. In detail, the skin region is extracted by classifying the skin color using a Red-Green-Blue (RGB) color model and a Hue-Saturation-Intensity (HSI) color model for a color image. One way to classify skin color in a color image, which is an input image, is to clearly define the border of a skin color region in a color space. In the embodiment of the present invention, the RGB color model capable of detecting the simplest and quickest skin color and the HSI color model are used for robustness against variations in illumination.

In the RGB color model, the skin region is defined as Equation (1).

The HSI color model is a color model based on hue, saturation, and intensity. The HSI color model can obtain each component value using the normalized R, G, and B values. The equation for obtaining the color component in the normalized RGB color model is shown in Equation (2).

In the embodiment of the present invention, a hue-resistant color component is used in the HSI color model. The skin area of the color component and the skin area of the RGB color model are used to extract the final skin area which is an intersection of the two. FIG. 2 is a diagram showing spectra of the RGB color model (a) and the HSI color model (b) used in the embodiment of the present invention.

FIG. 3 is a color image (a) and an extracted skin region image (b) input according to the non-contact multi-touch recognition method according to an embodiment of the present invention. (a) is a color image in which a user takes a visual touch with five fingers opened. FIG. 3 (a) is an image of FIG. 3 (b) showing the result of classifying the skin color using the RGB color model and the HSI color model. 3 (b) corresponds to an image of skin region classified only by skin color. Therefore, the result of judging that the face region and the hand region are included in the skin region and the region other than the actual user's skin in the color image is also determined as the skin region I have. In order to separate a hand, which is an object of visual touch, from a color image, only the hand region must be separated within the extracted skin region.

In the embodiment of the present invention, the edge information corresponding to the outline of the skin region is extracted using a Canny Edge Detector in order to separate the hand region from the extracted skin region.

The canyon edge detector provides a method for finding edges satisfying good detection, good localization, and clear response. Detectability refers to the ability to detect all edges in real time. Localness refers to the ability to minimize the difference between an actual edge and a detected edge. Response is the ability to have a single response to each edge. The process of extracting the edge information for separating the edge (the outline of the skin region) is shown in FIG.

FIG. 4 is a flowchart illustrating a process of separating an outline using the canyon edge detector in step S120 of FIG.

The binarized skin region image is input (S121), and Gaussian smoothing filtering is performed in order to minimize the influence of noise on the image (S122).

는 가중치

를 사용하여

위치에 적용되는 가우시안 함수를 의미하며,

는

위치의 픽셀 값을 의미하며,

는 가우시안필터를 적용한 결과를 의미한다. 가우시안 필터링을 적용한

에 1차 미분 연산자를 통한 에지검출은 수학식 4와 같다.Gaussian smoothing filtering can be calculated as shown in Equation (3). here

Weight

use with

The Gaussian function is applied to the position,

The

Pixel value of the position,

Means the result of applying the Gaussian filter. With Gaussian filtering applied

The edge detection through the first-order differential operator is expressed by Equation (4).

는 1차 미분 연산자를 통해 x축 에지를 검출한 결과이며,

는 y축 에지를 검출(S123)한 결과이다. 각 화소에 대한 기울기와 크기는 수학식 5와 같이 계산된다(S124).here

Is the result of detecting the x-axis edge through the first-order differential operator,

Is a result of detecting the y-axis edge (S123). The slope and the magnitude for each pixel are calculated as shown in Equation (5) (S124).

는 최종적으로 임계치(S125)를 통하여 피부영역의 에지정보를 추출(S126)할 수 있다. 도 5는 도 4의 추출된 피부영역 이미지(b)와 그로부터 상기 과정에 의해 추출된 피부영역의 외곽선 이미지(c)를 나타낸 것이다. here,

The edge information of the skin region can be finally extracted (S126) through the threshold value S125. FIG. 5 shows an extracted skin area image (b) of FIG. 4 and an outline image (c) of the skin area extracted from the extracted skin area image (b) of FIG.

Next, the complexity of the outline is analyzed using the spatial frequency technique for the extracted outline, and only the hand area is separated from the skin area.

In the conventional non-contact multi-touch recognition method, the labeling method is mainly used as a method of separating the hand and face from the extracted skin area image. The labeling method can easily extract the hand region by grouping the connected pixels. However, there is a limitation that the hand area must be larger than the face area in order to separate the hand and the face. In order to solve such a constraint, an embodiment of the present invention proposes a method of separating a hand and a face from a skin region extraction image using a spatial frequency technique.

A spatial frequency technique according to an embodiment of the present invention is a method of expressing a change of an image in a two-dimensional space with a frequency characteristic. In the edge image of the extracted skin region, the spatial frequency in the horizontal direction and the spatial frequency in the vertical direction are highest The frequency domain refers to the area in which the edge information is the most complex in the edge image of the skin region. The proposed spatial frequency technique is shown in Equation (6).

이며,

는 가로방향의 공간주파수를 의미하며,

는 세로방향의 공간주파수를 의미한다. 일반적으로 사람의 손영역은 얼굴영역보다 더 복잡한 에지 정보를 가지고 있으므로 가로방향과 세로 방향의 공간 주파수가 가장 높은 부분을 손영역으로 정의함으로써 효과적으로 손영역의 위치를 검출할 수 있다. 비접촉 멀티 터치(비주얼 터치)의 종류에 따라서 복잡도의 정도, 복잡도의 범위를 정의한다면, 본 발명의 상기 실시예에서 설명하고 있는 손가락으로 화면의 특정 영역을 터치하는 듯한 움직임을 식별하는 외에도 특징적인 손가락 제스처를 식별할 수 있을 것이다. here,

Lt;

Means the spatial frequency in the horizontal direction,

Means the spatial frequency in the vertical direction. Generally, since the human hand region has more complex edge information than the face region, it is possible to effectively detect the position of the hand region by defining the portion having the highest spatial frequency in the horizontal direction and the vertical direction as the hand region. If a degree of complexity and a range of complexity are defined according to the type of noncontact multi-touch (visual touch), in addition to identifying the movement of touching a specific area of the screen with the finger described in the above embodiment of the present invention, You will be able to identify the gesture.

FIG. 6 is a diagram illustrating a process of extracting a hand region using a spatial frequency in a non-contact multi-touch recognition method according to an embodiment of the present invention. Referring to FIG. 6, both the case (a) in which the hand region is larger than the face in the skin region and the case (b) in which the hand region is smaller than the face are all analyzed. Unlike the conventional labeling method, in which the hand region must be larger than the face region in order to separate the hand and the face, the method using the spatial frequency according to the embodiment of the present invention, Can be separated.

More specifically, FIG. 6A shows a histogram in which a vertical spatial frequency and a horizontal spatial frequency are applied in a skin region extracted image when a hand region is larger than a face region. It can be seen that the portion with the highest spatial frequency in the longitudinal and transverse directions is a more complex hand region than the face region. FIG. 6 (b) shows the result of applying the spatial frequency when the hand region is smaller than the face. As shown in FIG. 6 (a), it can be seen that the vertical and horizontal spatial frequencies are highest in the hand region. Therefore, it can be seen that the hand region and the face region can be easily separated from the input image by extracting the hand region using the spatial frequency of the embodiment of the present invention. In other words, the complexity of the outline is extracted through the spatial frequency analysis as a hand region with high complexity.

In step S130, the hand feature point of the hand region is extracted through the approximation of the outline in the hand region of the extracted color image. The feature points of the hand region can be extracted by approximating the hand outline within a predetermined threshold range using the Douglas Peucker algorithm. The feature point of the hand region is a significant peak in modeling the finger to recognize the contactless multi-touch. For example, the extracted feature point may be a finger end point or the like.

Since the color image extracted in step S120 has a curve of the hand outline, the Douglas Parker algorithm is first applied to extract the feature points. The Douglas Parker algorithm is an algorithm that approximates complex linear data within a set threshold. FIG. 7 is a diagram illustrating an example of a process of approximating the hand outline using the Douglas Parker algorithm in the non-contact multi-touch recognition method according to an embodiment of the present invention.

FIG. 7A shows the initial threshold setting and shows a simplification process from the start point pt0 to the end point pt15 of the bezel. Set the threshold value and remove pt1-pt14 belonging to the threshold value. In FIG. 7 (b), a threshold value is set to pt15-pt22 as in FIG. 7 (a), and pt16-pt21 belonging to the threshold value is removed. By repeating this process, the minutiae of the hand can finally be extracted. FIG. 8 is a conceptual illustration of a hand region image (a), an outline approximation image (b), and a feature point extraction result (c) extracted according to the noncontact multi-touch recognition method according to an embodiment of the present invention.

의 원소

로 정의한다. 여기서,

의 원소 중

는 손가락 끝점에 해당하는 특징점으로 비주얼 터치 인식과정에서 중요하게 사용된다. 본 발명에 따른 실시예들은 추출된 손영역 특징점을 입력으로 보로노이-테셀레이션 손모델을 생성하고, 최종적으로 손가락 끝점에 해당하는 손모델의 특징점을 이용하여 손가락 끝점의 터치를 인식함으로서 비접촉 멀티 터치(비주얼 터치)를 인식하게 되기 때문이다. As shown in (c) of FIG. 8,

Element of

. here,

Of the elements

Is a feature point corresponding to the finger end point and is used in the visual touch recognition process. Embodiments according to the present invention generate a Voronoi-tessellated hand model by inputting extracted hand region feature points and finally recognize the touch of a finger end point using feature points of a hand model corresponding to a finger end point, Visual touch).

In step S140, a hand model is generated by separately modeling the fingers by dividing the color image so that the respective feature points are included in different regions based on the feature points of the hand region. The color image is segmented by applying Voronoi tessellation to the feature points of the extracted hand region. Voronoi tessellation includes a Voronoi cell, a Voronoi polygon, a Voronoi space, and a Voronoi vertex. In the generated hand model, specific points are represented by Voronoi cells, Cell, and the divided region can be a Voronoi polygon.

가 있을 때, 한 부분집합

에 속하는 원소는 다른 부분 집합 에 속하지 않고, 집합

의 모든 원소는 임의의 한 부분집합

에 반드시 속하는 성질을 가지는 부분집합

을 말한다.Voronoi tessellation is a set of polygons consisting of lines bisecting a line segment with neighbors. Tessellation is a set

When there is a subset

The elements belonging to the < RTI ID = 0.0 > , But not set

All elements of an arbitrary subset

A subset having properties that necessarily belong to

.

의 원소를

라 하자. 집합

의 원소

에 대한 유클리디언 거리함수

일 때, 보로노이 부분집합

는 수학식 7을 만족하는

에 속하는 모든 원소의 집합이다.Voronoi tessellation is defined as follows. set

Element of

Let's say. set

Element of

Euclidean distance function for

, The Voronoi subset

Lt; RTI ID = 0.0 > (7)

Is a set of all elements belonging to.

를 의 보로노이 테셀레이션이라 정의한다. 이때,

의 집합을 보로노이 테셀레이션의 생성자(Generators) 또는 보로노이 셀이라 한다. 보로노이 부분집합

는 보로노이 다각형이라 한다. A set of Voronoi subsets

Is defined as Voronoi tessellation. At this time,

Are called generators or Voronoi cells of Voronoi tessellation. Voronoi subset

Is called a Voronoi polygon.

FIG. 9 is a diagram illustrating components extracted from the input color image according to the non-contact multi-touch recognition method according to an embodiment of the present invention and showing Voronoi tessellation. In Fig. 9, Voronoi tessellation is composed of Voronoi cell, Voronoi polygon, Voronoi space and Voronoi peak. The Voronoi space means a space within the Voronoi polygon, and the Voronoi vertex means a point where an area including an arbitrary vertex meets an area having two or more other vertices.

The Voronoi tessellation algorithm is suitable for expressing a range in which each vertex can have a maximum value without affecting other vertices around an arbitrary starting point. Thus, in the embodiment of the present invention, This is applied to the hand region segmentation step.

를 보로노이 테셀레이션 알고리즘의 입력 데이터로 사용한다. 그리고 입력데이터에 보로노이 테셀레이션을 적용시킨다. 적용된 결과로 컬러 이미지에서 특징점을 기반으로 각각 특징점들이 서로 다른 영역 내에 포함되도록 컬러 이미지를 분할할 수 있다. 분할된 컬러 이미지를 통해 손가락을 개별적으로 모델링한 손모델이 생성되는 것이다. In Fig. 8 (c), the feature point set of the simplified hand region

As the input data of the Voronoi tessellation algorithm. Then apply Voronoi tessellation to the input data. As a result of applying the color image, the color image can be divided so that the feature points are included in different areas based on the feature points in the color image. A hand model is created by individually modeling the fingers through a divided color image.

The hand model generated according to an embodiment of the present invention has the following features. First, the hand model uses the set of feature points of the hand region including the fingertip as input data. Next, all Voronoi cells do not interfere with each other. In addition, the Voronoi polygon has a maximum area as long as the Voronoi cells do not interfere with each other. Finally, the boundary of the Voronoi polygon is the equilibrium point of the facing Voronoi space. This can be defined as the Voronoi rule.

In step S150, the non-contact multi-touch (visual touch) is recognized using the hand model. Contact non-contact multi-touch is recognized using the change in the internal angle of the Voronoi polygon including the feature point corresponding to the end point of the finger among the minutiae points. This is based on the Voronoi rule. More specifically, the angle of the finger bending according to the change of the Voronoi polygon cabinet is calculated to determine the visual touch.

That is, in the embodiment of the present invention, a hand model based on Voronoi tessellation is used to recognize a visual touch by analyzing a color image photographed from a motion (visual touch) . One of the features of the Voronoi tessellation is that the Voronoi polygon, which is one of the features of the Voronoi tessellation mentioned above, utilizes the rule that the Voronoi cells have the largest area unless they interfere with each other.

10 is a view showing three kinds of manually operated operations recognized according to the non-contact multi-touch recognition method according to an embodiment of the present invention. FIG. 10 (a) shows a hand model in which five fingers are extended. Fig. 10 (b) shows a hand gesture in which the index finger is bent, and Fig. 10 (c) shows a finger gesture in which the stop is bent.

을 포함하는 보로노이 다각형은 내각이 좁아지는 특성을 보인다. 이는 손가락의 구부림으로 인해 보로노이 셀

과

의 거리가 근접할수록

과

와의 거리는 멀어지기 때문이다.

과

와의 거리가 멀어질수록 보로노이 다각형의 최대 영역을 확보하기 위해

의 내각이 좁아지게 된다. 보로노이 다각형의 내각 변화는 다음 수학식 8과 같이 제 2 코사인법칙을 이용한다. In Fig. 10 (b), when the index finger is bent,

And the Voronoi polygon, which contains the Voronoi polygon, show the narrowing of the cabinet. Because of the bending of the finger,

and

The closer the distance of

and

And the distance between them becomes far away.

and

The larger the distance between the Voronoi polygon

The cabinet of the present invention becomes narrower. The inner angle change of the Voronoi polygon uses the second cosine law as shown in the following equation (8).

를 포함하는 보로노이 다각형도 내각이 좁아지는 특성을 보인다. 손가락 특징점

을 포함한 보로노이 다각형 내각의 변화를 계산하여 구부림에 따른 각도

를 계산함으로써 화면을 터치하는 듯한 비주얼 터치를 판단한다. 이와 같이 손가락 하나 하나의 움직임을 구분하여 비주얼 터치를 인식하기 때문에 자연스러운 인터페이스로써 역할할 수 있으며, 생성된 손모델에 따라 비주얼 터치를 인식하는 연산이 비교적 기존의 방법에 비하여 간단해 연속적인 컬러 이미지를 입력받아도 실시간 처리가 가능하다. Similarly, in FIG. 10 (c), when the stop finger is bent,

And the Voronoi polygon, which includes the polygon, also shows the narrowing of the cabinet. Finger feature point

The angle of the bending is calculated by calculating the change of the Voronoi polygon cabinet including

To determine a visual touch that seems to touch the screen. In this way, it is possible to play a natural interface by recognizing the visual touch by dividing the movement of each finger, and the operation of recognizing the visual touch according to the generated hand model is relatively simple compared with the conventional method, Real-time processing is possible even if input is received.

Hereinafter, in order to describe the present invention in more detail, two experimental examples to which one embodiment of the present invention is applied will be described. Experimental environment was performed on i5-3570 3.4GHz CPU and 4GB RAM PC, and web camera of 640x480 pixels and 12 frames per second was used.

Experiments were carried out by dividing the hand into two parts: (1) the hand was larger than the face (Experiment 1) and (2) the hand was smaller than the face (Experiment 2) And applied to a natural visual touch interface in a computer environment. The experimental procedure consisted of Voronoi - tessellation hand model generation and hand motion recognition experiment.

11A is an experimental example showing an analysis process according to a non-contact multi-touch recognition method according to an embodiment of the present invention with respect to a color image in which a hand is larger than a face. This is a diagram showing the hand model generation process by applying Voronoi tessellation through Experiment (1). Referring to FIG. 11A, a skin region is extracted for a color image (1) when a hand is larger than a face (2), and only a hand region is extracted by applying a spatial frequency technique according to an embodiment of the present invention ③), extracting the hand feature points by applying the Douglas-Parker algorithm in the extracted hand region (④), and finally generating the Voronoisel model by inputting the extracted feature points (⑤).

FIG. 11B is a screen view showing a result of recognizing noncontact multi-touch in frame order according to an embodiment of the present invention for a color image continuously photographed when a hand is larger than a face. The visual touch is recognized for each of 10 frames, 230 frames, 350 frames, 420 frames, 610 frames, and 705 frames. As a result, a visual touch that draws a figure corresponding to a continuous motion can be estimated.

12A shows an example of an analysis process according to the non-contact multi-touch recognition method according to an embodiment of the present invention with respect to a color image in which a hand is smaller than a face, and shows the process and result of the experiment (2). Referring to FIG. 12A, a skin region is extracted for a color image (1) in which a hand is larger than a face (2), and only a hand region is extracted by applying a spatial frequency technique according to an embodiment of the present invention ③), extracting the hand feature points by applying the Douglas-Parker algorithm in the extracted hand region (④), and finally generating the Voronoisel model by inputting the extracted feature points (⑤).

FIG. 12B is a view showing a result of recognizing a noncontact multi-touch (visual touch) in a frame order according to an embodiment of the present invention, with respect to a color image continuously photographed when a hand is smaller than a face. The visual touch is recognized for each of 30 frames, 270 frames, 380 frames, 510 frames, 660 frames, and 745 frames, and the motion for drawing a figure corresponding to the continuous motion can be estimated as in FIG. 11B.

As can be seen from Figs. 11A, 11B and 12A and 12B, in the case where the hand is larger than the face (Experiment 1) and the case where the hand is smaller than the face (Experiment 2) It can be seen that the embodiment of the present invention proposes a method of recognizing a natural touch operation.

13 is a conceptual diagram showing a contactless multi-touch recognition system 1 according to another embodiment of the present invention. The noncontact multi-touch recognition system 1 receives a color image and outputs a recognized noncontact multi-touch (visual touch). The noncontact multi-touch recognition system 1 includes an input unit 10, an analysis unit 20, a hand model generation unit 30, And a determination unit (40). Each configuration of the non-contact multi-touch recognition system 1 corresponds to each step of the non-contact multi-touch recognition method of FIG. 1, and a detailed description thereof will not be repeated.

The input unit 10 acquires a color image of a non-contact multi-touch (visual touch) of the user. Various types of cameras can be connected to the noncontact multi-touch recognition system 1 by wire or wireless to receive color images. This corresponds to step S110 of FIG.

The analysis unit 20 extracts the skin region and the skin region outline through the color analysis in consideration of the illumination change in the input color image, and extracts the hand region using the complexity of the outline. Skin area is extracted by classifying skin color using Red-Green-Blue (RGB) color model and HSI (Hue-Saturation-Intensity) color model and skin area is extracted using Canny Edge Detector. And the complexity of the outline is analyzed using the spatial frequency technique for the extracted outline. At this time, unlike the conventional non-contact multi-touch recognition method using the labeling method, in the embodiment of the present invention, the hand region and the face region can be accurately separated in the skin region including a plurality of regions by using the spatial frequency technique.

The hand feature point of the hand region is extracted by approximating the outline in the hand region of the color image. The feature points of the hand region are extracted by approximating the hand outline within a predetermined threshold range using the Douglas Peucker algorithm. This is a configuration including step S120 and step S130 in FIG.

The hand model generating unit 30 generates a hand model in which fingers are separately modeled by dividing the color image so that the respective feature points are included in different areas based on the feature points of the hand region. Specifically, the color image can be divided by applying Voronoi tessellation to the feature points of the hand region.

The generated hand models include Voronoi cells, Voronoi polygons, Voronoi spaces and Voronoi peaks. The feature point becomes a Voronoi cell, and the divided area becomes a Voronoi polygon. In this case, all the Voronoi cells do not interfere with each other, and the Voronoi polygon has a maximum area as long as the Voronoi cells do not interfere with each other can do. This corresponds to step S140 in FIG.

The determination unit 40 recognizes the non-contact multi-touch (visual touch) using the hand model. The visual touch is identified by using the change of the internal angle of the Voronoi polygon including the feature point corresponding to the end point of the finger among the feature points. Specifically, it is possible to determine whether or not the noncontact multi-touch corresponds to the movement of touching the screen by calculating the angle of the bending of the finger according to the change of the Voronoi polygon cabinet. This corresponds to step S150 in FIG.

Embodiments of the present invention have proposed a method and system for recognizing a noncontact multi-touch (visual touch) for NUI that can naturally interact with a displayed object based on image information as a color image. Accordingly, when users move their fingers in an arbitrary space, they recognize and interpret the corresponding operation, thereby playing a role of a mouse instructing the operation of the existing computer. In other words, it will be possible to implement a natural interface that can remotely operate the computer without any contact device.

In addition, embodiments of the present invention solve the drawback which is sensitive to illumination by applying the RGB / HSI color model to a continuous input image obtained from a camera and propose a spatial frequency technique, And realizes natural visual touch recognition using hand model generated by Voronoi tessellation algorithm.

Meanwhile, the embodiments of the present invention can be embodied as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device and the like, and also a carrier wave (for example, transmission via the Internet) . In addition, the computer-readable recording medium may be distributed over network-connected computer systems so that computer readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily deduced by programmers skilled in the art to which the present invention belongs.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- And various modifications and changes may be made thereto by those skilled in the art to which the present invention pertains.

Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

1: Non-contact multi-touch recognition system
10: Input unit
20: Analytical Department
30: hand model generating unit
40:

Claims (16)

Acquiring a color image obtained by photographing a noncontact multi-touch of a user;
Extracting a skin region and an outline of the skin region through color analysis in consideration of illumination change in the color image, and extracting a hand region using the complexity of the outline;
Extracting a hand feature point of the hand region through approximation of an outline in a hand region of the color image;
Generating a hand model individually modeling the fingers by dividing the color image so that each feature point is included in different areas based on the feature points of the hand region; And
And recognizing the non-contact multi-touch using the hand model. The method according to claim 1,
The step of extracting the hand region
Wherein the skin region is extracted by classifying the skin color using a Red-Green-Blue (RGB) color model and a Hue-Saturation-Intensity (HSI) color model. The method according to claim 1,
The step of extracting the hand region
And extracting an outline of a skin region using a Canny Edge Detector. The method according to claim 1,
The step of extracting the hand region
And extracting only the hand region according to the complexity of the outline analyzed using the spatial frequency technique for the extracted outline. The method according to claim 1,
The step of extracting feature points of the hand region
Wherein the feature point of the hand region is extracted by approximating the hand outline within a predetermined threshold range using a Douglas Peucker algorithm. The method according to claim 1,
The step of generating the hand model
Wherein the color image is divided by applying Voronoi tessellation to the minutiae points of the hand region. The method according to claim 6,
Wherein the feature point is a Voronoi cell in the hand model, and the divided region is a Voronoi polygon. 8. The method of claim 7,
Wherein all the Voronoi cells do not interfere with each other, and the Voronoi polygon has a maximum area as long as the Voronoi cells do not interfere with each other. 8. The method of claim 7,
The step of recognizing the non-contact multi-touch
Wherein the noncontact multi-touch recognition method recognizes the noncontact multi-touch using a change in the internal angle of the Voronoi polygon including the feature point corresponding to the end point of the finger among the minutiae. A computer-readable recording medium storing a program for causing a computer to execute the method according to any one of claims 1 to 9. An input unit for receiving a color image of a noncontact multi-touch of the user;
Extracting a skin region and an outline of the skin region through color analysis in consideration of illumination change in the color image, extracting a hand region using the complexity of the outline, extracting a feature point of the hand region by approximating the outline in the hand region, a feature point extracting unit for extracting a feature point;
A hand model generating unit for generating a hand model in which fingers are individually modeled by dividing the color image so that respective feature points are included in different regions based on the feature points of the hand region; And
And a determination unit for recognizing the non-contact multi-touch using the hand model. 12. The method of claim 11,
The analyzing unit extracts a skin region by classifying the skin color using a Red-Green-Blue (RGB) color model and a Hue (Saturation-Intensity) color model,
Extracting an outline of a skin region using a Canny Edge Detector and extracting only the hand region according to the complexity of the outline analyzed using a spatial frequency technique with respect to the extracted outline. Touch recognition system. 12. The method of claim 11,
Wherein the analyzing unit approximates the hand outline within a predetermined threshold range using a Douglas Peucker algorithm to extract the feature points of the hand region. 12. The method of claim 11,
Wherein the hand model generating unit divides the color image by applying Voronoi tessellation to the minutiae points of the hand region. 15. The method of claim 14,
The feature point becomes a Voronoi cell, the divided region becomes a Voronoi polygon,
Wherein all the Voronoi cells do not interfere with each other, and the Voronoi polygon has a maximum area as long as the Voronoi cells do not interfere with each other. 15. The method of claim 14,
Wherein the determination unit recognizes the noncontact multi-touch using a change in the internal angle of the Voronoi polygon including the minutiae corresponding to the end point of the finger among the minutiae.

Images