CN102693025B - Touch finger identification method for multi-touch interaction system - Google Patents

Abstract

The invention discloses a touch hand recognition method of a multi-point touch interaction system, which belongs to the technical field of multi-point touch. The method is as follows: 1) Obtain the detected fingers F ₀ , F ₁ , ... F _n in a certain frame; 2) Fit each finger Fj in the frame to an ellipse E _j ; 3) According to each finger The corresponding ellipse and the set finger distance T cluster the fingers in the frame to obtain k-type fingers; 4) Calculate the center of each class H{F ₀ , F ₁ ,...,F _p } and the Convex hull region R of ; 5) Each class and its coverage area of the current frame is taken as a touch hand information (H{F ₀ , F ₁ ,...,F _p }, R). The present invention recognizes the touch hand information with high accuracy and good robustness, which makes the touch hand recognition more accurate in the multi-touch interactive system of the touch screen, and makes the touch screen more practical.

Description

A touch hand recognition method for a multi-touch interactive system

technical field

The invention belongs to the technical field of multi-point touch, and in particular relates to a touch hand recognition method of a multi-point touch interactive system.

Background technique

The multi-touch system has gradually become practical in recent years. In the multi-touch system, users usually use multiple fingers to perform operations, such as sliding, rotating, and translating operations with multiple fingers.

The traditional touch table can only recognize the fingers on the table, and cannot obtain the information of the touching hand and the motion information of the hand. However, the motion of the hand can provide more accurate translation and rotation information of the hand. If the motion information of the touching hand can be obtained directly, it will be convenient for users to use the system, and the harmony of human-computer interaction will be enhanced.

In a multi-touch system, when the desktop is large, the number of users operating and the number of fingers increases, how to effectively and accurately obtain the hand information corresponding to each finger, including whether each touching finger corresponds to the left hand or the right hand, and How the left and right hands move when touching, and the coverage area of the left and right touching hands, these information cannot be directly obtained. Therefore, there is an urgent need for a fast and stable touch hand recognition algorithm.

Contents of the invention

In order to solve the problem of identifying touch hand information in a multi-touch interactive system in the prior art, the purpose of the present invention is to identify touch hand information in a multi-touch interactive system.

In order to achieve the above object, the present invention provides a touch hand recognition method in a multi-touch interactive system. This method clusters the touch points in the initial state, identifies the left and right hands for the same type of touch fingers, and recognizes the left and right hands during the movement of the touch points. Contacts are tracked continuously and clustering results are dynamically adjusted. The maximum inter-finger distance in a single hand is set to T.

Technical scheme of the present invention is:

A touch hand recognition method of a multi-touch interactive system, the steps of which are:

1) Obtain the fingers F ₀ , F ₁ ,...F _n detected in a certain frame; where, each finger F _j ={f ₀ , f ₁ ,...f _m }, f _i is the pixel in the finger; where , m, n are positive integers;

2) fitting each finger F _j in the frame to an ellipse E _j ;

3) According to the ellipse corresponding to each finger and the set finger distance T, the fingers in the frame are clustered to obtain k-type fingers; wherein, k is a natural number;

4) Calculate the center of each class H{F ₀ , F ₁ ,...,F _p } and the convex hull area R of this class; where, p is a positive integer and p≤n;

5) Take each class and its coverage area of the current frame as a touch hand information (H{F ₀ , F ₁ ,...,F _p }, R).

Further, the method of fitting finger F _j to an ellipse E _j is:

1) Calculate the coordinates (x _c , y _c ) _of the center _point of the ellipse according to the coordinates (x i , y _i ) of the pixel point f i in the finger F _j ;

2) Normalize each coordinate of the pixel points in F _j , and calculate the covariance matrix of x _i and y _i after normalization, and obtain the major and minor axes a and b of the ellipse, and the relationship between the major axis of the ellipse and the x axis Angle θ; get the ellipse E _j (x _c , y _c , a, b, θ) fitted by the finger F _j .

Further, the clustering method of said step 3) is:

a) Initialize each finger in the frame as a class H _i ;

b) Calculate the distance L between any finger fitting ellipse E _m in class H _i and the center point L of any finger fitting ellipse E _n in class H _j , if the maximum value of L is less than the set finger distance T, then combine Hi _and H _j merged into one class;

c) Repeat step b) until there is no class that can be merged, and k class fingers are obtained.

Further, according to the center of the class H{F ₀ , F ₁ ,...,F _p } and the center of each finger in the class, the convex hull region R of the class is obtained.

Further, the calculation method of R is: the center of H{F ₀ , F ₁ ,...,F _p } and the center (x _c , y _c ) of each finger fitting ellipse in this class are composed The set of points determines the convex hull region R.

Further, the method for calculating the center of each class H{F ₀ , F ₁ ,...,F _p } is:

1) For the fitting ellipse E(x _c ,y _c ,a,b,θ) of each finger in the class H{F ₀ ,F ₁ ,...,F _p }, sort them counterclockwise according to the value of θ;

2) Calculate the major axis intersection points of all fitted ellipses in this class, and calculate the average value of all intersection points, which is the center (x _h , y _h ) of this class H{F ₀ ,F ₁ ,...,F _p } .

Main content of the present invention is:

Step S1: Assuming that fingers F ₀ , F ₁ ,...F _n are detected in a certain frame, each finger contains F _j ={f ₀ , f ₁ ,...f _m }, and f _i is a pixel in the finger. Fit an ellipse E _j (x _c , y _c , a, b, θ) to each finger F _j , where x _c , y _c are the coordinates of the center point of the ellipse, a, b are the major and minor axes of the ellipse, and θ is the The angle between the major axis and the x-axis.

Our fitting algorithm is as follows, which is low in complexity and easy to implement:

Let the coordinates of a pixel point f _i inside the finger F _j be (x _i , y _i ), and calculate (x _c , y _c ):

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

Normalize f _i :

x _i = x _i - x _c

y _i =y _i -y _c

Calculate the covariance matrix of x _i , y _i :

$\left[\begin{array}{cc}\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right]$

Calculate the eigenvalues of this matrix to get λ ₀ , λ ₁ , (λ ₀ ＞λ ₁ ), and λ ₀ corresponds to the eigenvector (v ₀ , v ₁ ), then

$\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}}$

$\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}$

θ=atan(v ₁ /v ₀ )

Step S2: According to the largest distance T between contacts, cluster F ₀ , F ₁ , ... F _n , and cluster them into H ₀ , H ₁ , ..., H _k , a total of k clusters.

Clustering Algorithm:

1. Initially each finger is a class H _i .

2. For each class H _i , find its nearest class H _j , for any F _m ∈ H _i , F _n ∈ H _j , if |F _m -F _n |<T(|F _m -F _n | represents the finger distance, which refers to the distance between the center point of the ellipse fitted to the touch area), then combine H _i and H _j into one class.

3. Repeat step 2 until there are no more classes to merge.

Step S3: θ in the fitted ellipse E(x _c , y _c , a, b, θ) for each finger in each class H is sorted in a counterclockwise manner. Calculate the intersection of all the major axes of the ellipse in this class, set it as (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), take the center of this class H; The center is (x _h ,y _h ): where

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

Calculate the convex hull region R of the center of H and the center of each finger in class H, representing the area of the touching hand.

Step S4: Take each class as a touching hand in the current frame, and obtain H{F ₀ ,F ₁ ,,F _n },R, as the touching hand information, which includes F ₀ ,F ₁ ,...,F _n fingers, whose centers are x _h , y _h , and the area covered by them is R.

After the initial clustering, each contact point is uniquely marked. During the continuous tracking process, the marked points are continuously tracked according to the closest distance between the two frame points before and after. When a new finger appears or the original finger disappears, the above-mentioned Algorithm recalculation.

The beneficial effects of the present invention are:

The invention provides a method for identifying touch hand information in a multi-point touch interaction system. You only need to get the area information of the touching finger, and quickly fit the ellipse to get the direction of the finger, cluster the fingers, and identify the touching hand information. By fitting ellipses and finger clustering, the touch hand information is recognized with high accuracy and good robustness. The invention enables more accurate identification of touching hands in a multi-point touch interactive system of a touch screen, and makes the touch screen more practical.

Description of drawings

Figure 1 is a schematic diagram of the system layout;

1. Desktop, 2. Infrared lamp, 3. Camera, 4. Projector, 5. Host;

Figure 2 is a finger fitting ellipse, clustering effect diagram;

Fig. 3 is a flowchart of the present invention.

Detailed ways

Various details involved in the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be pointed out that the described embodiments are only intended to facilitate the understanding of the present invention, rather than limiting it in any way.

In order to realize the method of the present invention, it is 1.6G to adopt a CPU during implementation, and internal memory is 512M, the computer of hard disk 320G or other types of computers, and the length of the multi-touch desktop is 1.5 meters, and the width is 1.15 meters. The schematic diagram of the system layout is shown in Figure 1. Figure 2 shows an illustration of our embodiment. It can be seen from Figure 2 that we have correctly clustered the three hands, and the rectangular bounding box represents the recognized result. Adopt Matlab to compile relevant program on computer, the flow chart of the inventive method is referring to Fig. 3, and the concrete implementation steps of the inventive method are as follows:

Step S1: Let the fingers detected in a certain frame be F ₀ , F ₁ ,...F ₁₅ , a total of 15 touching finger areas, where each finger contains F _j ={f ₀ ,f ₁ ,...f _i ,... f _m }, f _i is the pixel in the finger. Fit an ellipse E _j (x _c ,y _c ,a,b,θ) to each finger F _j

The fitting algorithm is as follows:

Let the coordinates of f _i be (x _i , y _i ), and calculate (x _c , y _c ):

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

Normalize f _i :

x _i = x _i - x _c

y _i =y _i -y _c

Calculate the covariance matrix of x _i , y _i :

$\left[\begin{array}{cc}\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}& \mathrm{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right]$

Calculate the eigenvalues of this matrix to get λ ₀ , λ ₁ , (λ ₀ ＞λ ₁ ), and λ ₀ corresponds to the eigenvector (v ₀ , v ₁ ), then

$\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}}$

$\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 2</span>}\sqrt{{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}$

θ=atan(v ₁ /v ₀ )

Step S2: Select the largest distance within the finger T=30 pixels, that is to say, if the distance exceeds 30 pixels, it is considered not to belong to a class; cluster F ₀ , F ₁ ,...F ₁₅ , and cluster them into H ₀ , H ₁ , H ₂ , a total of 3 classes.

Clustering Algorithm:

1. Initially each finger is a class H _i .

2. For each class H _i , find its nearest class H _j , for any F _m ∈ H _i , F _n ∈ H _j , if |F _m -F _n |<T(|F _m -F _n | represents the finger distance, which refers to the distance between the center point of the ellipse fitted to the touch area), then combine H _i and H _j into one class.

3. Repeat step 2 until there are no more classes to merge.

Step S3: For each finger of each class H (ie, H ₀ , H ₁ , H ₂ ), the θ in the fitted ellipse E(x _c ,y _c ,a,b,θ) is given by

Sort in a counterclockwise manner. Calculate the intersection of the major axes of each ellipse, which is the center x _h , y _h of H, and calculate the center of H

and the convex hull region R of the point set formed by the center of each finger in the class H, representing the area of the touching hand.

Step S4: Take each class as a touching hand in the current frame, and obtain H{F ₀ ,F ₁ ,,F _n },R, as the touching hand information, which includes F ₀ ,F ₁ ,,F _n fingers , whose center is x _h , y _h , and the covered area is R.

For the touching finger in Figure 2, fit an ellipse, as shown in Figure 2, each finger area contains an ellipse, cluster the 15 finger areas, and the results are clustered into three categories, which are drawn in a rectangular frame in Figure 2.

The above is only a specific implementation mode in the present invention, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technology can understand the conceivable transformation or replacement within the technical scope disclosed in the present invention. All should be covered within the scope of the present invention, therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (4)

1. A touch hand recognition method of a multi-touch interactive system, the steps of which are: 1) Obtain the fingers F ₀ , F ₁ ,...F _n detected in a certain frame; where, each finger F _j ={f ₀ , f ₁ ,...f _m }, f _i is the pixel in the finger; where , m, n are positive integers; 2) Fitting each finger F _j in the frame to an ellipse E _j ; wherein, the fitting method _{is: fitting an ellipse E j (} x _c ,y _c ,a,b, θ), x _c , y _c are the coordinates of the center point of the ellipse, a, b are the major and minor axes of the ellipse, and θ is the angle between the major axis of the ellipse and the x-axis; let the coordinates of a pixel f _i in the finger F _j be ( x _i ,y _i ), calculate (x _c ,y _c ): Normalize f _i : x _i =xi _{-x c} , y _i =y _i -y _c ; calculate the covariance matrix of x _i , y _i : $\left[\begin{array}{cc}\mathrm{\&Sigma;}{x}_i{x}_i& -\mathrm{\&Sigma;}{x}_i{\mathrm{the\; y}}_i\\ -\mathrm{\&Sigma;}{x}_i{\mathrm{the\; y}}_i& \mathrm{\&Sigma;}{\mathrm{the\; y}}_i{\mathrm{the\; y}}_i\end{array}\right],$ Calculate the eigenvalues of this matrix to get λ ₀ , λ ₁ , (λ ₀ ＞λ ₁ ), and λ ₀ corresponds to the eigenvector (v ₀ , v ₁ ), then θ=atan(v ₁ /v ₀ ); 3) According to the ellipse corresponding to each finger and the finger distance T set, the fingers in the frame are clustered to obtain k-type fingers; wherein, k is a natural number; the clustering method is: a) the frame Each finger in is initially a class H _i ; b) Calculate the distance L between the center point of any finger fitting ellipse E _m in class H _i and any finger fitting ellipse E _n in class H _j , if L is the maximum value is less than the set finger distance T, then merge H _i and H _j into one class; c) repeat step b) until there is no class that can be merged, and k class fingers are obtained; 4) Calculate the center of each class H{F ₀ , F ₁ ,...,F _p } and the convex hull area R of this class; where, p is a positive integer and p≤n; 5) Take each class and its coverage area of the current frame as a touch hand information (H{F ₀ , F ₁ ,...,F _p }, R). 2. The method according to claim 1, wherein the convex hull region R of the class is obtained according to the center of the class H{F ₀ , F ₁ ,...,F _p } and the center of each finger in the class . 3. The method according to claim 2, characterized in that the calculation method of R is: the center of H{F ₀ , F ₁ ,...,F _p } and each finger in the class are fitted to an ellipse The point set consisting of the center (x _c , y _c ) of determines the convex hull region R. 4. The method according to claim 3, characterized in that the method for calculating the center of each class H{F ₀ , F ₁ ,...,F _p } is: 1) For the fitting ellipse E(x _c ,y _c ,a,b,θ) of each finger in the class H{F ₀ ,F ₁ ,...,F _p }, sort them counterclockwise according to the value of θ; 2) Calculate the major axis intersection points of all fitted ellipses in this class, and calculate the average value of all intersection points, which is the center (x _h , y _h ) of this class H{F ₀ ,F ₁ ,...,F _p } .