KR20110047776A - Apparatus and method for recognizing NUI gesture - Google Patents

Abstract

PURPOSE: An apparatus and method for recognizing an NUI(Natural User Interface) gesture are provided to supply an developer environment which is able to implement various kinds of functions for NUI programming by registering and recognizing various gestures on a touch screen or a touch panel. CONSTITUTION: A coordinate row input unit(231) receives a touch data coordinate row, and a coordinate row offset unit(232) offsets the touch data coordinate row. A coordinate row normalizing unit(233) normalizes the gesture profile data coordinate rows and the touch data coordinate rows by selecting the reference coordinate row. A gesture recognizer(234) calculates the distance between the normalized touch data coordinate row and the gesture profile data coordinate row to recognize the gesture corresponding to the shortest gesture profile data coordinate row.

Description

Apparatus and method for recognizing NUI gesture

The present invention relates to a gesture recognition device and method for a natural user interface (NUI), and more particularly, to recognize a gesture for a natural user interface (NUI) used on a touch panel or a touch screen. An apparatus and method are provided.

The touch screen (touch panel) is not only a simple screen display such as a computer, a mobile phone, a PDA, a personal information terminal device, a product display terminal device, a subway or railway ticket vending machine, a bank ATM machine, an industrial monitor, but also a mutual interaction between the device and the user. It is a device that can receive predetermined information by detecting contact of a specific part on the screen in place of an input device such as a keypad, a keyboard, a mouse, and a joystick where an action is required. Such touch screens are mainly used in environments where it is difficult to install input devices such as keypads, keyboards, mice, and joysticks, or where intuitive interaction is required for users, and the demand is gradually increasing.

In this regard, the touch screen is a level that only allows input of a single point on the screen by a single touch method, but recently, a multi-point input technology called multi-touch has started to appear. It is now possible to enter several points on the screen, and the way of interaction between the device and the user is beginning to change dramatically.

On the other hand, Figure 1 is a diagram illustrating the implementation of the Natural User Interface (NUI) using such a touch screen.

Referring to FIG. 1, when the touch recognition device 10 recognizes a touch on a touch screen and transmits corresponding position information (input coordinates) to the NUI device 20, the NUI device 20 based on the touch. The predetermined gesture is recognized as compared with the registered gesture profile. The gesture recognized as described above is recognized as a predetermined command or input by the multi-touch application 30 or the GUI application 35 to perform a corresponding function.

However, according to the prior art, due to the limitations of the input device of the existing touch screen (for example, the existing operating system (OS) mainly supports only a single input), many limitations on the production of content to be reflected or excavated by the user's needs As a result, there was a problem that the needs of the program or application developer could not be faithfully reflected.

In addition, there is a lack of technology for efficiently registering and recognizing gestures through multi-touch, so that program developers have difficulty in implementing NUI based on multi-touch.

The present invention was devised to solve the above problems, and an object of the present invention is to provide an apparatus and method capable of efficiently registering and recognizing a gesture for an NUI used on a touch panel or a touch screen.

Another object of the present invention is to provide a gesture recognition apparatus and method for NUI that can easily register and recognize not only a single touch based gesture but also a multi touch based gesture.

Still another object of the present invention is to provide a gesture recognition apparatus and method for a program or application developer that can easily implement various functions of NUI programming by recognizing various gestures on a touch panel or a touch screen.

It is still another object of the present invention to provide a gesture recognition apparatus and method for NUI that can be implemented to be compatible by replacing a portion operated as a pointing device with a gesture using a touch with respect to an existing application.

For this purpose, a gesture recognition device for NUI of one embodiment of the present invention includes a coordinate string input unit for receiving a touch data coordinate string for gesture recognition; A coordinate column offset unit for offsetting the touch data coordinate column; A coordinate column normalization unit for selecting a reference coordinate column from the offset touch data coordinate column and the plurality of gesture profile data coordinate columns and normalizing the touch data coordinate column and the plurality of gesture profile data coordinate columns based on the reference coordinate column; And a gesture recognizer configured to calculate a distance between the normalized touch data coordinate column and the normalized plurality of gesture profile data coordinate columns, and recognize a gesture corresponding to the gesture profile data coordinate column having the shortest distance. .

The gesture recognition system for NUI of one embodiment of the present invention includes a touch detector for sensing a touch from an input device of a touch screen and generating input coordinates for a touch position; A location generator for indexing the input coordinates to generate a data coordinate column; And after offsetting the touch data coordinate column transmitted from the location generator, selecting a reference coordinate column from the offset touch data coordinate column and a plurality of gesture profile data coordinate columns, and based on the reference coordinate column, touch data. After normalizing a coordinate column and a plurality of gesture profile data coordinate columns, a distance corresponding to the gesture profile data coordinate column having the shortest distance is calculated by calculating a distance between the normalized touch data coordinate column and the normalized plurality of gesture profile data coordinate columns, respectively. Characterized in that it comprises a gesture recognizer for recognizing.

On the other hand, a gesture recognition method for NUI of one embodiment of the present invention, a) receiving a touch data coordinate string for gesture recognition, the step of offsetting the touch data coordinate column; b) selecting a reference coordinate column from the offset touch data coordinate column and the plurality of gesture profile data coordinate columns and normalizing the touch data coordinate column and the plurality of gesture profile data coordinate columns based on the reference coordinate column: and c) And calculating a distance between the normalized touch data coordinate column and the normalized plurality of gesture profile data coordinate columns, and recognizing a gesture corresponding to the gesture profile data coordinate column having the shortest distance.

According to the present invention, it is possible to easily and efficiently register and recognize a gesture for an NUI used on a touch panel or a touch screen.

In addition, according to the present invention, by extending the single input system to implement the gesture recognition function through the multi-point input (multi-touch) has an effect that can provide an intuitive NUI to the user.

In addition, according to the present invention, by registering and recognizing various gestures on a touch panel or a touch screen, it is possible to provide a developer environment that can easily implement various functions of NUI programming.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments. For reference, detailed descriptions of well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention will be omitted in the following description.

Prior to the detailed description of the present invention, when defining a gesture in the present invention, this may mean a general operation of a user on a touch screen. That is, a gesture includes a basic operation generally used and an application-specific gesture defined in a specific application.

FIG. 2 illustrates that a predetermined gesture is input on the touch screen, and the thumb and forefinger are pinched on the touch screen (see FIG. 2 (a)) and then pressed (see FIG. 2 (b)). Indicates a gesture of. 3 is a diagram illustrating various gestures that can be used for NUI, the present invention can provide a user-friendly development environment by allowing a developer or a user to define and register various gestures. .

4 illustrates a configuration diagram of a gesture recognition system for NUI according to an embodiment of the present invention.

Referring to FIG. 4, the gesture recognition system according to the present invention includes a touch recognition device 100 and an NUI device 200, and the touch recognition device 100 includes a touch detector 110, a location generator 120, And a location converter 130, and the NUI device 200 includes a gesture register 210, a gesture database 220, a gesture recognizer 230, a variable adjuster 240, a debugger 250, and the like.

In detail, the touch detector 110 detects a single touch (one-point input) or multi-touch (multi-point input) from an input device of a touch screen and obtains coordinates on a hardware-dependent touch screen. For reference, (a) of FIG. 5 illustrates a data format generated by the touch detector, where the center coordinate x, the center coordinate y, the radius x, and the radius y are repeated by the number of touches.

The position generator 120 indexes the data received from the touch detector 110. Here, performing indexing means generating continuous data from coordinate data measured at discrete time and converting the data into relevant data. Then, the operation of pressing, moving and releasing the finger is managed by the index. For reference, FIG. 5B illustrates a data format generated by the location generator 120, where the center coordinate x, the center coordinate y, the radius x, and the radius y are repeated by the number of touches, and the touch index is an angle. It refers to the number of points, and the action refers to the pressing, moving and releasing action.

The position converter 130 generates the system-dependent coordinates of the hardware-dependent multi-input information obtained from the position generator 120 to suit various monitor settings. In this case, the position converter 130 is implemented to support a variety of resolutions, it is preferable to implement to support multiple monitors, clone monitors. In addition, when the user changes the resolution of the monitor, the error of the multi-touch is not reflected by reflecting the change, and the coordinate transformation equation preferably uses a simple linear mapping equation and adds a coordinate correction equation if necessary.

Meanwhile, the gesture profiler 210 registers the gesture for the NUI through several trainings. That is, the gesture register 210 receives training data coordinate sequences for a plurality of trainings provided from the touch recognition apparatus 100, and generates profile data to be registered as a gesture based on the training data coordinate string.

For reference, FIG. 6 is a diagram illustrating a user interface for gesture registration that can be used in the present invention. The user presses the start button and then touch dragging the gesture input area. After repeating this several times and pressing the end button, the training data coordinate string is stored. For a more detailed description of the gesture register 210, refer to Patent No. 10-2009-0104479, "Gesture Registration Device and Method for NUI," filed with the same person as the present invention, and the patent application is included herein. Are combined.

The gesture database 220 stores a gesture profile generated through the gesture register 210, and provides the gesture profile 230 to the gesture recognizer 230 when a touch is recognized on the touch screen.

The gesture recognizer 230 receives a touch data coordinate string provided from the touch recognition apparatus 100 when there is a touch on the touch screen, and compares it with the gesture profile received from the gesture database 220. It recognizes which gesture the touch corresponds to.

In addition, the parameter regulator 240 may adjust values that may be changed in each part, thereby selecting values suitable for a specific device or an application program.

Finally, the debugger 250 can check the progress in each part, such as checking the input / output data and the state of the applied algorithm, so that problems in the entire development process can be identified.

Hereinafter, a gesture recognition apparatus and method for NUI according to the present invention will be described in detail with reference to FIGS. 7 to 11. For reference, the gesture recognition device (gesture recognizer) according to the present invention may be used as one component of the NUI device as shown in FIG. 4 or may be implemented as a separate independent device.

First, FIG. 7 is a block diagram of a gesture recognition apparatus for NUI according to an embodiment of the present invention. As shown, the gesture recognition apparatus 230 according to an embodiment of the present invention includes a coordinate column input unit 231, a coordinate column offset unit 232, a coordinate column normalization unit 233, a gesture recognition unit 234, and the like. It includes.

8 is a flowchart illustrating a gesture recognition method for NUI according to an embodiment of the present invention.

Hereinafter, a gesture recognition apparatus and method for NUI according to the present invention will be described with reference to FIGS. 7 and 8.

In operation S810, the gesture recognition unit 234 reads the profile data file from the gesture database 220. For reference, the step S810 represents that the gesture recognizing apparatus reads the profile data into the memory in advance in order to recognize the touch. In an actual implementation, after the touch data coordinate string input step S820 and the offset step S830, It may be carried out or in conjunction with these steps.

In operation S820, the coordinate string input unit 231 receives a touch data coordinate string provided from the touch recognition apparatus 100. Here, the data coordinate column means a set of input coordinates from the start of touch to the end of touch after dragging. As shown in FIG. 9, the single touch recognition device may have a format including the number of input coordinates and the corresponding input coordinates (see FIG. 9A). It may include a number, a corresponding input coordinate, and a touch type tag representing a single or multi-touch.

In operation S830, the coordinate column offset unit 232 calculates a minimum value of x and y with respect to the touch data coordinate column input through the coordinate column input unit 231, and offsets the touch data coordinate column based on the x, y minimum value. offset). FIG. 10 is a diagram illustrating this, and FIG. 10 (a) shows that x ₉ and y ₄ are obtained as minimum values of x and y, and FIG. 10 (b) shows minimum values x ₉ and y _{4 of} x and y. It indicates that the touch data coordinate column is offset based on.

In operation S840, the coordinate column normalization unit 233 selects the shortest data coordinate column among at least one profile data coordinate column read in step S810 and the touch data coordinate column obtained in steps S820 and S80 as a reference coordinate column. . For reference, the shortest data coordinate column means that the number of coordinates of the profile data or the touch data is the smallest.

In operation S850, the coordinate column normalization unit 233 normalizes the remaining profile data coordinate columns and / or touch data coordinate columns based on the shortest data coordinate column. Here, coordinate column normalization can be performed, for example, by fixing the size of the pattern to a constant size and extracting the necessary coordinate points according to the fixed size. FIG. 11 is a diagram illustrating this. If the profile data coordinate column shown in FIG. 11 (a) is selected as the reference, the touch data coordinate column shown in FIG. 11 (b) may have a predetermined pattern size (eg, reference). And the coordinate column of the index corresponding to the profile data coordinate column is extracted. For example, if i is the index of the reference coordinate column, j is the index of the target coordinate column, and n is the size of the reference coordinate column, and m is the size of the target coordinate column, the coordinates of the index corresponding to j = i * (m / n) Extracted from the target coordinate column.

Finally, in step S860, the gesture recognizing unit 234 calculates a distance from each of the normalized touch data coordinate strings as well as the normalized at least one profile data coordinate column, respectively, and selects profile data corresponding to the minimum distance. . In this case, the distance between the data coordinate columns may use the sum of the X-axis distance and the Y-axis distance of the corresponding coordinates in order to obtain an actual distance using Pythagorean theorem or to reduce the amount of calculation. The gesture recognition unit 234 recognizes the touch recognized by the touch recognition apparatus 100 as a gesture corresponding to the selected profile data.

For reference, in the above-described embodiment, the shortest data coordinate column is selected as the reference coordinate column among the touch data coordinate column and the at least one profile data coordinate column for the coordinate column normalization. You can also set the and size and normalize the touch coordinate column based on this.

Meanwhile, the gesture recognition method for NUI according to the present invention may be implemented through a computer readable recording medium including program instructions for performing operations implemented by various computers. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The recording medium may be one specially designed and configured for the present invention, or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the present invention has been described in detail with reference to preferred embodiments, it will be apparent to those skilled in the art that the present invention may be embodied in other specific various forms without changing the technical spirit or essential features of the present invention. One embodiment is to be understood in all respects as illustrative and not restrictive.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. .

1 is a diagram illustrating an implementation of an NUI using a touch screen.

2 illustrates an example of inputting a predetermined gesture on a touch screen.

3 is a diagram illustrating various gestures that may be used for NUI.

4 is a block diagram of a gesture recognition system for NUI according to an embodiment of the present invention.

5 illustrates a data format generated in the touch detector and the position generator, respectively.

6 is a diagram illustrating a user interface for gesture registration that can be used in the present invention.

7 is a block diagram of a gesture recognition apparatus for NUI according to an embodiment of the present invention.

8 is a flowchart illustrating a gesture recognition method for NUI according to an embodiment of the present invention.

9 is a diagram illustrating a format of a data coordinate column according to the present invention.

10 illustrates an offset of a touch data coordinate column according to the present invention.

FIG. 11 is a diagram illustrating distance calculation between a profile data coordinate column and a touch data coordinate column according to the present invention. FIG.

Claims (16)

Gesture recognition device for Natural User Interface (NUI), A coordinate column input unit receiving a touch data coordinate column for gesture recognition; A coordinate column offset unit for offsetting the touch data coordinate column; A coordinate column normalization unit selecting a reference coordinate column among the offset touch data coordinate columns and the plurality of gesture profile data coordinate columns and normalizing the touch data coordinate column and the plurality of gesture profile data coordinate columns based on the reference coordinate column; And And a gesture recognizing unit configured to calculate a distance between the normalized touch data coordinate column and the normalized plurality of gesture profile data coordinate columns, and recognize a gesture corresponding to the gesture profile data coordinate column having the shortest distance. The method of claim 1, And the coordinate column offset unit calculates a minimum value of x and y with respect to a touch data coordinate column, and offsets the touch data coordinate column based on the minimum value of x and y. The method according to claim 1 or 2, And the reference coordinate column is a shortest coordinate column among the offset touch data coordinate column and a plurality of gesture profile data coordinate columns. The method according to claim 1 or 2, The gesture column normalization unit fixes the touch pattern to a predetermined size and extracts a coordinate point corresponding to a reference coordinate column from the touch data coordinate column according to the fixed size. Device. The method according to claim 1 or 2, The format of the touch data coordinate column may include a coordinate number, a touch type tag, and a coordinate column represented by a matrix corresponding to the touch type tag. Gesture Recognition System for Natural User Interface (NUI) A touch detector for sensing a touch from an input device of the touch screen to generate input coordinates for the touch position; A location generator for indexing the input coordinates to generate a data coordinate column; And After offsetting the touch data coordinate column transmitted from the location generator, a reference coordinate column is selected from the offset touch data coordinate column and the plurality of gesture profile data coordinate columns, and the touch data coordinates are referenced based on the reference coordinate column. After normalizing the column and the plurality of gesture profile data coordinate columns, the distance between the normalized touch data coordinate column and the normalized plurality of gesture profile data coordinate columns is calculated, respectively, to generate a gesture corresponding to the gesture profile data coordinate column having the shortest distance. A gesture recognition system comprising a gesture recognizer to recognize. The method of claim 6, And the gesture recognizer calculates a minimum value of x and y with respect to the touch data coordinate column, and offsets the touch data coordinate column based on the minimum value of x and y. The method of claim 6, The gesture recognizer fixes the touch pattern to a predetermined size, and normalizes the touch data coordinate column by extracting coordinate points corresponding to the reference coordinate column from the touch data coordinate column according to the fixed size. . 9. The method according to any one of claims 6 to 8, And a gesture database that provides the plurality of gesture profile data. 10. The method of claim 9, Receiving a plurality of training data coordinate columns transmitted from the position generator, offsetting the plurality of training data coordinate columns, and selecting a reference coordinate column from the plurality of offset training data coordinate columns, And a gesture register configured to normalize the plurality of training data coordinate columns based on a reference coordinate column, and to generate gesture profile data by averaging the normalized plurality of training data coordinate columns. As a gesture recognition method for Natural User Interface (NUI), a) receiving a touch data coordinate string for gesture recognition, and offsetting the touch data coordinate string; b) selecting a reference coordinate column from the offset touch data coordinate column and the plurality of gesture profile data coordinate columns, and normalizing the touch data coordinate column and the plurality of gesture profile data coordinate columns based on the reference coordinate column: c) calculating a distance between the normalized touch data coordinate column and the normalized plurality of gesture profile data coordinate columns, and recognizing a gesture corresponding to the gesture profile data coordinate column having the shortest distance. Way. The method of claim 11, The offset of step a), the gesture recognition method, characterized in that to calculate the minimum value of x, y with respect to the touch data coordinate column, and to offset the touch data coordinate column based on the minimum value of the x, y. The method of claim 11, And the reference coordinate column is a shortest coordinate column among the offset touch data coordinate column and a plurality of gesture profile data coordinate columns. 14. The method according to any one of claims 11 to 13, In the coordinate column normalization of step b), the size of the touch pattern is fixed to a predetermined size, and a gesture point corresponding to a reference coordinate column is extracted from the touch data coordinate column according to the fixed size. 14. The method according to any one of claims 11 to 13, The distance calculation may be performed using a Pythagorean theorem or using a sum of the X-axis distance and the Y-axis distance of each coordinate. A computer-readable recording medium having recorded thereon a program for performing a gesture recognition method according to any one of claims 11 to 13.

Images