KR20110047735A - Apparatus and method for registering NUI gesture - Google Patents

Abstract

PURPOSE: An apparatus and method for registering an NUI(Natural User Interface) gesture are provided to supply intuitive NUI to a user by implementing a gesture recognition function through a multi-touch input. CONSTITUTION: A coordinate row input unit(211) receives training data coordinate rows for gesture registration, and a coordinate row offset unit(212) respectively offsets the training data coordinate rows. A coordinate row normalizing unit(213) selects a reference coordinate row from the offset training data coordinate rows, and normalizes the training data coordinate rows based on the reference coordinate rows. A profile data generator(214) generates gesture profile data by averaging the normalized training data coordinate rows.

Description

Gesture registration device and method for NUI {Apparatus and method for registering NUI gesture}

The present invention relates to a gesture registration apparatus and method for a natural user interface (NUI), and more particularly to a gesture for a natural user interface (NUI) used on a touch panel or a touch screen. An apparatus and method for registering can be provided.

The touch screen (touch panel) is not only a simple screen display such as a computer, a mobile phone, a PDA, a personal information terminal device, a product display terminal device, a subway or railway ticket vending machine, a bank ATM machine, an industrial monitor, but also a mutual interaction between the device and the user. It is a device that can receive predetermined information by detecting contact of a specific part on the screen in place of an input device such as a keypad, a keyboard, a mouse, and a joystick where an action is required. Such touch screens are mainly used in environments where it is difficult to install input devices such as keypads, keyboards, mice, and joysticks, or where intuitive interaction is required for users, and the demand is gradually increasing.

In this regard, the touch screen is a level that only allows input of a single point on the screen by a single touch method, but recently, a multi-point input technology called multi-touch has started to appear. It is now possible to enter several points on the screen, and the way of interaction between the device and the user is beginning to change dramatically.

On the other hand, Figure 1 is a diagram illustrating the implementation of the Natural User Interface (NUI) using such a touch screen.

Referring to FIG. 1, when the touch recognition device 10 recognizes a touch on a touch screen and transmits corresponding position information (input coordinates) to the NUI device 20, the NUI device 20 based on the touch. The predetermined gesture is recognized as compared with the registered gesture profile. The gesture recognized as described above is recognized as a predetermined command or input by the multi-touch application 30 or the GUI application 35 to perform a corresponding function.

However, according to the prior art, due to the limitations of the input device of the existing touch screen (for example, the existing operating system (OS) mainly supports only a single input), many limitations on the production of content to be reflected or excavated by the user's needs As a result, there was a problem that the needs of the program or application developer could not be faithfully reflected.

In addition, there is a lack of technology for efficiently registering and recognizing gestures through multi-touch, so that program developers have difficulty in implementing NUI based on multi-touch.

The present invention was devised to solve the above problems, and an object of the present invention is to provide an apparatus and method for efficiently registering a gesture for an NUI used on a touch panel or a touch screen.

Another object of the present invention is to provide a gesture registration apparatus and method for NUI that can easily register not only a single touch based gesture but also a multi touch based gesture.

Another object of the present invention is to provide a gesture registration apparatus and method for a program or application developer that can easily implement various functions of NUI programming by recognizing various gestures on a touch panel or a touch screen.

Another object of the present invention is to provide a gesture registration apparatus and method for NUI that can be implemented to replace the portion that was operated as a pointing device for a conventional application with a gesture using a touch.

For this purpose, a gesture registration device for NUI of one embodiment of the present invention includes: a coordinate string input unit for receiving a plurality of training data coordinate strings for gesture registration; A coordinate column offset unit for offsetting the plurality of training data coordinate columns, respectively; A coordinate column normalization unit for selecting a reference coordinate column from the plurality of offset training data coordinate columns and normalizing the plurality of training data coordinate columns based on the reference coordinate column; And a profile data generator for generating gesture profile data by averaging the normalized plurality of training data coordinate columns.

The gesture registration system for an NUI of one embodiment of the present invention includes a touch detector for sensing a touch from an input device of a touch screen and generating input coordinates for a touch position; A location generator for indexing the input coordinates to generate a data coordinate column; And offset each training data coordinate column transmitted from the location generator, and select a reference coordinate column from the plurality of offset training data coordinate columns, and based on the reference coordinate column, the plurality of training data coordinates. And a gesture register for normalizing the columns to generate gesture profile data.

On the other hand, a gesture registration method for an NUI of one embodiment of the present invention comprises the steps of: a) receiving a plurality of training data coordinate columns for gesture registration, and offsetting each of the plurality of training data coordinate columns; b) selecting a reference coordinate column among the offset plurality of training data coordinate columns, and normalizing the plurality of training data coordinate columns based on the reference coordinate column: and c) the normalized plurality of training data coordinate columns Generating a gesture profile data by averaging the mean.

According to the present invention, it is possible to easily and efficiently register a gesture for an NUI used on a touch panel or a touch screen.

In addition, according to the present invention, by extending the single input system to implement the gesture recognition function through the multi-point input (multi-touch) has an effect that can provide an intuitive NUI to the user.

In addition, according to the present invention, by registering various gestures on a touch panel or a touch screen, it is possible to provide a developer environment that can easily implement various functions of NUI programming.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments. For reference, detailed descriptions of well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention will be omitted in the following description.

Prior to the detailed description of the present invention, when defining a gesture in the present invention, this may mean a general operation of a user on a touch screen. That is, a gesture includes a basic operation generally used and an application-specific gesture defined in a specific application.

FIG. 2 illustrates that a predetermined gesture is input on the touch screen, and the thumb and forefinger are pinched on the touch screen (see FIG. 2 (a)) and then pressed (see FIG. 2 (b)). Indicates a gesture of. 3 is a diagram illustrating various gestures that can be used for NUI, the present invention can provide a user-friendly development environment by allowing a developer or a user to define and register various gestures. .

4 illustrates a configuration diagram of a gesture registration system for NUI according to an embodiment of the present invention.

Referring to FIG. 4, the gesture registration system according to the present invention includes a touch recognition device 100 and an NUI device 200, and the touch recognition device 100 includes a touch detector 110, a location generator 120, And a location converter 130, and the NUI device 200 includes a gesture register 210, a gesture database 220, a gesture recognizer 230, a variable adjuster 240, a debugger 250, and the like.

In detail, the touch detector 110 detects a single touch (one-point input) or multi-touch (multi-point input) from an input device of a touch screen and obtains coordinates on a hardware-dependent touch screen. For reference, FIG. 5A illustrates a data format generated by the touch detector, where the center coordinate x, the center coordinate y, the radius x, and the radius y are repeated by the number of touches.

The position generator 120 indexes the data received from the touch detector 110. Here, performing indexing means generating continuous data from coordinate data measured at discrete time and converting the data into relevant data. Then, the operation of pressing, moving, and releasing the finger is managed by the index. For reference, FIG. 5B illustrates a data format generated by the location generator 120, where the center coordinate x, the center coordinate y, the radius x, and the radius y are repeated by the number of touches, and the touch index is an angle. It refers to the number of points, and the action refers to the pressing, moving and releasing action.

The position converter 130 generates the system-dependent coordinates of the hardware-dependent multi-input information obtained from the position generator 120 to suit various monitor settings. In this case, the position converter 130 is implemented to support a variety of resolutions, it is preferable to implement to support multiple monitors, clone monitors. In addition, when the user changes the resolution of the monitor, the error of the multi-touch is not reflected by reflecting the change, and the coordinate transformation equation preferably uses a simple linear mapping equation and adds a coordinate correction equation if necessary.

Meanwhile, the gesture profiler 210 registers the gesture for the NUI through several trainings. That is, the gesture register 210 receives training data coordinate sequences for a plurality of trainings provided from the touch recognition apparatus 100, and generates profile data to be registered as a gesture based on the training data coordinate strings.

For reference, FIG. 6 is a diagram illustrating a user interface for gesture registration that can be used in the present invention. The user presses the start button and then touch dragging the gesture input area. After repeating this several times and pressing the end button, the training data coordinate string is stored. A more detailed description of the gesture register 210 will be described later with reference to FIGS. 7 through 11.

The gesture database 220 stores a gesture profile generated through the gesture register 210, and provides the gesture profile 230 to the gesture recognizer 230 when a touch is recognized on the touch screen.

The gesture recognizer 230 receives a touch data coordinate string provided from the touch recognition apparatus 100 when there is a touch on the touch screen, and compares it with the gesture profile received from the gesture database 220. It recognizes which gesture the touch corresponds to.

In addition, the parameter regulator 240 may adjust values that may be changed in each part, thereby selecting values suitable for a specific device or an application program.

Finally, the debugger 250 may check the progress in each part, such as checking input / output data and checking the state of an applied algorithm, so that problems in the entire development process may be confirmed.

Hereinafter, a gesture registration device and method for NUI according to the present invention will be described in detail with reference to FIGS. 7 to 11. For reference, the gesture registration device (gesture register) according to the present invention may be used as one component of the NUI device as shown in FIG. 4, or may be implemented as a separate independent device.

First, FIG. 7 is a configuration diagram of a gesture registration device for NUI according to an embodiment of the present invention. As shown, the gesture registration apparatus 210 according to an embodiment of the present invention is a coordinate column input unit 211, coordinate column offset unit 212, coordinate column normalization unit 213, profile data generator 214 And the like.

8 is a flowchart illustrating a gesture registration method for an NUI according to an embodiment of the present invention.

Hereinafter, the gesture registration apparatus and method for NUI according to the present invention will be described with reference to FIGS. 7 and 8.

In operation S810, the coordinate string input unit 211 receives a training data coordinate string provided from the touch recognition apparatus 100. Here, the data coordinate column means a set of input coordinates from the start of touch to the end of touch after dragging. As shown in FIG. 9, the single touch recognition device may have a format including the number of input coordinates and the corresponding input coordinates (see FIG. 9A). It may include a number, a corresponding input coordinate, and a touch type tag representing a single or multi-touch.

In step S820, the coordinate column offset unit 212 calculates a minimum value of x and y with respect to the training data coordinate column input through the coordinate column input unit 211, and in step S830, the training data coordinates based on the minimum value of x and y. Offset the column. FIG. 10 is a diagram illustrating this, and FIG. 10 (a) shows that x ₆ and y ₃ are obtained as minimum values of x and y, and FIG. 10 (b) shows minimum values x ₆ and y _{3 of} x and y. It indicates that the training data coordinate column offset.

On the other hand, in step S840 it is determined whether the training has ended, and if the start button is pressed again (see FIG. 6), the process proceeds to step S810 to receive the next training data coordinate string.

After finishing inputting a plurality of training data coordinate columns and pressing the end button, the process proceeds to step S850, and the coordinate column normalization unit 213 performs the shortest training data among the plurality of training data coordinate columns input in steps S810 to S830. Select the coordinate column as the reference coordinate column. For reference, the shortest training data coordinate column means the smallest number of coordinates input during training.

In step S860, the coordinate column normalization unit 213 normalizes the remaining training data coordinate columns based on the shortest training data coordinate column. Here, coordinate column normalization can be performed, for example, by fixing the size of the pattern to a constant size and extracting the necessary coordinate points according to the fixed size. FIG. 11 is a diagram illustrating this. If the first training data coordinate column shown in FIG. 11 (a) is selected as a reference, the second training data coordinate column shown in FIG. 11 (b) has a predetermined pattern size. After changing to (eg, the size of the first training pattern), the coordinate column of the index corresponding to the first training data coordinate column is extracted. For example, if i is the index of the reference coordinate column, j is the index of the target coordinate column, and n is the size of the reference coordinate column, and m is the size of the target coordinate column, the coordinates of the index corresponding to j = i * (m / n) Extracted from the target coordinate column.

Finally, in step S870, the profile data generator 214 generates profile data by averaging the plurality of normalized training data coordinate strings. For reference, the number of coordinates of the profile data of any gesture will be equal to the number of coordinates of the training data selected as a reference.

For reference, in the above-described embodiment, the shortest training data coordinate column is selected as a reference coordinate column for registering a certain gesture. In another embodiment of the present invention, the coordinate number and size of the reference coordinate column are determined in advance, and the reference is used as training data. Normalization of coordinate columns can also be performed.

Meanwhile, the gesture registration method for the NUI according to the present invention may be implemented through a computer readable recording medium including program instructions for performing various computer-implemented operations. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The recording medium may be one specially designed and configured for the present invention, or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. Examples of program instructions include machine language code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter.

Although the present invention has been described in detail with reference to preferred embodiments, it will be apparent to those skilled in the art that the present invention may be embodied in other specific various forms without changing the technical spirit or essential features of the present invention. One embodiment is to be understood in all respects as illustrative and not restrictive.

It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. .

1 is a diagram illustrating an implementation of an NUI using a touch screen.

2 illustrates an example of inputting a predetermined gesture on a touch screen.

3 is a diagram illustrating various gestures that may be used for NUI.

4 is a block diagram of a gesture registration system for NUI according to an embodiment of the present invention.

5 illustrates a data format generated in the touch detector and the position generator, respectively.

6 is a diagram illustrating a user interface for gesture registration that can be used in the present invention.

7 is a block diagram of a gesture registration device for NUI according to an embodiment of the present invention.

8 is a flowchart illustrating a gesture registration method for an NUI according to an embodiment of the present invention.

9 is a diagram illustrating a format of a data coordinate column according to the present invention.

10 is a diagram illustrating an offset of a training data coordinate column according to the present invention.

11 is a diagram illustrating normalization of a training data coordinate column according to the present invention.

Claims (16)

Gesture registration device for NUI (Natural User Interface), A coordinate string input unit configured to receive a plurality of training data coordinate strings for gesture registration; A coordinate column offset unit for offsetting the plurality of training data coordinate columns, respectively; A coordinate column normalization unit for selecting a reference coordinate column from the plurality of offset training data coordinate columns and normalizing the plurality of training data coordinate columns based on the reference coordinate column; And And a profile data generator for generating gesture profile data by averaging the normalized plurality of training data coordinate columns. The method of claim 1, And the coordinate column offset unit calculates minimum values of x and y for each training data coordinate column, and offsets the corresponding training data coordinate column based on the minimum values of x and y. The method according to claim 1 or 2, And the reference coordinate column is a shortest coordinate column among the plurality of offset training data coordinate columns. The method according to claim 1 or 2, The coordinate column normalization unit fixes the size of each training pattern to a predetermined size, and normalizes the training data coordinate column by extracting coordinate points corresponding to the reference coordinate column from the corresponding training data coordinate column according to the fixed size. Gesture registration device. The method according to claim 1 or 2, The format of the training data coordinate column may include a coordinate number, a touch type tag, and a coordinate column represented by a matrix corresponding to the touch type tag. Gesture registration system for Natural User Interface (NUI), A touch detector for sensing a touch from an input device of the touch screen to generate input coordinates for the touch position; A location generator for indexing the input coordinates to generate a data coordinate column; And After each offset of the training data coordinate column transmitted from the position generator, a reference coordinate column is selected from the plurality of offset training data coordinate columns, and the plurality of training data coordinate columns based on the reference coordinate column. A gesture registration system comprising a gesture register for generating gesture profile data by normalizing. The method of claim 6, And the gesture register calculates minimum values of x and y for each training data coordinate column, and offsets the corresponding training data coordinate column based on the minimum values of x and y. The method of claim 6, The gesture register may fix the size of each training pattern to a predetermined size and normalize the training data coordinate column by extracting coordinate points corresponding to the reference coordinate column from the corresponding training data coordinate column according to the fixed size. Registration system. 9. The method according to any one of claims 6 to 8, And a gesture database for storing the generated gesture profile data. 10. The method of claim 9, And a gesture recognizer for recognizing a gesture corresponding to the touch data coordinate string by comparing the touch data coordinate string transmitted from the position generator with gesture profile data stored in the gesture database. As a gesture registration method for the Natural User Interface (NUI), a) receiving a plurality of training data coordinate columns for gesture registration, and offsetting the plurality of training data coordinate columns, respectively; b) selecting a reference coordinate column from the offset plurality of training data coordinate columns, and normalizing the plurality of training data coordinate columns based on the reference coordinate columns: and and c) generating gesture profile data by averaging the normalized plurality of training data coordinate sequences. The method of claim 11, The offset of step a) is, the gesture registration method characterized in that for calculating the minimum value of x, y for each training data coordinate column, and offset the corresponding training data coordinate column based on the minimum value of the x, y. The method of claim 11, And the reference coordinate column is a shortest coordinate column among the plurality of offset training data coordinate columns. 14. The method according to any one of claims 11 to 13, In the coordinate column normalization of step b), the gesture registration is characterized in that the size of each training pattern is fixed to a predetermined size, and a coordinate point corresponding to the reference coordinate column is extracted from the corresponding training data coordinate column according to the fixed size. Way. 14. The method according to any one of claims 11 to 13, d) storing the generated profile data in a gesture database. A computer-readable recording medium having recorded thereon a program for performing the gesture registration method according to any one of claims 11 to 13.

Images