KR101348763B1 - Apparatus and method for controlling interface using hand gesture and computer-readable recording medium with program therefor - Google Patents

Abstract

The present invention provides an apparatus and method for manipulating an interface of an electronic device by recognizing only a movement (gesture) according to a user's hand gesture using an image captured by a camera without directly selecting a selection menu plate displayed on a display screen. At least two gestures according to the number of points of the user's body recognized in the image output from the image acquisition unit; Generating a sequence sequence using a gesture recognition unit for recognizing a gesture, a gesture information comparison unit for detecting gesture information matched with each other in comparison with the recognized gesture among prestored gesture information, and a sequence sequence using the detected gesture information, Operation command message of the electronic device corresponding to the generated sequence Detected in the storage unit that is pre-stored in it consists of including the function processor and outputting to the main control unit of the electronic apparatus.

Description

Apparatus and method for controlling interface using hand gesture and computer-readable recording medium with program therefor}

The present invention relates to an apparatus and a method for manipulating an interface of an electronic device by recognizing a user's movement (gesture) using an image captured by a camera, and in particular, generated according to a hand gesture distinguished by recognizing a user's finger point. An apparatus and method for manipulating an interface of an electronic device by defining operation commands in an ordered sequence.

Recently, electronic devices having a touch panel such as a smart phone are widely used. This touch panel technology does not need to display a pointer on the display, compared to electronic devices such as computers that can be operated with a conventional mouse. That is, the user does not need to move the pointer (cursor of the computer) to a corresponding position (for example, a program icon) in order to operate the electronic device. Instead, the user directly moves a finger over the icon and touches it.

In such a touch panel technology, a pointer movement step, which is essential as an existing electronic device control unit, is unnecessary, so that the electronic device can be quickly operated.

However, despite the above convenience, the touch panel technology cannot be used remotely because the user needs to touch the display surface directly. Therefore, even an electronic device employing a touch panel technology can not but rely on a device such as a conventional remote control for remote operation.

Recently, as a remote electronic control device, a device capable of generating a pointer at an accurate point, such as touch panel technology, that is, a straight line of the user's eyes and fingers is displayed from the captured image after photographing the front of the display using two cameras. A technique for generating a pointer at a point meeting with has been disclosed in the published patent application document (Korean Patent Publication No. 2010-0129629 2010.12.09).

However, these conventional technologies must display a selection menu panel through a remote operation on the display screen, and the user must directly select (touch) the menu panel displayed on the display screen while moving the pointer generated on the screen. Accordingly, the selection menu panel displayed on the display screen must be distinguishable from a distance, and the size of the selection menu panel displayed on the display screen must be accurately selected from a long distance among the menus displayed on the selection menu panel. Should be displayed large and clear.

Accordingly, not only the entire screen or a large portion of the display screen needs to be covered by the selection menu panel, but also the display screen on which the selection menu panel is displayed also needs to be large. At this time, the display screen is large, there is a problem that can not be applied to a small electronic device or difficult to use.

Accordingly, the present invention has been made to solve the above problems, by using only the image captured by the camera without direct selection of the selection menu panel displayed on the display screen to recognize the movement (gesture) according to the user's hand gestures electronic devices It is an object of the present invention to provide an apparatus and method for operating the interface of the.

Another object of the present invention is to define a user's hand gestures, configure a sequence of the defined hand gestures as a matrix, and define a single command by combining the indices of the matrix, thereby performing a command matching a previously stored sequence. The present invention provides an apparatus and method for manipulating an interface of an electronic device for guiding a result of a corresponding command by voice.

In order to achieve the above object, a feature of an interface manipulation apparatus by a hand gesture according to the present invention is an image acquisition unit for continuously outputting a photographed image of a movement of a hand movement based on the selection menu plate, and in the image acquisition unit. A gesture recognition unit for recognizing at least two or more gestures according to the number of points of the user's body recognized in the output image, and a gesture for detecting gesture information matched with each other in comparison with the recognized gestures among previously stored gesture information Generates an order sequence using the information comparison unit and the detected gesture information, detects an operation command message of the electronic device corresponding to the generated order sequence in a storage unit that is stored in advance, and outputs it to the main control unit of the corresponding electronic device. It is configured to include a function processing unit.

Preferably, the apparatus further comprises a screen processing unit which provides the display unit with a selection menu plate for selecting an interface operation of a corresponding electronic device from among previously stored selection menu plate information.

Preferably, the display unit is integrally provided with an electronic device, or is an independent device provided separately.

Preferably, the point recognized by the gesture recognition unit is a fingertip point.

Preferably, the function processing unit configures the sequence as a matrix, and expresses an index of the matrix as gesture information to detect a manipulation command message in the storage unit by combining gesture information of the matrix.

Preferably, the function processing unit may provide an output of the main control unit of the electronic device using an optical wireless communication method or a short-range wired or wireless communication method.

Preferably, the gesture recognition unit may further include a voice guide unit configured to output the recognition result of the gesture recognized according to the number of points to the outside through voice.

Preferably, the voice guide unit outputs an operation command message of the electronic device output from the function processor to the outside through voice.

Preferably, the interface manipulation apparatus includes a selection menu plate information DB for storing selection menu plate information for interface manipulation of at least one or more electronic devices, a gesture information DB for storing gesture information defined according to the number of fingertips, and gesture information. The apparatus further includes a storage unit including a manipulation command information DB for storing a corresponding manipulation command message for each of a plurality of electronic devices according to an order sequence, and a voice information DB storing five hand gesture recognition and voice information corresponding to the manipulation command of the electronic device. It is characterized by.

Features of the interface operation method by the hand gesture according to the present invention for achieving the above object is (A) recognizing at least two or more gestures in accordance with the number of points of the user's body recognized through the movement of the user, (B) repeating step (A) until the recognized gesture becomes the last gesture of a predefined sequence, and (C) generating the sequence using the recognized gesture up to the last And detecting an operation command message of a matched electronic device in comparison with an operation command message of an electronic device previously stored in correspondence with the generated sequence, and (D) converting the operation command message of the detected electronic device into a corresponding message. And outputting to the main control unit of the electronic device.

Preferably, the step of activating the display unit and the camera before step (A), displaying a selection menu panel for selecting a pre-stored interface operation on the display unit, and photographing a user located in front of the camera It characterized in that it further comprises.

Preferably, in step (A), when the user's movement is detected, step (A1) recognizes a fingertip through the detected user's movement, and (A2) five hand gestures according to the number of the recognized fingertips ( Gesture), and (A3) performing gesture recognition by matching with the previously stored gesture information.

Preferably, the step (A1) is a step of determining the change of the fingertip point recognized for a predetermined time, and if the change of the fingertip point, as a result of the determination, determining that the fingertip recognition is successful, and the determination As a result, when there is a change in the fingertip, it is determined that the recognition of the fingertip has failed, and the above step (A) is repeated.

Preferably, when matching is not performed in the step (A3), it is determined that gesture recognition has failed, and the step (A) is repeated.

The method may further include outputting the recognized at least two gestures to the outside by voice.

Preferably, the step (C) comprises the step of constructing the generated sequence sequence in a matrix,

Setting the first recognized gesture as an index of a column using the first recognized gesture, setting the index of a row using the second recognized gesture as a second matrix, the first matrix and the second And detecting an operation command message of one electronic device selected by the matrix combination of the gesture information of the matrix.

The method may further include outputting an operation command message of the electronic device output to the main controller of the electronic device to the outside through a voice.

In addition, the present invention is characterized in that a computer-readable recording medium having recorded thereon a program for executing each step of the method for operating an interface by hand operation described above.

As described above, the apparatus and method for manipulating an interface by hand according to the present invention recognizes only a movement (gesture) according to a user's hand gesture using an image captured by a camera without directly selecting a selection menu panel displayed on a display screen. The interface of the electronic device can be operated to reduce the size of the selection menu displayed on the display screen or to display it with semi-transparency that is somewhat indistinguishable, thereby minimizing the occlusion of the existing image displayed on the display screen. have.

In addition, since the interface of the electronic device can be precisely manipulated through a small selection menu board, it can be easily applied to a small electronic device.

1 is a block diagram showing the configuration of the interface operation apparatus by the hand gesture according to a preferred embodiment of the present invention
2 is a view for explaining a process of classifying and defining five hand gestures using the number of fingertips by using a captured image in the gesture recognition unit according to the present invention.
3 is a flowchart illustrating a method of interface manipulation by hand gesture according to a preferred embodiment of the present invention.
Fig. 4 is a first embodiment for explaining the method of operating an interface by hand operation according to the present invention by arranging a number keypad of a mobile telephone or a landline telephone.
Figure 5 is a second embodiment for explaining the method of operating the interface by the hand gesture according to the present invention applied to the telephone number search keypad using the character input

Other objects, features and advantages of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

Referring to the accompanying drawings, a preferred embodiment of the interface operation apparatus and method by a hand gesture according to the present invention will be described. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. It is provided to let you know. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

1 is a block diagram showing the configuration of an interface manipulation apparatus by hand operation according to a preferred embodiment of the present invention.

As shown in FIG. 1, the interface manipulation apparatus 100 includes a screen processing unit 110, an image acquisition unit 120, a gesture recognition unit 130, a gesture information comparison unit 140, a function processing unit 150, , The voice guide unit 160 and the storage unit 170 are configured.

The screen processing unit 110 provides the display unit 200 with a selection menu board for selecting an interface operation of the electronic device 300 among the selection menu board information previously stored in the storage unit 170. In this case, the display unit 200 may be a display unit integrally provided with the electronic device, but may be an independent device configured separately from the electronic device 300.

The image acquisition unit 120 outputs a photographed image continuously by capturing a motion (gesture) of a hand movement moving on the basis of the selection menu panel displayed on the display unit 200. In this case, the image acquisition unit 120 does not output a fixed captured picture from the captured image, but outputs a video of real-time shooting of continuous hand movements.

The gesture recognition unit 130 recognizes a fingertip point in the image captured by the image acquisition unit 120. That is, as illustrated in FIG. 2, five hand gestures are defined according to the number of recognized fingertip points.

The gesture information comparison unit 140 detects gesture information that is matched with each other in comparison with five hand gestures classified according to the number of the recognized fingertip points among gesture information previously stored in the storage unit 170.

In this case, the gesture information indicates five hand gestures according to the number of fingertip points. When the number of fingertip points is one of five hand gestures as shown in FIG. 2 (a), the gesture information indicates the first gesture, and FIG. 2 (b). As shown in FIG. 2 (c), when the number of fingertips is two, five fingers indicate a second gesture. As shown in FIG. As shown in d), when the number of the fingertips is four among the five hand gestures, the fourth gesture is shown. When the number of the fingertips is five among the five hand gestures as shown in FIG.

The function processor 150 generates an order sequence using the gesture information detected by the gesture information comparison unit 140, and corresponds to the generated order sequence of the electronic device previously stored in the storage unit 170. The operation command message is detected and output to the main controller 300 of the electronic device. In this case, the sequence is configured by a matrix of (X, Y), and the index of the matrix is expressed by gesture information. The gesture information located in the X matrix indicates an order sequence of columns, and the gesture information located in the Y matrix indicates an order sequence of rows, thereby defining one operation command by a combination of gesture information of the X matrix and the Y matrix.

On the other hand, the output from the function processing unit 150 to the main controller 300 of the corresponding electronic device using an optical wireless communication method such as infrared communication used in the general remote control, or using a separate known short-range wired or wireless communication method To provide.

When the voice guide unit 160 recognizes the fingertip point from the image captured by the gesture recognition unit 130, voice recognition results of five hand gestures divided according to the number of the recognized fingertip point to the outside through the speaker Will output In addition, the function processing unit 150 outputs an operation command message of the electronic device, and at this time, outputs the operation command information of the output electronic device to the outside through a speaker.

The storage unit 170 includes a selection menu plate information DB 171 for storing selection menu plate information for interface manipulation of at least one or more electronic devices, and a gesture information DB for storing gesture information defined according to the number of fingertips. 172, an operation command information DB 173 for storing operation command messages corresponding to gesture information for each of a plurality of electronic devices according to a sequence, and voice information corresponding to five hand gesture recognition and operation commands of the electronic device; Voice information DB 174 to be included.

Referring to the accompanying drawings, the operation of the interface operation by the hand motion according to the present invention configured as described above will be described in detail as follows. Like reference numerals in FIG. 1 or FIG. 2 denote the same members performing the same function.

3 is a flowchart illustrating a method of interface manipulation by hand gesture according to a preferred embodiment of the present invention.

Referring to FIG. 3, first, the screen processor 110 and the image acquirer 120 activate the display 200 and the camera (the image acquirer 120) (S10). Accordingly, the display unit 200 displays a selection menu panel for selecting an interface operation previously stored in the storage unit 170, and the camera 120 captures a user located at the front surface (S20).

When the gesture recognition unit 130 detects a user's movement through the captured image (S30), the gesture recognition unit 130 recognizes a fingertip through the detected user's movement, and the gesture information comparing unit 140 recognizes the fingertip of the recognized fingertip. After dividing into five hand gestures (gestures) according to the number, gesture recognition is performed by matching with the gesture information previously stored in the storage unit 170 (S40). In this case, if no match is made, the gesture recognition is treated as a failure.

On the other hand, in the recognition of the gesture, when the fingertips are continuously recognized only by the user's movement, there is a problem that the recognition can be made wrong through the user's meaningless movement. This problem may not ensure the accuracy of gesture recognition. Therefore, it is preferable to determine that the gesture recognition is successful only when the fingertip recognized for a predetermined time for gesture recognition does not change. For example, if the result of the gesture recognition (the number of fingertips) is changed within 15 frames or within 1 second, it is determined that the gesture recognition has failed, and again from the process of detecting the user's movement again in the selection menu screen S20 (S30). To be performed.

And the gestures according to the five hand gestures recognized as described above are output to the outside by a voice (not shown) by the voice guide unit 160, so that the user's gesture is recognized by the interface manipulation apparatus 100. Tells (S50). In this way, by informing the user that the gesture according to the user's movement has been successfully recognized, the user can easily recognize when to input the next hand gesture, and if the voice gesture guides the hand gesture that can come to the next hand gesture, The number of cases is reduced, reducing the amount of unnecessary information and allowing more accurate commands to be entered.

The process (S20 ~ S50) is repeated until the recognized gesture is the last gesture of the predefined sequence (S60). In the present specification, since the sequence sequence is composed of a matrix and the index of the matrix is represented by gesture information, only the gesture information of the X matrix and the Y matrix is needed, and thus the processes S20 to S50 are performed until the second gesture is recognized. Will be repeated. However, this is for the description of the preferred embodiment and not for the limitation, and various embodiments are possible within the scope of the technical idea of the present invention.

Meanwhile, the gestures recognized through the five hand gestures are the first gestures according to the number of fingertip points recognized through the gesture recognition unit 130, as shown in FIGS. 2 (a) to 2 (e). To the fifth gesture.

Therefore, the function processor 150 may generate an order sequence using the gesture recognized until the last time, and compare the generated command sequence with an operation command message of the electronic device previously stored in the storage unit 170 in correspondence to the generated order sequence. It is possible to detect the operation command message of the matching electronic device (S70). In this case, the sequence sequence is configured as a matrix, the first recognized gesture is set as an index of a column using the X matrix, and the second (last) recognized gesture is set as an index of a row using the Y matrix, The operation command message of one electronic device selected by the matrix combination of the gesture information of the X matrix and the Y matrix is detected.

A method of detecting an operation command message of an electronic device through the sequence will be described in detail with reference to various embodiments below.

In addition, the detected operation command message of the electronic device outputs a voice to the speaker (not shown) to the outside by the voice guide unit 160 to inform the user of the command to be performed by the electronic device (S80). In addition, the function processing unit 150 outputs to the corresponding electronic device main control unit 300 (S90).

Meanwhile, a method of detecting an operation command message of the electronic device through the sequence described briefly above will be described in detail with reference to the following preferred embodiments. The following first and second embodiments illustrate only the case where the order sequence is configured as a matrix, but as one preferred embodiment, various embodiments using the order sequence may be possible within the scope of the inventive concept.

1st Example

Fig. 4 is a first embodiment for explaining the method of operating an interface by hand operation according to the present invention in the arrangement of the number keypad of a cellular phone or a landline.

At this time, the direct input of the telephone number can be defined as 10 commands for the input of numbers from 0 to 9, and four commands for dialing, deleting one number, deleting two numbers, and at least fourteen. . And because the phone number must be entered, the number keypad design of the existing mobile phone or landline phone is expressed in a matrix as it is, and is configured in the form that is most easily recognized.

A case of directly performing a telephone number input command through the number keypad design as described above will be described with reference to FIG. 4.

First, when the phone number 1 is input, the gesture to be input may be a combination of indexes of the matrix by continuously inputting <gesture A, gesture A> as shown in FIG. In the case of inputting the number 5, the command may provide a combination of indices of the matrix by continuously inputting <gesture B and gesture B> as shown in FIG. 4 (b). When making a phone call after all phone number input is completed, as shown in FIG. 4 (c), <gesture D and gesture C> may be continuously input to provide an index combination of the matrix.

In addition, since direct input of a telephone number can be recognized by a user incorrectly or by another gesture, it is necessary to define a command corresponding to deletion or termination as well as input.

Therefore, the command is represented as 'Action_op', and can be defined as in Equation 1 below. Therefore, if the index combination of the matrix defined in Equation 1 is input continuously, the defined command is executed.

Second Example

FIG. 5 is a second embodiment for explaining a method of operating an interface by hand operation according to the present invention to a telephone number search keypad using character input.

At this time, the text search should be able to search the desired items with the minimum command, and should be able to move up and down the list. In addition, Hangul can be sorted according to the combination of consonants and vowels, and by using such characteristics, we constructed a matrix for efficient searching.

As shown in FIG. 5 (a), the command composed of <gesture A, gesture A> is selected by searching for the first one among items starting with 'a' in the phone book. In addition, as shown in FIG. 5 (b), the command composed of <gesture B, gesture C> is selected by searching for the first one among items starting with 'ㅅ' (swear), and as shown in FIG. , Command C> will select an item one level below the currently selected item. After successfully searching for all the desired items, if a command consisting of <gesture X, gesture C> is selected as shown in FIG.

On the other hand, the process of the interface operation method by the hand gesture shown in Figure 3 of the present invention can be written as a program that can be executed in a computer. That is, the various steps involved in the method according to the present invention may be stored in a computer readable recording medium. The medium may be a magnetic storage medium such as a ROM, a floppy disk, a hard disk, etc., an optical reading medium such as CD-ROM or DVD, a digital storage medium such as a USB memory, XD), etc.) and carrier waves (e.g., transmission over the Internet).

Although the technical spirit of the present invention described above has been described in detail in a preferred embodiment, it should be noted that the above-described embodiment is for the purpose of description and not of limitation. It will be apparent to those skilled in the art that various modifications may be made without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (18)

An image acquisition unit for outputting a continuous photographed image of a moving gesture based on a selection menu displayed on a display unit;
A gesture recognition unit recognizing at least two or more gestures according to the number of points of the user's body recognized in the image output from the image acquisition unit;
A gesture information comparison unit which detects gesture information matched with each other by comparing with the recognized gesture among gesture information stored in advance;
A function processor configured to generate an order sequence using the detected gesture information, and detect an operation command message of the electronic device corresponding to the generated order sequence in a storage unit that is stored in advance and output the same to the main controller of the corresponding electronic device; Interface operation apparatus by a hand gesture, characterized in that configured. The method of claim 1,
And a screen processing unit which provides a display unit with a selection menu panel for selecting an interface operation of a corresponding electronic device from among selection menu panel information stored in advance. 3. The method of claim 2,
The display unit is a handheld interface operation apparatus, characterized in that the electronic device is provided as an integrated device or separately provided. The method of claim 1,
Points recognized by the gesture recognition unit is an interface manipulation device by a hand gesture, characterized in that the fingertip point. The method of claim 1,
The function processing unit configures the sequence as a matrix, and expresses an index of the matrix as gesture information to detect a single operation command message in the storage unit by combining gesture information of the matrix. . The method of claim 1,
And the function processing unit provides an output of the main controller of the electronic device using an optical wireless communication method or a short-range wireless or wireless communication method. The method of claim 1,
And a voice guide unit which outputs a result of recognizing a gesture recognized by the gesture recognition unit according to the number of points to the outside through a voice. The method of claim 7, wherein
And the voice guide unit outputs an operation command message of the electronic device output from the function processor to the outside through voice. The apparatus of claim 1, wherein the interface manipulation device is
A selection menu plate information DB for storing selection menu plate information for interface operation of at least one or more electronic devices, a gesture information DB for storing gesture information defined according to the number of fingertips, and an operation command message corresponding to the gesture information According to the hand gesture further comprises a storage unit including a manipulation command information DB for storing a plurality of electronic devices according to the column, five hand gesture recognition and a voice information DB for storing voice information corresponding to the operation command of the electronic device Interface operation device. (A) recognizing at least two or more gestures according to the number of points of the user's body recognized through the user's movement,
(B) repeating step (A) until the recognized gesture becomes the last gesture of a predefined sequence;
(C) generating an order sequence using the gesture recognized up to the last, and detecting an operation command message of the matched electronic device in comparison with an operation command message of an electronic device previously stored corresponding to the generated order sequence; Steps,
And (D) outputting an operation command message of the detected electronic device to a corresponding main controller of the electronic device. 11. The method of claim 10,
By activating the display unit and the camera included in the electronic device before the step (A), the display unit displays a selection menu panel for selecting a pre-stored interface operation, the user located in front of the camera Interface operation method according to the hand gesture, characterized in that it further comprises the step of photographing. The method of claim 10, wherein step (A)
(A1) when the movement of the user is detected, recognizing a fingertip through the detected movement of the user;
(A2) dividing into five hand gestures (gestures) according to the number of fingertip points recognized;
(A3) A method of operating an interface by a hand gesture, comprising performing gesture recognition through matching with respect to previously stored gesture information. The method of claim 12, wherein step (A1)
Determining a change in the fingertip recognized for a preset time;
Judging that the recognition of the fingertip is successful when there is no change of the fingertip as a result of the determination;
If the result of the determination is that there is a change in the fingertip, it is determined that the recognition of the fingertip has failed, and the step (A) is repeated. 13. The method of claim 12,
If the matching is not performed in the step (A3), it is determined that the gesture recognition has failed, and repeating the step (A), characterized in that for repeating the step (A). 11. The method of claim 10,
And outputting the recognized at least two or more gestures to the outside by voice. 11. The method of claim 10, wherein step (C)
Organizing the generated sequence into a matrix;
Setting the index of the column using the first recognized gesture as the first matrix,
Setting the second recognized gesture as the index of the row using the second matrix;
And detecting an operation command message of one electronic device selected from a matrix combination of gesture information of the first matrix and the second matrix. 11. The method of claim 10,
And outputting an operation command message of the electronic device output to the main controller of the electronic device to the outside through a voice. A computer-readable recording medium having recorded thereon a program for executing each step of the interface operating method by hand gestures according to any one of claims 10 to 17.

Images