KR20110032429A - Apparatus and method for recognzing gesture in mobile device - Google Patents

Abstract

PURPOSE: An apparatus for improving gesture recognizing ratio in a mobile device is provided to generate integrated specification information from a plurality of input data, and recognize a gesture. CONSTITUTION: A gesture recognition unit(102) comprises a data acquisition unit(104), a feature extraction unit(106), and an integrated information generator(108). The data acquisition unit acquires a plurality of input data through cameras. The feature extraction unit performs pre-processing about a plurality of input data. The feature information extraction unit classifies the domain capable of gesture recognition. The integrated information generator mixes the extracted feature information to one integrated feature information.

Description

Apparatus and method for recognizing a gesture in a mobile device {APPARATUS AND METHOD FOR RECOGNZING GESTURE IN MOBILE DEVICE}

The present invention relates to an apparatus and method for gesture recognition in a portable terminal, and more particularly, to an apparatus and method for increasing a gesture recognition rate by using integrated feature information using a plurality of input data in a portable terminal.

Recently, with the rapid development of the portable terminal, a portable terminal capable of wireless voice call and information exchange has become a necessity. In the early days of portable terminals, it was recognized that they could simply be carried and wirelessly talked. However, with the development of the technology and the introduction of wireless Internet, the portable terminals are not only for the purpose of telephone calls but also for watching games, satellite broadcasting, and short-range communication. The range of applications such as remote control using a digital camera, image capture by a digital camera, and schedule management is increasing to meet the needs of users.

The digital camera function has a function of capturing a moving picture for a moving subject as well as a still picture. In particular, a portable image capturing device is widely used for capturing an image of a person. .

When taking a still image of a person using the portable terminal as described above, it is not easy to select the position of the person. When a person wants to take a front face while the person is constantly moving, the front face captures the moment when the screen comes in and accurately releases the shutter. There is a problem that it is difficult to press.

In order to solve the above problems, researches on constructing facial expression recognition system and user's recognition / adaptation HCI system have been actively conducted in recent years, but accurate face detection is required for facial recognition. Detecting gestures in an area is never easy.

In order to detect a gesture in a general portable terminal, a specific region is recognized by analyzing an image input through a camera, and a method of extracting feature information from an image and recognizing it through a time series data recognition algorithm is mainly used. Hidden Markov Model Algorithm (HMM) is widely used as a time-series data recognition algorithm.

Assuming that the specific region is composed of a finite state set in the probability space, the hidden Markov model determines the gesture by the transition probability between these states and the observations representing the states when the states are determined. Done.

The above method can be performed by a number of drills, but it is difficult to obtain stable observations due to noise present in the signal. In order to improve this problem, a method using a covariance matrix of images has been proposed, but this method is less stable for gesture recognition with complex shape changes, and it does not contain information with different dimensions such as motion data. Is low. In addition, there is a problem in that the recognition rate is reduced by using this as simple pattern information, which is sensitive to noise.

The present invention was derived to solve the above problems, and an object of the present invention is to provide an apparatus and method for improving a gesture recognition rate in a portable terminal.

Another object of the present invention is to provide an apparatus and method for increasing a gesture recognition rate by using integrated feature information using a plurality of input data in a portable terminal.

It is still another object of the present invention to provide an apparatus and method for accurately classifying observations through effective and stable distance measurement using probability density parameters by mapping feature information integrated in a portable terminal into a probability space. .

According to a first aspect of the present invention for achieving the above objects, a device for recognizing a gesture in a portable terminal is a combination of a plurality of input data to extract a plurality of feature information, the integration of the extracted feature information into one And a gesture recognition unit generating information.

According to a second aspect of the present invention for achieving the above object, a method for recognizing a gesture in a portable terminal is a process of receiving a plurality of input data, the process of extracting a plurality of feature information from the input data And generating the integrated information by integrating the extracted feature information into one.

As described above, the present invention relates to an apparatus and a method for improving a gesture recognition rate in a portable terminal, and by generating feature information extracted from a plurality of input data as one integrated feature information to recognize a gesture, The gesture recognition rate is improved and enables the recognition of complex gestures.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

In the following description, an apparatus and method for generating feature information extracted from a plurality of input data as one integrated feature information in order to improve a gesture recognition rate in a portable terminal will be described. In the following description, the gesture refers to a continuous motion or pose (a specific posture of a human body) of an object having an abstract meaning.

1 is a block diagram illustrating a configuration of a portable terminal for improving a gesture recognition rate according to the present invention.

Referring to FIG. 1, the portable terminal may include a control unit 100, a gesture recognition unit 102, a memory unit 110, an input unit 112, a display unit 114, and a communication unit 116. The gesture recognition unit 102 may include a data acquisition unit 104, a feature extraction unit 106, and an integrated information generation unit 108.

First, the controller 100 of the portable terminal controls the overall operation of the portable terminal. For example, the processor 100 performs processing and control for voice call and data communication, and in addition to the usual function, when the controller 100 performs gesture recognition in the portable terminal, By combining the extracted feature information into one feature information, the feature is processed to increase the gesture recognition rate.

Subsequently, the controller 100 enables the complex gesture recognition by recognizing a gesture using the integrated feature information.

The gesture recognition unit 102 extracts feature information required for gesture recognition under the control of the controller 100, and integrates feature information extracted from a plurality of input data to increase gesture recognition rate according to the present invention. Generate information.

As described above, the gesture recognizing unit 102 integrating the feature information extracted from the plurality of input data into one feature information may process the data obtaining unit 104 to acquire a plurality of input data including the feature information. do.

Thereafter, the gesture recognition unit 102 extracts feature information from a plurality of input data obtained by the data obtaining unit 104, which causes the feature information extracting unit 106 to perform the feature information.

In addition, the gesture recognition unit 102 causes the integrated information generation unit 108 to integrate the feature information obtained by the feature information extraction unit 106 into one feature information.

The data acquisition unit 104 of the gesture recognition unit 102 is a block for acquiring a plurality of input data, and is composed of a general 2D camera and a 3D camera (a stereo camera or a camera using a time of flight (TOF) principle). Acquire three-dimensional image data, three-dimensional image data, and the like.

The feature information extractor 106 of the gesture recognizer 102 performs Gaussian filtering, smoothing, gamma correction, and image on a plurality of input data acquired by the data acquirer 104. After performing preprocessing operations such as image equalization, image recovering or image correction to enable stable signal processing, an area capable of gesture recognition is classified. That is, the feature information extractor 106 classifies specific areas such as a hand area, a face area, and a body area by using color information and distance information from the preprocessed data, and masks the classified parts. Do this. Here, the feature information extracting unit 106 sets the mask and extracts feature information from the set mask area in order to limit the calculation area of the specific part.

The integrated information generation unit 108 combines the feature information extracted by the feature information extraction unit 106 into one integrated feature information. At this time, the integrated information generation unit 108 is used to express the motion, and processes to combine the feature information by combining information necessary for gesture recognition.

The memory unit 110 of the portable terminal includes a read only memory (ROM), a random access memory (RAM), and a flash ROM. The ROM stores microcodes and various reference data of a program for processing and controlling the controller 100 and the gesture recognition unit 102.

The RAM is a working memory of the controller 100, and stores temporary data generated during execution of various programs. In addition, the flash ROM stores a phone book, a calling message, and a receiving message.

The input unit 112 receives data of a still image or a video, and the input unit 112 includes 0 to 9 numeric key buttons, a menu button (menu), a cancel button (erase), a confirmation button, and a call button (TALK). ), A plurality of function keys such as an end button (END), an internet access button, navigation key (or direction key) buttons, and a text input key, and input key input data corresponding to a key pressed by the user to the controller 100. to provide.

The display unit 114 displays state information generated during the operation of the portable terminal, a limited number of characters, a large amount of video and still images, and the like. The display unit 114 may use a color liquid crystal display (LCD), and the display unit 114 may include a touch input device to be used as an input device when applied to a portable terminal of a touch input method. .

The communication unit 116 transmits and receives a radio signal of data input / output through an antenna (not shown). For example, in the case of transmission, after performing channel coding and spreading on the data to be transmitted, RF processing is performed to transmit the data. In the case of reception, the received RF signal is converted into a baseband signal. The baseband signal is de-spreaded and channel decoded to restore data.

The role of the gesture recognition unit 102 may be performed by the control unit 100 of the portable terminal. However, the configuration of the gesture recognition unit 102 is an exemplary configuration for convenience of description, and the scope of the present invention will never be limited. It is not intended to be limiting and those skilled in the art will recognize that various modifications are possible within the scope of the invention. For example, the controller 100 may be configured to process all of them.

2 is a block diagram showing in detail the configuration of the feature information extraction unit according to an embodiment of the present invention.

Referring to FIG. 2, the feature information extractor 200 may include a mask generator 202, a feature extractor 204, and a feature combiner 206.

The mask generator 202 classifies a specific part (eg, a body, a hand, a face, etc.) in order to limit a calculation area in a plurality of input data preprocessed by the gesture recognition unit 102, and is required for gesture recognition. This process separates the area containing the feature information and generates a mask.

The feature extractor 204 checks a mask area set by the mask generator 202, extracts feature information included in the area where the mask is set, and the feature combiner 206 is the feature extractor. Confirm the information for combining the feature information extracted by the into one unified feature information.

The above description has been made of a device for improving a gesture recognition rate in a portable terminal, and in the following description, the gesture recognition rate is generated by generating feature information extracted from a plurality of input data as one integrated feature information using the device according to the present invention. A method for improving will be described.

3 is a diagram illustrating a process of combining feature information extracted from a plurality of input data in an integrated information generating unit according to an exemplary embodiment of the present invention.

Referring to FIG. 3, the integrated information generating unit 108 generates 301 a plurality of feature information 300 extracted by the feature information extracting unit 106 as time series information, and then needs data for expressing motion. Check.

Herein, the data necessary for expressing the motion is data used for recognizing a gesture from the extracted feature information, and in the case of 1-dimensional data, the maximum point of the signal or the amount of change in power spectrum, etc., and in the case of 1-dimensional data, Bean value of histogram of image gradient, gradient value for pixels in x and y direction, or 3D surface normal vector for 3D data vector of surface polygons, tangent vectors of vertex areas, motion information in the case of motion vectors from new locations to new locations, acceleration, consistency of motion vectors between each pixel in the area, and changes in area. Etc. can be used.

Thereafter, the integrated information generation unit 108 integrates the feature information extracted from the plurality of input data using the checked data (eg, the acceleration 303, the slope 305, and the maximum change amount 307). The combination 309 is used as feature information.

4 is a diagram illustrating a process of combining feature information extracted from motion information in an integrated information generating unit according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the integrated information generator 108 selects a specific region from the acquired image data 401 and performs a mask operation (402).

Thereafter, the integrated information generation unit 108 extracts feature information using data (data required for expressing a motion) used to recognize a gesture.

In this case, the integrated information generation unit 108 extracts (403) feature information from the pixels of the masked region 404 using data used to recognize the gesture.

If the data used to recognize the gesture is used as color, edge, and slope, the integrated information generation unit 108 extracts the elements of the color, edge, and slope for all pixels in the mask area of the image data. And extracts the extracted element into one unified information (405).

5 is a flowchart illustrating a process of recognizing a gesture in a portable terminal according to the present invention.

Referring to FIG. 5, the portable terminal first acquires a plurality of input data in step 501. Here, the plurality of input data refers to two-dimensional image data, three-dimensional image data, etc. obtained using a general 2D camera and a 3D camera (a stereo camera or a camera using a time of flight (TOF) principle). Unlike the conventional portable terminal, the portable terminal according to the present invention will increase the gesture recognition rate by using the features extracted from the plurality of input data as one integrated feature information.

In step 503, the portable terminal extracts feature information from the plurality of input data acquired in step 501. In this case, the portable terminal checks an area required for gesture recognition to extract feature information included in a specific region, and extracts feature information by processing a mask in the corresponding region.

In this case, the portable terminal calculates motion information by using previous frame information and current frame information in order to use feature information using shape information and motion information according to the present invention.

In step 505, the portable terminal processes the feature information extracted in the step to be integrated into one feature information.

Herein, the portable terminal integrates feature information extracted from a plurality of input data into one to increase the gesture recognition rate, which will be described in detail with reference to FIGS. 6 to 7.

In step 507, the portable terminal performs a process of classifying a pattern through a learning process.

Here, the portable terminal performs a general learning process for modeling parameters assuming a Gaussian distribution of one feature information by combining feature information extracted from a plurality of input data.

In this case, the portable terminal obtains a principal component by eigen-valu decomposition to classify the feature information combined into the one to perform the learning process, and classifies the information into categories by a clustering method. Thereafter, the portable terminal obtains parameters by assuming a Gaussian distribution for the images belonging to the classified category and records them as a state model. In this case, if the covariance matrix of the principal components obtained by eigenvalue decomposition is used, the calculation becomes simple and the calculation is possible by mapping it to the log-Euclidean space.

The portable terminal performing the learning process as described above stores the pattern of the operation using a model obtained through the learning process as a model. In this case, when a signal to be recognized is newly input, the portable terminal calculates a parameter in the same manner as described above, calculates a parameter value and a distance of the previously stored training data and inputs a model closest to the distance in the pattern space. It is classified into a specific pattern of. This is called a pattern classification process. When the model having the closest distance is difficult to find by the pattern classification process, the signal is not re-learned by random or pattern classification.

In step 509, the portable terminal performs a process of recognizing the corresponding gesture using the classified pattern.

In this case, the portable terminal stores the parameters for each gesture by learning using a plurality of methods for interpreting time series information such as an HMM and an improved HMM, and measures parameter values for a newly inputted input. The model having a may be selected as a gesture of the input signal.

The portable terminal then terminates this algorithm.

6 is a flowchart illustrating a process of extracting feature information from a plurality of input data in a portable terminal according to the present invention.

Referring to FIG. 6, the process of extracting the feature information is for step 503 of FIG. 5 described above, and the portable terminal proceeds to step 601 in order to extract feature information. Do this.

Herein, the portable terminal is different from the conventional portable terminal as described above by using a general 2D camera and a 3D camera (stereo camera or a camera using a TOF (Time of Flight) principle), etc., two-dimensional image data, three-dimensional image Preprocessing tasks such as Gaussian filtering, smoothing, gamma correction, image equalization, image recovering or image correction are performed by receiving a plurality of input data such as data. To enable stable signal processing.

In step 603, the portable terminal classifies a region in which gesture recognition is possible from a plurality of preprocessed input data. That is, the portable terminal classifies specific areas such as hand area, face area, and body area by using color information and distance information from the preprocessed data, and proceeds to step 605 for the specific areas classified in step 603. Perform masking work. Herein, in step 605, the portable terminal sets a mask on a specific part in order to limit the calculation area of the specific part classified in step 603.

In step 607, the portable terminal extracts feature information from the mask area set in step 605 and then performs step 505 of FIG. 5.

In addition, the portable terminal calculates a value using previous frame information and current frame information in order to calculate motion information with a technique of simultaneously integrating shape information and motion information according to the present invention.

7 is a flowchart illustrating a process of integrating feature information extracted from a plurality of input data in a portable terminal according to the present invention.

Referring to FIG. 7, the process of integrating the feature information is the process of step 505 of FIG. 5 described above, and the portable terminal first starts operation 607 of FIG. 6 in step 701 in order to integrate the feature information. After the feature information extracted in the step is generated as time series information, the process proceeds to step 703 to check the data necessary for expressing the motion.

Herein, the data necessary for expressing the motion is data used for recognizing a gesture from the extracted feature information, and in the case of 1-dimensional data, the maximum point of the signal or the amount of change in power spectrum, etc., and in the case of 2-dimensional data, Bean value of histogram of image gradient, gradient value for pixels in x and y direction, or 3D surface normal vector for 3D data vector of surface polygons, tangent vectors of vertex areas, motion information in the case of motion vectors from new locations to new locations, acceleration, consistency of motion vectors between each pixel in the area, and changes in area. And the like, and the portable terminal needs to fill in the motion expression through a number of experiments in order to improve the performance of gesture recognition. A data center may be used by using only the specific data with the data needed to represent motion, or put all the features available to the input principal component analysis selecting the most influential element is greater by a statistical method such as (principle component analysis).

In step 705, the portable terminal combines the feature information extracted from the plurality of input data using the identified data into one integrated feature information, and then proceeds to step 507 of FIG. 5. Perform pattern classification process.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the scope of the following claims, but also by the equivalents of the claims.

1 is a block diagram showing the configuration of a portable terminal for improving a gesture recognition rate according to the present invention;

2 is a block diagram showing in detail the configuration of the feature information extraction unit according to an embodiment of the present invention;

3 is a diagram illustrating a process of combining feature information extracted from a plurality of input data in an integrated information generating unit according to an exemplary embodiment of the present invention;

4 is a diagram illustrating a process of combining feature information extracted from motion information in an integrated information generating unit according to an embodiment of the present invention;

5 is a flowchart illustrating a process of recognizing a gesture in a portable terminal according to the present invention;

6 is a flowchart illustrating a process of extracting feature information from a plurality of input data in a portable terminal according to the present invention;

7 is a flowchart illustrating a process of integrating feature information extracted from a plurality of input data in a portable terminal according to the present invention.

Claims (12)

An apparatus for recognizing a gesture in a portable terminal, And a gesture recognition unit configured to receive a plurality of input data, extract a plurality of feature information, and generate the extracted feature information as integrated information in one. The method of claim 1, The gesture recognition unit, And a plurality of input data including one-dimensional data, two-dimensional data, shape information of three-dimensional data, or image information. The method of claim 1, The gesture recognition unit, Selecting a specific region from the input data, setting a mask on the selected region to limit the calculation region, and then extracting the plurality of feature information from the limited region. The method of claim 3, wherein The gesture recognition unit, And extracting a plurality of pieces of characteristic information from the input data by extracting specific information on motion information using previous frame information and current frame information. The method of claim 1, The gesture recognition unit, And generating the extracted feature information as time series information, and then combining information corresponding to data required for motion representation among the generated time series information to generate the extracted feature information as integrated information. The method of claim 5, The data required for the motion representation is This data is used to recognize gestures, and in the case of 1-dimensional data, it is the maximum point of the signal or the amount of change in the power spectrum. Value, normal vector of 3D surface for 3D data, tangential vector of vertex area, motion vector for new position from previous position for acceleration information, acceleration, consistency of motion vector between each pixel in the area, area of And at least one of the changes. In the method for recognizing a gesture in a portable terminal, Receiving a plurality of input data; Extracting a plurality of feature information from the input data; And generating the integrated information into which the extracted feature information is integrated into one. The method of claim 7, wherein The plurality of input data, And shape information or image information of one-dimensional data, two-dimensional data, and three-dimensional data. The method of claim 7, wherein The process of extracting a plurality of feature information from the received input data, Selecting a specific part from the input data; Limiting the calculation area by setting a mask on the selected area; And extracting feature information from the restricted area. The method of claim 9, The process of extracting a plurality of feature information from the received input data, And extracting specific information on the motion information by using previous frame information and current frame information. The method of claim 7, wherein The process of generating the extracted feature information as a unified information in one, Generating the extracted feature information as time series information; And combining information corresponding to data required for motion representation among the generated time series information. The method of claim 11, The data required for the motion representation is This data is used to recognize gestures, and in the case of 1-dimensional data, it is the maximum point of the signal or the amount of change in the power spectrum. Value, 3D data, normal vector of 3D surface, tangential vector of vertex region, motion vector of movement information from previous position to new position, acceleration, consistency of motion vector between each pixel in the region, area of At least one of the changes.

Images