KR20200080419A - Hand gesture recognition method using artificial neural network and device thereof - Google Patents

Abstract

An embodiment of the present invention relates to a method and an apparatus for recognizing a hand gesture based on an artificial neural network. An objective of the present invention is to provide a method and an apparatus for efficiently recognizing a hand gesture by combining a restricted coulomb energy (REC) neural network algorithm, which may be learned in real-time, and a dynamic time warping (DTW) distance measuring algorithm which may measure the likelihood by reflecting the characteristic of data varying on a time axis. To this end, according to the present invention, disclosed is the method for recognizing the hand gesture based on the artificial neural network, including: a feature data extracting step of extracting feature data from input data of the hand gesture; and a learning and recognizing step of learning and recognizing the hand gesture by inputting feature data, wherein the learning and recognizing step of the hand gesture includes: a hand gesture learning step including a feature data input step to sequentially input the feature data, learning the hand gesture, and a DTW distance measuring step of calculating a distance between the feature data and a neural central point through a DTW distance measuring algorithm; and a hand gesture recognizing step including a DTW distance measuring step for measuring the distance between the feature data and the neural central point through the DTW algorithm, an activated neuron determining step of determining a neuron having a feature present inside a radius of the relevant neuron, as an activated neuron, and a neural level output step of outputting a label of a neuron having the minimum distance value, among neurons activated, and a method thereof.

Description

Hand gesture recognition method using artificial neural network and device thereof

An embodiment of the present invention relates to a method and apparatus for recognizing a hand gesture based on an artificial neural network.

Recently, as the growth of the smart phone market has slowed, wearable devices are emerging as a new growth engine to lead the Internet of Things (IoT) industry. Wearable devices that are attached to or worn on the body are devices that acquire information by processing and processing user-related information or user-related information through sensors, and have been commercialized in various forms such as watches, glasses, and headsets. The application of the wearable device includes safety-security equipment that combines data-based exercise such as sleep time, heart rate, and stress level with data from health management, biometrics, and user behavior, and a human machine that recognizes human movements and handles the machine. interaction). Among them, the HMI system that can freely control the device with only movement without distracting the user's attention is becoming a necessity in various applications such as smart cars and smart homes. Research on a hand gesture recognition system that can be controlled is actively being conducted.

The hand gesture recognition system is divided into a non-contact method of acquiring user's motion information from image information using a camera and a contact method of acquiring data by attaching a sensor or device to the body. The non-contact method using a camera is a method of recognizing using depth information of an image, and various studies have been performed. However, since the video information of the camera is sensitive to the surrounding environment, it is difficult to design a reliable system in all environments. On the other hand, the contact method using a wearable device can obtain relatively accurate motion information because the sensor is directly attached, and has the advantage that there is no restriction on the surrounding environment, such as the field of view and weather, like a camera.

Various sensors are used in the sensor-based hand gesture recognition system. Representatively, there are an inertial measurement sensor, an EMG sensor, an EEG sensor, an ECG sensor, and a radar sensor. Among them, the inertial measurement sensor is not limited by the environment, and is embedded in various wearable devices such as a smart device, a smart watch, and an inertial pen, and thus has characteristics suitable for a hand gesture recognition system as compared to other sensor-based systems. The hand motion data extracted by the inertial measurement sensor has various and extensive patterns according to the user's physical characteristics and environment. Therefore, a machine learning based algorithm capable of real-time learning and recognition is needed to respond to various users as well as the state and environment of the sensor.

Machine learning algorithms used for inertial sensor-based hand gesture recognition include template-based dynamic time warping (DTW) algorithms and multiple layer perceptrons (MLPs). DTW is an algorithm that determines the similarity between two patterns by comparing a representative pattern with a given input pattern, and various application systems have been proposed due to its excellent recognition performance. In addition, it was suggested that the system using MLP supports high recognition performance and can be used for hand gesture recognition applications. However, in the case of DTW and MLP, real-time learning is impossible because it requires a lot of time in the process of requiring a complex optimization technique-based learning algorithm or finding an optimal template.

On the other hand, the RCE (restricted coulomb energy) neural network algorithm has an advantage of being able to support real-time learning because the learning method is relatively simple. The RCE neural network is an algorithm that generates a classification boundary that can be classified in a given feature space based on the input feature data and the distance information between the stored center points. The network structure is not fixed, and the network structure is actively modified according to the learning process. It is possible to support various sensor applications.

However, since the existing RCE neural network algorithm measures distance information with a simple difference value without considering similarity between data on a time axis, in the inertial sensor-based hand gesture recognition application with a large change over time, performance is deteriorated. have.

The above-described information disclosed in the technology that is the background of the present invention is only to improve the understanding of the background of the present invention, and thus may include information that does not constitute the prior art.

The problem to be solved according to an embodiment of the present invention is to provide an efficient hand gesture recognition method and apparatus that combines an RCE neural network algorithm capable of real-time learning and a DTW distance measurement algorithm that measures similarity by reflecting the characteristics of data changing on a time axis. have.

An artificial neural network based hand gesture recognition method according to an embodiment of the present invention includes a feature extraction step of extracting feature data from hand gesture input data; And a learning and recognition step of learning and recognizing hand gestures by inputting feature data, wherein the learning and recognition step comprises: feature data input step of sequentially receiving feature data; And a DWT distance measurement step of calculating a distance between the feature data and the neuron center point using a dynamic time warping (DTW) distance measurement algorithm; And a DWT distance measurement step of calculating a distance between the feature data and the neuron center point using a DTW distance measurement algorithm. An activated neuron determination step of determining a neuron having a characteristic inside a radius of the corresponding neuron as an activated neuron; And a hand gesture recognition step having a neuron label output step of outputting a label of a neuron having a minimum distance value among activated neurons.

The hand gesture learning step may further include a new neuron generation step of generating a new neuron when there is no learned neuron or no activated neuron.

The hand gesture learning step may further include a neuron radius reduction step of reducing the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron.

In the hand gesture learning step, when the label of the input data and the label of the activated neuron are the same, a new neuron may not be generated.

The learning and recognition step may be performed in a restricted coulomb energy (RCE) neural network.

The RCE neural network comprises an input layer composed of characteristic data; It is connected in parallel to the input layer, and consists of a number of neurons having a center point and a radius forming a specific model, and each neuron calculates the distance between the feature data received and the center point of the neuron, and compares the radius with the input data. A hidden layer that determines whether the neuron is included and delivers the result; And an output layer that outputs the class value of the model closest to the input feature data using information received from the hidden layer.

Certain models of neurons can create decision boundaries for classifying data in a feature space in a circle or Gaussian.

을 포함하고, 뉴런

는 아래 수학식을 만족하는 중심점

와 반경

를 포함할 수 있다.The hidden layer is a set of neurons

Containing, neurons

Is the center point that satisfies the following equation

And radius

It may include.

는 각 뉴런의 인덱스이고,

은 학습 과정에서 생성된 뉴런의 총 개수이며,

는 입력된 특징 벡터의 차원이다.here,

Is the index of each neuron,

Is the total number of neurons generated during the learning process,

Is the dimension of the input feature vector.

는, 입력된 특징 벡터의 차원이

일 때, 반경

와 특징 벡터의 차원과 같은

차원의 중심점

를 저장할 수 있다.Each neuron

Is, the dimension of the input feature vector

When, radius

Such as the dimension of the feature vector and

Center of dimension

Can be saved.

차원의 특징 벡터의 개수를

이라 할때, 특징 벡터 집합

을 포함하고, 특징 벡터

는 아래 수학식을 포함할 수 있다.The hidden layer is entered in the learning process

The number of dimensional feature vectors

Set of feature vectors

Features vector

May include the following equation.

는 특징 벡터

를 포함하는

번째 특징 데이터이다.here,

Features vector

Containing

The second feature data.

에 순차적으로 입력되면, 특징 벡터와 뉴런 중심점 사이의 거리 값을 DWT 거리 측정 알고리즘을 이용하여 연산할 수 있다.The hidden layer has a feature vector through each input layer.

If sequentially input to, the distance value between the feature vector and the neuron center point can be calculated using the DWT distance measurement algorithm.

In the DWT distance measurement algorithm, the similarity between at least two sequence data is measured, so that the optimum distance between data can be calculated.

및

이고, 두개의 시퀀스 데이터에 대한 유사도는 아래 수학식을 이용하여 행렬을 만들고, 행렬로부터

값을 구하여 측정될 수 있다.Each of the two sequence data

And

, And the similarity between two sequence data creates a matrix using the following equation,

It can be measured by obtaining a value.

와 반경

를 아래 수학식과 같이 비교하여, 해당 특징 벡터가 각 뉴런

를 활성화시켰는지 여부를 판단할 수 있다.Calculated distance value

And radius

Is compared to the following equation, and the corresponding feature vector is for each neuron.

It can be determined whether or not activated.

에서

를 만족하는 경우 해당 뉴런이 활성화되었다고 판단할 수 있다.Specific neurons

in

If is satisfied, it can be determined that the corresponding neuron is activated.

에 대해 모든 뉴런이

를 만족하지 않는 경우 중심점

를 갖는 새로운 뉴런

을 생성하고, 총 뉴런 개수

을 1 증가시킬 수 있다.Feature vector

About all neurons

If not satisfied, the center point

New neurons having

And the total number of neurons

Can be increased by 1.

Hand gesture input data can be obtained from an inertial measurement sensor.

The gesture input data can be obtained from a smart device, a smart watch or an inertial measurement sensor mounted on an inertial pen.

A method for recognizing a gesture based on an artificial neural network according to an embodiment of the present invention includes a gesture learning step of sequentially receiving feature data and calculating a distance between the feature data and the central point of a neuron with a dynamic time warping (DTW) distance measurement algorithm; And the distance between the feature data and the central point of the neuron is calculated with a DTW distance measurement algorithm, and a neuron having a feature inside the radius of the neuron is determined as an activated neuron, and a label of a neuron having a minimum distance value among the activated neurons is output. It may include a gesture recognition step.

The hand gesture learning step may further include a new neuron generation step of generating a new neuron when there is no learned neuron or no activated neuron.

The hand gesture learning step may further include a neuron radius reduction step of reducing the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron.

In the hand gesture learning step, when the label of the input data and the label of the activated neuron are the same, a new neuron may not be generated.

The hand gesture learning step may be performed in a restricted coulomb energy (RCE) neural network.

An apparatus for recognizing a hand gesture based on an artificial neural network according to an embodiment of the present invention includes a feature extracting unit for extracting feature data from hand gesture input data; And a learning and recognition unit for learning and recognizing hand gestures by inputting feature data, wherein the learning and recognition unit sequentially receives feature data; And a DWT distance measuring unit that calculates a distance between the feature data and the neuron center point using a dynamic time warping (DTW) distance measuring algorithm. And a DWT distance measuring unit that calculates a distance between the feature data and the neuron center point using a DTW distance measurement algorithm. An activated neuron judging unit for determining a neuron having a characteristic inside a radius of the corresponding neuron as an activated neuron; And a hand gesture recognition unit having a neuron label output unit for outputting a label of a neuron having a minimum distance value among activated neurons.

The hand gesture learning unit may further include a new neuron generating unit that generates new neurons when there is no learned neuron or no activated neurons.

The hand gesture learning unit may further include a neuron radius reduction unit that reduces the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron.

The hand gesture learning unit may not generate a new neuron if the label of the input data and the label of the activated neuron are the same.

The learning and recognition unit may include a restricted coulomb energy (RCE) neural network.

The RCE neural network comprises an input layer composed of characteristic data; It is connected in parallel to the input layer, and consists of a number of neurons having a center point and a radius forming a specific model, and each neuron calculates the distance between the feature data received and the center point of the neuron, and compares the radius with the input data. A hidden layer that determines whether the neuron is included and delivers the result; And an output layer that outputs the class value of the model closest to the input feature data using information received from the hidden layer.

Certain models of neurons can create decision boundaries for classifying data in a feature space in a circle or Gaussian.

을 포함하고, 뉴런

는 아래 수학식을 만족하는 중심점

와 반경

를 포함할 수 있다.The hidden layer is a set of neurons

Containing, neurons

Is the center point that satisfies the following equation

And radius

It may include.

는 각 뉴런의 인덱스이고,

은 학습 과정에서 생성된 뉴런의 총 개수이며,

는 입력된 특징 벡터의 차원이다.here,

Is the index of each neuron,

Is the total number of neurons generated during the learning process,

Is the dimension of the input feature vector.

는, 입력된 특징 벡터의 차원이

일 때, 반경

와 특징 벡터의 차원과 같은

차원의 중심점

를 저장할 수 있다.Each neuron

Is, the dimension of the input feature vector

When, radius

Such as the dimension of the feature vector and

Center of dimension

Can be saved.

차원의 특징 벡터의 개수를

이라 할때, 특징 벡터 집합

을 포함하고, 특징 벡터

는 아래 수학식을 포함할 수 있다.The hidden layer is entered in the learning process

The number of dimensional feature vectors

Set of feature vectors

Features vector

May include the following equation.

는 특징 벡터

를 포함하는

번째 특징 데이터이다.here,

Features vector

Containing

The second feature data.

에 순차적으로 입력되면, 특징 벡터와 뉴런 중심점 사이의 거리 값을 DWT 거리 측정 알고리즘을 이용하여 연산할 수 있다.The hidden layer has a feature vector through each input layer.

If sequentially input to, the distance value between the feature vector and the neuron center point can be calculated using the DWT distance measurement algorithm.

In the DWT distance measurement algorithm, the similarity between at least two sequence data is measured, so that the optimum distance between data can be calculated.

및

이고, 두개의 시퀀스 데이터에 대한 유사도는 아래 수학식을 이용하여 행렬을 만들고, 행렬로부터

값을 구하여 측정될 수 있다.Each of the two sequence data

And

, And the similarity between two sequence data creates a matrix using the following equation,

It can be measured by obtaining a value.

와 반경

를 아래 수학식과 같이 비교하여, 해당 특징 벡터가 각 뉴런

를 활성화시켰는지 여부를 판단할 수 있다.Calculated distance value

And radius

Is compared to the following equation, and the corresponding feature vector is for each neuron.

It can be determined whether or not activated.

에서

를 만족하는 경우 해당 뉴런이 활성화되었다고 판단할 수 있다.Specific neurons

in

If is satisfied, it can be determined that the corresponding neuron is activated.

에 대해 모든 뉴런이

를 만족하지 않는 경우 중심점

를 갖는 새로운 뉴런

을 생성하고, 총 뉴런 개수

을 1 증가시킬 수 있다.Feature vector

About all neurons

If not satisfied, the center point

New neurons having

And the total number of neurons

Can be increased by 1.

Hand gesture input data can be obtained from an inertial measurement sensor.

The gesture input data can be obtained from a smart device, a smart watch or an inertial measurement sensor mounted on an inertial pen.

The apparatus for recognizing a hand motion based on an artificial neural network according to an embodiment of the present invention includes a hand motion learning unit that sequentially receives feature data and calculates a distance between the feature data and a central point of a neuron by a dynamic time warping (DTW) distance measurement algorithm; And the distance between the feature data and the central point of the neuron is calculated with a DTW distance measurement algorithm, and a neuron having a feature inside the radius of the neuron is determined as an activated neuron, and a label of a neuron having a minimum distance value among the activated neurons is output. It may include a hand gesture recognition unit.

The hand gesture learning unit may further include a new neuron generating unit that generates new neurons when there is no learned neuron or no activated neurons.

The hand gesture learning unit may further include a neuron radius reduction unit that reduces the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron.

The hand gesture learning unit may not generate a new neuron if the label of the input data and the label of the activated neuron are the same.

The hand gesture learning unit may include a restricted coulomb energy (RCE) neural network.

An embodiment of the present invention can provide an efficient hand gesture recognition method and apparatus combining a RCE neural network algorithm capable of real-time learning and a DTW distance measurement algorithm that measures similarity by reflecting the characteristics of data changing on a time axis.

As an example, an embodiment of the present invention performed a performance evaluation with 5-fold cross validation on a 3D numeric data set, and the performance evaluation result showed a recognition accuracy of approximately 97.10%, similar to the existing RCE neural network algorithm. Veterinary neurons showed an improved recognition accuracy of approximately 4.1%.

In addition, since the embodiment of the present invention can be implemented with low power and low area compared to other artificial neural network structures, system-on-chip (SoC) integration is possible, and thus it is applicable to wearable and IoT devices.

In addition, the embodiments of the present invention are capable of learning hand gestures optimized for a user and an application environment due to a real-time learning function, so that hand gestures can be controlled, such as smart appliances, advanced vehicle driver assistance systems (ADAS), virtual reality, and personal information related services. It can be used in the field.

1 is a schematic diagram showing an example of an RCE neural network structure.
2 is a table showing an example of a distance measurement method by the DTW algorithm.
3 is a learning flowchart illustrating an example of a hand gesture recognition method according to an embodiment of the present invention.
4 is a recognition flowchart illustrating an example of a hand gesture recognition method according to an embodiment of the present invention.
5 is a block diagram showing an example of an artificial neural network based hand gesture recognition device according to an embodiment of the present invention.
6 is a table showing the accuracy of a conventional hand gesture recognition method based on RCE technique.
6 is a table showing the accuracy of the hand gesture recognition method based on the RCE technique according to an embodiment of the present invention.

The present invention as described above will be described in detail through the accompanying drawings and embodiments.

It should be noted that the technical terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. In addition, technical terms used in the present invention should be interpreted as meanings generally understood by a person having ordinary knowledge in the technical field to which the present invention belongs, unless otherwise defined in the present invention. It should not be interpreted as a meaning or an excessively reduced meaning. In addition, when the technical term used in the present invention is a wrong technical term that does not accurately represent the spirit of the present invention, it should be understood as being replaced by a technical term that can be correctly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted as defined in the dictionary or in context before and after, and should not be interpreted as an excessively reduced meaning.

In addition, the singular expression used in the present invention includes a plural expression unless the context clearly indicates otherwise. In the present invention, terms such as “consisting of” or “comprising” should not be construed to include all of the various components, or various stages, described in the invention, including some of the components or some stages It may or may not be construed as further comprising additional components or steps.

Further, terms including ordinal numbers such as first and second used in the present invention may be used to describe elements, but the elements should not be limited by terms. The terms are only used to distinguish one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, but the same or similar elements will be given the same reference numbers regardless of the reference numerals, and redundant descriptions thereof will be omitted.

In addition, in the description of the present invention, when it is determined that detailed descriptions of related known technologies may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention and should not be interpreted as limiting the spirit of the present invention by the accompanying drawings.

In particular, the present invention relates to an artificial neural network learning device having a logical structure, and the artificial neural network and a learning device for learning the same are implemented by hardware components, software components, and/or combinations of hardware components and software components. Can be. For example, the systems, devices, and components described in the embodiments include, for example, a processor commonly referred to as a controller, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and an FPA. (field programmable array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include physical hard disks, solid state drives (SSDs), web hards, mass storage means such as cloud storage, magnetic media such as floppy disks and magnetic tapes ( magnetic media, optical media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and ROM, RAM, Hardware devices specifically configured to store and execute program instructions, such as flash memory, are included. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes made by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

1, an example of an RCE neural network structure 10 is shown.

As illustrated in FIG. 1, the RCE neural network 10 may be composed of a total of three layers: an input layer 12, a hidden layer 13, and an output layer 14. The input layer 12 is composed of feature data into which the feature set 11 is input, and may be connected in parallel to each neuron of the hidden layer 13. The hidden layer 13 may be composed of neurons having a center point and a radius forming a specific model. The model of each neuron can generate a decision boundary for classifying data in a feature space such as a circle or Gaussian. Each neuron calculates the distance between the input feature data and the center point of the neuron and compares it with a radius to determine whether the input data is included in the neuron and deliver the result to the output layer 14. Thereafter, the output layer 14 may output the class value of the model closest to the input feature data by using the received information.

로 구성될 수 있으며, 각 뉴런

는 아래의 수학식 1의 정보(즉, 중심점

와 반경

)를 포함할 수 있다.The hidden layer 13 is a set of neurons, for example,

May consist of, each neuron

Is the information in Equation 1 below (that is, the center point)

And radius

).

는 각 뉴런의 인덱스이고,

은 학습 과정에서 생성된 뉴런의 총 개수이며,

는 입력된 특징 벡터의 차원일 수 있다.here,

Is the index of each neuron,

Is the total number of neurons generated during the learning process,

May be the dimension of the input feature vector.

는, 입력된 특징 벡터의 차원이

일 때, 반경

와 특징 벡터의 차원과 같은

차원의 중심점

를 저장할 수 있다.Also, here, each neuron

Is, the dimension of the input feature vector

When, radius

Such as the dimension of the feature vector and

Center of dimension

Can be saved.

Meanwhile, the learning algorithm by the RCE neural network 10 will be described as follows.

차원의 특징 벡터 수를

이라 할 때, 특징 벡터 집합은

으로 표현할 수 있다. 여기서

는 특징 벡터를 나타내며, 특징 벡터는 수학식 2와 같이 구성될 수 있다.Being entered in the learning process

The number of dimensional feature vectors

This set of feature vectors

Can be expressed as here

Denotes a feature vector, and the feature vector may be configured as in Equation 2.

는 특징 벡터

를 포함하는

번째 특징 데이터일 수 있다.

Features vector

Containing

It may be the second feature data.

에 순차적으로 입력되며 아래 수학식 3과 같이 뉴런 중심점과의 거리 값을 연산할 수 있다. Feature vector for each neuron

It is sequentially input to and the distance value from the neuron center point can be calculated as shown in Equation 3 below.

를 비교함으로써, 해당 특징 벡터가 각 뉴런

영역에 활성화되는지 여부를 판단할 수 있다.Then, the calculated distance value and radius as shown in Equation 4 below

By comparing, the corresponding feature vector is for each neuron.

It can be determined whether or not the region is activated.

에서 수학식 4를 만족하는 경우, 해당 뉴런이 활성화되었다고 판단하는 반면, 특징 벡터

에 대해 모든 뉴런이 수학식 4를 만족하지 못하는 경우 중심점

을 갖는 새로운 뉴런

을 생성하고, 총 뉴런 개수

을 1 증가시킬 수 있다.Specific neurons

In Equation 4, it is determined that the corresponding neuron is activated, whereas the feature vector

For all neurons that do not satisfy Equation 4, the center point

New neurons having

And the total number of neurons

Can be increased by 1.

However, data extracted using the inertial sensor may be shifted or scaled on the time axis depending on the speed and method of performing the hand motion. However, since the algorithm of the RCE neural network 10 measures distance information with a simple difference value as shown in Equation 3, the recognition performance may be deteriorated because it does not correspond to the characteristics of the inertial sensor data.

Therefore, in the embodiment of the present invention, considering the characteristics of the inertial sensor data in hand gesture recognition, a DTW distance measurement algorithm is used instead of the distance measurement method of the general RCE neural network 10.

This DTW distance measurement algorithm is a method of measuring the similarity between two sequence data, and it is an algorithm that finds the optimal distance between data by allowing the data sequence to be expanded or reduced on the time axis. Therefore, the DTW distance measurement algorithm is suitable for measuring the similarity of data shifted or scaled on the time axis.

및

일 수 있고, 두개의 시퀀스 데이터에 대한 유사도는 아래 수학식 5을 이용하여 도 2에 도시된 것과 같은 혼동 행렬(confusion matrix, contingency tabel 또는 an error matrix)을 만들고, 그 행렬로부터

값을 구하여 측정할 수 있다.As an example, the two sequence data are each

And

It can be, the similarity of the two sequence data is to create a confusion matrix (confusion matrix, contingency tabel or an error matrix) as shown in Figure 2 by using Equation 5 below, from the matrix

It can be measured by obtaining a value.

Here, in the case of a hand gesture recognition method using a general DTW distance measurement algorithm, a process of finding one sequence data representing each label is performed in the learning process. Thereafter, in the recognition process, a similarity between sequence data and input data of each label is measured using a DTW distance measurement algorithm, and the label of the most similar sequence data is output.

At this time, in order to obtain high recognition performance, it is necessary to select optimal sequence data that can represent each label in the learning process. That is, since it is necessary to find the optimal sequence data while changing the representative sequence data based on a plurality of hand gesture data prepared in advance, it takes a lot of time and real-time learning is impossible.

Accordingly, an embodiment of the present invention provides a hand gesture recognition method and apparatus that combines an algorithm of a RCE neural network 10 capable of real-time learning with a DTW distance measurement algorithm suitable for measuring similarity of sequence data.

, 학습 뉴런 반경은

로 정의되고,

로부터

까지 반복 수행된다.Referring to FIG. 3, an example of a hand gesture recognition method according to an embodiment of the present invention is illustrated. here,

, The learning neuron radius is

Is defined as,

from

Until it is repeated.

가 입력층(12)을 통하여 은닉층(13)에 순차적으로 전달되고, 이때 데이터 라벨

도 함께 전달된다.In step S1, feature data

Is sequentially transmitted to the hidden layer 13 through the input layer 12, wherein the data label

Is also delivered.

인지 판단된다.

이면 단계(S3)를 수행하고,

이 아니면 단계(S6-1)가 수행된다.In step S2,

Is judged.

Perform the back side step (S3),

Otherwise, step S6-1 is performed.

In step S3, it is determined whether an activated neuron exists. If it is determined that the activated neuron exists, the process is terminated. If it is determined that the activated neuron does not exist, step S4 is performed.

).In step S4, a new neuron is generated (

).

을 수행한다.Then, in step S5,

To perform.

로부터

까지 반복되는데, 우선 단계(S6-1)에서 DWT 기반 거리 측정이 수행된다(

). 단계(S6-2)에서, 뉴런 중심점 거리가 반경 내에 존재하는지 판단한다(

). 뉴런 중심점 거리가 반경 내에 있으면, 단계(S6-3)을 수행하고, 뉴런 중심점 거리가 반경 내에 있지 않으면 종료처리한다. 단계(S6-3)에서, 입력 데이터의 라벨과 활성화된 뉴런의 라벨이 같은지의 여부를 판단한다(

). 입력 데이터의 라벨과 활성화된 뉴런의 라벨이 같으면 종료 처리하고 다르면 단계(S6-4)를 수행한다. 단계(S6-4)에서, 해당 뉴런(

)의 반경을 줄여준다.Meanwhile, in step S6-1,

from

It is repeated until, first, DWT-based distance measurement is performed in step S6-1 (

). In step S6-2, it is determined whether the neuron center point distance is within a radius (

). If the neuron center point distance is within a radius, step S6-3 is performed, and if the neuron center point distance is not within a radius, processing is terminated. In step S6-3, it is determined whether the label of the input data is the same as the label of the activated neuron (

). If the label of the input data and the label of the activated neuron are the same, the process is terminated, and if different, step S6-4 is performed. In step (S6-4), the corresponding neuron (

).

That is, in the hand gesture learning method according to an embodiment of the present invention, the feature data extracted through the hand gesture is sequentially input to obtain a distance between the neuron center points using a DTW distance measurement algorithm, wherein the learned neuron does not exist or is activated. In the absence of this, a new neuron is created, and when the label of the input data is different from the label of the activated neuron, the radius of the corresponding neuron is reduced. On the other hand, if the label of the input data and the label of the activated neuron are the same, no new neuron is generated.

4, an example of a hand gesture recognition method according to an embodiment of the present invention is shown.

)가 입력층(12)을 통하여 은닉층(13)에 전달된다. 이하에서,

부터

까지 반복된다.In step S11, the feature data (

) Is transferred to the hidden layer 13 through the input layer 12. Below,

from

Repeat until

). 뉴런 중심점 거리가 반경 내에 존재한다면 단계(S12-2)가 수행된다. 단계(S12-2)에서, 최소 거리값이 계산된다.In step S12-1, the distance between the feature data and the neuron center point is calculated by the DTW distance measurement algorithm, and then it is determined whether the neuron center point distance is within the radius (

). If the neuron center point distance is within a radius, step S12-2 is performed. In step S12-2, the minimum distance value is calculated.

In step S13, the label of the neuron having the minimum distance value is output through the output layer 14.

That is, in the hand gesture recognition method according to an embodiment of the present invention, the distance between the feature data extracted through the hand gesture and the central point of the neuron is obtained by applying a DTW distance measurement algorithm, and the neuron having the feature inside the radius of the neuron is activated as the activated neuron. It determines, and outputs the label of the neuron having the minimum distance value among the activated neurons.

Referring to FIG. 5, an example of an artificial neural network based hand gesture recognition apparatus 100 according to an embodiment of the present invention is illustrated.

As illustrated in FIG. 5, the apparatus 100 for recognizing a hand gesture based on an artificial neural network according to an embodiment of the present invention may include a feature extraction unit 110 and a learning and recognition unit 120.

The feature extraction unit 110 may perform pre-processing on the hand gesture input data to convert and extract feature data for easy recognition. In some examples, the feature extraction unit 110 may include an IMU-sensor that is an inertial sensor, a data acquisition unit that is a preprocessing member, and a data preprocessor.

The learning and recognition unit 120 inputs feature data to learn and recognize hand gestures. In some examples, the learning and recognition unit 120 may include a receiving module (UART RX module), a memory (FPGA memory), and an RCE neural network 10 (see FIG. 1 ).

In addition, the learning and recognition unit 120 may include a hand gesture learning unit 121 and a hand gesture recognition unit 126.

The hand gesture learning unit 121 may include a feature data input unit 122 and a DWT distance measurement unit 123. The feature data input unit 122 serves to sequentially receive feature data, and the DWT distance measurement unit 123 can calculate the distance between the feature data and the central point of the neuron by the DTW distance measurement algorithm described above.

The hand gesture learning unit 121 may further include a new neuron generation unit 124 and a neuron radius reduction unit 125. The new neuron generating unit 124 may generate a new neuron when there is no learned neuron or no activated neuron. The neuron radius reduction unit 125 may reduce the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron.

Here, the hand gesture learning unit 121 does not generate a new neuron when the label of the input data and the label of the activated neuron are the same.

Meanwhile, the hand gesture recognition unit 126 may include a DWT distance measurement unit 127, an activated neuron determination unit 128, and a neuron label output unit 129. The DWT distance measurement unit 127 may calculate the distance between the feature data and the neuron center point using the DTW distance measurement algorithm described above. The activated neuron determination unit 128 may determine a neuron having a characteristic within a radius of the corresponding neuron as an activated neuron. The neuron label output unit 129 may output a label of a neuron having a minimum distance value among activated neurons.

As described above, the learning and recognition unit 120 may include an RCE neural network 10 (see FIG. 1 ). The RCE neural network 10 includes an input layer 12 composed of characteristic data; It is connected in parallel to the input layer 12 and is composed of a plurality of neurons having a center point and a radius forming a specific model, and the neurons calculate the distance between the input characteristic data and the center point of the neuron, respectively, and compare it with the radius. A hidden layer 13 that determines whether input data is included in the corresponding neuron and delivers the result; And an output layer 14 that outputs the class value of the model closest to the input feature data using the information received from the hidden layer 13.

The hand gesture recognition method by combining the algorithm of the RCE neural network 10 capable of real-time learning and the DTW distance measurement algorithm reflecting the similarity of data on the time axis has been described above, and thus is omitted here.

An inertial sensor-based 3D numeric data set was generated using the hand gesture recognition apparatus 100 according to an embodiment of the present invention, and the algorithm of the general distance measurement method-based RCE neural network 10 and an embodiment of the present invention are generated using the generated data set Performance evaluation of the hand gesture recognition method was performed and compared.

The number of trained neurons and recognition accuracy were evaluated by 5-fold cross validation, and as a result of performance evaluation, a typical RCE neural network algorithm uses 11.36 neurons on average, as shown in the confusion matrix of FIG. 6, with an average of 93.0%. Recognition accuracy was shown, and the hand gesture recognition method according to an embodiment of the present invention showed an average recognition accuracy of 97.10% as in the confusion matrix of FIG. 7 with an average of 11.96 neurons.

That is, the hand gesture recognition method/device according to an embodiment of the present invention showed a recognition performance improvement of about 4.1% with a similar number of neurons for a 3D numeric data set.

In this way, in the embodiment of the present invention, a hand gesture recognition method is implemented that combines the algorithm of the RCE neural network 10 capable of real-time learning and the DTW distance measurement algorithm reflecting the similarity of data on the time axis. In order to evaluate the performance of the hand gesture recognition method according to an embodiment of the present invention, the hand gesture recognition device 100 is configured, and performance evaluation is performed by 5-fold cross validation on the 3D numeric data set generated by the hand gesture recognition device 100. Conduct. As a result of the performance evaluation, it was found that the recognition accuracy was 97.10%, and that the number of neurons similar to the algorithm of the general RCE neural network 10 was improved by about 4.1%.

What has been described above is only one embodiment for implementing the artificial neural network-based hand gesture recognition method and apparatus according to the present invention, and the present invention is not limited to the above-described embodiment, as claimed in the following claims Without departing from the gist of the present invention, any person having ordinary knowledge in the field to which the present invention pertains will have the technical spirit of the present invention to the extent that various changes can be implemented.

10; RCE neural network 11; Feature set
12; Input layer 13; Hidden
14; Output layer
100; Hand gesture recognition device based on artificial neural network
110; Feature extraction unit 120; Learning and Recognition Department
121; Hand gesture learning unit 122; Features data input
123; DWT distance measuring unit 124; New neuron generation
125; New Year radius reduction portion 126; Hand gesture recognition
127 DWT distance measuring unit 128; Activated neuron judgment
129; Neuron label output

Claims (46)

A feature extraction step of extracting feature data from hand gesture input data; And a learning and recognition step of learning and recognizing hand gestures by inputting feature data,
Learning and recognition stages
A feature data input step of sequentially receiving feature data; And a DWT distance measurement step of calculating a distance between the feature data and the neuron center point using a dynamic time warping (DTW) distance measurement algorithm; And
DWT distance measurement step of calculating the distance between the feature data and the neuron center point with a DTW distance measurement algorithm; An activated neuron determination step of determining a neuron having a characteristic inside a radius of the corresponding neuron as an activated neuron; And a hand gesture recognition step having a neuron label output step of outputting a label of a neuron having a minimum distance value among activated neurons. According to claim 1,
The hand gesture learning step further includes a new neuron generation step of generating a new neuron when there is no learned neuron or there is no activated neuron. According to claim 1,
The gesture learning step further includes a neuron radius reduction step of reducing the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron. According to claim 1,
The hand gesture learning step does not generate a new neuron when the label of the input data and the label of the activated neuron are the same. According to claim 1,
The learning and recognition step is performed in a restricted coulomb energy (RCE) neural network, an artificial neural network based hand gesture recognition method. The method of claim 5,
The RCE neural network comprises an input layer composed of characteristic data; It is connected in parallel to the input layer and consists of a number of neurons with a center point and a radius forming a specific model, the neurons each calculate the distance between the input feature data and the center point of the neuron, and compare the radius with the input data A hidden layer that determines whether the neuron is included and delivers the result; And an output layer that outputs the class value of the model closest to the input feature data using the information received from the hidden layer. The method of claim 6,
A method of recognizing a gesture based on an artificial neural network, in which a specific model of a neuron generates a decision boundary for classifying data in a feature space in a circle or Gaussian. The method of claim 6,
The hidden layer is a set of neurons

Containing, neurons

Is the center point that satisfies the following equation

And radius

Including, artificial neural network based hand gesture recognition method.

here,

Is the index of each neuron,

Is the total number of neurons generated during the learning process,

Is the dimension of the input feature vector. The method of claim 8,
Each neuron

Is, the dimension of the input feature vector

When, radius

Such as the dimension of the feature vector and

Center of dimension

Storing, artificial neural network based hand gesture recognition method. The method of claim 9,
The hidden layer is entered in the learning process

The number of dimensional feature vectors

Set of feature vectors

Features vector

A method of recognizing a gesture based on artificial neural network, which includes the following equation.

here,

Features vector

Containing

The second feature data. The method of claim 10,
The hidden layer has a feature vector through each input layer.

When sequentially input to the, the distance between the feature vector and the central point of the neuron is calculated using a DWT distance measurement algorithm, artificial neural network based hand gesture recognition method. The method of claim 11,
The DWT distance measurement algorithm is an artificial neural network-based hand gesture recognition method in which an optimum distance between data is calculated by measuring the similarity between at least two sequence data. The method of claim 12,
Each of the two sequence data

And

, And the similarity between two sequence data creates a matrix using the following equation,

A method for recognizing hand gestures based on artificial neural networks, which is measured by obtaining a value.

The method of claim 13,
Calculated distance value

And radius

Is compared to the following equation, and the corresponding feature vector is for each neuron.

To determine whether or not to activate, artificial neural network based hand gesture recognition method.

The method of claim 14,
Specific neurons

in

If it satisfies, it is determined that the neuron is activated, artificial neural network based hand gesture recognition method. The method of claim 14,
Feature vector

About all neurons

If not satisfied, the center point

New neurons having

And the total number of neurons

A method for recognizing hand gestures based on an artificial neural network, which increases 1. According to claim 1,
The hand motion input data is obtained from an inertial measurement sensor, and the artificial neural network based hand motion recognition method. According to claim 1,
The gesture input data is obtained from an inertial measurement sensor mounted on a smart device, a smart watch, or an inertial pen. A hand gesture learning step of sequentially receiving feature data and calculating a distance between the feature data and the neuron center point with a dynamic time warping (DTW) distance measurement algorithm; And
The distance between the feature data and the central point of a neuron is calculated by a DTW distance measurement algorithm, and a neuron having a feature within a radius of the neuron is determined as an activated neuron, and a label of a neuron having a minimum distance value among the activated neurons is output. A gesture recognition method based on an artificial neural network including a gesture recognition step. The method of claim 19,
The hand gesture learning step further includes a new neuron generation step of generating a new neuron when there is no learned neuron or there is no activated neuron. The method of claim 19,
The gesture learning step further includes a neuron radius reduction step of reducing the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron. The method of claim 19,
The hand gesture learning step does not generate a new neuron when the label of the input data and the label of the activated neuron are the same. The method of claim 19,
The hand gesture learning step is performed in a restricted coulomb energy (RCE) neural network. A feature extraction unit for extracting feature data from hand gesture input data; And a learning and recognition unit for learning and recognizing hand gestures by inputting feature data,
Department of Learning and Recognition
A feature data input unit that sequentially receives feature data; And a DWT distance measuring unit that calculates a distance between the feature data and the neuron center point using a dynamic time warping (DTW) distance measuring algorithm. And
DWT distance measuring unit for calculating the distance between the feature data and the neuron center point with a DTW distance measurement algorithm; An activated neuron judging unit for determining a neuron having a characteristic inside a radius of the corresponding neuron as an activated neuron; And a hand gesture recognition unit having a neuron label output unit for outputting a label of a neuron having a minimum distance value among activated neurons. The method of claim 24,
The hand gesture learning unit further includes a new neuron generating unit that generates a new neuron when there is no learned neuron or there is no activated neuron. The method of claim 24,
The gesture learning unit further includes a neuron radius reduction unit for reducing the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron. The method of claim 24,
The hand motion learning unit does not generate new neurons when the label of the input data and the label of the activated neuron are the same. The method of claim 24,
The learning and recognition unit includes an artificial neural network-based hand gesture recognition device including a restricted coulomb energy (RCE) neural network. The method of claim 28,
The RCE neural network comprises an input layer composed of characteristic data; It is connected in parallel to the input layer and consists of a number of neurons with a center point and a radius forming a specific model, the neurons each calculate the distance between the input feature data and the center point of the neuron, and compare the radius with the input data A hidden layer that determines whether the neuron is included and delivers the result; And an output layer outputting the class value of the model closest to the input feature data using the information received from the hidden layer. The method of claim 29,
A specific model of a neuron is an artificial neural network-based hand gesture recognition device that generates a decision boundary for classifying data in a feature space in a circle or Gaussian. The method of claim 29,
The hidden layer is a set of neurons

Containing, neurons

Is the center point that satisfies the following equation

And radius

Including, artificial neural network based hand gesture recognition device.

here,

Is the index of each neuron,

Is the total number of neurons generated during the learning process,

Is the dimension of the input feature vector. The method of claim 31,
Each neuron

Is, the dimension of the input feature vector

When, radius

Such as the dimension of the feature vector and

Center of dimension

Hand gesture recognition device based on artificial neural network for storing. The method of claim 32,
The hidden layer is entered in the learning process

The number of dimensional feature vectors

Set of feature vectors

Features vector

A gesture recognition device based on an artificial neural network including the following equation.

here,

Features vector

Containing

The second feature data. The method of claim 33,
The hidden layer has a feature vector through each input layer.

When sequentially input to the, the neural network-based hand gesture recognition device calculates the distance value between the feature vector and the neuron center point using a DWT distance measurement algorithm. The method of claim 34,
The DWT distance measurement algorithm is an artificial neural network-based hand gesture recognition device in which an optimum distance between data is calculated by measuring the similarity between at least two sequence data. The method of claim 35,
Each of the two sequence data

And

, And the similarity between two sequence data creates a matrix using the following equation,

An artificial neural network-based hand gesture recognition device that is measured by obtaining a value.

The method of claim 36,
Calculated distance value

And radius

Is compared to the following equation, the corresponding feature vector is each neuron

Hand gesture recognition device based on the artificial neural network to determine whether or not to activate.

The method of claim 37,
Specific neurons

in

If it satisfies, it is determined that the neuron is activated, artificial neural network based hand gesture recognition device. The method of claim 37,
Feature vector

About all neurons

If not satisfied, the center point

New neurons having

And the total number of neurons

A gesture recognition device based on artificial neural network that increases 1 The method of claim 24,
The hand gesture input data is obtained from an inertial measurement sensor. The method of claim 24,
The gesture input data is obtained from an inertial measurement sensor mounted on a smart device, a smart watch or an inertial pen, and an artificial neural network-based gesture recognition device. A hand gesture learning unit that sequentially receives feature data and calculates a distance between the feature data and the neuron center point by a dynamic time warping (DTW) distance measurement algorithm; And
The distance between the feature data and the central point of a neuron is calculated by a DTW distance measurement algorithm, and a neuron having a feature within a radius of the neuron is determined as an activated neuron, and a label of a neuron having a minimum distance value among the activated neurons is output. An artificial neural network based hand gesture recognition device including a hand gesture recognition unit. The method of claim 42,
The hand gesture learning unit further includes a new neuron generating unit for generating a new neuron if there is no learned neuron or there is no activated neuron. The method of claim 42,
The gesture learning unit further includes a neuron radius reduction unit for reducing the radius of the corresponding neuron when the label of the input data is different from the label of the activated neuron. The method of claim 42,
The hand motion learning unit does not generate new neurons when the label of the input data and the label of the activated neuron are the same. The method of claim 42,
The hand motion learning unit includes an artificial neural network-based hand motion recognition device including a restricted coulomb energy (RCE) neural network.

Images