KR102343963B1 - CNN For Recognizing Hand Gesture, and Device control system by hand Gesture - Google Patents

Abstract

The present invention relates to a convolutional neural network for detecting hand gestures and a device control system based on hand gestures. The gesture classifier according to the present invention is a gesture classifier for learning parameters of a convolutional neural network for detecting hand gestures. a convolutional neural network comprising a plurality of convolutional layers for calculating a feature map by performing an operation and a fully connected layer for classifying a detected image by analyzing the feature maps calculated from the plurality of convolutional layers; and a learning engine that trains the convolutional neural network to calculate parameters optimized for hand gesture detection, wherein the plurality of convolutional layers reduce the size of a feature map calculated as a result of a convolution operation based on a detection image. a first convolutional layer including a sub-sampling layer; a second convolutional layer implemented as a non-subsampling layer to repeat a convolution operation based on an output of the first convolutional layer; A third convolution layer including a sub-sampling layer that reduces the size of a feature map calculated as a result of a convolution operation based on the output of the second convolution layer; and a type of kernel filter that performs a convolution operation, The number and size are independently configured for each of the plurality of convolutional layers, and the type, number, and size of kernel filters included in the plurality of convolutional layers are individually calculated by learning of the learning engine.

Description

CNN For Recognizing Hand Gesture, and Device control system by hand Gesture

The present invention relates to a system for controlling a device using hand gestures. Specifically, a convolutional neural network (“CNN”) structure optimized for extracting hand gesture features is designed, and the convolutional neural network It relates to a hand gesture-based device control system that classifies hand gestures using a classifier having a (CNN) structure and controls peripheral devices.

Recently, a natural user interface (NUI) that recognizes a gesture, a natural human movement, away from an input device such as a mouse or keyboard, and enables communication between a user and a computing device through the recognition result ) is being actively studied.

Gesture recognition technology can be roughly divided into rule-based recognition technology and learning-based recognition technology. Rule-based recognition technology sets a certain threshold from the center of the palm and recognizes hand shapes according to the number of finger tips that exceed the threshold. Learning-based recognition technology is a method of recognizing a hand shape through a model created by acquiring a DB for a hand shape to be recognized and learning it.

The rule-based recognition technique has difficulty in determining the optimal threshold value (r) because the hand size is different for each person. When an environmental change occurs, it may be necessary to reset the threshold value in order to set the optimal threshold value (r). There may be problems with degradation. And the rule-based recognition technology has limitations in recognizing various hand shapes compared to the learning-based recognition technology.

Learning-based recognition technology is a technology based on deep learning that clusters or classifies a plurality of data by a learning structure designed to accurately classify gestures. In particular, in the field of object recognition, a technology called Convolutional Neural Network (hereinafter “CNN”), which is a kind of deep learning, is in the spotlight. It is a model that simulates the brain function of a person based on the assumption that basic features of In a convolutional neural network (CNN), various filters for extracting image features through a convolution operation and a pooling or non-linear activation function to add non-linear characteristics are basically included. used

However, even when such a neural network technique is used, the performance result sharply varies depending on the type of applied function and how the structure of the operation is designed. Therefore, properly designing a convolutional neural network (CNN) for a purpose is a very important issue directly related to performance.

Korean Patent Application Laid-Open No. 10-2010-0129629 "Method for controlling operation of electronic device by motion detection and device employing the same"

It has been devised to solve the problems of the prior art,

An object of the present invention is to design a convolutional neural network (CNN) structure optimized for classifying various hand shapes and hand gestures, and to learn the designed convolutional neural network (CNN) to automatically extract various parameters. to provide a control system.

Another object of the present invention is to accurately classify long-distance non-contact hand gestures by using a convolutional neural network (CNN) optimized for classifying hand gestures and a classifier composed of parameters extracted by learning, so that device control performance by hand gestures It is to provide a device control system by a hand gesture that raises the

Another object of the present invention is to set the desired position and desired control area by the user, rather than having a fixed position, a predetermined control area, or a device to be controlled already set, and also set peripheral devices and control signals to be controlled. It is to provide a device control system by hand gestures.

It has been devised to solve the problems of the prior art,

A gesture classifier according to an aspect is a gesture classifier for learning parameters of a hand gesture detection convolutional neural network, a plurality of convolutional layers for calculating a feature map by performing a convolution operation, and the plurality of convolutional layers. a convolutional neural network composed of fully connected layers that classify detected images by analyzing the calculated feature maps; and a learning engine that trains the convolutional neural network to calculate parameters optimized for hand gesture detection, wherein the plurality of convolutional layers reduce the size of a feature map calculated as a result of a convolution operation based on a detection image. a first convolutional layer including a sub-sampling layer; a second convolutional layer implemented as a non-subsampling layer to repeat a convolution operation based on an output of the first convolutional layer; A third convolution layer including a sub-sampling layer that reduces the size of a feature map calculated as a result of a convolution operation based on the output of the second convolution layer; and a type of kernel filter that performs a convolution operation, The number and size are independently configured for each of the plurality of convolutional layers, and the type, number, and size of kernel filters included in the plurality of convolutional layers are individually calculated by learning of the learning engine.

The first convolution layer and the third convolution layer perform padding together so that the size of the output is maintained the same as the size of the input when performing the convolution operation, and the padding parameter is set for each of the plurality of convolutional layers. It is independently configured and the parameter size of the padding is calculated individually for each of the plurality of convolutional layers by learning of the learning engine.

The first convolutional layer and the third convolutional layer are characterized in that the output obtained by sequentially performing normalization and nonlinear function application on the feature map calculated by the convolution operation is transmitted to the subsampling layer.

The second convolution layer performs padding together so that the size of the output remains the same as the size of the input when performing the convolution operation, and performs normalization and application of a non-linear function on the feature map calculated by the convolution operation in sequence. A plurality of convolutional layers for calculating an output are included, and the plurality of convolutional layers constituting the second convolutional layer are configured by serial coupling.

The number of the plurality of convolutional layers constituting the second convolutional layer is calculated by learning of the learning engine.

The second convolution layer may include: a first parallel layer for calculating a first feature map by performing normalization on a feature map calculated by a convolution operation on an output of the first convolution layer; A first layer that sequentially applies normalization and a nonlinear function to a feature map calculated by a convolution operation on the output of the first convolution layer, and a feature map calculated by a convolution operation on the output of the first layer a second parallel layer comprising a second layer for calculating a second feature map by applying normalization, wherein the first layer and the second layer are serially combined; a fusion layer for performing a sum operation on the first feature map and the second feature map; and a noise reduction layer that applies a nonlinear function to the output of the fusion layer.

The second layer performs padding together so that the size of the output remains the same as the size of the input when the convolution operation is performed, and the padding parameter is the size of the padding parameter as the learning engine learns the plurality of convolutions. It is characterized in that it is calculated individually for each solution layer.

A gesture recognition device according to another aspect is a gesture recognition device for recognizing a hand gesture to control a peripheral device. a gesture use verification unit that calculates a correspondence relationship between the gesture display area and the monitor so as to be displayed in corresponding coordinates; a gesture registration unit for registering a user-designated device to be controlled and a gesture used as a control signal; and a device control unit configured to detect and analyze a gesture from the gesture image detected in the gesture display area and control the device according to a control command corresponding to the gesture registered by the gesture registration unit; wherein the gesture display area includes: It is a user-specified area that transmits hand gesture information, and it is characterized in that it is created by user definition at a location spaced apart from the monitor by a certain distance.

The correspondence between the gesture display area and the monitor is calculated based on the coordinates of the monitor displayed on the monitor and the reference coordinates extracted by analyzing the image for the coordinates of the hand area displayed by the user on the gesture display area along the monitor coordinates. characterized by being

The device controller may include a gesture classifier that detects a gesture from a gesture image and analyzes a gesture type, wherein the gesture classifier is implemented using a gesture detection convolutional neural network including learned parameters.

The gesture classifier is a convolution comprising a plurality of convolutional layers that calculate a feature map by performing a convolution operation, and a fully connected layer that classifies a detected image by analyzing the feature maps calculated by the plurality of convolutional layers neural network; and a learning engine for calculating parameters optimized for hand gesture detection by learning the convolutional neural network, wherein the plurality of convolutional layers include a sub-sampling layer that reduces the size of the feature map calculated based on the detected image. a first convolutional layer comprising; a second convolutional layer that repeats a convolution operation based on an output of the first convolutional layer, including a non-subsampling layer; A third convolution layer including a sub-sampling layer that reduces the size of the feature map calculated based on the output of the second convolution layer; includes, and the type, number, and size of kernel filters performing convolution operation It is characterized in that each of the plurality of convolutional layers is independently configured, and the type, number, and size of kernel filters included in the plurality of convolutional layers are individually calculated by learning of the learning engine.

The first convolution layer and the third convolution layer perform padding together so that the size of the output is maintained the same as the size of the input when performing the convolution operation, and the padding parameter is set for each of the plurality of convolutional layers. It is independently configured and the parameter size of the padding is calculated individually for each of the plurality of convolutional layers by learning of the learning engine.

The first convolutional layer and the third convolutional layer are characterized in that the output obtained by sequentially performing normalization and nonlinear function application on the feature map calculated by the convolution operation is transmitted to the subsampling layer.

The second convolution layer performs padding together so that the size of the output remains the same as the size of the input when performing the convolution operation, and performs normalization and application of a non-linear function on the feature map calculated by the convolution operation in sequence. A plurality of convolutional layers for calculating an output are included, and the plurality of convolutional layers constituting the second convolutional layer are configured by serial coupling.

The number of the plurality of convolutional layers constituting the second convolutional layer is calculated by learning of the learning engine.

The second convolution layer may include: a first parallel layer for calculating a first feature map by performing normalization on a feature map calculated by a convolution operation on an output of the first convolution layer; A first layer that sequentially applies normalization and a nonlinear function to a feature map calculated by a convolution operation on the output of the first convolution layer, and a feature map calculated by a convolution operation on the output of the first layer a second parallel layer comprising a second layer for calculating a second feature map by applying normalization, wherein the first layer and the second layer are serially combined; a fusion layer for performing a sum operation on the first feature map and the second feature map; and a noise reduction layer that applies a nonlinear function to the output of the fusion layer.

The second layer performs padding together so that the size of the output remains the same as the size of the input when the convolution operation is performed, and the padding parameter is the size of the padding parameter as the learning engine learns the plurality of convolutions. It is characterized in that it is calculated individually for each solution layer.

The present invention has the following effects by the above configuration.

The present invention has the effect of providing a design structure of a customized convolutional neural network (CNN) optimized for hand shape and hand gesture classification.

The present invention has the effect of automatically extracting various parameters optimized for hand gesture classification so that a control signal can be generated even with a hand shape or gesture of various people by learning a hand gesture custom convolutional neural network (CNN) .

The present invention provides a classifier composed of a convolutional neural network (CNN) optimized for classifying hand gestures, thereby accurately classifying hand gestures caused by remote non-contact, thereby increasing device control performance by hand gestures.

The present invention has the effect that the user can set the desired position and the desired control area by himself/herself, and also freely set the peripheral devices and control signals to be controlled.

1 is a block diagram showing the configuration of a gesture classifier according to an embodiment.
2 is a diagram illustrating a structure of a convolutional neural network (CNN) according to an embodiment.
3 is a conceptual diagram illustrating a convolution operation performed by the convolution layer of FIG. 2 .
FIG. 4 is a conceptual diagram illustrating pooling performed by the sub-sampling layer of FIG. 2 .
FIG. 5 is an exemplary diagram in which parameters and the number of convolutional layers are calculated by learning for the convolutional neural network (CNN) of FIG. 2 .
6 is a diagram illustrating a structure of a convolutional neural network (CNN) according to another embodiment.
7 is an exemplary diagram in which parameters are calculated by learning for the convolutional neural network (CNN) of FIG. 6 .
8 is an exemplary diagram in which a plurality of second convolutional layers are connected in series in the structure of the convolutional neural network (CNN) of FIG. 6 .
9 is an exemplary diagram of a learning image used for learning of a convolutional neural network (CNN) according to an embodiment.
10 is an exemplary diagram of a final classification result derived by a hand gesture classifier using a convolutional neural network (CNN) structure according to an embodiment.
11 is an overall conceptual diagram illustrating a device control system using a gesture according to another embodiment.
12 is a block diagram illustrating the gesture recognition apparatus of FIG. 11 .
13 is a diagram illustrating an example of gesture detection according to an embodiment.

Hereinafter, an embodiment of the present invention will be described in more detail with reference to the accompanying drawings. Embodiments of the present invention may be modified in various forms, and the scope of the present invention should not be construed as being limited to the following embodiments. This embodiment is provided to more completely explain the present invention to those of ordinary skill in the art. In addition, although specific terms have been used in the drawings and the specification of the present invention, these are used only for the purpose of describing the present invention and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Next, a convolutional neural network for detecting a hand gesture of the present invention and a device control system using a hand gesture will be described in detail with reference to the drawings.

1 is a block diagram showing the configuration of a gesture classifier according to an embodiment.

Referring to FIG. 1 , the gesture classifier 1 includes a convolutional neural network 11 and a learning engine 13, and these components may be implemented as hardware or software, or may be implemented through a combination of hardware and software. . In addition, the gesture classifier 1 may include a memory and one or more processors, and the functions of the convolutional neural network 11 and the learning engine 13 are stored in the memory and executed by the one or more processors. can be implemented in the gesture classifier (1).

The convolutional neural network 11 is deeply learned by the learning engine 13, and according to an embodiment, can recognize a hand gesture image with high precision. The convolutional neural network 11 according to an embodiment is a kind of deep learning in the field of object recognition, and in particular, a CNN (Convolutional Neural Network) structure optimized for recognizing hand gestures or hand shapes can be designed as

The learning engine 13 may calculate the parameters by learning the convolutional neural network 11 . The hand can take various changes in the shape of the fingers, such as opening and closing the hand, rapid movement of the hand, rotation, and the shape of the fingers. Accordingly, the structure of the convolutional neural network 11 according to the embodiment is presented, and the learning engine 13 learns the convolutional neural network 11 to calculate parameters optimized for hand gesture detection to perform various hand shapes or hand gestures. The structure of the convolutional neural network 11 that accurately classifies can be completed. Here, the parameter includes not only the type, number, and size of the filter (eg, a kernel filter performing a convolution operation), but also the number of layers to be described below.

Hereinafter, the structure of the convolutional neural network 11 according to various embodiments will be described with reference to FIGS. 2 to 8 , and examples of parameters optimized for the structure of the convolutional neural network 11 will be described.

FIG. 2 is a diagram illustrating the structure of a convolutional neural network (CNN) according to an embodiment, FIG. 3 is a conceptual diagram illustrating a convolution operation performed on the convolution layer of FIG. 2, and FIG. 4 is a sub of FIG. It is a conceptual diagram illustrating pooling performed by a sampling layer, and FIG. 5 is an exemplary diagram in which parameters and the number of convolutional layers are calculated by learning for the convolutional neural network (CNN) of FIG. 2 .

Referring to FIG. 2 , the convolutional neural network includes a convolutional layer 21 and a fully connected layer 23 .

The convolution layer 21 performs a convolution operation on an input image using a convolution filter (or a kernel, a mask) and generates a feature map. Here, the convolution operation extracts all possible n×n subregions (or receptive fields) from the entire input image, and each value of the n×n subregion and n corresponding to the size of the subregion This means that each unit element of the convolution filter composed of ×n parameters is multiplied and then summed (ie, the sum of the dot product products between the filter and the partial region). In addition, the feature map means image data in which various features of the input image are expressed, and the number of calculated feature maps does not necessarily correspond to the number of convolution filters, and may not correspond depending on the method of convolution operation. .

The convolutional layer 21 includes a plurality of convolutional layers L ₁ , L ₂ , L ₃ , …, L _N , and the plurality of convolutional layers L ₁ , L ₂ , L ₃ , …, L _N ) is a first convolutional layer 211 : L ₁ , a second convolutional layer 213 : L ₂ , L ₃ , ..., L _{N -1} ), and a third convolutional layer 215 : L according to a function. _N ) can be distinguished.

Referring to FIG. 2 , in the convolutional layer 21 according to an embodiment, the first convolutional layer 211 and the third convolutional layer 215 are feature maps by subsampling or pooling. A process (POOL) of reducing the size of is performed, but the second convolution layer 213 does not perform a process (POOL) of reducing the size of the feature map. Therefore, after performing the convolution operation and the pooling process in the first convolution layer 211, only the convolution operation is repeated several times without the pooling process in the second convolution layer 213, and the number of output feature maps increases. In-depth learning is possible while maintaining each characteristic of hand gesture images to be learned and classified.

The first convolution layer 211 receives an image to be classified (hereinafter, a detection image) as an input image (input) and performs a convolution operation (CONV) to generate a feature map (L1a) ) and a sub-sampling layer L1b that reduces the size of the feature map through sampling or pooling. According to an embodiment, the convolution layer L1a performs padding together when performing the convolution operation so that the image size before and after the convolution operation is maintained to be the same. In addition, the convolution layer L1a transmits the result of sequentially applying normalization (NORM) and nonlinear function (RELU or PRELU) to the feature map calculated by the convolution operation to the sub-sampling layer L1b. . That is, the convolution layer L1a transmits the feature map to which the normalization and nonlinear function RELU or PRELU are applied to the sub-sampling layer L1b.

If the size of the input image is mxm, if all nxn partial regions (or receptive fields) are extracted and convolution operation (CONV) is performed, the size of one output image is (m - (n - 1)) x (m - (n - 1)). Accordingly, the output (output image) of the convolution operation is reduced in width and length by n - 1, respectively, compared to the input image. For example, if a convolution operation is applied by extracting all subregions with a size of 3 x 3 to an input of size 6 x 6, the output has a size of (6 - (3 - 1)) x (6 - (3) - 1)) = 4 x 4. Therefore, according to an embodiment, the first convolution layer 211 prevents the size of the output from being reduced and performs a padding technique so that the size of the input and the size of the output are the same. Padding means using an odd number of n to cover the top, bottom, left, and right sides of the input image with [n / 2] thick spaces, respectively. Here, square brackets indicate Gaussian symbols (or floor functions).

In addition, the distance between adjacent subregions (or receptive fields) is referred to as a stride, and when the stride is greater than 1, the horizontal and vertical lengths of the output become smaller than the horizontal and vertical lengths of the input, respectively. For example, if the stride is 2, the width and height of the output will be half the width and height of each input.

The second convolution layer 213 does not include a sub-sampling layer that reduces the size of the feature map through sampling or pooling, but performs a convolution operation (CONV) to generate the feature map. It includes a plurality of convolutional layers (L ₂ , L ₃ , ..., L _N-1 ). The plurality of convolutional layers L ₂ , L ₃ , ..., L _N-1 are connected in series so that the output of the previous layer becomes the input of the next layer.

Each of the convolution layers L ₂ , L ₃ , ..., L _N-1 constituting the second convolution layer 213 maintains the size of the output equal to the size of the input when the convolution operation CONV is performed. As much as possible, padding is performed together, and normalization (NORM) and nonlinear function (RELU or PRELU) are sequentially applied to the feature map calculated by the convolution operation to calculate the output, that is, the feature map.

The third convolution layer 215 receives the output of the second convolution layer 213 as an input, and performs a convolution operation (CONV) to generate a feature map (L _NA ) and sampling ( _{A sub-sampling layer (L Nb} ) for reducing the size of the feature map through sampling or pooling is included. The convolution layer (L _Na ) performs padding to keep the size of the output equal to the size of the input when performing the convolution operation (CONV) according to an embodiment, and a feature map calculated by the convolution operation The output obtained by sequentially applying normalization (NORM) and applying a nonlinear function (RELU or PRELU) to the sub-sampling layer (L _Nb ) is transferred. In this case, the sub-sampling layer L _Nb reduces the size of the image and transmits it to the fully connected layer 23 .

The fully connected layer 23 analyzes the video image received from the convolutional layer 21 to finally determine which category it belongs to. The fully connected layer 23 may be implemented as a global average pooling or a fully-connected layer. For example, the fully connected layer 23 analyzes the output received from the convolutional layer 21, that is, the feature maps, and finally determines whether the hand gesture displayed on the detected image corresponds to scissors, rock, paper, or other hand shapes. can judge

Referring to FIG. 3 , according to an embodiment, the first convolutional layer (L ₁ ) includes ℓ feature maps (W ₁ ), the second convolution layer (L ₂ ) includes m feature maps (W ₂ ), and the third convolution The solution layer (L ₃ ) has n feature maps (W ₃ ), … , the N-th convolutional layer (L _N ) shows that g feature maps (W _N ) are calculated. According to an embodiment, for one input 100 , the first convolution layer (L ₁ ) generates ℓ feature maps as a result of convolution operation, and the second convolution layer (L ₂ ) is the first convolution layer It receives ℓ feature maps (W ₁ ) as the output of (L ₁ ) and performs convolution operation to generate m feature maps (W _{2 ).} Similarly, the third convolutional layer (L ₃ ) receives m feature maps (W ₂ ), which are the outputs of the second convolution layer (L ₂ ), and performs a convolution operation to generate _{n feature maps (W 3 )} and repeat the convolution operation while passing through each convolution layer (L) in the same way. When the convolution operation of the last N-th convolutional layer (L _N ) is completed, the final g feature maps (W _N ) are generated. In this case, the input 100, which is an input image, may have one or more channels. For example, when the input 100, which is an input image, is an 8-bit image, it is 1 channel, and in the case of a 32-bit image, it has 3 channels. Here, the ℓ feature maps (W ₁ ) mean image data in which various features of the input 100 are expressed.

A convolution filter (or kernel filter) may have a different type, number, size (n×n), etc. for each convolutional layer (L), and a plurality of convolutions implemented within the same convolutional layer (L). Filters may also be implemented in different types. The type of the convolution filter may be a color-related filter such as red, green, or blue, or may be implemented as a filter having characteristics for finding various other hand features.

Referring to FIG. 4 , according to an embodiment, the first convolutional layer 211 : L ₁ includes a convolutional layer 211a and a sub-sampling layer 211b, and includes a plurality of The convolutional layer 213: L ₂ , L ₃ , …, L _N-1 consists of only a convolution layer without a sub-sampling layer, and the last N-th convolution layer 215: L _N is a convolution layer 215a ) and a sub-sampling layer 215b.

For example, in the first convolution layer 211: L ₁ , the convolution layer 211a utilizes a kernel filter to perform a convolution operation (CONV) for one input 100 , resulting in three feature maps 101 . , the sub-sampling layer 211b performs pooling or sampling on the three feature maps 101 to calculate the feature map 102 with a reduced size. Thereafter, the convolution operation is repeatedly performed on the _{plurality of convolutional layers 213: L 2} , L ₃ , ..., L _N-1 from the second to the N-1 th for the feature map 102, so that the number increases. In the last convolution layer 215: L _N , when the convolution operation is completed in the convolution layer 215a and the final g feature maps 100' are calculated, the sub-sampling layer 215b is the feature map 100 ') by performing pooling or sampling to calculate a feature map 100” with a reduced size.

5 shows learning data learned by the learning engine 13 through a convolutional neural network designed with the structure shown in FIG. The number of layers N of the layer 21, that is, the number of second convolutional layers 213 (L ₂ , L ₃ , ..., L _N-1 ) is also included.

Referring to FIG. 5 , the convolutional layer 21 is composed of a total of 10 (N=10) layers, and the second convolutional layer 213 is composed of 8 layers connected in series.

According to an embodiment, in the first convolution layer 211: L ₁ , the convolution layer L _1a performs a convolution operation (CONV) using a kernel filter having a size of 3×3 (ker 3), Normalization (NORM) and nonlinear function (PRELU) are sequentially applied to the map calculated by the convolution operation, thereby yielding 16 final feature maps (out 16). At this time, the convolution layer (L _1a ) is padding (ex, zero padding) overlaid with [n/2=1/2] thick spaces on the top, bottom, left, and right of the feature map, which is the result of the convolution operation, when performing the convolution operation. , n = 1). Thereafter, the sub-sampling layer L _1b has a size of 3×3 (ker 3), an interval between adjacent receptive fields is 2 (stride 2), and performs pooling to extract a maximum value (max) to increase the size of the feature map. reduce Accordingly, for one input (detected image), the number of outputs (ie, feature maps) of the first convolutional layer 211 is 16.

The second convolution layer 213: L ₂ , L ₃ , ..., L ₉ repeatedly performs a convolution operation. That is, the 2nd to 9th convolutional layers 213: L ₂ , L ₃ , …, L ₉ are connected in series so that the output of the previous layer is input as the input of the subsequent layer, and all convolution operations (CONV) ), normalization (NORM), and nonlinear function (PRELU) are sequentially applied to calculate an output, that is, a feature map.

According to an embodiment, the second layer (L ₂ ) performs a convolution operation (CONV) using a kernel filter having a size of 1×1 (ker 1) and performs padding (n=0) to obtain 16 feature maps. (out 16) is created. The third layer (L ₃ ) generates 32 feature maps (out 32) by performing a convolution operation (CONV) and padding (n=0) using a kernel filter having a size of 1×1 (ker 1). The fourth layer (L ₄ ) generates 64 feature maps (out 64) by performing convolution operation (CONV) and padding (n=0) using a kernel filter having a size of 1×1 (ker 1). The fifth layer (L ₅ ) generates 64 feature maps (out 64) by performing convolution operation (CONV) and padding (n=1) using a kernel filter having a size of 3×3 (ker 3). . The sixth layer (L ₆ ) generates 64 feature maps (out 64) by performing convolution operation (CONV) and padding (n=0) using a kernel filter having a size of 1×1 (ker 1). . The seventh layer (L ₇ ) generates 128 feature maps (out 128) by performing convolution operation (CONV) and padding (n=1) using a kernel filter having a size of 3×3 (ker 3). . The eighth layer (L ₈ ) generates 128 feature maps (out 128) by performing convolution operation (CONV) and padding (n=0) using a kernel filter having a size of 1×1 (ker 1). . The ninth layer (L ₉ ) generates 256 feature maps (out 256) by performing convolution operation (CONV) and padding (n=1) using a kernel filter having a size of 3×3 (ker 3). . The parameters are learning data learned by the learning engine 13 through a convolutional neural network designed with the structure shown in FIG. 2 , and are an embodiment optimized for hand gesture detection.

Accordingly, in the second convolution layer 213 : L ₂ , L ₃ , ..., L ₉ , the convolution operation is repeatedly performed. At this time, when the convolution operation is performed in each of the plurality of convolution layers 213: L ₂ , L ₃ , ..., L ₉ , a space of [n/2] thickness is formed on the top, bottom, left, and right sides of the feature map that is the result of the convolution operation. Overlay padding (ex, zero padding) is also performed. According to an embodiment, only the convolution operation is repeated without pooling or sampling processing in the second convolution layer 213 .

The last tenth convolutional layer (L ₁₀ ), that is, the convolutional layer (L _10a ) of the third convolutional layer 215 , uses the output (feature map) of the second convolutional layer 213 as an input to include 256 kernels. The convolution operation by the filter (W ₁₀ ) is performed, and the normalization (NORM) and the nonlinear function (PRELU) are sequentially applied to the output calculated by the convolution operation (CONV), and the final output (feature map) for 256 to calculate In this case, the convolution layer (L _10a ) is padding (ex, zero padding) overlaid with [n/2=1/2] thick spaces on the top, bottom, left, and right sides of the feature map, which is the result of the convolution operation, when performing the convolution operation. , n = 1). Thereafter, the sub-sampling layer (L _10b ) has a size of 5×5 (ker 5), an interval between adjacent receptive fields is 2 (stride 2), and performs a pooling (POOL) of pulling an average value to obtain the above characteristics Reduce the size of the maps.

For one detection image, the output of the final convolutional layer 21, that is, feature maps, is 256, and the feature maps are transmitted to the fully connected layer 23 (FC) to determine which category the detected image belongs to. becomes a material for

The learning engine 13 learns through the convolutional neural network composed of the design structure shown in Fig. 2, and as shown in Fig. 5, the number of layers constituting the convolutional layer 21 most suitable for hand gesture classification ( N=10) can be generated. In addition, the type, number (out), and size (ker) of the kernel filter (W) performing the convolution operation are the plurality of convolutional layers (L ₁ , L ₂ , L ₃ , ..., L ₉ , L ₁₀ ) Kernels included in the _{plurality of convolutional layers (L 1} , L ₂ , L ₃ , …, L ₉ , L ₁₀ ) by learning of the learning engine 13 are not identically configured, but may be configured independently. As discussed above, parameters such as the type, number (out), and size (ker) of the filter (W) may be individually calculated. In addition, when performing a convolution operation, the parameter (n) of the padding is also individually calculated for each _{of the plurality of convolutional layers (L 1} , L ₂ , L ₃ , ..., L ₉ , L _{10 ) by learning of the learning engine 13 .} can

6 is a diagram showing the structure of a convolutional neural network (CNN) according to another embodiment, FIG. 7 is an exemplary diagram in which parameters are calculated by learning for the convolutional neural network (CNN) of FIG. 6, and FIG. 6 is an exemplary diagram in which a plurality of second convolutional layers are connected in series in the structure of the convolutional neural network (CNN).

Referring to FIG. 6 , a convolutional neural network according to another embodiment includes convolutional layers 21 ( 211 , 213 , 215 ) and a fully connected layer 23 .

The convolutional layer 21 includes five convolutional layers L ₁ , L ₂ , L ₃ , L ₄ , L ₅ , and the five convolutional layers L ₁ , L ₂ , L ₃ , L ₄ , L ₅ ) is a first convolutional layer ( 211 : L ₁ ), a second convolutional layer ( 213 : L ₂ , L ₃ , L ₄ ), and a third convolutional layer ( 215 : L ₅ ) according to functions can be distinguished as

The convolutional neural network shown in FIG. 6 may be designed such that only the second convolutional layer 213 is different from the convolutional neural network shown in FIG. 2 , and the rest of the configuration is the same. That is, a first convolutional layer of Figure 6 (211: L ₁₎ and the third convolution layer (215: L ₅₎ is a first convolutional layer shown in Figure 2 (211: L ₁₎ and the third convolution Each corresponds to the layer 215: L _N , but parameters of the corresponding layer may be calculated differently through learning of the learning engine 13 . Therefore, referring to FIG. 6 , the first convolution layer 211 : L ₁ and the third convolution layer 215 : L _{5 perform a} subsampling or pooling process for reducing the size of the feature map. However, the second convolution layer 213 (L ₂ , L ₃ , L ₄ ) does not perform a process of reducing the size of the feature map.

Compared with FIG. 2, the convolutional neural network designed as shown in FIG. 6 increases the processing speed by reducing the number of convolution operations performed, and at the same time, not only reduces the number of convolution operations, but also features map and It is designed so that the accuracy of hand gesture recognition can be maintained by combining the feature maps that have performed two convolution operations into one map and then using the combined feature map in the next step. The first convolutional layer 211: L ₁ and the third convolutional layer 215: L ₅ of FIG. 6 are the first convolutional layer 211: L ₁ and the third convolutional layer ( Since each corresponds to 215 : L ₇ , the second convolutional layer 213 : L ₂ , L ₃ , and L ₄ will be described in detail below.

The second convolutional layer 213: L ₂ , L ₃ , L ₄ includes a first parallel layer 2131 , a second parallel layer 2133: 2133a, 2133b , a fusion layer 2135 , and a noise reduction layer ( 2137). The first parallel layer 2131 and the second parallel layer 2133 (2133a, 2133b) are connected in parallel.

The first parallel layer 2131 performs a convolution operation (CONV) with the output of the first convolution layer 211: L ₁ as an input, and performs normalization (NORM) on the feature map calculated by the convolution operation. to calculate the first feature map.

The second parallel layer 2133: 2133a, 2133b _{performs a convolution operation (CONV) with the output of the first convolution layer 211: L 1} as an input, and normalizes the feature map calculated by the convolution operation (NORM) ) and a first layer 2133a that sequentially applies a nonlinear function (ex, PRELU or RELU), and a convolution operation (CONV) using the output of the first layer 2133a as an input and a second layer 2133b for calculating a second feature map by applying normalization (NORM) to the calculated feature map. The first layer 2133a and the second layer 2133b are connected in series.

The fusion layer 2135 generates one map by performing a fusion operation (fusion, concentration) on the first feature map and the second feature map. According to an embodiment, when the horizontal and vertical sizes of the first feature map and the second feature map are the same, but the number of times and output sizes of the convolution operation used for extracting each feature map are different, the first feature map and the second feature map The 3D size of the map may be different. By fusion or concentration of the first feature map and the second feature map obtained by using a structure in which the number of times of convolution operation is different, it is possible to extract features from a single video image in various aspects. That is, since features are extracted using different convolution operations (number of operations performed, weights), multi-faceted feature extraction is possible for one input, and as a result, hand gesture classification performance can be improved. In addition, since the first feature map and the second feature map are each normalized (NORM) before combining, the properties of the first feature map and the second feature map are maintained.

The noise reduction layer 2137 applies a nonlinear function (eg, PRELU) to a feature map combined by fusion of the first feature map and the second feature map. The noise reduction layer 2137 transmits the output to the third convolution layer 215 : L ₅ , and the third convolution layer 215 : L ₅ performs a convolution operation and pooling.

7 shows learning data learned by the learning engine 13 through a convolutional neural network designed with the structure shown in FIG. 6 , and the learning data is implemented as parameters optimized for hand gesture detection.

As shown in FIGS. 2 and 5, if a convolution operation is repeatedly performed by connecting a series of convolution layers one after another, deep learning that increases accuracy is possible, while processing speed may be slowed due to an increase in the amount of operation. have. However, if the number of convolution operations performed (learning depth), that is, the number of kernel filters and feature maps, is somewhat reduced in the learning structure as shown in FIGS. This may not be possible, and then, the accuracy of hand gesture classification may be lowered. Accordingly, according to another embodiment, a design structure capable of improving processing speed without significantly lowering classification accuracy is shown in FIG. 6 , and parameters capable of classifying hand gestures with high accuracy when learning through this design structure are determined. 7 shows. In addition, according to another embodiment, a convolutional neural network designed by connecting the second convolutional layer 213 according to another embodiment shown in FIG. 6 in series a plurality of times is shown in FIG. 8 .

Referring to FIG. 8 , in the convolutional neural network according to another embodiment, the structure of the second convolutional layer 213 (L ₂ , L ₃ , L ₄ ) according to another embodiment shown in FIG. 6 is serially repeated a plurality of times. connected, the first convolutional layer 211 : L ₁ and the third convolutional layer 215 : L5 are configured in the same manner.

Since the convolutional neural network shown in FIG. 8 is configured such that the second convolutional layer 213: L ₂ , L ₃ , L ₄ is repeated multiple times, it classifies a plurality of objects (ex, hand gestures) having similar shapes or characteristics. effective when That is, a structure in which the second convolutional layer 213: L ₂ , L ₃ , L ₄ is inserted several times may take longer to process learning and classification than when the second convolutional layer 213 is inserted once, but since deep learning is possible, more It is effective in classifying many kinds of objects.

9 is an exemplary diagram of a learning image used for learning on a convolutional neural network (CNN) according to an embodiment, and FIG. 10 is a final classification derived by a hand gesture classifier using a convolutional neural network (CNN) structure according to an embodiment. This is an example of the result.

The image shown in FIG. 9 shows an example of an input used for learning when the learning engine 13 learns through the convolutional neural network designed according to the embodiment. The learning engine 13 can learn by using various images that have been changed by performing various processing on one image, such as size, rotation, inversion, histogram, smoothing, blurring, gamma transformation, brightness change, and perspective distortion.

Referring to FIG. 10 , a hand gesture classifier using a convolutional neural network (CNN) structure according to an embodiment shows a result of analyzing a hand image. Referring to FIG. 10(a) , a gesture candidate region is detected by a solid line box, and the analysis table of FIG. 10(b) shows the final result of the classifier analyzing N feature maps for the candidate region. Examining the analysis table of FIG. 10( b ), it is possible to determine a gesture having the highest probability and a probability of 0.5 or more in the final classification result of the classifier as the final result. With respect to the candidate region, since gesture 1 is ranked first with a probability of 99.89% or more, the final result can be determined as gesture 1. For example, gesture 1 may correspond to 'bo' among various hand gestures (eg, scissors, rock, and paper).

11 is an overall conceptual diagram illustrating a device control system using a gesture according to another embodiment.

Referring to FIG. 11 , a device control system using a gesture may include an image input device 1 , a monitor 2 , a gesture display area 3 , and a gesture recognition device 4 .

The image input device 1 acquires a detection image to recognize a user's hand shape or hand gesture. For example, the image input device 1 may be implemented as a depth recognition camera, a stereo camera, or a color camera (eg, a kinect camera), and may acquire a moving image and a still image as a detection image. When the detected image is a moving picture, it may be composed of a plurality of consecutive frames. Also, the detection image may include a color image, a depth image, and a color-depth (RGB-C) image.

The monitor 2 displays an image corresponding to the shape, motion, and position of the hand that the user moves on the gesture display area 3 . Accordingly, the user can check through the monitor 2 whether the user's intended hand shape and hand gesture are transmitted to the gesture recognition device 4 within the range of the gesture display area 3 without departing from the gesture display area 3 . . If a hand shape or hand gesture different from that intended by the user is transmitted to the gesture recognition device 4 and displayed on the monitor 2, the user corrects the hand shape or hand action and displays it in the gesture display area 3 so that it is transmitted again. can be displayed

The gesture display area 3 is an area that transmits information about a user's hand shape or hand gesture, and is created remotely by user definition. According to an embodiment, the user may create the gesture display area 3 by defining an arbitrary area for remotely displaying a control signal at a location T designated by the user rather than a fixed location.

Referring to FIG. 11 , when the four monitor coordinates M1, M2, M3, and M4 are displayed on the monitor 2, the user selects the four monitor coordinates M1, M2, M3 at the remote location T designated by the user. , mark the hand region coordinates (G) at four points in the air along M4). The image input device 1 acquires an image in which hand region coordinates G are displayed, and the gesture recognition device 4 analyzes the image acquired by the image input device 1 to obtain reference coordinates B1, B2, B3, B4) can be derived. According to an embodiment, the gesture display area 3 may correspond to an area connected to the reference coordinates B1, B2, B3, and B4. In addition, since the reference coordinates (B1, B2, B3, B4) and the monitor coordinates (M1, M2, M3, M4) are inverted left and right, the hand area coordinates (G) for the four points obtained from the image are inverted left and right to create the reference coordinates (B1, B2, B3, B4). The remote location (T) designated by the user is a point that is a certain distance (D) away from the image input device 1 and the monitor 2, and is a location that is not smaller or greater than the threshold distance, and at the same time that the image input device 1 It is defined as a position that does not deviate from the range of angle of view that can be obtained.

The gesture recognition apparatus 4 analyzes the user's hand shape, hand gesture, and various combinations thereof from the image acquired by the image input apparatus 1 , and controls various devices according to the analysis result. According to an embodiment, the gesture recognition apparatus 4 can accurately recognize a hand shape, a hand motion, and various combinations thereof using the previously learned hand shape and hand motion classifier, and the classifier according to various embodiments described above. It may be implemented using a convolutional neural network (CNN). Therefore, the classifier can be realized as a hand gesture classifier with high accuracy and high analysis speed.

Meanwhile, in the following description, a hand gesture (gesture) that is a part of the user's body is described as an example, but the shape or motion of other parts of the face, arm, or other various body parts is not excluded. In addition, the hand gesture described below does not refer only to the hand motion itself, but is defined as including the hand shape.

12 is a block diagram illustrating the gesture recognition apparatus of FIG. 11 , and FIG. 13 is a diagram illustrating an example of gesture detection according to an embodiment.

Referring to FIG. 12 , the gesture recognition apparatus 4 may include a gesture use verification unit 41 , a gesture registration unit 43 , and a device control unit 45 , and these components may be implemented as hardware or software, or hardware It can be implemented through the combination of and software. In addition, the gesture recognition device 4 may include a memory and one or more processors, and the functions of the gesture use verification unit 41 , the gesture registration unit 43 , and the device control unit 45 are stored in the memory, and the It may be implemented in the gesture recognition device 4 in the form of a program executed by one or more processors.

The gesture use verification unit 41 verifies whether a peripheral device can be controlled using a gesture at a remote location T arbitrarily designated by the user, referring to FIGS. 11 and 12 . In addition, the gesture use verification unit 41 calculates a transformation matrix indicating the correspondence between the monitor 2 and the gesture display area 3 .

The remote location T may be defined as a location that does not deviate from a threshold distance within a predetermined area of the front of the monitor 2 . Accordingly, when the user's current location does not satisfy the defined remote location (T), the gesture use verification unit 41, for example, the current location is out of the threshold distance, or the image input device 1 is If it is out of an angle at which an image can be acquired, it may be determined that the current position cannot be controlled. Hereinafter, the description assumes that the user's current location corresponds to a defined remote location and that controllability is verified.

The gesture use verification unit 41 induces a user's gesture by displaying four monitor coordinates M1, M2, M3, and M4 on the monitor 2 with reference to FIGS. 11 and 12 . When the coordinates (G) are displayed at four points in the air along the four monitor coordinates (M1, M2, M3, M4) at the remote location (T) designated by the user, the gesture use verification unit 41 enters the image An image of the hand region coordinates G obtained by the device 1 is analyzed to derive a correspondence between the monitor 2 and the gesture display region 3 . The correspondence between the monitor 2 and the gesture display area 3 is a transformation between the monitor coordinates M1, M2, M3, M4 and the reference coordinates B1, B2, B3, B4 of the gesture display area 3 It may be realized in the form (T), but is not limited thereto. Here, the transformation matrix T enables the coordinates for the movement of the hand gesture at the remote location T to be implemented as coordinates on the monitor 2 . For example, if we put a matrix (M) for four monitor coordinates (M1, M2, M3, M4) and a matrix (B) for the detected reference coordinates (B1, B2, B3, B4), M = T × Equation B is derived, and the transformation matrix (T) can be derived from ^{T = M × B - 1 from the arithmetic operation.}

The gesture registration unit 43 may register a gesture to be used as a control signal on the gesture display area 3 and a device to be controlled by the gesture based on a user's selection. In this case, the user can control the device with various control signals by selecting different types of gestures for controlling one device.

The device control unit 45 transmits the gesture image displayed by the user in the region of the reference coordinates B1, B2, B3, and B4 set by the gesture use verification unit 41, that is, the gesture display region 3 to the image input device 1 ) to detect and classify gestures. Also, the device controller 45 may control the device according to the classified gesture. A user can generate an event by taking a gesture, and through a combination of gestures of various shapes, control signals such as On, Off, sound reproduction, volume control, etc. can be generated, and a device to be controlled can be remotely controlled.

According to an embodiment, the device controller 45 may implement a detector using a convolutional neural network (CNN) according to various embodiments described above. From the image acquired by the image input device 1 , the device controller 45 can detect and classify hand gestures with high precision by using the detector (classifier).

Referring to FIG. 12 , the device controller 45 detects motion in the image within the gesture display area 3 , that is, the reference coordinates B1 , B2 , B3 and B4 in the entire image acquired by the image input device 1 . explore the part The device control unit 45 may search for a motion part in the image based on a dense optical flow using the difference between the previous frame and the current frame, and an algorithm (Lucas) for extracting a motion vector based on the optical flow field. -Kanade or Gunnar Farneback) can be used.

In more detail, the device control unit 45 extracts the magnitude and angle of the motion vector from the block in which there is motion, and the number of the motion vectors larger than the threshold by identifying the magnitude of the motion vector within an arbitrary block. If is greater than a certain value, it is identified as a moving block. When it is determined that the block has a stop while the movement is in progress, the device controller 45 regards the area as an area including a target to be detected. According to an exemplary embodiment, when selecting a candidate region including a detection target, the device controller 45 performs detection as a region including a detection target based on the topmost block or topmost motion vector coordinates among blocks where motion is stopped. can do. This reflects the characteristic that the hand is positioned in the uppermost block area due to the movement characteristics of the arm and hand in the standing posture. The device control unit 45 extracts a feature from a block area determined to contain a detection target, or detects a candidate area using a sliding window method in the block area, and then applies a classification method to recognize a gesture. can

Referring to FIG. 13 , a table 451 shows an example of motion block coordinates and optical flow field motion vector values. For example, one feature point can be extracted per 10×10, and when the magnitude of the motion vector is 4.5 or more, it can be determined that the point has a movement. The screen 453 shows an example of a partial tube detection target block and a detected hand region with a change in motion.

While this specification contains many features, such features should not be construed as limiting the scope of the invention or the claims. Also, features described in individual embodiments herein may be implemented in combination in a single embodiment. Conversely, various features described herein in a single embodiment may be implemented in various embodiments individually, or may be implemented in appropriate combination.

Although acts are described in a particular order in the drawings, it should not be understood that the acts are performed in the particular order as shown, or that all of the described acts are performed in a continuous order, or to obtain a desired result. . Multitasking and parallel processing can be advantageous in certain circumstances. In addition, it should be understood that the division of various system components in the above-described embodiments does not require such division in all embodiments. The program components and systems described above may generally be implemented as a package in a single software product or multiple software products.

The method of the present invention as described above may be implemented as a program and stored in a computer-readable form in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.). Since this process can be easily performed by a person skilled in the art to which the present invention pertains, it will not be described in detail any longer.

The present invention described above, for those of ordinary skill in the art to which the present invention pertains, various substitutions, modifications and changes are possible within the scope without departing from the technical spirit of the present invention. It is not limited by the drawings.

2: Convolutional Neural Network 21: Convolutional Layer
211: first convolutional layer 213: second convolutional layer
215: third convolutional layer 23: fully connected layer

Claims (17)

A gesture classifier for learning parameters of a hand gesture detection convolutional neural network, comprising:
a convolutional neural network comprising a plurality of convolutional layers for calculating a feature map by performing a convolution operation and a fully connected layer for classifying a detected image by analyzing the feature maps calculated from the plurality of convolutional layers; and
and a learning engine that trains the convolutional neural network to calculate parameters optimized for hand gesture detection.
The plurality of convolution layers may include: a first convolution layer including a sub-sampling layer that reduces a size of a feature map calculated as a result of a convolution operation based on a detection image; a second convolutional layer implemented as a non-subsampling layer so as to maintain the same size of the feature map and repeating a convolution operation based on an output of the first convolutional layer; A third convolution layer including a sub-sampling layer that reduces the size of the feature map calculated as a result of a convolution operation based on the output of the second convolution layer;
The type, number, and size of filters performing a convolution operation are independently configured for each of the plurality of convolutional layers, and the types, number, and size of filters included in the plurality of convolutional layers are individually determined by learning of the learning engine. Gesture classifier, characterized in that calculated. According to claim 1,
The first convolution layer and the third convolution layer perform padding together so that the size of the output remains the same as the size of the input when the convolution operation is performed,
The parameter of the padding is independently configured for each of the plurality of convolutional layers, and the size of the parameter of the padding is individually calculated for each of the plurality of convolutional layers by learning of the learning engine. 3. The method of claim 2,
The first convolutional layer and the third convolutional layer are
A gesture classifier characterized in that the output obtained by sequentially applying the normalization and the nonlinear function to the feature map calculated by the convolution operation is transmitted to the sub-sampling layer. 4. The method of claim 3,
The second convolutional layer is
A plurality of convolutions in which padding is performed so that the size of the output remains the same as the size of the input when the convolution operation is performed, and normalization and application of a nonlinear function are sequentially performed on the feature map calculated by the convolution operation to calculate the output contains layers,
A plurality of convolutional layers constituting the second convolutional layer are configured by serial combination. 5. The method of claim 4,
The number of the plurality of convolutional layers constituting the second convolutional layer is,
Gesture classifier, characterized in that calculated by the learning of the learning engine. 4. The method of claim 3,
The second convolutional layer is
a first parallel layer for calculating a first feature map by performing normalization on a feature map calculated by a convolution operation on the output of the first convolutional layer;
A first layer that sequentially applies normalization and a nonlinear function to a feature map calculated by a convolution operation on the output of the first convolution layer, and a feature map calculated by a convolution operation on the output of the first layer a second parallel layer comprising a second layer for calculating a second feature map by applying normalization, wherein the first layer and the second layer are serially combined;
a fusion layer for performing a sum operation on the first feature map and the second feature map; and
and a noise reduction layer that applies a nonlinear function to the output of the fusion layer. 7. The method of claim 6,
The second layer performs padding together so that the size of the output remains the same as the size of the input when the convolution operation is performed,
The padding parameter is a gesture classifier, characterized in that the size of the padding parameter is calculated individually for each of the plurality of convolutional layers by learning of the learning engine. delete delete delete delete delete delete delete delete delete delete

Images