KR20180130869A - CNN For Recognizing Hand Gesture, and Device control system by hand Gesture - Google Patents

Abstract

The present invention relates to a convolution neural network for detecting a hand gesture, and a device control system by a hand gesture. According to the present invention, a gesture classifier for learning a parameter of a convolution neural network for detecting a hand gesture, comprises: a convolution neural network including a plurality of convolution layers for calculating a feature map by performing a convolution operation, and a complete connection layer for classifying a detected image by analyzing feature maps calculated from the convolution layers; and a learning engine for learning the convolution neural network to calculate a parameter optimized for detecting a hand gesture. The convolution layers include: a first convolution layer including a subsampling layer for reducing a size of the feature map calculated as a result of a convolution calculation on the basis of the detected image; a second convolution layer realized by a non-subsampling layer, and repeating the convolution operation on the basis of an output of the first convolution layer; and a third convolution layer including a subsampling layer for reducing a size of the feature map calculated as a result of the convolution calculation on the basis of an output of the second convolution layer. According to the present invention, a type, number, and size of a kernel filter performing a convolution operation are independently configured for each of the convolution layers, and the type, number, and size of the kernel filter included in the convolution layers are individually calculated by learning of the learning engine.

Description

Technical Field [0001] The present invention relates to a hand gesture detecting device, a convolutional neural network detecting a hand gesture, and a device control system using a hand gesture,

The present invention relates to a system for controlling a device using a hand gesture, and more particularly, to a congestion neural network (hereinafter referred to as " CNN ") structure optimized for extracting characteristics of a hand gesture, The present invention relates to a device control system using a hand gesture that classifies hand gestures using a classifier having a CNN structure and controls peripheral devices.

A natural user interface (NUI) that recognizes a gesture, which is a natural movement of a human being, out of an input device such as a mouse or a keyboard and enables communication between a user and a computing device through the recognition result, ) Have been actively studied.

Gesture recognition techniques can be broadly classified into two categories: rule-based recognition technology and learning-based recognition technology. Rule-based recognition technology is a method of setting a certain threshold value from the center of the palm of a hand and recognizing the hand shape according to the number of finger tips over the threshold value. Learning - based recognition technology is a method of acquiring a DB of the hand shape to be recognized and recognizing the hand shape through the model generated by learning it.

Rule-based recognition techniques have difficulty in determining the optimal threshold value (r) because the hand size varies from person to person. When an environment change occurs, a threshold value may need to be reset in order to set an optimal threshold value (r). If the determined threshold value (r) is not an optimal threshold value, The problem may be degraded. And rule - based recognition technology has limitations in recognition of various hand shapes compared to learning - based recognition technology.

Learning-based cognitive technology is a technology based on deep learning that clusters or classifies a plurality of data by a learning structure designed to classify gestures accurately. Particularly, in the field of object recognition, a technology called Convolutional Neural Network (CNN), which is a kind of deep learning, is attracting attention. Convolution Neural Network (CNN) Is a model that simulates the human brain function based on the assumption that the basic features of the brain are extracted and then subjected to complex calculations in the brain to recognize objects based on the results. Convolutional neural networks (CNN) basically include various filters for extracting image features through convolution operations, and pooling or non-linear activation functions to add nonlinear characteristics. Is used.

However, even when using such a neural network technique, the performance result is rapidly changed depending on the type of applied function and how the structure of the computation is designed. Therefore, designing the convolutional neural network (CNN) appropriately for the purpose is a very important problem directly related to performance.

Korean Patent Laid-Open Publication No. 10-2010-0129629 " Method of controlling electronic device operation by motion detection and apparatus employing the same "

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art,

It is an object of the present invention to provide a hand gesture device that designs a convolutional neural network (CNN) structure optimized for classifying various hand shapes and hand gestures and automatically learns various parameters by learning a designed convolution neural network (CNN) And to provide a control system.

Another object of the present invention is to accurately classify hand gestures by remote non-contact by using a convolutional neural network (CNN) optimized for classifying hand gestures and a classifier composed of parameters extracted by learning, And to provide a device control system based on a hand gesture for increasing a hand gesture.

It is a further object of the present invention to provide an apparatus and method for setting and controlling a desired position and a desired control area of a user, A hand gesture capable of controlling a hand gesture.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art,

A gesture classifier according to one aspect is a gesture classifier that learns parameters of a hand gesture detection convolutional neural network, comprising: a plurality of convolution layers for performing a convolution operation to calculate a feature map; A convolution neural network consisting of a completely connected layer for classifying the detected images by analyzing the calculated feature maps; And a learning engine that learns the convolutional neural network and calculates parameters optimized for hand gesture detection, wherein the plurality of convolutional layers reduce the size of the feature map calculated as a result of the convolution operation based on the detected image A first convolution layer comprising a subsampling layer; A second convolution layer, implemented as a non-subsampling layer, that repeats the convolution operation based on the output of the first convolution layer; And a third convolution layer including a sub-sampling layer for reducing the size of the feature map calculated as a result of the convolution operation based on the output of the second convolution layer, wherein the type of the kernel filter performing the convolution operation, The number and size of the kernel filters are independently configured for each of the plurality of convolution layers, and the type, number, and size of the kernel filters included in the plurality of convolution layers are individually calculated by learning of the learning engine.

Wherein the first convolution layer and the third convolution layer together perform padding such that the size of the output is kept equal to the size of the input when performing the convolution operation and the parameter of the padding is determined for each of the plurality of convolution layers Characterized in that the parameter size of the padding is calculated separately for each of the plurality of convolution layers by learning of the learning engine.

The first convolution layer and the third convolution layer transmit the output, which is obtained by successively performing normalization and nonlinear function application to the feature map calculated by the convolution operation, to the subsampling layer.

The second convolution layer performs padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed, normalizes the feature map calculated by the convolution operation, and applies the nonlinear function in order And a plurality of convolution layers for calculating an output, wherein a plurality of convolution layers constituting the second convolution layer are constituted by series coupling.

And the number of the plurality of convolution layers constituting the second convolution layer is calculated by the learning of the learning engine.

Wherein the second convolution layer includes: a first parallel layer for performing a normalization on a feature map calculated by a convolution operation on an output of the first convolution layer to calculate a first feature map; A first layer that sequentially performs normalization and nonlinear function application on the feature map calculated by the convolution operation on the output of the first convolution layer and a feature map that is calculated by convolution operation on the output of the first layer A second parallel layer including a second layer for calculating a second feature map by applying normalization to the first layer and the second layer, the first layer and the second layer being formed by serial combination; A fusion layer for performing a sum operation on the first feature map and the second feature map; And a noise reduction layer for applying a non-linear function to the output of the fusion layer.

Wherein the second layer performs padding so that the size of the output is kept equal to the size of the input when performing a convolution operation, and the parameter of the padding is determined by learning of the learning engine, And is calculated separately for each of the effect layer.

A gesture recognition apparatus according to another aspect is a gesture recognition apparatus for recognizing a hand gesture and controlling a peripheral device, the gesture recognition apparatus comprising: a verification unit for verifying a gesture display area at a user- A gesture usage verification unit operable to calculate a corresponding relationship between the gesture display area and the monitor so as to display the corresponding coordinates; A gesture registration unit for registering a device to be controlled by a user and a gesture used as a control signal; And a device controller for detecting and analyzing a gesture in the gesture image detected in the gesture display area and controlling the device in accordance with a control command corresponding to the gesture registered by the gesture registering unit, And a user-defined area for transmitting hand gesture information. The user-defined area is generated at a position spaced a certain distance from the monitor.

The correspondence relation between the gesture display area and the monitor is calculated by analyzing the monitor coordinates displayed on the monitor and the image of the hand area coordinates displayed on the gesture display area by the user along the monitor coordinates and based on the extracted reference coordinates .

The apparatus control unit includes a gesture classifier for detecting a gesture in a gesture image and analyzing a gesture type, and the gesture classifier is implemented using a gesture detection convolutional neural network including learned parameters.

Wherein the gesture classifier comprises convolution layers for calculating a feature map by performing a convolution operation and convolution layers composed of a complete connection layer for classifying detected images by analyzing feature maps calculated by the plurality of convolution layers Neural network; And a learning engine that learns the convolutional neural network and calculates parameters optimized for hand gesture detection, and the plurality of convolution layers include a sub-sampling layer for reducing the size of the feature map calculated on the basis of the detected image A first convolution layer comprising: A second convolution layer that includes a non-subsampling layer and repeats the convolution operation based on the output of the first convolution layer; And a third convolution layer including a sub-sampling layer for reducing the size of the feature map calculated on the basis of the output of the second convolution layer. The type, number, and size of kernel filters for performing the convolution operation are And the type, number, and size of the kernel filters included in the plurality of convolution layers are independently calculated by learning the learning engine, independently of the plurality of convolution layers.

Wherein the first convolution layer and the third convolution layer together perform padding such that the size of the output is kept equal to the size of the input when performing the convolution operation and the parameter of the padding is determined for each of the plurality of convolution layers Characterized in that the parameter size of the padding is calculated separately for each of the plurality of convolution layers by learning of the learning engine.

The first convolution layer and the third convolution layer transmit the output, which is obtained by successively performing normalization and nonlinear function application to the feature map calculated by the convolution operation, to the subsampling layer.

The second convolution layer performs padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed, normalizes the feature map calculated by the convolution operation, and applies the nonlinear function in order And a plurality of convolution layers for calculating an output, wherein a plurality of convolution layers constituting the second convolution layer are constituted by series coupling.

And the number of the plurality of convolution layers constituting the second convolution layer is calculated by the learning of the learning engine.

Wherein the second convolution layer includes: a first parallel layer for performing a normalization on a feature map calculated by a convolution operation on an output of the first convolution layer to calculate a first feature map; A first layer that sequentially performs normalization and nonlinear function application on the feature map calculated by the convolution operation on the output of the first convolution layer and a feature map that is calculated by convolution operation on the output of the first layer A second parallel layer including a second layer for calculating a second feature map by applying normalization to the first layer and the second layer, the first layer and the second layer being formed by serial combination; A fusion layer for performing a sum operation on the first feature map and the second feature map; And a noise reduction layer for applying a non-linear function to the output of the fusion layer.

Wherein the second layer performs padding so that the size of the output is kept equal to the size of the input when performing a convolution operation, and the parameter of the padding is determined by learning of the learning engine, And is calculated separately for each of the effect layer.

The present invention has the following effects with the above-described configuration.

The present invention has the effect of providing a design structure of a customized convolutional neural network (CNN) optimized for hand and hand gesture classification.

The present invention has an effect of automatically extracting various parameters optimized for hand gesture classification so that a control signal can be generated by hand shapes or gestures of various people by learning a hand gesture customized convolution neural network (CNN) .

The present invention provides a classifier composed of a convolutional neural network (CNN) optimized for classifying hand gestures, so that it is possible to classify the hand gestures by the distance noncontact accurately, thereby enhancing the device control performance by the hand gesture.

The present invention has the effect of freely setting peripherals and control signals for setting and controlling a desired position and a desired control area by the user.

1 is a block diagram showing the configuration of a gesture classifier according to an embodiment.
2 is a diagram illustrating the structure of a convolutional neural network (CNN) according to an embodiment.
3 is a conceptual diagram illustrating a convolution operation performed by the convolution layer of FIG.
4 is a conceptual diagram illustrating pulling performed by the subsampling layer of FIG.
FIG. 5 is a diagram illustrating an example in which the number of parameters and convolutional layers is calculated by learning the convolutional neural network (CNN) of FIG. 2. FIG.
6 is a diagram showing the structure of a convolutional neural network (CNN) according to another embodiment.
FIG. 7 is a diagram illustrating parameters calculated by learning the convolutional neural network (CNN) of FIG. 6;
8 is an exemplary diagram illustrating a plurality of second convolution layers connected in series in the structure of the convolutional neural network CNN of FIG.
FIG. 9 is an illustration of a learning image used for learning of a convolutional neural network (CNN) according to an embodiment.
10 is an illustration of the final classification results derived by the hand gesture classifier using the convolutional neural network (CNN) structure according to an embodiment.
11 is an overall conceptual diagram showing a device control system using a gesture according to another embodiment.
12 is a block diagram illustrating the gesture recognition apparatus of Fig.
13 is a diagram showing an example of gesture detection according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments of the present invention can be modified in various forms, and the scope of the present invention should not be construed as being limited to the following embodiments. This embodiment is provided to more fully describe the present invention to those skilled in the art. Furthermore, although specific terms have been used in the drawings and specification of the present invention, they have been used for the purpose of describing the present invention only and not for limiting the scope of the present invention described in the claims or the claims. Therefore, those skilled in the art will appreciate that various modifications and equivalent embodiments are possible without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Hereinafter, a convolution neural network for detecting a hand gesture according to the present invention and a device control system using a hand gesture will be described in detail with reference to the drawings.

1 is a block diagram showing the configuration of a gesture classifier according to an embodiment.

1, the gesture classifier 1 includes a convolutional neural network 11, and a learning engine 13, which may be implemented in hardware or software or through a combination of hardware and software . The function of the convolutional neural network 11, the learning engine 13 may also be stored in the memory so that the program type executed by the one or more processors In the gesture classifier 1 as shown in FIG.

The convolutional neural network 11 is learned in depth by the learning engine 13, and according to an embodiment, a hand gesture image can be recognized with high accuracy. Convolutional neural network 11 according to one embodiment is a kind of deep learning in the field of object recognition and is particularly applicable to CNN (Convolutional Neural Network) structure optimized for hand gesture or hand shape recognition . ≪ / RTI >

The learning engine 13 can learn the convolutional neural network 11 and calculate parameters. The hand can take various changes of the hand opening and gathering, the quick movement of the hand, the rotation, the finger shape, and the change of the shape can change quickly and largely, and there are cases where various hand gestures are used at the same time. Accordingly, the structure of the convolutional neural network 11 according to the embodiment is presented, and the learning engine 13 learns the convolutional neural network 11 to calculate parameters optimized for the hand gesture detection, thereby obtaining various hand shapes or hand gestures It is possible to complete the structure of the convolutional neural network 11 which correctly classifies it. Here, the parameter includes not only the type, number and size of the filter (ex, kernel filter performing convolution operation), but also the number of layers to be described below.

Hereinafter, in Figs. 2 to 8, the structure of the convolutional neural network 11 according to various embodiments will be described, and a parameter example optimized for the structure of the convolutional neural network 11 will be described.

FIG. 2 is a diagram illustrating a structure of a convolutional neural network (CNN) according to an embodiment. FIG. 3 is a conceptual diagram illustrating a convolution operation performed on the convolution layer of FIG. 2, FIG. 5 is an exemplary diagram illustrating the number of parameters and convolutional layers calculated by learning the convolutional neural network (CNN) of FIG. 2. FIG. 5 is a conceptual diagram illustrating pulling performed by the sampling layer.

Referring to Fig. 2, the convolutional neural network includes a convolution layer 21, and a full connection layer 23. [

The convolution layer 21 performs a convolution operation on the input image by using a convolution filter (or a kernel, a mask), and generates a feature map. In this case, the convolution operation extracts all n × n partial areas (or reception areas) that are possible in the entire area of the input image, and extracts n (n × n) (I.e., the sum of the inverse product between the filter and the partial area) after multiplying each unit element of the convolution filter composed of x n parameters. In addition, the feature map means image data in which various features of the input image are expressed, and the number of calculated feature maps does not necessarily correspond to the number of convolution filters and may not correspond according to the method of convolution operation .

The convolution layer 21 includes a plurality of convolution layers L ₁ , L ₂ , L ₃ , ..., L _N and a plurality of convolution layers L ₁ , L ₂ , L ₃ , L _N may have a first convolution layer 211 L ₁ , a second convolution layer 213 L ₂ , L ₃ , ... L _{N -1} , a third convolution layer 215 L _N ). ≪ / RTI >

Referring to FIG. 2, in the convolution layer 21 according to an embodiment, the first convolution layer 211 and the third convolution layer 215 are sub-sampled or pooled, The second convolution layer 213 does not perform the process of reducing the size of the feature map (POOL). Accordingly, after the convolution operation and the pulling operation are performed in the first convolution layer 211, the second convolution layer 213 repeats the convolution operation only several times without performing the pulling operation to increase the number of feature maps, i.e., So that depth learning is possible while maintaining the characteristics of the hand gesture images to be learned and classified.

The first convolution layer 211 includes a convolution layer L1a for inputting an image to be classified (hereinafter referred to as a detection image) as an input image (input) and performing a convolution operation (CONV) And a subsampling layer L1b that reduces the size of the feature map through sampling or pooling. The convolution layer L1a performs padding in performing the convolution operation so that the image sizes before and after the convolution operation are kept the same, according to an embodiment. In addition, the convolution layer L1a transfers the result of performing the normalization (NORM) and the nonlinear function (RELU or PRELU) on the feature map calculated by the convolution operation in turn to the subsampling layer L1b . That is, the convolution layer L1a transfers the feature map to the subsampling layer L1b to which the normalization and the nonlinear function RELU or PRELU are applied.

If the size of the input image is mxm, the size of one output (output, output image) is (m - (n - 1)) when a partial area x (m - (n - 1)). As a result, the output (output image) for the convolution operation is reduced by n - 1 in the horizontal and vertical directions, respectively, compared with the input image. For example, if you extract all of the subregions of size 3 x 3 on an input of size 6 x 6 and apply a convolution operation, the output will have a size of (6 - (3 - 1) - 1)) = 4 x 4. Thus, according to one embodiment, the first convolution layer 211 prevents the output from decreasing in size and performs a padding scheme to equalize the size of the input and the size of the output. Padding implies overlaying [n / 2] thickness of white space on each side of the input image using an odd number n. Where the square brackets represent the Gaussian symbols (or floor functions).

Also, the spacing between adjacent partial regions (or receiving spaces) is referred to as a stride, and if the stride is greater than 1, the horizontal and vertical lengths of the output are respectively smaller than the horizontal and vertical lengths of the input. For example, if the stride is 2, the horizontal and vertical length of the output is half of the horizontal and vertical length of each input.

The second convolution layer 213 does not include a subsampling layer that reduces the size of the feature map through sampling or pooling and performs a convolution operation (CONV) to generate a feature map And a plurality of convolution layers (L ₂ , L ₃ , ..., L _N-1 ). The plurality of convolutional layers L ₂ , L ₃ ,..., L _N-1 are connected in series so that the output of the preceding layer is the input of the next layer.

The convolution layers L ₂ , L ₃ , ..., L _N-1 constituting the second convolution layer 213 are set such that the size of the output is maintained equal to the size of the input during the convolution operation (CONV) (NORM) and applying the nonlinear function (RELU or PRELU) to the feature map calculated by the convolution operation in order to calculate the output, that is, the feature map.

Third keonbol convolutional layer 215 is a second keonbol receives the output of the convolutional layer 213 as input, and keonbol convolution operation keonbol convolution layer (L _NA) by performing (CONV) for generating a feature map, sampled ( and a subsampling layer (L _Nb ) that reduces the size of the feature map through sampling or pooling. The convolution layer L _Na may perform padding so that the size of the output is kept equal to the size of the input in performing the convolution operation (CONV) according to an embodiment, (NORM) to the subsampling layer (L _Nb ), and applying the nonlinear function (RELU or PRELU) to the subsampling layer (L _Nb ). At this time, the subsampling layer (L _Nb ) reduces the size of the image and transfers the reduced size to the complete connection layer (23).

The complete connection layer 23 analyzes the video image transmitted from the convolution layer 21 and finally judges to which category it belongs. The full connection layer 23 may be implemented as a global average pooling or a fully-connected layer. For example, the complete connection layer 23 analyzes the output, that is, the feature maps, received from the convolution layer 21 and determines whether the hand gesture displayed on the detected image corresponds to scissors, rock, It can be judged.

Referring to FIG. 3, according to one embodiment, the first convolution layer L ₁ includes l feature maps W ₁ , the second convolution layer L ₂ includes m feature maps W ₂ , The routing layer L ₃ includes n feature maps W ₃ , ..., , And the N-th convolution layer (L _N ) shows the calculation of g feature maps (W _N ). According to an embodiment, a first convolution layer (L ₁ ) for one input (100) generates a feature map of l results of a convolution operation, a second convolution layer (L ₂ ) (L) feature maps (W ₁ ), which are output of the feature map (L ₁ ), and performs convolution operation to generate m feature maps (W ₂ ). Similarly, the third convolution layer L ₃ receives m feature maps W ₂ , which are outputs of the second convolution layer L ₂ , and performs convolution operation to generate n feature maps W ₃ And repeats the convolution operation while passing through each convolution layer L in the same way. When the convolution operation of the last N-th convolution layer L _N is completed, the last g feature maps W _N are generated. At this time, the input 100, which is an input video, may have one or more channels. For example, when the input video 100 is an 8-bit video, it is 1-channel, and in the case of a 32-bit video, it is 3-channel. Here, the l feature maps W ₁ indicate image data in which various features of the input 100 are expressed.

The convolution filter (or the kernel filter) may be different in kind, number, size (nxn), etc. for each convolution layer L, and may also include a plurality of convolutional layers (L) The filters may also be implemented in different types. The type of the convolution filter may be a color-related filter such as red, green, or blue, or a filter having characteristics to find various other hand features.

4, the first convolution layer 211 (L ₁ ) includes a convolution layer 211a and a subsampling layer 211b, and a plurality of (N-1) th convolution layer _{(213: L 2, L 3} , ..., L N-1) is of only convolutional layer without subsampling layer, and the last N-th convolution layer (215: L _N) is a convolution layer (215a And a subsampling layer 215b.

For example, in the first convolution layer 211 (L ₁ ), the convolution layer 211a uses the kernel filter to convolute the convolution operation (CONV) for one input 100, The subsampling layer 211b performs pooling or sampling on the three feature maps 101 to calculate the feature map 102 whose size has been reduced. Thereafter, the convolution operation is repeatedly performed on the feature map 102 from the second to the (N-1) th convolution layers 213 (L ₂ , L ₃ , ..., L _N-1 ) The number increases. When the convolution operation is completed in the convolution layer 215a and the last g feature maps 100 'are calculated in the last convolution layer 215 (L _N ), the subsampling layer 215b generates the feature map 100 ') To calculate a reduced feature map 100''.

5 shows the learning data learned by the learning engine 13 through the convolutional neural network designed with the structure shown in FIG. 2, and the learning data includes parameters optimized for hand gesture detection, The number of layers N of the layer 21, that is, the number of the second convolution layers 213 (L ₂ , L ₃ , ..., L _N-1 ) is also included.

Referring to FIG. 5, the convolution layer 21 is composed of 10 layers (N = 10) in total, and the second convolution layer 213 is formed by connecting eight layers in series.

According to an embodiment, in the first convolution layer 211 (L ₁ ), the convolution layer L _1a performs a convolution operation (CONV) using a kernel filter of size 3x3 (ker 3) (NORM) and nonlinear function (PRELU) are sequentially applied to the map calculated by the convolution operation to calculate 16 final feature maps (out 16). At this time, the convolution layer L _1a performs padding (ex, zero padding) to cover spaces of [n / 2 = 1/2] on the top, bottom, right and left sides of the feature map, which is the result of the convolution operation, , n = 1). Thereafter, the subsampling layer L _1b performs pooling to extract the maximum value max, which is 3 × 3 (ker 3), the interval between adjacent storage spaces is 2 (stride 2) . Therefore, for one input (detection image), the output (i.e., feature map) of the first convolution layer 211 is sixteen.

The second convolution layer 213 (L ₂ , L ₃ , ..., L ₉ ) performs the convolution operation repeatedly. That is, the convolution layers 213 (L ₂ , L ₃ , ..., L ₉ ) from the second to the ninth are connected in series to each other so that the output of the preceding layer is input to the input of the subsequent layer, ), Normalization (NORM), and application of the nonlinear function (PRELU) in order to calculate the output, that is, the feature map.

According to one embodiment, the second layer L ₂ performs a convolution operation (CONV) using a kernel filter of 1 × 1 (ker 1) size and performs padding (n = 0) (out 16). The third layer L ₃ performs a convolution operation (CONV) and padding (n = 0) using a kernel filter having a size of 1 × 1 (ker 1) to generate 32 feature maps (out 32). The fourth layer L ₄ performs a convolution operation (CONV) and padding (n = 0) using a kernel filter having a size of 1 × 1 (ker 1) to generate 64 feature maps (out 64). The fifth layer L ₅ performs a convolution operation (CONV) and padding (n = 1) using a kernel filter having a size of 3 × 3 (ker 3) to generate 64 feature maps (out 64) . The sixth layer L ₆ performs a convolution operation (CONV) and padding (n = 0) using a kernel filter having a size of 1 × 1 (ker 1) to generate 64 feature maps (out 64) . The seventh layer L ₇ performs a convolution operation (CONV) and padding (n = 1) using a kernel filter having a size of 3 × 3 (ker 3) to generate 128 feature maps (out 128) . The eighth layer L ₈ performs a convolution operation (CONV) and padding (n = 0) using a kernel filter having a size of 1 × 1 (ker 1) to generate 128 feature maps (out 128) . The ninth layer L ₉ performs a convolution operation (CONV) and padding (n = 1) using a kernel filter having a size of 3 × 3 (ker 3) to generate 256 feature maps (out 256) . The parameters are learning data learned by the learning engine 13 through the convolutional neural network designed in the structure shown in FIG. 2, and are examples optimized for hand gesture detection.

Therefore, the convolution operation is repeatedly performed in the second convolution layer 213 (L ₂ , L ₃ , ..., L ₉ ). At this time, when the convolution operation is performed in each of the plurality of convolution layers 213 (L ₂ , L ₃ , ..., L ₉ ), a space of [n / 2] Overlapped padding (ex, zero padding) is also performed. According to the embodiment, in the second convolution layer 213, only the convolution operation is repeated without pooling or sampling processing.

The convolution layer L _10a of the last convolution layer L ₁₀ , that is, the convolution layer 215 of the third convolution layer 215, has 256 kernels as the input of the output (characteristic map) of the second convolution layer 213, performing the convolution operation by a filter (W _10), and convolution operation (CONV), the final output for the 256 and the calculated output do the normalization (NORM) and the non-linear function (PRELU) applied sequentially to the (characteristic map) . At this time, the convolution layer L _10a performs padding (ex, zero padding) to cover spaces of [n / 2 = 1/2] in the top, bottom, left and right of the feature map, which is the result of the convolution operation, , n = 1). Subsequently, the subsampling layer L _10b performs pooling (POOL) in which the size is 5 × 5 (ker 5), the interval between adjacent storage spaces is 2 (stride 2), and an average value is extracted, Reduce the size of the maps.

For one detected image, the output of the final convolution layer 21, that is, the feature map is 256, and the feature maps are transmitted to the completely connected layer 23 (FC) to judge to which category the detected image belongs .

The learning engine 13 learns through the convolutional neural network composed of the design structure shown in FIG. 2 and calculates the number of layers constituting the convolution layer 21 most suitable for the hand gesture classification N = 10). In addition, convolution operation type of kernel filter (W) to perform, the number (out), the size (ker) has the plurality of convolution layer _{(L 1, L 2, L} 3, ..., L 9, L 10) the kernel included in the same is not configured differently, and may be configured independently, the learning engine 13 is learning a plurality of convolutional layer in the _{(L 1, L 2, L} 3, ..., L 9, L 10) each Parameters such as the type, the number of outs, and the size ker of the filter W can be separately calculated as discussed above. In addition, the convolution operation performed when the padding of the parameter (n) also learning engine 13 is learning a plurality of convolution layers of be separately calculated for each _{(L 1, L 2, L} 3, ..., L 9, L 10) .

FIG. 6 is a diagram showing a structure of a convolutional neural network (CNN) according to another embodiment, FIG. 7 is an example of a parameter calculated by learning about a convolutional neural network (CNN) of FIG. 6, 6 is a diagram illustrating an example in which a plurality of second convolution layers are connected in series in a structure of a convolutional neural network CNN.

6, the convolutional neural network according to another embodiment includes a convolution layer 21 (211, 213, 215), and a full connection layer 23.

The convolution layer 21 includes five convolution layers L ₁ , L ₂ , L ₃ , L ₄ and L ₅ and the five convolution layers L ₁ , L ₂ , L ₃ , L _4, L ₅₎ of the first convolutional layer (211 according to the function: L _1), the second convolutional layer _{(213: L 2, L 3} , L 4), the third convolution layer (215: L ₅₎ ≪ / RTI >

The convolutional neural network shown in Fig. 6 may be designed so that the second convolution layer 213 differs from the convolutional neural network shown in Fig. 2 only, and the remaining configurations are the same. That is, a first convolutional layer of Figure 6 (211: L ₁₎ and the third convolution layer (215: L ₅₎ is a first convolutional layer shown in Figure 2 (211: L ₁₎ and the third convolution And the layer 215 (L _N ), respectively. However, the parameters of the layer may be calculated differently depending on the learning of the learning engine 13. 6, the first convolution layer 211 (L ₁ ) and the third convolution layer 215 (L ₅ ) perform a subsampling or a pooling process to reduce the size of the feature map , But the second convolution layer 213 (L ₂ , L ₃ , L ₄ ) does not reduce the size of the feature map.

The convolutional neural network designed as shown in FIG. 6 has a feature map that performs a convolution operation at a time instead of reducing the convolution operation number only by increasing the processing speed by reducing the convolution operation execution frequency It is designed to combine feature maps that have been subjected to two convolution operations into a single map and to use the combined feature map in the next step so that the accuracy of hand gesture recognition can be maintained. The first convolution layer 211 (L ₁ ) and the third convolution layer 215 (L ₅ ) of FIG. 6 correspond to the first convolution layer 211 (L ₁ ) and the third convolution layer 215: L ₇ ). Therefore, the second convolution layer 213 (L ₂ , L ₃ , L ₄ ) will be described in detail below.

The second convolution layer 213 (L ₂ , L ₃ , L ₄ ) includes a first parallel layer 2131, a second parallel layer 2133: 2133a, 2133b, a fusion layer 2135, 2137). The first parallel layer 2131 and the second parallel layer 2133 (2133a, 2133b) are connected in parallel.

The first parallel layer 2131 performs a convolution operation (CONV) with the output of the first convolution layer 211 (L ₁ ) as an input, performs normalization (NORM) on the feature map calculated by the convolution operation And calculates a first feature map.

Second parallel layer (2133: 2133a, 2133b) includes: a first convolutional layer: normalized to the the output of the (211 L ₁₎ to the input to perform a convolution operation (CONV) is calculated by convolution operation characteristic map (NORM And a nonlinear function (ex, PRELU or RELU), and a first layer 2133a performing an output of the first layer 2133a as an input and performing a convolution operation (CONV) And a second layer 2133b for applying a normalization (NORM) to the calculated feature map to calculate a second feature map. The first layer 2133a and the second layer 2133b are connected in series.

The fusion layer 2135 performs fusion and concentration on the first feature map and the second feature map to generate one map. According to the embodiment, when the first feature map and the second feature map have the same horizontal and vertical sizes, but when the number of convolution operations used for each feature map extraction is different from the output size, the first feature map and the second feature The 3D size of the map may be different. It is possible to extract features from various aspects of one image image by fusion or concentration of the first feature map and the second feature map obtained by utilizing structures having different convolution operation execution times. In other words, by extracting features using different convolution operations (operation execution frequency and weight), it is possible to extract multi-faceted features for a single input, and as a result, hand gesture classification performance can be improved. Also, since the first feature map and the second feature map are normalized (NORM) before coupling, the characteristics of the first feature map and the second feature map are maintained.

The noise reduction layer 2137 performs a nonlinear function (ex, PRELU) application to the feature maps combined by fusion of the first feature map and the second feature map. A noise reduction layer (2137) is the third layer, the convolution output: and passed to (215 L _5), the third convolution layer: performs a convolution operation and pooling (215 L _5).

FIG. 7 shows learning data learned by the learning engine 13 through the convolutional neural network designed with the structure shown in FIG. 6, and the learning data is implemented with parameters optimized for hand gesture detection.

As shown in FIG. 2 and FIG. 5, if the convolution operation is repeatedly performed by successively connecting a series of convolution layers, it is possible to perform deep learning that increases the accuracy, while the processing speed becomes slow have. However, if the number of convolution operation execution times (learning depth), that is, the number of kernel filters and feature maps in the learning structure shown in FIGS. 2 and 5 is somewhat reduced, the processing speed becomes faster. However, May be impossible, and then the accuracy of the handset classification may be lowered. Thus, according to another embodiment, a design structure capable of improving the processing speed without lowering the accuracy of the classification is shown in Fig. 6, and a parameter capable of classifying the hand gesture with high accuracy during learning through this design structure Is shown in Fig. In addition, according to another embodiment, a convolutional neural network designed by connecting a second convolution layer 213 according to another embodiment shown in FIG. 6 in series a plurality of times is shown in FIG.

8, the convolutional neural network according to another embodiment includes a second convolution layer 213 (L ₂ , L ₃ , L ₄ ) structure according to another embodiment shown in FIG. 6, And the first convolution layer 211 (L ₁ ) and the third convolution layer 215 (L ₅ ) are constructed in the same manner.

The convolutional neural network shown in FIG. 8 is configured such that the second convolution layer 213 (L ₂ , L ₃ , L ₄ ) is repeated a plurality of times so that a plurality of objects (eg, hand gestures) Effective. In other words, since the structure in which the second convolution layer 213 (L ₂ , L ₃ , L ₄ ) is inserted once may have a longer time for processing the learning and classification than the case where the second convolution layer 213 It is effective to classify many kinds of objects.

FIG. 9 is an exemplary view of a learning image used for learning of a convolutional neural network (CNN) according to an embodiment, and FIG. 10 is a diagram illustrating an example of a learning image using a convolutional neural network (CNN) Fig.

The image shown in Fig. 9 shows an example of an input used for learning during learning through the convolutional neural network designed according to the embodiment of the learning engine 13. The learning engine 13 can learn by using various images that have undergone various processes by performing various processes on one image such as size, rotation, inversion, histogram, smoothing, blurring, gamma conversion, brightness change, and perspective distortion.

Referring to FIG. 10, a hand gesture classifier using a convolutional neural network (CNN) structure according to an embodiment shows a result of analyzing a hand image image. 10 (a), a gesture candidate region is detected by a solid line box, and the analysis table of FIG. 10 (b) shows the final result of analyzing the N feature maps for the candidate region by the classifier. By examining the analysis table of FIG. 10 (b), it is possible to determine a gesture having a probability of 0.5 or more and the highest probability as the final result in the final classification result of the classifier. Since the gesture 1 has a probability of 99.89% or more with respect to the candidate region, the final result can be judged as the gesture 1. For example, gesture 1 may correspond to a 'beam' of various hand gestures (ex, scissors, rock, beam).

11 is an overall conceptual diagram showing a device control system using a gesture according to another embodiment.

11, a device control system using a gesture may include a video input device 1, a monitor 2, a gesture display area 3, and a gesture recognition device 4. [

The image input apparatus 1 acquires a detection image to recognize a hand shape or a hand gesture of a user. For example, the image input device 1 may be implemented as a depth recognition camera, a stereo camera, and a color camera (ex, kinect camera), and may acquire moving images and still images as detected images. When the detected image is a moving image, it may be composed of a plurality of consecutive frames. Also, the detected image may include a color image, a depth image, and a color-depth (RGB-C) image.

The monitor 2 displays an image corresponding to a shape, an operation, and a hand position of a hand moving by the user on the gesture display area 3. [ Therefore, the user can confirm via the monitor 2 whether the hand shape and the hand operation intended by the user are transmitted to the gesture recognition device 4 within the range of the gesture display area 3 without leaving the gesture display area 3 . If a hand shape or a hand gesture different from that intended by the user is transmitted to the gesture recognition device 4 and displayed on the monitor 2, the user modifies the hand gesture or hand gesture, Can be displayed.

The gesture display area (3) is created as a user-defined area remotely as an area for conveying information on the hand shape or the hand movements of the user. According to the embodiment, the user can create a gesture display area 3 by defining an arbitrary area that displays a control signal remotely at a position (T) designated by himself, rather than at a fixed position.

11, when four monitor coordinates (M1, M2, M3, M4) are displayed on the monitor 2, the user sets the four monitor coordinates M1, M2, M3 , M4), the hand area coordinate (G) is displayed at four points in the air. The gesture recognition device 4 analyzes the image obtained by the image input device 1 and obtains the reference coordinates B1, B2, B3, B4). According to one embodiment, the gesture display area 3 may correspond to a region connecting the reference coordinates B1, B2, B3, and B4. Since the reference coordinates B1, B2, B3 and B4 and the monitor coordinates M1, M2, M3 and M4 are inverted left and right, the hand area coordinates G for the four points acquired from the image are reversed To generate reference coordinates (B1, B2, B3, B4). The remote location T specified by the user is a position that is less than or not greater than a critical distance from the image input device 1 and the monitor 2 to a point located a certain distance D away from the monitor 2. At the same time, Is defined as a position that does not deviate from the range of the angle of view that can be obtained.

The gesture recognition device 4 analyzes a hand shape, a hand operation, and various combinations of the user's hand in the image acquired by the image input device 1, and controls various devices according to the analysis result. According to the embodiment, the gesture recognition device 4 can accurately recognize the hand shape, the hand gesture and various combinations thereof using the learned hand shape and the hand gesture classifier, and the classifier according to the above-described various embodiments Can be implemented using a convolutional neural network (CNN). Thus, the classifier can be realized as a hand gesture classifier with high accuracy and fast analysis speed.

In the following description, a hand gesture (gesture) that is a part of the user's body is described, but it does not exclude the shape or motion of a part of the body, such as other faces, arms, or the like. In addition, the hand gesture described below is not limited to the hand operation itself, but is defined as including a hand shape.

FIG. 12 is a block diagram illustrating the gesture recognition apparatus of FIG. 11, and FIG. 13 is a diagram illustrating an example of gesture detection according to an embodiment.

12, the gesture recognition apparatus 4 may include a gesture use verification unit 41, a gesture registration unit 43, and a device control unit 45, which may be implemented by hardware or software, And a combination of software. The functions of the gesture usage verification unit 41, the gesture registration unit 43, and the device control unit 45 are stored in the memory, and the functions of the gesture recognition unit 4, May be implemented in the gesture recognition device 4 in the form of a program executed by one or more processors.

11 and 12, the gesture use verification unit 41 verifies whether the user can control the peripheral device using the gesture at the arbitrarily specified remote position T. [ The gesture usage verification unit 41 also calculates a conversion matrix indicating a correspondence relationship between the monitor 2 and the gesture display area 3. [

The remote position T may be defined as a position that does not deviate from a critical distance within a predetermined area of the front surface of the monitor 2. [ Therefore, the gesture usage verification unit 41 determines whether the current position of the user does not satisfy the defined remote position T, for example, when the current position is out of the critical distance, or the image input apparatus 1 If it is out of the angle at which the image can be acquired, it can be determined that it can not be controlled at the current position. Hereinafter, the description is based on the assumption that the user's current position corresponds to the defined remote position and thus the controllability is verified.

Referring to FIGS. 11 and 12, the gesture use verification unit 41 displays four monitor coordinates (M1, M2, M3, and M4) on the monitor 2 to guide the user's gesture. The gesture use verification unit 41 displays the coordinates G at four positions of the air space along the four monitor coordinates M1, M2, M3 and M4 at the remote position T which the user himself designates, The image of the hand area coordinate G obtained by the device 1 is analyzed to derive the correspondence between the monitor 2 and the gesture display area 3. [ The correspondence between the monitor 2 and the gesture display area 3 is determined by the conversion between the monitor coordinates M1, M2, M3, M4 and the reference coordinates B1, B2, B3, B4 of the gesture display area 3 (T), but is not limited thereto. Here, the transformation matrix T allows the coordinates of the movement of the hand gesture at the remote position T to be implemented in the coordinates of the monitor 2 phase. For example, if the matrix M for the four monitor coordinates (M1, M2, M3, M4) and the matrix B for the detected reference coordinates (B1, B2, B3, B4) From the arithmetic operation, the transformation sequence T can be derived from T = M x B ^{- 1} .

The gesture registering unit 43 can register a gesture to be used as a control signal and a device to be controlled by the gesture on the gesture display area 3 based on the user's selection. At this time, the user can control the device with various control signals by selecting different kinds of gestures for controlling one device.

The device control unit 45 displays the gesture image displayed by the user in the area of the reference coordinates B1, B2, B3 and B4 set in the gesture use verification unit 41, that is, in the gesture display area 3, ) To detect and classify the gesture. Further, the device control section 45 can control the device according to the classified gesture. The user can take a gesture and generate an event, and can generate a control signal such as On, Off, Sound reproduction, Balance control and the like through a combination of gestures of various shapes, and can remotely control the device to be controlled.

According to the embodiment, the device control unit 45 may implement the detector using the convolutional neural network (CNN) according to the various embodiments described above. The device control unit 45 can detect and classify the hand gesture with high precision using the detector (classifier) in the image acquired by the video input device 1. [

12, the device control unit 45 controls the movement of the entire image acquired by the image input device 1 in the image within the area of the gesture display area 3, that is, the reference coordinates B1, B2, B3, B4 Navigate to the part where you are. The device control unit 45 can search for a moving part in an image based on a dense optical flow using a difference between a previous frame and a current frame, -Kanade or Gunnar Farneback) are available.

More specifically, the device control unit 45 extracts the magnitude and angle of the motion vector in the motion block, grasps the size of the motion vector in an arbitrary block, and determines the number of the motion vectors larger than the threshold Is greater than a certain value, the block is identified as a motion block. If the device control unit 45 determines that the motion is progressing and there is a block with a stop, the device control unit 45 regards the block as an area including an object to be detected. According to the embodiment, when selecting the candidate region including the detection target, the device control unit 45 performs detection based on the uppermost block or the region including the detection target on the basis of the uppermost motion vector coordinate among the blocks in which the motion stops can do. This reflects the characteristic that the hand is positioned in the block region located at the uppermost position in the standing posture in terms of arm and hand motion characteristics. The device control unit 45 extracts features from the block region determined to include the detection target, or detects the candidate region using the sliding window method in the block region, and then applies the classification method to recognize the gesture .

Referring to FIG. 13, a table 451 shows examples of coordinates of a motion block and an optical flow field motion vector value. For example, one feature point can be extracted per 10 × 10, and if the magnitude of the motion vector is 4.5 or more, it can be determined that the motion is a pointer. A screen 453 shows an example of a partial tube detection target block and a detected hand area with a change in motion.

While the specification contains many features, such features should not be construed as limiting the scope of the invention or the scope of the claims. In addition, the features described in the individual embodiments herein may be combined and implemented in a single embodiment. Conversely, various features described in the singular < Desc / Clms Page number 5 > embodiments herein may be implemented in various embodiments individually or in combination as appropriate.

Although the operations have been described in a particular order in the figures, it should be understood that such operations are performed in a particular order as shown, or that all described operations are performed to obtain a sequence of sequential orders, or a desired result . In certain circumstances, multitasking and parallel processing may be advantageous. It should also be understood that the division of various system components in the above embodiments does not require such distinction in all embodiments. The above-described program components and systems can generally be implemented as a single software product or as a package in multiple software products.

The method of the present invention as described above can be implemented by a program and stored in a computer-readable recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto optical disk, etc.). Such a process can be easily carried out by those skilled in the art and will not be described in detail.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. The present invention is not limited to the drawings.

2: Convolution Neural Network 21: Convolution Layer
211: first convolution layer 213: second convolution layer
215: third convolution layer 23: fully connected layer

Claims (17)

A gesture classifier for learning parameters of a hand gesture detection convolutional neural network,
A convolution neural network consisting of a plurality of convolutional layers for calculating a feature map by performing a convolution operation and a complete connection layer for classifying detected images by analyzing feature maps calculated in the plurality of convolutional layers; And
And a learning engine that learns the convolution neural network to calculate parameters optimized for hand gesture detection
Wherein the plurality of convolution layers comprise a first convolution layer including a sub-sampling layer for reducing a size of a feature map calculated as a result of a convolution operation based on a detected image; A second convolution layer, implemented as a non-subsampling layer, that repeats the convolution operation based on the output of the first convolution layer; And a third convolution layer including a sub-sampling layer for reducing the size of the feature map calculated as a result of the convolution operation based on the output of the second convolution layer,
The type, number, and size of filters for performing the convolution operation are configured independently for each of the plurality of convolution layers, and the types, numbers, and sizes of the filters included in the plurality of convolution layers are individually The gesture classifier comprising: The method according to claim 1,
The first convolution layer and the third convolution layer perform padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed,
Wherein the parameter of the padding is configured independently for each of the plurality of convolution layers, and the parameter size of the padding is calculated separately for each of the plurality of convolution layers by the learning of the learning engine. 3. The method of claim 2,
Wherein the first convolution layer and the third convolution layer comprise:
Wherein the output of the normalization and the application of the nonlinear function to the feature map calculated by the convolution operation are transferred to the subsampling layer. The method of claim 3,
Wherein the second convolution layer comprises:
A plurality of convolution operations for performing padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed, performing the normalization on the feature map calculated by the convolution operation, and applying the non- Layers,
And the plurality of convolution layers constituting the second convolution layer are configured in a serial combination. 5. The method of claim 4,
Wherein the number of convolution layers constituting the second convolution layer is a number of convolution layers,
Wherein the gesture classifier is calculated by learning the learning engine. The method of claim 3,
Wherein the second convolution layer comprises:
A first parallel layer for performing a normalization on a feature map calculated by a convolution operation on the output of the first convolution layer to calculate a first feature map;
A first layer that sequentially performs normalization and nonlinear function application on the feature map calculated by the convolution operation on the output of the first convolution layer and a feature map that is calculated by convolution operation on the output of the first layer A second parallel layer including a second layer for calculating a second feature map by applying normalization to the first layer and the second layer, the first layer and the second layer being formed by serial combination;
A fusion layer for performing a sum operation on the first feature map and the second feature map; And
And a noise reduction layer that performs a non-linear function application to the output of the fusion layer. The method according to claim 6,
The second layer performs padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed,
Wherein the parameters of the padding are computed separately for each of the plurality of convolution layers as the parameter size of the padding due to learning of the learning engine. A gesture recognition device for recognizing a hand gesture and controlling a peripheral device,
A gesture usage verification unit for verifying a gesture display area at a user-specified remote location and calculating a correspondence relationship between the gesture display area and monitors so that coordinates of the gesture motion in the gesture display area are displayed in corresponding coordinates on a monitor;
A gesture registration unit for registering a device to be controlled by a user and a gesture used as a control signal; And
And a device controller for detecting and analyzing the gesture in the gesture image detected in the gesture display area and controlling the device according to a control command corresponding to the gesture registered by the gesture registering unit
Wherein the gesture display area is created by user definition at a position spaced a certain distance from the monitor as a user defined area in which the user delivers hand gesture information. 9. The method of claim 8,
Wherein the correspondence relationship between the gesture display area and the monitor is determined by:
Wherein the gesture recognition device is calculated on the basis of monitor coordinates displayed on a monitor and an extracted reference coordinate by analyzing an image of the hand area coordinates displayed by the user in the gesture display area along the monitor coordinates. 10. The method of claim 9,
Wherein the device control unit includes a gesture classifier for detecting a gesture in the gesture image and analyzing the type of the gesture,
Wherein the gesture classifier is implemented using a gesture detection convolutional neural network that includes learned parameters. 11. The method of claim 10,
The gesture classifier comprising:
A convolution neural network consisting of a plurality of convolutional layers for performing a convolution operation to calculate a feature map and a complete connection layer for classifying detected images by analyzing feature maps calculated by the plurality of convolutional layers; And
And a learning engine that learns the convolution neural network to calculate parameters optimized for hand gesture detection
Wherein the plurality of convolution layers comprise: a first convolution layer including a sub-sampling layer for reducing a size of a feature map calculated based on a detected image; A second convolution layer that includes a non-subsampling layer and repeats the convolution operation based on the output of the first convolution layer; And a third convolution layer including a sub-sampling layer for reducing the size of the feature map calculated on the basis of the output of the second convolution layer,
The type, number, and size of filters for performing the convolution operation are configured independently for each of the plurality of convolution layers, and the types, numbers, and sizes of the filters included in the plurality of convolution layers are individually The gesture recognition apparatus comprising: 12. The method of claim 11,
The first convolution layer and the third convolution layer perform padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed,
Wherein the parameter of the padding is independently configured for each of the plurality of convolution layers, and the parameter size of the padding is calculated separately for each of the plurality of convolution layers by the learning of the learning engine. 13. The method of claim 12,
Wherein the first convolution layer and the third convolution layer comprise:
Wherein the output of the normalization and the application of the nonlinear function to the feature map calculated by the convolution operation are transferred to the subsampling layer. 14. The method of claim 13,
Wherein the second convolution layer comprises:
A plurality of convolution operations for performing padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed, performing the normalization on the feature map calculated by the convolution operation, and applying the non- Layers,
And the plurality of convolution layers constituting the second convolution layer are constituted by series combination. 15. The method of claim 14,
Wherein the number of convolution layers constituting the second convolution layer is a number of convolution layers,
And the gesture recognition device is calculated by learning the learning engine. 14. The method of claim 13,
Wherein the second convolution layer comprises:
A first parallel layer for performing a normalization on a feature map calculated by a convolution operation on the output of the first convolution layer to calculate a first feature map;
A first layer that sequentially performs normalization and nonlinear function application on the feature map calculated by the convolution operation on the output of the first convolution layer and a feature map that is calculated by convolution operation on the output of the first layer A second parallel layer including a second layer for calculating a second feature map by applying normalization to the first layer and the second layer, the first layer and the second layer being formed by serial combination;
A fusion layer for performing a sum operation on the first feature map and the second feature map; And
And a noise reduction layer that performs a nonlinear function application to the output of the fusion layer. 17. The method of claim 16,
The second layer performs padding so that the size of the output is kept equal to the size of the input when the convolution operation is performed,
Wherein the parameter of the padding is calculated separately for each of the plurality of convolution layers by the learning of the learning engine.

Images