CN110046544A - Digital gesture identification method based on convolutional neural networks - Google Patents

Abstract

The invention relates to a digital gesture recognition method based on a convolutional neural network, comprising the following steps: using a Kinect depth camera to collect 10 types of digital gesture images, and filtering the gesture images; Sample set, the method is as follows: perform morphological preprocessing on the filtered gesture image; classify and mark image information, obtain sample set of each digital gesture representation, and classify into training set and test set; construct convolutional neural network-CNN; Input the training sample set, extract image features, and perform classification training; use the trained convolutional neural network to identify the images in the test data set.

Description

Digital gesture recognition method based on convolutional neural network

technical field

The invention relates to the fields of deep learning and image processing, in particular to a gesture recognition based on a convolutional neural network (CNN).

Background technique

Gesture recognition has always been a popular research topic. Digital gesture recognition needs to solve data collection, image processing and selection, selection of input sample expression, selection of pattern recognition classifiers, and supervised training of recognizers based on sample sets. and many other issues.

Gestures are an inseparable part of human-to-human communication, and gesture recognition technology has also opened up a new situation in the interaction between humans and machines, devices or computers. The era of data gloves has developed to the stage of pattern classification based on computer vision. The current popular vision-based gesture recognition is divided into three stages: segmentation, feature extraction and recognition. Gesture segmentation is the basis of gesture recognition, and the goal is to segment images from complex backgrounds. Gestures. Since skin color has certain clustering characteristics in color space, most of the current gesture segmentation methods use color features (YUV, HSV, YCbCr, etc.) or geometric features (such as ellipse model, graph model) of skin color. Finish.. The main direction of the current research is: the current research work is to separate gesture detection and recognition. On the basis of continuous refinement of recognition technology, how to apply new technologies of mathematical morphology, neural network algorithm and genetic algorithm to gestures being identified.

The biggest difficulty in gesture recognition research is that in the data processing stage, the system performs frame separation processing on the video captured by the camera, separates a single gesture image from the video frame, and preprocesses the data such as smoothing and sharpening. Then it is detected whether there is a gesture image, and if a gesture image is detected, the gesture image is separated from the background. In the gesture analysis stage, the gesture features are detected, and then the selected gesture model is used to estimate the

corresponding characteristic parameters. In the identification and classification stage, through feature extraction and model parameter estimation, various classification algorithms are used to classify points or trajectories in the parameter space into different subspaces, and finally the identification information is converted into specific meanings for practical application. The influence of illumination and pixels will have different degrees of influence on the accuracy of the recognition system. The Kinect depth image has nothing to do with ambient lighting and shadows, and the pixels can clearly express the surface geometry of the scene. The Kinect depth camera is a motion sensing input device created by Microsoft for its Xbox360 game console and Windows platform PC. As a somatosensory Peripheral, it is actually a 3D somatosensory camera using a new spatial positioning technology (Light Coding). Kinect has three lenses, the middle lens is an RGB color camera, and the left and right lenses are respectively composed of infrared emitters and infrared CMOS cameras. 3D depth sensor. Utilize real-time motion capture, image recognition, microphone input, voice recognition, community interaction, and more, allowing players to interact with the Xbox 360 through natural user interface technology using body gestures and voice commands.

As an important part of computer intelligent interface, digital gesture recognition is of great significance. The improvement of this technology can greatly improve the efficiency of computer use. It is the most ideal input method in the fields of office automation, smart home, and robot interactive control in the future. At present, the problems of gesture recognition are mainly reflected in three aspects: 1) the problem of data set collection; 2) the problem of preprocessing of gesture images, how to accurately separate the gesture pose detection in the picture is one of the problems in image detection. The main aspects; 3) Combining digital gesture recognition and neural network to achieve the best recognition effect.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a digital gesture automatic recognition method based on a convolutional neural network with better recognition effect. The technical solution is as follows:

A digital gesture recognition method based on convolutional neural network, comprising the following steps:

(1) Use the Kinect depth camera to collect 10 types of digital gesture images, and filter the gesture images; use the filtered images to establish a sample set for each digital gesture representation, the method is as follows: perform morphological preprocessing on the filtered gesture images ; Classify and mark image information, obtain sample sets of digital gesture representations, and classify them into training sets and test sets;

(2) Build a convolutional neural network—CNN:

(2a) Import the digital gesture images of each category into the convolutional neural network as the inputs layer, whose size is [320, 320, 3, 59];

(2b) Construct an 8-layer convolutional neural network, and perform operations such as convolution, downsampling, and pooling on each pixel of the input image to obtain the maps feature map of each layer;

(2c) Use the output of each layer as the input of the next layer, go through 8 layers before and after, and finally converge on the fully connected fc layer, and output the result through the output layer softmax classifier;

(3) Input the training sample set, extract image features, and perform classification training;

(3a) Using the softmax classifier to classify the image feature vector;

(3b) using the convolutional neural network algorithm to train the training sample set to obtain the model .mat file after training;

(4) Use the trained convolutional neural network to identify images in the test dataset.

Wherein, the filtering method of step (1) is preferably as follows: based on the depth image filtering algorithm of joint bilateral filter, the depth image and the color image of the gesture image collected by the Kinect lens at the same moment are used as input, and the depth image is calculated with a Gaussian kernel function. The spatial distance weight and the grayscale weight of the RGB color image are multiplied to obtain a joint filtering weight, and a joint bilateral filter is designed. The filtering result of this filter is used to perform a convolution operation with the noise image. Kinect depth image filtering.

The gesture recognition based on the convolutional neural network (CNN) of the present invention can efficiently realize the acquisition of gesture feature depth images and further denoising processing, as well as the automatic recognition and output of gesture digital feature images, and the recognition accuracy can reach about 93%. . The algorithm has a certain robustness to illumination changes, simple geometric deformation, and additional noise, and can be used in the related fields of digital gesture feature recognition; after the algorithm is extended, it can also be used for automatic recognition of other gesture features.

Description of drawings

Fig. 1 is the algorithm flow chart of the present invention.

Figure 2 The preprocessed dataset.

Figure 3 shows the overall training diagram of the network.

Figure 4 shows the recognition results of the images in the test set.

Detailed ways

The purpose of the present invention is to use the Kinect depth camera to collect the data set, to realize the automatic recognition of digital gestures based on the constructed convolutional neural network after denoising through image morphological preprocessing, so as to meet the practical requirements. It mainly includes the following steps:

(1) Obtain a sample set of digital gesture representations;

(1a) Collect 10 types of digital representation data sets;

(1b) Morphological preprocessing is performed on the collected images;

(1c) classify and mark image information, and classify the collected digital gesture data set into training set and test set;

(2) Build a convolutional neural network—CNN:

(2a) Import the digital gesture images of each category into the convolutional neural network as the inputs layer, whose size is [320, 320, 3, 59];

(2b) Construct an 8-layer convolutional neural network, and perform operations such as convolution, downsampling, and pooling on each pixel of the input image to obtain the maps feature map of each layer;

(2c) Use the output of each layer as the input of the next layer, go through 8 layers before and after, and finally converge on the fully connected fc layer, and output the result through the output layer softmax classifier;

(3) Input the training sample set, extract image features, and perform classification training;

(3a) Using the softmax classifier to classify the image feature vector;

(3b) using the convolutional neural network algorithm to train the training sample set to obtain the model .mat file after training;

(4) Use the trained convolutional neural network to automatically identify the pictures in the test data set.

Input the test sample set into the trained convolutional neural network, that is, test the test sample set by inputting the trained model .mat file, realize the automatic recognition of each digital gesture picture, and output the test result.

The specific steps of the present invention are described as follows in conjunction with accompanying drawing 1:

(1) Obtain a sample set of each digital gesture feature;

10 types of images containing digital gesture features are collected as datasets, including 1050 digital gestures representing 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9, for a total of 10,500 training datasets. The digital gesture feature maps of each category are collected from different angles and lights using the Kinect depth camera. In view of the fact that the depth images collected by the Kinect lens generally have noise and black holes, and the effect of direct application is poor, we use the depth image filtering algorithm based on the joint bilateral filter to take the depth image and color image collected by the Kinect lens at the same time as the input. . First, use the Gaussian kernel function to calculate the spatial distance weight of the depth image and the grayscale weight of the RGB color image, then multiply these two weights to obtain the joint filtering weight, and use the fast Gaussian transform to replace the Gaussian kernel function design out a joint bilateral filter. Finally, the filtering result of this filter is used to perform convolution operation with the noise image to realize the Kinect depth image filtering. Then randomly select 10,000 of them as the training sample set, carry out manual classification and labeling, and the remaining 500 are used as the test sample set; finally, 10,000 training sample sets and 500 test sample sets are obtained.

(2) Construct Convolutional Neural Network - CNN:

The digital gesture images of each category are imported into the convolutional neural network as the inputs layer, the size of which is [320, 320, 3, 59]; an 8-layer convolutional neural network is constructed, and each pixel of the input image is convolved, down-sampled, Pooling and other operations are performed to obtain the maps feature map of each layer; the output of each layer is used as the input of the next layer, and after 8 layers before and after, it is finally converged in the fully connected fc layer, and the result is output through the output layer softmax classifier;

Input the training sample set, extract the image features, and perform classification training; use the softmax classifier to classify the image feature vector; use the convolutional neural network algorithm to train the training sample set, and obtain the trained model .mat file;

The basic flow of the convolutional neural network algorithm is as follows: randomly initialize the network weights and neuron thresholds; carry out forward propagation according to formula (1):

The input and output of hidden layer neurons and output neurons are calculated hierarchically; where E represents the output error, d represents the authenticity, w _jk , vi _ij represent the weights and thresholds of each layer, respectively.

Error backpropagation is performed according to formula (2):

Among them, θ is the learning rate parameter of the backpropagation algorithm (theta=0.001 in the present invention), n represents the number of input vectors (n=320*320*3*59 in the present invention), m represents the output vector of the hidden layer The number (in the present invention m changes with the change of the output vector of the convolution layer), l represents the number of output vectors of the output layer (in the present invention l=1*1*4096*59), the negative sign in the above formula represents Gradient descent in the weight space, that is, the weights that make the value of E decrease change direction. The weights and thresholds are modified by the above formulas until the termination conditions are met.

(3) Using the trained convolutional neural network to automatically identify the digital gesture feature set.

Input the test sample set into the trained convolutional neural network, that is, test the test sample set by inputting the trained model .mat file, realize the automatic recognition of each digital gesture picture, and output the test result.

Compared with the prior art, the present invention has the following features and advantages:

First: the present invention applies the convolutional neural network to digital gesture feature recognition, the data set includes: 10 types of digital gesture feature images, including 0, 1, 2, 3, 4, 5, 6, 7, 8, There are 1050 digital gestures of 9 each, a total of 10500 training data sets. The digital gesture feature maps of each category are collected from different angles and light by the Kinect depth camera, and then the collected 10500 images are morphologically processed. Filtering and denoising processing. The test results of the test set show that most of the gesture feature images are correctly recognized, as shown in Figure 4. Table 2 is the comparison of the recognition accuracy of the current traditional algorithm and the method of the present invention. It can be seen from the table that the recognition accuracy of the method of the present invention is relatively good.

Second, the present invention builds a parallel Pooling layer. The advantage of this structure is that it can effectively reduce top-1 (the correct answer with the highest probability) and top-5 (the top 5 the highest probability contains the correct answer). In the structure of CNN, the feature extraction layer can connect the input of each neuron with the local receptive field of the previous layer, and extract the local features of this layer at the same time. Once a local feature is extracted, its positional relationship with other feature vectors is also determined, which is helpful for feature vector extraction.

Third, the present invention adopts the Kinect depth image filtering algorithm based on joint bilateral filter when collecting digital gesture images, which can better retain the relevant features of the original image and help to improve the recognition accuracy.

Claims (2)

1. A digital gesture recognition method based on convolutional neural network, comprising the following steps: (1) Use the Kinect depth camera to collect 10 types of digital gesture images, and filter the gesture images; use the filtered images to establish a sample set for each digital gesture representation, the method is as follows: perform morphological preprocessing on the filtered gesture images ; Classify and mark image information, obtain sample sets of digital gesture representations, and classify them into training sets and test sets; (2) Build a convolutional neural network—CNN: (2a) Import the digital gesture images of each category into the convolutional neural network as the inputs layer, whose size is [320, 320, 3, 59]; (2b) Construct an 8-layer convolutional neural network, and perform operations such as convolution, downsampling, and pooling on each pixel of the input image to obtain the maps feature map of each layer; (2c) Use the output of each layer as the input of the next layer, go through 8 layers before and after, and finally converge on the fully connected fc layer, and output the result through the output layer softmax classifier; (3) Input the training sample set, extract image features, and perform classification training; (3a) Using the softmax classifier to classify the image feature vector; (3b) using the convolutional neural network algorithm to train the training sample set to obtain the model .mat file after training; (4) Use the trained convolutional neural network to identify images in the test dataset. 2. method according to claim 1 is characterized in that, the filtering method of step (1) is as follows: based on the depth image filtering algorithm of joint bilateral filter, the depth image and the color image of the gesture image that Kinect lens collects at the same moment As the input, use the Gaussian kernel function to calculate the spatial distance weight of the depth image and the grayscale weight of the RGB color image, multiply these two weights to obtain the joint filtering weight, design a joint bilateral filter, and use this filter The result of filtering is convolved with the noise image to realize the Kinect depth image filtering.