CN108062170A - Multi-class human posture recognition method based on convolutional neural networks and intelligent terminal - Google Patents

Abstract

The invention discloses a multi-category human gesture recognition method based on a convolutional neural network and an intelligent terminal, comprising the following steps: Step 1, collecting triaxial acceleration sensor data of a mobile intelligent terminal device, and recording corresponding action categories; Step 2, After preprocessing the triaxial acceleration sensor data, divide the data into two categories, one is training samples and the other is testing samples; step 3, train the convolutional neural network with training samples, and test its accuracy with test samples and Constantly adjust according to demand; step 4, transplant the trained convolutional neural network model to the mobile smart terminal; step 5, use the mobile smart terminal to collect the three-axis acceleration sensor data, preprocess it, and input it to the trained convolutional neural network Neural network model to obtain the result of human gesture recognition. This method has high recognition accuracy and can recognize many types.

Description

Multi-class Human Pose Recognition Method Based on Convolutional Neural Network and Intelligent Terminal

technical field

The invention belongs to the field of artificial intelligence research, relates to the field of wearable intelligent monitoring, in particular to a method for recognizing human body gestures by using sensors.

Background technique

Human gesture recognition technology has been widely used in virtual reality, mobile games, healthcare, human-computer interaction, image recognition and other fields. Generally, gesture recognition technology is divided into two types: non-wearable and wearable. As the name implies, non-wearable technology refers to human gesture recognition technology in which gesture recognition equipment does not come into direct contact with the human body, such as image recognition technology. Compared with non-wearable body posture recognition technology, wearable body posture recognition technology has the advantage of unlimited space, and has a better development space in research and application. Due to the diversity of human body postures and the differences in individual movements, how to establish a posture recognition model with high recognition accuracy is a research topic that has been discussed and paid attention to at present.

Usually, in order to maintain a high recognition accuracy, multiple sensor devices are placed on multiple joints of the human body. Although this method can intuitively discover the acceleration characteristics of various actions, it is inconvenient to require the user to carry multiple sensors in practical applications. How to perform high-accuracy human pose recognition with fewer or even a set of sensors is a very practical research problem.

Using the built-in sensors of smart phones or smart watches for human body posture recognition has already had many research applications at home and abroad. At present, most smart bracelet watches and mobile phones on the market have gesture recognition applications APP. The vast majority of such human gesture recognition methods are threshold detection methods, which classify action types by judging whether the raw or processed data of the sensor is greater than or less than a preset good threshold. This method is simple to calculate and takes up less memory of smart mobile devices, but at the same time, its shortcomings are also obvious: the accuracy rate of different products is uneven, and the types of actions that can be recognized are also very limited. On the one hand, this is the reason for the technical gap of R&D personnel in various companies, and more importantly, on the one hand, it is the limitation of such methods. The more categories of actions that need to be recognized, the more complex the algorithm is to build.

Deep learning has great prospects for development in pattern recognition. Deep Learning originated from the research of Artificial Neural Network (ANN). The convolutional neural network is a neural network containing a convolutional layer. Convolutional neural networks have received great attention in the field of computer vision. Convolutional neural networks can not only process one-dimensional data (such as text), but are also particularly suitable for processing two-dimensional data (such as images) and three-dimensional data (such as video and The three-dimensional acceleration data mentioned in this patent). The convolutional neural network belongs to the category of artificial intelligence. It is more efficient than the traditional method in the construction of the pattern recognition classifier, and it is easy to expand. It can realize more recognition models than the traditional method of action recognition.

Contents of the invention

The purpose of the present invention is to provide a multi-category human gesture recognition method based on convolutional neural network and intelligent terminal, which has high recognition accuracy and can recognize many types.

In order to achieve the above object, the solution of the present invention is:

A multi-category human gesture recognition method based on convolutional neural network and intelligent terminal, comprising the following steps:

Step 1, collect the three-axis acceleration sensor data of the mobile smart terminal device, and record the corresponding action category;

Step 2, after preprocessing the triaxial acceleration sensor data, divide the data into two categories, one is training samples and the other is testing samples;

Step 3, use the training samples to train the convolutional neural network, and use the test samples to test its accuracy and adjust it according to the needs;

Step 4, transplanting the trained convolutional neural network model to the mobile smart terminal;

Step 5: Use the mobile smart terminal to collect the data of the three-axis acceleration sensor, and after preprocessing, input it into the trained convolutional neural network model to obtain the human body posture recognition result.

In the above step 1, the sampling frequency is set to 25Hz.

In the above step 2, the data is preprocessed, including filtering and normalizing the data, and adjusting the data to the input format of the convolutional neural network.

In the above step 2, 75% of the data are used as training samples, and 25% of the data are used as test samples.

The specific steps of the above step 3 are:

a, Establish a multi-layer convolutional neural network model;

b. Import training samples to adjust the parameters of the convolutional neural network model to obtain a high-accuracy model.

In the above step b, the adjustment of the parameters of the convolutional neural network model includes the adjustment of the number of neurons in each layer, the adjustment of the loss function and the convolution kernel.

In the above step a, the structure of the convolutional neural network model includes: an input layer, two layers of convolution and maximum pooling layers, a fully connected layer and an output layer.

In the above convolutional neural network model, the size of the convolution kernel is 3*3, the number of neurons in the two convolution layers is 96 and 198 respectively, and the data size of the convolution kernel in the first layer is (5,5,3) , a total of 96 convolution kernels; the pooling kernels of the whole experiment are (2,2), the pooling step size is 2, and the maximum pooling strategy is used; the convolution kernel data size of the second convolutional layer is (3,3,96), a total of 198 convolution kernels; the fully connected layer contains 1024 hidden nodes; the learning rate is 0.0001; the drop-out is 1.

After adopting the above solution, due to the advantages of the convolutional neural network, as long as the number of samples is sufficient, the present invention can expand the action categories that can be classified to more by adjusting the parameters. The invention has important practical application significance in aspects such as intelligent monitoring and human body gesture recognition.

The present invention has the following advantages:

(1) The present invention utilizes the artificial intelligence-convolutional neural network identification method, which has high identification accuracy and many types that can be identified;

(2) The number of actions recognized by the recognition method of the present invention is scalable, and the expansion operation is simple and easy for developers to operate;

(3) Compared with video or image recognition methods, the present invention can effectively protect user privacy;

(4) The present invention can be applied to Android smart phones and smart watches commonly used by people, and has good promotion.

Description of drawings

Fig. 1 is a flow chart of the present invention;

Fig. 2 is a schematic diagram of the present invention;

Fig. 3 is a schematic diagram of the direction of the three-axis acceleration sensor of the mobile phone;

Fig. 4 is a schematic diagram of partial acceleration data waveforms corresponding to different actions;

Figure 5 is a diagram of the change of cross entropy (cross_entropy) with the number of training.

Detailed ways

The technical solutions and beneficial effects of the present invention will be described in detail below in conjunction with the accompanying drawings.

The present invention provides a multi-category human posture recognition method based on convolutional neural network and intelligent terminal, comprising the following steps:

Step 1, collect the three-axis acceleration sensor data of the mobile smart terminal device under the supervision and recording of a third party, and pre-attach the action category label, and use it as a sample for human body posture recognition model training;

Step 2, preprocessing the data of the three-axis acceleration sensor, including filtering and normalizing the data, and adjusting the data into the input format of the convolutional neural network, dividing the data into two categories, one is training samples, One is the test sample;

Step 3, use the training samples to train the convolutional neural network, and use the test samples to test its accuracy and adjust it according to the needs; specifically include:

a, Establish a multi-layer convolutional neural network model;

b. Importing training samples to adjust convolutional neural network model parameters to obtain a high-accuracy model; wherein, the convolutional neural network model parameter adjustment includes: adjustment of the number of neurons in each layer, loss function and convolution kernel adjustment.

Step 4, transplanting the trained convolutional neural network model (human gesture recognition model) to the mobile smart terminal to realize the real-time terminal gesture recognition processing function;

Step 5: Use the mobile smart terminal to collect the data of the three-axis acceleration sensor, and after preprocessing, input it into the trained convolutional neural network model to obtain the human body posture recognition result.

The present invention obtains a human body posture recognition model based on a preset training set and convolutional neural network structure training, and can recognize seven postures of walking, running, going upstairs, going downstairs, sit-ups, sweeping the floor, and wiping.

Figure 1 is the flow chart of target processing. After collecting the three-dimensional acceleration time series of human body movement from the smart mobile terminal, after integration and processing, it is input to the initial convolutional neural network for model training, and the trained model that meets the design requirements is output to the mobile On the terminal, it can recognize human body movements offline on the mobile smart terminal.

Figure 2 is a structural diagram of a convolutional neural network, which mainly includes: an input layer, two convolutional and maximum pooling layers, a fully connected layer and an output layer. The input is the preprocessed three-axis acceleration data x, y, z, for example, each acceleration direction of a smart phone is shown in Figure 3.

Optionally, the sampling frequency of collecting the smart terminal can be set to 25Hz. Part of the motion acceleration data waveform collected at this frequency is shown in Figure 4. Optionally, the example of the present invention defines an action sample every 2.56 seconds, that is, every 64 sets of data is a sample. Of course, the sampling frequency can be set to an appropriate value according to actual needs, which is not limited here.

In order to train the convolutional neural network, the present invention divides the collected samples into two categories: training samples and testing samples. The training samples are used as the input of the convolutional neural network for model training, and the test samples are used as the basis for considering the recognition accuracy. By default, 75% of the dataset is used as the training set and 25% of the dataset is used as the test set.

As input to a convolutional neural network, the acceleration data needs to be folded. In the example of the present invention, the three-axis acceleration data size is set to (8,8,3), representing length, width and depth respectively. The data matrix form of each axis is as follows:

In this way, the three-axis acceleration data in each short period of time can be shaped like a pixel picture to adapt to the training of the convolutional neural network. Of course, you can set an appropriate value according to actual needs, and there is no limitation here.

The formula of neuron, the basic unit of neural network, is as follows:

Among them, x is the neuron input, n is the number of input parameters, b is the bias, h _W,b (x) is the neuron output.

The difference between a convolutional neural network and an ordinary neural network is that the convolutional neural network contains a feature extractor composed of a convolutional layer and a subsampling layer. In a convolutional layer of a convolutional neural network, a neuron is only connected to some neurons in neighboring layers. In a convolutional layer of CNN, it usually contains several feature planes (featureMap). Each feature plane is composed of some neurons arranged in a rectangle. The neurons of the same feature plane share weights. The shared weights here are convolutions. nuclear. The convolution kernel is generally initialized in the form of a random decimal matrix, and the convolution kernel will learn to obtain reasonable weights during the training process of the network. The direct benefit of sharing weights (convolution kernels) is to reduce the connections between layers of the network while reducing the risk of overfitting.

In this part of the present invention, only the size of the convolution kernel and the number of neurons need to be set. The value of the convolution kernel size and the number of neurons is an empirical value, and there is no fixed value method. In the example of the present invention, the convolution kernel size is 3*3, and the neurons of the two convolution layers are respectively 96 and 198, this data is for reference only.

Subsampling is also called pooling, and usually has two forms: mean pooling and max pooling. Subsampling can be seen as a special kind of convolution process. Convolution and subsampling greatly simplify the model complexity and reduce the parameters of the model.

The final specific experimental parameters of the model are listed as follows: the data size of the first layer of convolution kernels is (5,5,3), and there are 96 convolution kernels in total; the pooling kernels of the entire experiment are (2,2), and the pooling The step size is 2, and the maximum pooling strategy is used; the convolution kernel data size of the second convolutional layer is (3,3,96), and there are 198 convolution kernels in total; the fully connected layer contains 1024 hidden nodes; The learning rate is 0.0001; the drop-out is 1.

If the amount of training data is not large enough, the data needs to be reused. Each time, 50 pieces of data are input into the neural network for training, and the recognition accuracy and cross-entropy are measured every 50 times. The change diagram of the cross-entropy is shown in Figure 5.

When the trained convolutional neural network meets the design requirements, the model can be extracted to the mobile smart terminal for use. If the trained convolutional neural network does not meet the design requirements, the number of neurons in each hidden layer needs to be modified. Which value is appropriate to modify the number of neurons needs to be tested repeatedly. If the above method of modifying the number of neurons in each hidden layer has little effect on the recognition accuracy, it is recommended to add the number of hidden layers or increase the number of training samples.

It should be noted that the human body gesture recognition device in the embodiment of the present invention may be integrated into a smart mobile terminal, and the above-mentioned smart terminal may specifically be a smart phone, a smart watch and other terminals, which are not limited here.

It should be understood that the human body posture recognition device in the embodiment of the present invention can realize all the technical solutions in the above-mentioned method embodiments, and the functions of each functional module can be specifically realized according to the methods in the above-mentioned method embodiments, and its specific implementation process can refer to the above-mentioned Relevant descriptions in the embodiments will not be repeated here.

It can be seen from the above that the human body posture recognition device in the embodiment of the present invention collects the acceleration data of the smart terminal, based on the collected acceleration data of the smart terminal, and inputs the preprocessed data into the trained human body posture recognition model , to get the result of human gesture recognition. Since the human body posture recognition model is trained based on the preset training set convolutional neural network, the recognition of human body posture can be realized by preprocessing the acceleration data and inputting it into the trained human body posture recognition model, thereby realizing Human pose recognition based on non-visual means of acceleration data.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The above embodiments are only to illustrate the technical ideas of the present invention, and can not limit the protection scope of the present invention with this. All technical ideas proposed in accordance with the present invention, any changes made on the basis of technical solutions, all fall within the protection scope of the present invention. Inside.

Claims (8)

1. a kind of multi-class human posture recognition method based on convolutional neural networks and intelligent terminal, it is characterised in that including such as Lower step：

Step 1, the 3-axis acceleration sensor data of mobile intelligent terminal equipment are gathered, and record corresponding action classification；

Step 2, two classes are splitted data into after being pre-processed to 3-axis acceleration sensor data, one kind is training sample, one Class is test sample；

Step 3, with training sample training convolutional neural networks, and test its accuracy rate with test sample and constantly adjust according to demand It is whole；

It step 4, will be on trained convolutional neural networks model transplantations to mobile intelligent terminal；

Step 5,3-axis acceleration sensor data are gathered using mobile intelligent terminal, after being pre-processed, is input to and trains Convolutional neural networks model, obtain human body attitude recognition result.

2. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as described in claim 1, It is characterized in that：In the step 1, sample frequency is set as 25Hz.

3. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as described in claim 1, It is characterized in that：In the step 2, data are pre-processed, including being filtered to data, normalized, and by data It is adjusted to the input format of convolutional neural networks.

4. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as described in claim 1, It is characterized in that：In the step 2, using 75% in data as training sample, using 25% in data as test sample.

5. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as described in claim 1, It is characterized in that：The step 3 comprises the concrete steps that：

A establishes the convolutional neural networks model of multilayer；

B imports training sample and adjusts convolutional neural networks model parameter, obtains the model of high-accuracy.

6. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as claimed in claim 5, It is characterized in that：In the step b, adjust convolutional neural networks model parameter and adjusted including each layer neuronal quantity, loss function And convolution kernel is adjusted.

7. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as claimed in claim 5, It is characterized in that：In the step a, the structure of convolutional neural networks model includes：Input layer, two layers of convolution and maximum pond layer, One full articulamentum and an output layer.

8. the multi-class human posture recognition method based on convolutional neural networks and intelligent terminal as claimed in claim 7, It is characterized in that：In the convolutional neural networks model, convolution kernel size is 3*3, and the neuron number of two convolutional layers is respectively 96 and 198, the data size of first layer convolution kernel is (5,5,3), shares 96 convolution kernels；The Chi Huahe entirely tested is (2,2), pond step-length are all 2, all using maximum pondization strategy；The convolution kernel data size of second layer convolutional layer for (3,3, 96) 198 convolution kernels, are shared；Full articulamentum includes 1024 concealed nodes；Learning rate is 0.0001；Drop-out is 1.