CN113762082B - Unsupervised skeleton action recognition method based on cyclic graph convolution automatic encoder - Google Patents

Abstract

The invention relates to an unsupervised skeleton action recognition method based on a cyclic graph convolution autoencoder, which is characterized in that it includes: inputting a human skeleton action sequence into a cyclic graph convolution encoder; and outputting the action from the cyclic graph convolution encoder. The representation vector of the sequence; the representation vector of the action sequence is calculated through the weighted nearest neighbor classification algorithm to obtain the recognition category of the human skeleton action sequence; the cyclic graph convolution encoder includes: a multi-layer spatial joint attention module, which is used to combine the human skeleton action sequence and The hidden layer of the recurrent graph convolution encoder adaptively measures the importance of different joints in different actions to obtain a weighted skeleton sequence; the multi-layer graph convolution gated recurrent unit layer is used to integrate the connection relationship features of the weighted skeleton sequence. Get the representation vector of the action sequence. Compared with the existing technology, the present invention can significantly improve the recognition accuracy of the unsupervised action recognition system and has broad application prospects.

Description

Unsupervised skeleton action recognition method based on recurrent graph convolutional autoencoder

Technical field

The invention relates to the technical fields of computer vision and action recognition, and in particular to an unsupervised skeleton action recognition method based on a cyclic graph convolution autoencoder.

Background technique

The movements of natural organisms such as humans and animals, including whole body movements and partial movements of the body such as head, limbs, hands, eyes, etc., are usually called biological movements. These forms of movement are critical for humans to perceive dynamic environmental changes and infer the intentions of other people or other species. Recognizing and understanding the actions of an observed individual is a basic attribute of human visual perception, and the ability to recognize actions in different scenarios is also crucial. For the above reasons, the human action recognition task has attracted the attention of a large number of researchers in the field of computer vision. Since action recognition tasks are widely used, such as in video surveillance, human-computer interaction, motion analysis and other fields, it has gradually developed into an important research direction. Research on human action recognition can be traced back to 1973. Through experimental observations, Johansson found that human actions are mainly realized through the movement of several key skeletal points of the body. The combination and tracking of 10-12 key nodes can describe such things as walking, running, Dancing and other movements can realize the recognition of human body movements.

In recent years, with the advent and rapid development of depth sensors such as Kinect and RealSense, humans can more easily obtain the RGB information, depth information and skeleton information of images. This has also brought huge development to the field of action recognition. Most of the early action recognition methods were based on video sequences, but they faced shortcomings such as high computational complexity and vulnerability to other factors. However, skeleton information is very robust to factors such as human body appearance, environmental interaction, and perspective changes, and has low computational complexity. , data is easy to store. Action recognition based on skeleton data has become a rapidly developing research direction. Effective action recognition can be carried out by using the change information of these key points.

At present, research in the field of action recognition based on skeleton data is also changing rapidly with the development of deep learning technology, from the earliest recognition using manually extracted features to the current recognition using recurrent neural networks and convolutional neural networks. However, these methods cannot utilize the topological characteristics of the skeleton data itself, so the recognition accuracy needs to be improved.

Contents of the invention

The purpose of the present invention is to provide an unsupervised skeleton action recognition method based on a cyclic graph convolution autoencoder in order to overcome the shortcomings of the above-mentioned existing technologies.

The object of the present invention can be achieved through the following technical solutions:

An unsupervised skeleton action recognition method based on recurrent graph convolution autoencoder, including the following steps:

S1. Input the human skeleton action sequence to the cyclic graph convolution encoder;

S2. The cyclic graph convolution encoder outputs the representation vector of the action sequence;

S3. Calculate the representation vector of the action sequence through the weighted nearest neighbor classification algorithm to obtain the recognition category of the human skeleton action sequence;

The cycle graph convolution encoder includes: a multi-layer spatial joint attention module, which is used to combine the human skeleton action sequence and the hidden layer of the cycle graph convolution encoder to adaptively measure the importance of different joints in different actions to obtain a weighted Skeleton sequence; a multi-layer graph convolution gated recurrent unit layer is used to integrate the connection relationship features of the weighted skeleton sequence to obtain the representation vector of the action sequence.

Furthermore, in the spatial joint attention module, the weighted skeleton sequence calculation expression is:

x′ _t =(α _t +1)·x _t

s _t =U _s φ(W _x x _t +W _h h _t-1 +b _s )+b _u

In the formula, x′ _t represents the weighted skeleton sequence, α _t represents the importance of each joint, s _t represents the importance score of each joint, represents the sequence coordinates of N joints at time t, h _t-1 represents the hidden layer information, W _x and W _h represent the learnable parameter matrix, φ represents the activation function, and b _s and b _u represent the bias.

Furthermore, in the graph convolution gated recurrent unit layer, the expression for integrating the connection relationship features of the weighted skeleton sequence is:

In the formula, H ^(l+1) represents the output of the l+1th layer of graph convolution, Represents the symmetric adjacency matrix with spin, A represents the adjacency matrix, I represents the identity matrix,/> is the degree matrix, τ represents the activation function, H ^(l) represents the output of the l-th layer of graph convolution, and Θ ^(l) represents a learnable parameter matrix of the l-th layer.

Furthermore, the expression of the graph convolution gated recurrent unit layer is:

In the formula, z _t represents the update gate, r _t represents the reset gate, Represents the candidate activation vector, /> represents the graph convolution corresponding to H ^{(l +1)} , W _xz , W _hz , W _xr , W _hr , W _xh and W _hh represent the parameter matrices in different gates, and ⊙ represents the Hadamard multiplier.

Further, the training steps of the recurrent graph convolutional encoder include

A1. Input the training action sequence set to the cyclic graph convolution encoder to obtain the representation vector of the action sequence;

A2. Input the representation vector and hidden layer vector of the action sequence to the decoder, and perform sequence restoration to obtain the reconstructed action sequence set;

A3. Compare the reconstructed action sequence set and the training action sequence set, and calculate the loss function value through the reconstruction loss function;

Repeat the above steps A1 to A3 until the loss function value reaches the preset cutoff condition.

Further, the hidden layer vector is a vector with a value of zero and the same length as the human skeleton action sequence.

Further, the expression of the reconstruction loss function is:

In the formula, Represents a set of training action sequences,/> represents the reconstructed action sequence set, ||·|| _F represents the Frobenius norm, and L represents the loss function value.

Furthermore, the cyclic graph convolutional encoder is trained using a gradient descent method.

Further, the spatial joint attention module and the graph convolution gated recurrent unit layer are both three layers.

Further, in the weighted nearest neighbor classification algorithm, after obtaining the k closest samples, k is a set value, the number of votes for each category is calculated, and the recognition result is obtained through weighted voting, where the calculation expression of the weight is :

In the formula, w _i and d _i represent the voting weight and cosine distance of sample i respectively.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention applies the cycle graph convolution encoder to skeleton action recognition, and sets up a multi-layer spatial joint attention module in the cycle graph convolution encoder, so that the recognition process takes into account the spatial topological relationship of the skeleton sequence data, and uses the action sequence The spatio-temporal dependence improves the recognition accuracy; at the same time, the present invention adopts the weighted nearest neighbor classification algorithm as the classifier for final recognition, and uses the idea of exponential explosion to ensure that samples that are beneficial to the result have greater voting weight, further improving the recognition accuracy. accuracy.

Description of drawings

Figure 1 is a schematic diagram of the overall process of the present invention.

Figure 2 is a schematic diagram of the spatial joint attention module.

Figure 3 is a schematic diagram of the graph convolution gated recurrent unit layer.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention and provides detailed implementation modes and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

This embodiment provides an unsupervised skeleton action recognition method based on a cyclic graph convolution autoencoder to solve the problem that existing unsupervised action recognition methods ignore the spatial dependence of action sequences and improve the accuracy of action recognition.

As shown in the straight-line process in Figure 1, the specific steps of this embodiment are as follows:

Step S1: After preprocessing, the human skeleton action sequence is input to the cyclic graph convolution encoder.

Step S2: The cyclic graph convolution encoder outputs the representation vector of the action sequence. The cycle graph convolution encoder includes: a multi-layer spatial joint attention module, which is used to combine the human skeleton action sequence and the hidden layer of the cycle graph convolution encoder to adaptively measure the importance of different joints in different actions to obtain a weighted skeleton. Sequence; a multi-layer graph convolution gated recurrent unit layer (graph convolution GRU layer) is used to integrate the connection relationship features of the weighted skeleton sequence to obtain the representation vector of the action sequence. The spatial joint attention module and the graph convolution gated recurrent unit layer are preferably three layers.

Step S3: Calculate the representation vector of the action sequence through the weighted nearest neighbor classification algorithm to obtain the recognition category of the human skeleton action sequence, completing the action recognition process.

The training steps of the loop graph convolutional encoder are described in the dotted line process in Figure 1. The overall training is performed using the gradient descent method, as follows:

Step A1: Input the training action sequence set to the recurrent graph convolution encoder to obtain the representation vector of the action sequence.

Step A2: Input the representation vector and hidden layer vector of the action sequence to the decoder, perform sequence restoration, and obtain a reconstructed action sequence set;

Step A3: Compare the reconstructed action sequence set and the training action sequence set, and calculate the loss function value through the reconstruction loss function.

Steps Repeat the above steps A1 to A3 until the loss function value reaches the preset cutoff condition.

During the above training process, the expression of the trained reconstruction loss function is:

In the formula, Represents a set of training action sequences,/> represents the reconstructed action sequence set, ||·|| _F represents the Frobenius norm, and L represents the loss function value.

Next, this embodiment will be described in detail in several parts.

1. The spatial joint attention module is shown in Figure 2. The spatial joint attention module is used to combine the human skeleton action sequence x _t and the hidden layer h _t-1 of the recurrent graph convolution encoder, adaptively measure the importance of different joints in different actions, and obtain a weighted skeleton sequence x′ _t . The details of calculating the weighted skeleton sequence x′ _t are as follows:

First, calculate the importance score s _t of each joint, and its calculation expression is:

s _t =U _s φ(W _x x _t +W _h h _t-1 +b _s )+b _u

In the formula, represents the sequence coordinates of N joints at time t, h _t-1 represents the hidden layer information, W _x and W _h represent the learnable parameter matrix, φ represents the activation function, and b _s and b _u represent the bias.

Then, calculate the importance α _t of each joint, and its calculation expression is:

Finally, the weighted skeleton sequence x′ _t is calculated, and its calculation expression is:

x′ _t =(α _t +1)·x _t

Among them, · represents dot product.

2. The multi-layer graph convolution gated recurrent unit layer is shown in Figure 3. The multi-layer graph convolution gated recurrent unit layer is used to integrate the connection relationship features of the weighted skeleton sequence, make full use of the spatial dependence between the joints of each frame, while retaining the characteristics of the time dimension, and obtain the representation vector of the action sequence.

In the graph convolution gated recurrent unit layer, the expression for integrating the connection relationship features of the weighted skeleton sequence is:

In the formula, H ^(l+1) represents the output of the l+1th layer of graph convolution, Represents the symmetric adjacency matrix with spin, A represents the adjacency matrix of the graph, I represents the identity matrix,/> is the degree matrix, τ represents the activation function, H ^(l) represents the output of the l-th layer of graph convolution, and Θ ^(l) represents a learnable parameter matrix of the l-th layer.

The graph convolution gated recurrent unit layer combines graph convolution and gated recurrent units, and its expression is:

In the formula, z _t represents the update gate, r _t represents the reset gate, Represents the candidate activation vector, /> represents the graph convolution corresponding to H ^{(l +1)} , W _xz , W _hz , W _xr , W _hr , W _xh and W _hh represent the parameter matrices in different gates, and ⊙ represents the Hadamard multiplier.

3. In the training step of the recurrent graph convolutional encoder, the input of the decoder is the representation vector of the action sequence and a hidden layer vector. In order to make the decoder completely rely on the state passed by the encoder, thus forcing the encoder to learn For better feature representation, the hidden layer vector is a vector that is all 0 and has the same size as x _t .

4. This embodiment uses the weighted nearest neighbor classification algorithm as the classifier, and obtains the recognition category of the human skeleton action sequence from the representation vector of the action sequence. Specifically, after obtaining the top k closest samples, the number of votes for each category is calculated, and the recognition result is obtained through weighted voting. In this example k=9. The calculation expression of the weight is:

In the formula, w _i and d _i represent the voting weight and cosine distance of sample i respectively.

In order to support and verify the performance of the action recognition method proposed by the present invention, this embodiment uses recognition accuracy as the evaluation index on three public standard data sets, and compares the present invention with other conventional unsupervised skeleton action recognition methods, including LongT GAN (Long-Term Dynamics GAN, long-term dynamic generative adversarial network), P&C (Predict&Cluster, prediction and aggregation), MS2L (Multi-Task Self-Supervised Learning, multi-task self-supervised learning) were compared.

Table 1 shows the comparison of the recognition accuracy of the present invention and other skeleton-based unsupervised action recognition methods on the NTU-RGB+D 60 data set. Among them, CS (Cross-Subject) and CV (Cross-View) represent two different testing methods for this data set. CS refers to dividing the training set and test set based on the different volunteers who collected the data, and CV refers to dividing the training set and the test set based on the different volunteers who collected the data. The training set and test set are divided based on the results of different viewing angle cameras used to collect data.

Table 1 Comparison of recognition accuracy (%) on NTU-RGB+D 60 data set

As can be seen from Table 1, on the two test methods CS and CV of the NTU-RGB+D 60 data set, the unsupervised human action recognition method based on the cyclic graph convolution autoencoder proposed by the present invention is better than the existing one. method, which is 1.8 and 2.9 percentage points higher than the existing method respectively.

Table 2 shows the comparison of the recognition accuracy between the present invention and other skeleton-based unsupervised action recognition methods on the NW-UCLA data set.

Table 2 Comparison of recognition accuracy (%) on NW-UCLA data set

As can be seen from Table 2, although the existing method has achieved more than 80% accuracy on the NW-UCLA data set, the method proposed by the present invention can still further improve the recognition accuracy.

Table 3 shows the comparison of recognition accuracy between the present invention and other skeleton-based unsupervised action recognition methods on the UWA3D data set. Among them, V3 and V4 represent two testing methods on the UWA3D data set.

Table 3 Comparison of recognition accuracy (%) on UWA3D data set

As can be seen from Table 3, compared with other skeleton-based unsupervised action recognition methods, this method has better recognition accuracy in all test methods on the UWA3D data set. Under the test conditions of V4, Improved recognition accuracy by 2.4%. The examples on these three data sets jointly illustrate that the unsupervised human skeleton action recognition method based on cyclic graph convolution autoencoder proposed by the present invention can stably achieve excellent recognition in different data sets and different test conditions. Accuracy.

The preferred embodiments of the present invention are described in detail above. It should be understood that those skilled in the art can make many modifications and changes based on the concept of the present invention without creative efforts. Therefore, any technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention and on the basis of the prior art should be within the scope of protection determined by the claims.

Claims (7)

1. An unsupervised skeleton action recognition method based on a cyclic graph convolution automatic encoder is characterized by comprising the following steps:

s1, inputting a human skeleton action sequence to a cyclic graph convolution encoder;

s2, outputting and obtaining a characterization vector of the action sequence by a cyclic graph convolution encoder;

s3, calculating a characterization vector of the action sequence through a weighted nearest neighbor classification algorithm to obtain an identification category of the human skeleton action sequence;

the cyclic graph convolution encoder includes: the multi-layer space joint attention module is used for combining the human skeleton action sequence and a hidden layer of the cyclic graph convolution encoder, and adaptively measuring the importance of different joints with different actions to obtain a weighted skeleton sequence; the multi-layer diagram convolution gating circulating unit layer is used for integrating the connection relation characteristics of the weighted skeleton sequences to obtain the characterization vector of the action sequence;

in the spatial joint attention module, the weighted skeleton sequence calculation expression is:

x′ _t ＝(α _t +1)·x _t

s _t ＝U _s φ(W _x x _t +W _h h _t-1 +b _t )+b _u

wherein x 'is' _t Representing a weighted skeleton sequence, alpha _t Representing the importance of each joint s _t Representing the importance score of each joint,representing the sequence coordinates of N joints at time t, h _t-1 Representing hidden layer information, W _x And W is _h Represents a matrix of learnable parameters, phi represents an activation function, b _s And b _u Representing the bias;

in the graph convolution gating circulating unit layer, the expression of the connection relation characteristic of the integrated weighted skeleton sequence is as follows:

wherein H is ^(l+1) Indicating that the output of layer l +1 is convolved,representing a symmetric adjacency matrix with spin, A represents the adjacency matrixI represents an identity matrix,>for the degree matrix, τ represents the activation function, H ^(l) Representing the output of layer I of the graph convolution, Θ ^(l) A matrix of learnable parameters representing the first layer;

the representation of the graph convolution gating loop unit layer is as follows:

wherein z is _t Representing an update gate, r _t Representing a reset gate and,activation vector representing candidate->Representative graph convolution sum H ^(l+1) Correspondingly, W _xz 、W _hz 、W _xr 、W _hr 、W _xh And W is _hh Representing the parameter matrix in different gating, +..

2. The method for recognizing actions of an unsupervised skeleton based on a cyclic graph convolution automatic encoder according to claim 1, wherein the training step of the cyclic graph convolution encoder comprises

A1, inputting a training action sequence set to a cyclic graph convolution encoder so as to obtain a characterization vector of an action sequence;

a2, inputting the characterization vector and the hidden layer vector of the action sequence to a decoder, and performing sequence restoration to obtain a reconstructed action sequence set;

a3, comparing the reconstructed action sequence set with the training action sequence set, and calculating a loss function value through a reconstruction loss function;

repeating the steps A1 to A3 until the loss function value reaches the preset cut-off condition.

3. The method for recognizing the actions of the unsupervised skeleton based on the automatic loop-chart convolution encoder according to claim 2, wherein the hidden layer vector is a vector with a value of zero and the same length as the action sequence of the human skeleton.

4. The method for recognizing the actions of the unsupervised skeleton based on the automatic loop-chart convolution encoder according to claim 2, wherein the expression of the reconstruction loss function is:

in the method, in the process of the invention,representing training action sequence set,/->Representing a set of reconstructed action sequences, I.I _F Representing the Frobenius norm and L representing the loss function value.

5. The method for recognizing actions of an unsupervised skeleton based on a cyclic graph convolution automatic encoder according to claim 2, wherein the cyclic graph convolution encoder is trained by a gradient descent method.

6. The method for recognizing actions of an unsupervised skeleton based on a loop-based convolution automatic encoder according to claim 1, wherein the spatial joint attention module and the loop-based convolution gating loop unit layer are three layers.

7. The method for recognizing the actions of the unsupervised skeleton based on the automatic loop-chart convolution encoder according to claim 1, wherein in the weighted nearest neighbor classification algorithm, k is a set value after k nearest samples are obtained, the number of votes in each category is calculated, and recognition results are obtained through weighted voting, wherein the calculation expression of the weight is:

wherein w is _i And d _i The voting weights and cosine distances of the samples i are shown, respectively.