CN106909938B - Perspective-independent behavior recognition method based on deep learning network - Google Patents

Abstract

The present invention provides a method for recognizing perspective-independent behavior based on a deep learning network, which includes the following steps: recording a video frame image from a certain perspective, and using deep learning to extract and process underlying features; Modeling, the cube model is obtained in chronological order; the cube model of all perspectives is transformed into a perspective-invariant cylindrical feature space map, and then it is input into the classifier for training, and the video behavior perspective-independent classifier is obtained. The technical scheme of the present invention adopts a deep learning network to analyze human behavior under multiple perspectives, which improves the robustness of the classification model; it is especially suitable for training and learning based on big data, and can well exert its advantages.

Description

Perspective-independent behavior recognition method based on deep learning network

technical field

The present invention relates to the technical field of computer vision, in particular to a method for recognizing perspective-independent behavior based on a deep learning network.

Background technique

With the rapid development of information technology, computer vision has ushered in the best period of development with the emergence of concepts such as VR, AR, and artificial intelligence. As the most important video behavior analysis in the field of computer vision, it is increasingly being studied by scholars at home and abroad. favor. In a series of fields such as video surveillance, human-computer interaction, medical care, and video retrieval, video behavior analysis occupies a large proportion. For example, in the popular driverless car project, video behavior analysis is very challenging. Due to the complexity and diversity of human actions, coupled with the influence of human self-occlusion, multi-scale, and perspective rotation and translation from multiple perspectives, it is very difficult to recognize video behaviors. How to accurately identify human behavior from multiple angles in real life and analyze human behavior has always been a very important research topic, and the society has higher and higher requirements for behavior analysis.

Traditional research methods include the following:

Based on spatiotemporal feature points: extract the spatiotemporal feature points in the extracted video frame images, then model and analyze the spatiotemporal feature points, and finally classify them.

Based on human skeleton: extract human skeleton information through algorithms or depth cameras, and then describe and model the skeleton information, and then classify video behaviors.

The behavior analysis method based on spatiotemporal feature points and skeleton information has achieved remarkable results in the traditional single-view or single-person mode, but for areas with high pedestrian traffic such as streets, airports, and stations, or human occlusion and illumination changes The appearance of a series of complex problems such as the transformation of perspective, the simple use of these two analysis methods in real life often fails to meet people's requirements, and sometimes the robustness of the algorithm is also very poor.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned defects in the prior art, the present invention proposes a method for recognizing perspective-independent behavior based on a deep learning network, which adopts a deep learning network to analyze human behaviors from multiple perspectives, so as to improve the robustness of the classification model; The deep learning network is suitable for training and learning based on big data, and can give full play to its advantages.

The technical scheme of the present invention is realized as follows:

A perspective-independent behavior recognition method based on a deep learning network, comprising a training process of obtaining a classifier by using a training sample set and a recognition process of using the classifier to identify test samples;

The training process includes the following steps:

S1) the video frame images Image 1 to Image i under a certain viewing angle are input in chronological order;

S2) Use CNN (Convolutional Neural Network, convolutional neural network) to extract the underlying features of the image input in step S1) and pool them, and use STN (Spatial Transform Networks, spatial transformation network) for the pooled underlying features. strengthen;

S3) pooling the enhanced feature image (Feature Map) in step S2) and inputting the RNN (Recurrent Neural Network, recurrent neural network layer) for time modeling to obtain a cube model associated with time series;

S4) Repeat steps S1) to S3) to obtain a space cube model of the same behavior under multiple viewing angles, convert the space cube model of each viewing angle into a cylindrical feature space map with a constant viewing angle, and use it as the model of this type of behavior The training samples are input into the classifier for training;

S5) Repeat the above steps to obtain perspective-independent classifiers of various behaviors;

The identification process includes the following steps:

S6) input the video frame image under a certain angle of view, adopt above-mentioned steps S1) to S3) to carry out underlying feature extraction and modeling to it, obtain the space cube model under this angle of view;

S7) Convert the space cube model obtained in step S6) into a perspective-invariant cylindrical feature space map, and input it into a classifier for identification to obtain a video behavior category.

In the above technical solution, step S2) preferably adopts a three-layer convolution operation to extract underlying features; steps S2) and S3) preferably adopt a maximum pooling method to perform dimension reduction operation on the feature image.

In the above technical solution, step S3) obtains a space cube model of the same behavior under a certain viewing angle, and repeats steps S1) to S3) to obtain space cube models of the same behavior under multiple viewing angles.

In the technical solution of the present invention, the LSTM network (Long-Short Term Memory, LSTM for short) is preferably used for time modeling, because the back-propagation process of the deep learning network adopts the stochastic gradient descent method, and adopts the special gate operation in the LSTM. , which can prevent the gradient disappearance problem of each layer.

In the above technical solution, step S4) specifically includes:

S41) Repeat the operation steps S1) to S3) to obtain the same spatial cube model of each viewing angle, and integrate it into the cylindrical space with x, y, z as the coordinate axes, and the cylindrical space represents the movement under each viewing angle Trajectory description of the feature;

S42) adopt the formula to the model obtained in step S41):

Perform polar coordinate transformation to obtain an angle-invariant cylinder space mapping.

In the above technical solution, it also includes: S0) constructing a data set, and the present invention preferably adopts the IXMAS data set.

Compared with the prior art, the technical scheme of the present invention has the following differences:

1. Use the CNN method to perform feature extraction on the underlying features to obtain global features instead of key points obtained by traditional methods.

2. Use the STN method to perform feature enhancement on the obtained global features instead of directly modeling the obtained features.

3. Use the LSTM network to model the global features after strengthening and dimensionality reduction operations, and add important time information to make it temporally relevant.

4. Use polar coordinate transformation to perform coordinate transformation on the space cube model of the same behavior and different viewing angles to obtain a cylindrical space mapping with constant angle, and then complete the training and classification and recognition by CNN.

The advantages of the present invention are: the global high-level features are obtained by using the CNN method, which has good robustness to the video in real life after the feature enhancement of STN, and then uses the RNN network to establish time information, and finally passes through the polar coordinates. The transformation fuses different features in multiple perspectives, and uses CNN to train and classify the obtained angle-invariant descriptors without using traditional skeleton and key point extraction operations, and the global features obtain more comprehensive features; RNN network obtains Inter-frame time information makes the behavior description more complete and more applicable.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of a training process of the present invention;

Fig. 2 is the schematic flow chart of the identification process of the present invention;

Figure 3 is a schematic diagram of a general human behavior recognition process;

Fig. 4 is a simplified flow chart of extraction and modeling of underlying features;

Fig. 5 is the processing flow chart of general CNN;

Figure 6 is a simplified structural diagram of a general RNN;

Figure 7 is a block diagram of LSTM;

Fig. 8 is the flow chart of fusion classification to each angle of view;

FIG. 9 is a schematic diagram of the model of the Motion History Volume in FIG. 8 after polar coordinate transformation.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in FIG. 1 and FIG. 2 , the method for recognizing perspective-independent behavior based on a deep learning network of the present invention includes a training process for obtaining a classifier by using a training sample set and a recognition process for using the classifier to identify test samples;

The training process is shown in Figure 1 and includes the following steps:

S1) the video frame images Image 1 to Image i under a certain viewing angle are input in chronological order;

S2) using CNN to extract and pool the underlying features of the image input in step S1), and using STN to strengthen the underlying features after the pooling;

S3) pooling the enhanced feature image in step S2) and inputting RNN for time modeling to obtain a cube model associated with time sequence;

S4) Repeat steps S1) to S3) to obtain a space cube model of the same behavior under multiple viewing angles, convert the space cube model of each viewing angle into a cylindrical feature space map with a constant viewing angle, and use it as the model of this type of behavior The training samples are input into the classifier for training;

S5) Repeat the above steps to obtain perspective-independent classifiers for various behaviors.

The identification process is shown in Figure 2 and includes the following steps:

S6) input the video frame image under a certain angle of view, adopt above-mentioned steps S1) to S3) to carry out underlying feature extraction and modeling to it, obtain the space cube model under this angle of view;

S7) Convert the space cube model obtained in step S6) into a perspective-invariant cylindrical feature space map, and input it into a classifier for identification to obtain a video behavior category.

In the above technical solution, step S2) preferably adopts a three-layer convolution operation to extract underlying features; steps S2) and S3) preferably adopt a maximum pooling method to perform dimension reduction operation on the feature image.

In the above technical solution, step S3) obtains a space cube model of the same behavior under a certain viewing angle, and repeats steps S1) to S3) to obtain space cube models of the same behavior under multiple viewing angles.

In the technical solution of the present invention, the LSTM network (Long-Short Term Memory, LSTM for short) is preferably used for time modeling, because the back-propagation process of the deep learning network adopts the stochastic gradient descent method, and adopts the special gate operation in the LSTM. , which can prevent the gradient disappearance problem of each layer.

In the above technical solution, step S4) specifically includes:

S41) Repeat the operation steps S1) to S3) to obtain the same spatial cube model of each viewing angle, and integrate it into the cylindrical space with x, y, z as the coordinate axes, and the cylindrical space represents the movement under each viewing angle Trajectory description of the feature;

S42) adopt the formula to the model obtained in step S41):

Perform polar coordinate transformation to obtain an angle-invariant cylinder space mapping.

In the above technical solution, it also includes: S0) constructing a data set.

The present invention preferably adopts the IXMAS data set, the data set includes five different perspectives, 12 people each with 14 actions, and each action is repeated three times. Use 11 of them as the training dataset and the remaining 1 as the test dataset.

Specifically, for example, to identify the behavior of "running", first collect running videos of 12 people from five perspectives, of which 11 people's running videos are used as the training data set, and the remaining 1 person is used as the verification data set. First, operate the running video frame image from a perspective of a certain person according to the above steps S1) to S3), and finally obtain a cube model related to the time series of the "running" video behavior in this perspective, that is, in this perspective The space cube model of the "running" behavior; then repeat steps S1) to S3) to obtain the space cube model of the "running" behavior under the other four perspectives in turn; transform the space cube model of the "running" behavior under the above five perspectives into a The perspective-invariant cylindrical feature space mapping, and use it as the training sample of the person's "running" category of behavior, and input it into classifier training; after training with different people's training samples, the "running" behavior is obtained View-independent classifier. Similarly, view-independent classifiers for various video behaviors can be constructed.

When performing identification, the above steps S6) and S7) are performed, firstly, the video frame image under a certain viewing angle of a person in the test sample is operated according to the above steps S1) to S3), and the space of the behavior under the viewing angle is obtained. The cube model is then transformed into a cylindrical feature space map through polar coordinate transformation, which is input into the classifier to identify the behavior category. The identification process for other perspectives is the same.

In order to better understand and illustrate the technical solutions of the present invention, the following describes and analyzes the related technologies involved in the above-mentioned technical solutions in detail.

The method model of the present invention includes two main stages, one is to extract and model the underlying features; the second is to fuse and classify various perspectives, and the main innovative work is as follows.

The general process of human behavior recognition is shown in Figure 3. In this figure, the feature extraction and feature representation stage is the focus of behavior recognition. The results of this stage will ultimately affect the accuracy of recognition and the robustness of the algorithm. The present invention adopts Feature extraction using deep learning methods.

Figure 4 shows a simplified low-level feature extraction and modeling flowchart.

In the technical solution of the present invention, the adopted deep learning framework is Caffe, and the video frames Image 1 to Image i under a certain viewing angle in FIG. 4 are input into the network in chronological order. First, use CNN to extract features from the input image, then use STN to enhance the features to make it robust to translation, scale changes, and angle changes, and then perform the pooling operation on the feature image (Feature Map). The method is the maximum pooling method, and then the feature images that have undergone the pooling operation are input into the RNN layer for temporal modeling, and finally a feature image sequence (Feature Maps Sequences) with temporal correlation between frames is obtained.

The technical scheme of the present invention adopts three-layer convolution operation to extract the underlying features, and then performs dimension reduction operation on the features through the maximum pooling method. The feature image after pooling is input into the STN layer to enhance the feature. The function of the STN network is to make the obtained feature robust to translation, rotation and scale changes. Then, the feature images output by STN are subjected to maximum pooling, and the dimensionality reduction process is performed again, and then input to the RNN network to place time information. Finally, the obtained Feature Maps are combined into space cubes in chronological order. The RNN network used in the present invention is the LSTM network, because the back propagation process of the deep learning network adopts the stochastic gradient descent method, and the special gate operation in the LSTM can prevent the gradient disappearance problem of each layer.

Among the above technical solutions, CNN is an efficient identification method that has been developed in recent years and has attracted attention. In the 1960s, Hubel and Wiesel discovered that their unique network structure can effectively reduce the complexity of the feedback neural network when they studied the neurons used for local sensitivity and direction selection in the cat cerebral cortex, and then proposed CNN. Now, CNN has become one of the research hotspots in many scientific fields, especially in the field of pattern classification, because the network avoids the complex pre-processing of the image and can directly input the original image, so it has been more widely used.

Generally, the basic structure of CNN includes two layers, one of which is a feature extraction layer, the input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local feature is extracted, the positional relationship between it and other features is also determined; the second is the feature mapping layer, each computing layer of the network consists of multiple feature maps, each feature map is a plane, All neurons in the plane have equal weights.

In the technical scheme of the present invention, a feature mapping layer is used to extract the global underlying features in the video frame images, and then the underlying features are processed in a deeper level.

The generalized processing flow of CNN is shown in Figure 5.

The layer to be used in the technical solution of the present invention is the Feature Map obtained after convolution, and we ignore the subsequent pooling and fully connected layers. What CNN obtains is the feature information of a single image, and the video information to be processed, so time information needs to be introduced, so the simple use of CNN cannot meet the requirements of processing video behavior.

In the above technical solutions, RNN or recurrent neural network is developed on the basis of Feed-forward Neural Networks (FNNs for short). Unlike traditional FNNs, RNNs introduce directed loops, which are able to deal with the contextual correlation between those inputs. RNN contains input units (Input units), the input sets are labeled {x ₀ , x ₁ , ..., x _t-1 , x _t , x _t+1 , ...}, and the output units of the output units are Labeled {o ₀ , o ₁ , ..., o _t-1 , o _t , o _t+1 , ... }. The RNN also contains hidden units, and we label its output set as {s ₀ , s ₁ , …, s _t-1 , s _t , S _t+1 , …}, these hidden units complete the most Main job.

Figure 6 shows the simplified structure of a general RNN. In Figure 6, there is a one-way flow of information from the input unit to the hidden unit, while another one-way flow of information arrives from the hidden unit output unit. In some cases, RNN will break the latter limitation and guide the information from the output unit back to the hidden unit, these are called "BackProjections", and the input of the hidden layer also includes the state of the previous hidden layer, that is, the hidden layer. Nodes within the containing layer can be self-connected or interconnected. Therefore, the connection of time information is realized in the hidden layer, and there is no need to additionally consider the problem of time information. This is also a big advantage of RNNs when dealing with video behavioral features. Therefore, the processing with time series information is generally handed over to RNN in deep learning.

On the basis of RNN, a new model for processing time information is developed: Long-Short Term Memory (LSTM). Because of the stochastic gradient descent method used in the back-propagation of the deep learning network, RNN will have a problem of gradient disappearance, that is, the nodes at the later time are less sensitive to the nodes at the previous time. So a core element introduced by LSTM is Cell. The rough block diagram of LSTM is shown in Figure 7.

FIG. 8 is a flow chart showing the fusion and classification of various perspectives.

According to the method in Figure 4, the space cube model of the same action under multiple viewing angles is obtained, and then the space cube model of each viewing angle is integrated into the cylindrical space with x, y, and z as the coordinate axes. The trajectory description of the motion feature, and then use the mathematical method to perform polar coordinate transformation, and convert it into the space of r, θ, z coordinate axes, the formula is as follows:

Then, the Invariant Cylinder Space Map is obtained, and finally the obtained cylinder space map is input into the classifier to obtain the behavior category. Here, the CNN method is used for classification, which is different from the SVM classifier, because CNN was originally used for classification. The Motion History Volume in Figure 8 and the model after polar coordinate transformation are shown in Figure 9.

The underlying information extracted by the deep learning method in the technical solution of the present invention is more advanced than the spatiotemporal feature points and skeleton information of the traditional method, and the bone stickiness is better.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (6)

1. a method for recognizing behavior that is independent of perspective based on a deep learning network, comprising using a training sample set to obtain a training process of a classifier and using a classifier to identify a test sample identification process; it is characterized in that: The training process includes the following steps: S1) the video frame images Image 1 to Image i under a certain viewing angle are input in chronological order; S2) using CNN to extract and pool the underlying features of the image input in step S1), and using STN to strengthen the underlying features after the pooling; S3) pooling the enhanced feature image in step S2) and inputting RNN for time modeling to obtain a cube model associated with time sequence; S4) Repeat steps S1) to S3) to obtain a space cube model of the same behavior under multiple viewing angles, convert the space cube model of each viewing angle into a cylindrical feature space map with a constant viewing angle, and use it as the model of this type of behavior The training samples are input into the classifier for training; S5) Repeat steps S1)-S4) for various behaviors of other different categories to obtain perspective-independent classifiers corresponding to various behaviors; The identification process includes the following steps: S6) input the video frame image under a certain angle of view, adopt above-mentioned steps S1) to S3) to carry out underlying feature extraction and modeling to it, obtain the space cube model under this angle of view; S7) Convert the space cube model obtained in step S6) into a cylindrical feature space map, and input it into the classifier for identification to obtain the video behavior category. 2. the perspective-independent behavior identification method based on deep learning network according to claim 1, is characterized in that: Step S2) adopts a three-layer convolution operation to extract the underlying features. 3. the perspective-independent behavior identification method based on deep learning network according to claim 2, is characterized in that: Steps S2) and S3) use the maximum pooling method to perform dimension reduction operations on the feature images. 4. the perspective-independent behavior identification method based on deep learning network according to claim 1, is characterized in that: Step S3) adopts LSTM network to perform temporal modeling. 5. the perspective-independent behavior identification method based on deep learning network according to claim 1, is characterized in that, step S4) specifically comprises: S41) Repeat the operation steps S1) to S3) to obtain the same spatial cube model of each viewing angle, and integrate it into the cylindrical space with x, y, z as the coordinate axes, and the cylindrical space represents the movement under each viewing angle Trajectory description of the feature; S42) adopt the formula to the model obtained in step S41):

Perform polar coordinate transformation to obtain an angle-invariant cylinder space mapping. 6. The perspective-independent behavior identification method based on a deep learning network according to claim 1, characterized in that, further comprising: S0) Construct the dataset.

Images