CN114972874A - Three-dimensional human body classification and generation method and system for complex action sequence - Google Patents

Abstract

The invention discloses a three-dimensional human body classification and generation method and system for complex action sequences, which are applied to the field of virtual reality and include: acquiring and preprocessing complex action videos to construct a data set; The key points of the human body and the pose information of the action sequence are used as the training set; the classification and coding model of complex action sequences based on 3D geometry is constructed, and its input and output are combined into a sequence for encoding and decoding training, and a complex action sequence generation based on 3D geometry is constructed. Model; input the test set into the model to obtain action sequences of various test set action categories. By encoding the standard three-dimensional geometric sequence into geometric parameters containing time information, the invention not only enhances the network's learning of the distribution of different action categories in the latent space, but also realizes that in the case of complex actions, the action type can be accurately identified and a reasonable Action sequences improve the accuracy of recognition and the diversity of actions.

Description

A three-dimensional human body classification and generation method and system for complex action sequences

technical field

The invention relates to the field of virtual reality, in particular to a three-dimensional human body classification and generation method and system for complex action sequences.

Background technique

In the field of human 3D reconstruction, human motion prediction is an extremely challenging task. Based on CAVE, the semantic labels of actions are used as prior conditions, and are put into the network training together with the action sequences, and an infinite number of 3D human action sequences are generated according to the labels, making these sequences look more realistic. Many previous works are based on motion sequences, through structured prediction, to implicitly model the spatial structure of the human skeleton and apply it to individual joints. Most deep learning methods for action recognition today use shallow convolutional networks. With convolutional neural networks, stochastic gradient descent can be used to learn features and end-to-end learning with classification mitigation, and limit some methods related to support vector machines and handcrafted features. Extracted features are learned from training data in direct convolution filters. The main benefits are: a very simple and convenient end-to-end way to optimize feature extractor and classifier parameters, and the extracted features are adaptive for specific attributes. optimization. Multi-label convolutional neural networks are superior to support vector machines because they can learn the relationship between attributes more comprehensively.

The purpose of 3D human action sequence generation is to generate a 3D human body that meets the expected action type under given conditions. Methods based on deep learning have been dominant in the field of recognition. These methods can be roughly divided into two categories: the first category mainly focuses on extracting action features from 3D information, in order to obtain the distribution of different actions in the latent space, and generate complex and diverse action sequences through different distributions; the second category is to use style The method of migration is to predict the bones from the image under the premise of given action data, and bind the weights to generate a single action of the human body with diverse styles.

The existing 3D human action small datasets (NTU RGBD, HumanAct12, UESTC, BABEL) are all obtained through the ordinary camera or depth camera to obtain the human motion sequence, and then obtain the information of the key points of the human body, posture, posture and other information through VIBE processing. The largest dataset is only tens of thousands of images. In the face of different shooting angles of the scene, the process of obtaining the camera perspective conversion function model by learning the feature changes is more complicated.

At present, the technology of 3D action generation is still in the development stage. When the existing model has many action categories, the action type generated is relatively simple; or it can only generate action sequences of a single action. In complex action sequences, related actions will overlap, which will further reduce the accuracy of actions and make it more difficult to generate corresponding action sequences.

Therefore, how to provide a three-dimensional human body classification and generation method and system that can accurately identify, classify and generate complex action sequences of human body sequences corresponding to actions in the case of multiple complex action types is an urgent need for those skilled in the art to solve The problem.

SUMMARY OF THE INVENTION

In view of this, the present invention proposes a three-dimensional human body classification and generation method and system for complex action sequences. By using VIBE to process the picture data, the key points of the human body and the pose information of the action sequence in each frame of the picture can be obtained, which can reduce the influence of lens distortion on the action generation as much as possible; by using two linear full connections for the standard 3D geometric sequence layer, and a long-short-term memory neural network with time information to obtain geometric parameters containing time information, and then input them as a priori into the complex action sequence generation model based on 3D geometry, which can enhance the network’s ability to understand the distribution of different action categories in the latent space. In addition, the parameterized processing of complex actions in the data set can still accurately identify the action type in the case of complex actions, generate a reasonable action sequence, and improve the accuracy of recognition and the diversity of actions.

In order to achieve the above object, the present invention adopts the following technical solutions:

A three-dimensional human body classification and generation method for complex action sequences, comprising:

Step (1): Obtain complex action videos, preprocess the complex action videos, and construct a dataset.

Step (2): Perform key point recognition on the data set to obtain human body key points and action sequence pose information as a training set.

Step (3): construct a three-dimensional geometry-based complex action sequence classification and coding model, take the training set as input, and obtain the output of geometric parameters containing time information corresponding to the action categories of the training set.

Step (4): Combine the input and output of the three-dimensional geometry-based complex action sequence classification and encoding model into a sequence for encoding and decoding training, and construct a three-dimensional geometry-based complex action sequence generation model.

Step (5): input the test set of actions to be generated into the described three-dimensional geometry-based complex action sequence classification and coding model, obtain geometric parameters containing time information corresponding to the action categories of the test set, and then pass the trained described Based on the complex action sequence generation model of three-dimensional geometry, various action sequences of the test set action categories are obtained.

Optionally, in step (1), preprocessing the complex action video includes: editing, truncating frames, converting video data into picture data, and selecting picture data with rich motion features to construct a data set.

Optionally, in step (2), the key point identification of the data set is specifically: using VIBE to process the picture data to obtain the key points of the human body and the posture information of the action sequence in each frame of the picture.

Optionally, in step (3), the three-dimensional geometry-based complex action sequence classification and coding model constructs a trained complex action sequence classification model based on categorical coding, which specifically includes the following steps:

Use the dataset function to process the training set and corresponding labels to obtain the length of the training set;

Obtain the tensor of the corresponding 3D geometric model by iterating over the training set in the data iterator;

The categorical encoding is transferred to the GPU, and the geometric parameters containing time information are obtained through two linear fully connected layers and a long-short-term memory neural network with time information.

Optionally, in step (4), a three-dimensional geometry-based complex action sequence generation model is constructed based on the transformer model, including two stages: encoding training and decoding training.

Optionally, the encoding training is specifically:

Combining the training set and the geometric information parameters with time information corresponding to the training set into a sequence, and then inputting it to the transformer encoding after passing through the gated cyclic unit to obtain the Gaussian distribution of different action categories in the latent space;

The decoding training is specifically:

By learning the variance and mean of different action categories in the latent space, through the transformer decoder, the three-dimensional information and action pose of the human body are obtained, and the SMPL model is used to render the parameters to obtain the complete human body sequence.

Optionally, the training loss function of the three-dimensional geometric complex action sequence generation model based on the transformer model is:

Among them: Vt represents the three-dimensional geometric information of the input data, and Pt represents the predicted human joint points and human action posture information.

Optionally, it also includes introducing a DropPath module in the process of constructing a complex action sequence generation model based on three-dimensional geometry based on the transformer model, so that the two stages of encoding training and decoding training are alternately performed.

The present invention also provides a three-dimensional human body classification and generation system for complex action sequences, including:

Acquisition module: used to acquire complex action videos, preprocess complex action videos, and construct data sets;

Data recognition module: used to process image data using VIBE, identify key points in the data set, and obtain the key points of the human body and the posture information of the action sequence in each frame of the picture, as a training set;

The first building module: Based on categorical, coding and training the 3D geometric model corresponding to the training set, obtaining geometric information parameters containing time information, and constructing a 3D geometry-based complex action sequence classification and coding model;

The second building module is used to combine the training set and the geometric information parameters with time information corresponding to the training set into a sequence, and perform encoding and decoding training based on the transformer model until the complex action sequence of constructing three-dimensional geometry based on the transformer model is generated. The training loss function of the model converges, and a complex action sequence generation model based on 3D geometry is constructed;

Input generation module: used to input the test set of actions to be generated into the three-dimensional geometry-based complex action sequence classification and coding model, obtain geometric parameters containing time information corresponding to the action categories of the test set, and then pass the trained The complex action sequence generation model based on 3D geometry can obtain a variety of action sequences of the test set action categories.

It can be known from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for classifying and generating a 3D human body with complex action sequences. By using VIBE to process the picture data, the key points of the human body and the pose information of the action sequence in each frame of the picture can be obtained, which can reduce the influence of lens distortion on the action generation as much as possible; by using two linear full connections for the standard 3D geometric sequence layer, and a long-short-term memory neural network with time information to obtain geometric parameters containing time information, and then input them as a priori into the complex action sequence generation model based on 3D geometry, which can enhance the network’s ability to understand the distribution of different action categories in the latent space. In addition, the parameterized processing of complex actions in the data set can still accurately identify the action type in the case of complex actions, generate a reasonable action sequence, and improve the accuracy of recognition and the diversity of actions.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

FIG. 1 is a schematic flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of a single action type training process of the present invention.

FIG. 3 is a schematic diagram of the system structure of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

Embodiment 1 of the present invention discloses a three-dimensional human body classification and generation method for complex action sequences, including:

Step (1): obtaining complex action videos, preprocessing the complex action videos, including: editing, truncating frames, converting video data into picture data, and selecting picture data with rich motion features to construct a data set, specifically:

Capture action videos of various action types through the camera.

Edit the action video to obtain a video with a human body in each frame.

The frame truncation operation is performed on each frame of video, the video data is converted into picture data, and the data set is constructed by using pictures with rich motion features.

Step (2): Use VIBE to process the image data, identify the key points of the data set, and obtain the key points of the human body and the posture information of the action sequence in each frame of the picture, as a training set.

Further, regarding the collection and construction of the data set: use the rear camera of a common mobile phone to collect data. The collected video needs to meet the human body in the viewing angle, no occlusion, no movement (in-situ action), if you collect data at will, many data are invalid. In order to solve this problem, the high-efficiency of each video action is ensured by collecting videos at high frequency in a short time. In order to obtain better experimental results, a video with a resolution of 1080*1920 and an average of 30 frames per second was collected, and the duration of each video was controlled at about 15-20 seconds. At the same time, based on the 12 basic action types in the HumanAct12 dataset, we innovatively define videos of five complex action types:

1. Walk, sit, drink water

2. Squat down, stand up, walk

3. Sit down and answer the phone

4. Running, jumping

5. Eating and throwing things

Further, in order to reduce the scale of the network, the input is simplified to human key points and action sequence poses in the video. Therefore, in order to improve the quality of the dataset, the VIBE method is used to process the dataset in this embodiment.

Step (3): construct a three-dimensional geometry-based complex action sequence classification and coding model based on categorical coding, specifically:

Use the dataset function to process the training set and corresponding labels to obtain the length of the training set.

The tensors of the corresponding 3D geometric model are obtained by iterating over the training set in the data iterator.

The categorical encoding is transferred to the GPU, and the geometric parameters containing time information are obtained through two linear fully connected layers and a long-short-term memory neural network with time information.

Taking the training set as input, the output of geometric parameters containing time information corresponding to the action category of the training set is obtained.

Step (4): Combine the input and output of the three-dimensional geometry-based complex action sequence classification and encoding model into a sequence, perform encoding and decoding training based on the transformer model, and construct a three-dimensional geometry-based complex action sequence generation model.

The encoding training is specifically:

The training set and the geometric information parameters containing time information corresponding to the training set are combined into a sequence, which is input to the transformer encoding after passing through the gated recurrent unit, and the Gaussian distribution of different action categories in the latent space is obtained.

The decoding training is specifically:

By learning the variance and mean of different action categories in the latent space, through the transformer decoder, the three-dimensional information and action pose of the human body are obtained, and the SMPL model is used to render the parameters to obtain the complete human body sequence.

The training loss function of the three-dimensional geometric complex action sequence generation model based on the transformer model is:

Among them: Vt represents the three-dimensional geometric information of the input data, and Pt represents the predicted human joint points and human action posture information.

It also includes introducing the DropPath module in the process of building a complex action sequence generation model based on three-dimensional geometry based on the transformer model, so that the two stages of encoding training and decoding training are alternately performed, specifically: 1. The Join layer will be randomly discarded with a certain probability, But it must be ensured that at least one branch is open. 2. Globally discard, and randomly select a branch.

Step (5): Input the test set of the action to be generated into the described three-dimensional geometry-based complex action sequence classification and coding model, and obtain the geometric parameters (corresponding to the test set action category) containing time information corresponding to the test set action category. Model latent space coding), and then through the trained 3D geometry-based complex action sequence generation model (transformer variational autoencoder), action sequences of various test set action categories are obtained.

In addition, the model can be detected, and the detections can be trained individually for each individual action type. The test data and results are shown in Table 1. First, divide the videos of the same action type in the data set, train and output the training results separately, and then count the recognition accuracy of each action model and the accuracy of the generated action to judge the recognition and generation of the corresponding action by each sub-model. precise.

Table 1 Detection data and results of single action

Example 2

Embodiment 2 of the present invention discloses a three-dimensional human body classification and generation system for complex action sequences, including:

Acquisition module: used to acquire complex action videos, preprocess the complex action videos, and construct a data set;

Data recognition module: used to use VIBE to process image data, perform key point recognition on the data set, and obtain the human body key points and action sequence posture information in each frame of pictures, as a training set;

The first building module: based on categorical, coding and training the three-dimensional geometric model corresponding to the training set, obtaining geometric information parameters containing time information, and constructing a three-dimensional geometry-based complex action sequence classification and coding model;

The second building module: used to combine the training set and the geometric information parameters containing time information corresponding to the training set into a sequence, and perform encoding and decoding training based on the transformer model until the complex action of constructing three-dimensional geometry based on the transformer model The training loss function of the sequence generation model converges, and a complex action sequence generation model based on 3D geometry is constructed;

Input generation module: used to input the test set of actions to be generated into the three-dimensional geometry-based complex action sequence classification and coding model, and obtain geometric parameters containing time information corresponding to the action categories of the test set (and the action categories of the test set). Corresponding model latent space coding), and then through the trained three-dimensional geometry-based complex action sequence generation model, a variety of action sequences of the test set action categories are obtained.

In addition, a single-action detection module is included: used to train detections individually for each individual action type.

The invention discloses a three-dimensional human body classification and generation method and system for complex action sequences. By using VIBE to process the picture data, the key points of the human body and the pose information of the action sequence in each frame of the picture can be obtained, which can reduce the influence of lens distortion on the action generation as much as possible; by using two linear full connections for the standard 3D geometric sequence layer, and a long-short-term memory neural network with time information to obtain geometric parameters containing time information, and then input them as a priori into the complex action sequence generation model based on 3D geometry, which can enhance the network’s ability to understand the distribution of different action categories in the latent space. In addition, the parameterized processing of complex actions in the data set can still accurately identify the action type in the case of complex actions, generate a reasonable action sequence, and improve the accuracy of recognition and the diversity of actions.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (9)

1. a three-dimensional human body classification and generation method of complex action sequence, is characterized in that, comprises: Step (1): obtaining complex action videos, preprocessing the complex action videos, and constructing a data set; Step (2): carry out key point identification on the data set, obtain human body key points and action sequence posture information, as a training set; Step (3): construct a complex action sequence classification coding model based on three-dimensional geometry, take the training set as an input, obtain the output of the geometric parameter containing time information corresponding to the action category of the training set; Step (4): combine the input and output of the described three-dimensional geometry-based complex action sequence classification coding model into a sequence for encoding and decoding training, and construct a three-dimensional geometry-based complex action sequence generation model; Step (5): input the test set of the action to be generated into the described three-dimensional geometry-based complex action sequence classification coding model, obtain the geometric parameters containing time information corresponding to the action category of the test set, and then pass the trained The complex action sequence generation model based on the three-dimensional geometry obtains a variety of action sequences of the test set action categories. 2. the three-dimensional human body classification and generation method of a kind of complex action sequence according to claim 1, is characterized in that, in step (1), preprocessing described complex action video comprises: editing, truncating frame, converting video The data is converted into image data, and image data with rich motion features is selected to construct a dataset. 3. the three-dimensional human body classification and the generation method of a kind of complex action sequence according to claim 2, is characterized in that, in step (2), described data set is carried out key point identification specifically: adopt VIBE to carry out image data After processing, the key points of the human body and the pose information of the action sequence in each frame of the picture are obtained. 4. the three-dimensional human body classification and generation method of a kind of complex action sequence according to claim 1, is characterized in that, in step (3), the described three-dimensional geometry-based complex action sequence classification coding model is based on categorical coding, and constructs training The completed complex action sequence classification model includes the following steps: The training set and the corresponding label are processed by using the data set function to obtain the length of the training set; Obtain the tensor of the corresponding 3D geometric model by iterating over the training set in the data iterator; The categorical encoding is transferred to the GPU, and geometric parameters containing time information are obtained through two linear fully connected layers and a long-short-term memory neural network with time information. 5. the three-dimensional human body classification and the generation method of a kind of complex action sequence according to claim 1, is characterized in that, in step (4), build the described complex action sequence generation model based on three-dimensional geometry based on transformer model, comprising: There are two stages of encoding training and decoding training. 6. the three-dimensional human body classification and generation method of a kind of complex action sequence according to claim 5, is characterized in that, described coding training is specifically: Combining the training set and the geometric information parameters containing time information corresponding to the training set into a sequence, and inputting it into the transformer encoding after passing through the gated loop unit to obtain the Gaussian distribution of different action categories in the latent space; The decoding training is specifically: By learning the variance and mean of different action categories in the latent space, through the transformer decoder, the three-dimensional information and action pose of the human body are obtained, and the SMPL model is used to render the parameters to obtain the complete human body sequence. 7. the three-dimensional human body classification and generation method of a kind of complex action sequence according to claim 6, is characterized in that, the described training loss function that the complex action sequence generation model of three-dimensional geometry is constructed based on transformer model is:

Among them: Vt represents the three-dimensional geometric information of the input data, and Pt represents the predicted human joint points and human action posture information. 8. the three-dimensional human body classification and generation method of a kind of complex action sequence according to claim 6, is characterized in that, also comprises introducing DropPath module in the process of constructing described three-dimensional geometry-based complex action sequence generation model based on transformer model , the encoding training and the decoding training are alternately performed. 9. A three-dimensional human body classification and generation system for complex action sequences, characterized in that, comprising: Acquisition module: used to acquire complex action videos, preprocess the complex action videos, and construct a data set; Data recognition module: used to use VIBE to process image data, perform key point recognition on the data set, and obtain the human body key points and action sequence posture information in each frame of pictures, as a training set; The first building module: based on categorical, coding and training the three-dimensional geometric model corresponding to the training set, obtaining geometric information parameters containing time information, and constructing a three-dimensional geometry-based complex action sequence classification and coding model; The second building module: used to combine the training set and the geometric information parameters containing time information corresponding to the training set into a sequence, and perform encoding and decoding training based on the transformer model until the complex action of constructing three-dimensional geometry based on the transformer model The training loss function of the sequence generation model converges, and a complex action sequence generation model based on 3D geometry is constructed; Input generation module: used to input the test set of actions to be generated into the three-dimensional geometry-based complex action sequence classification and coding model, obtain geometric parameters containing time information corresponding to the action categories of the test set, and then pass the trained The complex action sequence generation model based on 3D geometry can obtain a variety of action sequences of the test set action categories.

Images