CN111222459B - Visual angle independent video three-dimensional human body gesture recognition method - Google Patents

Abstract

The invention relates to a visual angle independent video three-dimensional human body gesture recognition method, which comprises the following steps: step 1: virtual data generation phase: synthesizing virtual camera parameters based on a human body posture data set containing three-dimensional labels at will, and then generating a two-dimensional/three-dimensional data tuple; step 2: model training stage: training a first module of a modularized neural network for obtaining a model with camera view generalization capability and a second module of the modularized neural network for obtaining a model capable of protecting inter-frame motion continuity by using the generated two-dimensional/three-dimensional data elements respectively; step 3: unconstrained video reasoning phase: and (3) predicting the video acquired by any unconstrained acquisition by utilizing the multi-module depth neural network trained in the step (2) to acquire a three-dimensional human body posture recognition result. Compared with the prior art, the method is based on the modularized neural network combined training method, and the generalization capability of three-dimensional human body gesture recognition is effectively improved.

Description

A view-independent method for 3D human pose recognition in video

technical field

The present invention relates to three-dimensional human body posture recognition technology in the field of computer vision technology, in particular to a method for unknown viewing angle data synthesis, modularized neural network training and preprocessing for video tasks, that is, a viewing angle-independent video three-dimensional human body posture recognition method.

Background technique

In recent decades, with the development of artificial intelligence and deep learning related technologies, the subject of human gesture recognition has also made great progress. Video human pose recognition, especially 3D human pose recognition for video has long been an important content in the field of computer vision and intelligent human-computer interaction; it integrates digital image processing, human-computer interaction, computer graphics, computer vision, etc. With the popularity of portable mobile electronic devices such as security monitoring networks, intelligent robots, smart phones, and tablet computers, it has further integrated into people's lives.

The existing 3D human body pose recognition algorithm can be divided into single-stage human body pose recognition and multi-stage human body pose recognition according to the prediction target: the former generally refers to the direct regression of RGB or RGB-D images to obtain the key point position or parameterized model of the 3D human body The parameter information of this method has the advantage of using more hidden information in the picture, and the accuracy is higher in the laboratory environment. However, due to the lack of RGB image data with three-dimensional annotations, it cannot be separated from the laboratory collection environment, so the generalization ability Poor, it is difficult to transform into a product with strong usability to generate commercial value; the latter generally refers to estimating the key point positions of the two-dimensional human body posture in stages, and then estimating the three-dimensional human body posture. The advantage of this method is that it can Using manual labeling to collect a large number of Internet unconstrained pictures for 2D human body pose estimation training, and the 2D to 3D prediction problem has also been proved to be a relatively easy task by Martinez et al. In order to facilitate the transformation, the present invention generally uses the framework of multi-stage human body posture recognition. However, on the basis of the existing strong two-dimensional human body key point detection model, the existing method is still limited by the lack of perspective of the three-dimensional collected data. It is easier to overfit to the data set camera parameters.

Contents of the invention

The purpose of the present invention is to overcome the above-mentioned defects in the prior art and to provide an angle-independent video three-dimensional body pose recognition method, and to propose a virtual angle synthesis method, which uses a camera angle enhancement module to generate a random angle of view, and cooperates with the camera projection relationship to obtain a two-dimensional /3D data tuple, used for neural network multi-module training and generalization ability verification; in addition, it is proposed to use the 2D human body posture detection frame to normalize the input of 3D prediction, so that the 3D human body pose estimation method in an unconstrained environment is separated from The internal and external parameters of the camera are limited, so it has stronger generalization ability.

The purpose of the present invention can be achieved through the following technical solutions:

A viewing angle-independent video 3D human gesture recognition method, the recognition method comprising:

Step 1: Virtual data generation stage: based on any human body pose dataset containing 3D annotations, generate 2D/3D data tuples after synthesizing virtual camera parameters;

Step 2: Model training phase: Use the generated 2D/3D data tuples to train the first module of the modular neural network for obtaining a model with camera view generalization capability and the first module for obtaining a model that can protect the continuity of motion between frames. Model the second module of the modular neural network;

Step 3: Unconstrained video inference stage: For any unconstrained collected video, the 3D human pose recognition result is obtained by using the multi-module deep neural network trained in step 2 to predict.

Further, the step 1 specifically includes: for any human body pose dataset containing 3D annotations, using a camera view enhancement module to synthesize virtual camera parameters, and using projection relationships to generate 2D/3D data tuples.

Further, the camera parameters include external parameters for determining the position and orientation of the camera and internal parameters for determining the focal length and frame size of the camera projection.

Further, the first module in step 2 uses single frame data tuples to perform viewing angle enhancement training.

Further, the second module in step 2 uses continuous sequence of data tuples to perform time series model training.

The first module and the second module only need to satisfy that the first module is a single-frame 2D to 3D prediction module, and the second module is a sequential 3D to 3D correction module. The modules are connected in series to complete the prediction from 2D to 3D.

Further, in step 2 and step 3, before inputting the neural network, it also includes a camera-independent two-dimensional detection normalization preprocessing process for the two-dimensional detection results, and its corresponding description formula is:

表示原二维点坐标，

表示二维检测框的中心坐标，w^d,h^d分别为二维检测框的宽度和高度。In the formula, K _{x, y} represent the two-dimensional point coordinates after two-dimensional detection normalization preprocessing,

Indicates the original two-dimensional point coordinates,

Represents the center coordinates of the two-dimensional detection frame, w ^d , h ^d are the width and height of the two-dimensional detection frame, respectively.

Further, the video obtained by the unconstrained acquisition in step 3 specifically includes acquisition under natural conditions, or a video sequence that has been scaled, cropped, changed in speed, and transformed by other color adjustments.

Compared with the prior art, the present invention has the following advantages:

(1) In the perspective-independent video three-dimensional human body posture recognition method proposed by the present invention, in the virtual data generation stage, a reasonable random perspective is used to replace the camera perspective used in the original fixed perspective training to overcome the data set The internal and external parameters of the camera are dependent; in the model training phase, the modular design can train two separate modules separately, or complete training on video stream data tuples serially. The tasks of the two separate modules are clear and can be verified separately. Strong generalization ability.

(2) And because the present invention utilizes time series model training, it can obtain a longer period of prediction prompts on the basis of controlling the receptive field; in its unconstrained video reasoning stage, due to the effective normalization frame design and selection, decoupling Depending on the projection relationship, good prediction results can be obtained for a large number of videos collected from the Internet that lack camera parameters, have extreme proportions of people (often manifested as the proportion of people in the original video is too small), and have been processed by cropping.

(3) The present invention proposes a viewing angle-independent video 3D human pose recognition method, using a large number of 2D/3D data tuples enhanced by camera viewing angles to train the multi-module neural network, and using a camera-independent 2D The detection and normalization method performs preprocessing of two-dimensional input; the first module in the present invention can adapt to the unconstrained three-dimensional human body pose estimation task, and obtains a strong camera generalization ability; the second module can effectively use the time series The continuous features make the prediction key points obtain better spatial stability, and make the whole prediction achieve a more ideal accuracy.

Description of drawings

Fig. 1 is a method structure flowchart of the present invention;

Fig. 2 is a schematic diagram of rotation (attitude angle) control when the camera parameters in the method of the present invention are generated;

Fig. 3 is the structural example diagram of the first module neural network and the second module neural network in the method of the present invention;

Fig. 4 is a schematic diagram of the projection relationship in the method of the present invention;

Fig. 5 is a schematic diagram of a method for normalizing two-dimensional detection frames in the method of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

As shown in Figure 1, it is a flow chart of the overall structure of a viewing angle-independent video three-dimensional human body gesture recognition method according to the present invention, which mainly includes the following three stages: virtual data generation stage, model training stage, unconstrained video reasoning stage, and also includes A camera-independent 2D detection normalization method used in both training and inference phases;

Virtual data generation stage: For any public 3D human body pose academic data set, or the 3D human body pose data set collected by the motion capture system, the corresponding 2D projection and 3D stereoscopic data are generated through the camera perspective synthesis principle and projection transformation proposed by the present invention attitude data tuple;

Model training stage: train the neural network model, for the first module proposed by the present invention, use large-scale viewing angle enhancement single-frame data tuples for training, obtain better viewing angle resistance and decouple from camera parameters; The second module uses video stream data containing 3D annotations for time-series learning and prediction to obtain spatial continuity in time series and improve the accuracy of gesture recognition;

Unconstrained video reasoning stage: For the video data stream obtained in the general environment (in-the-wild), and the module based on any two-dimensional human key point detection, a special two-dimensional human body detection key point proposed by the present invention The normalization method of the point results is used for preprocessing, and the processed two-dimensional data is sequentially predicted through the first and second modules to obtain the human body posture based on the three-dimensional key point representation.

The human body posture data representation method mentioned in the present invention is mainly the key point-skeleton representation method; the first module proposed by the present invention is mainly used to improve the generalization ability of the viewing angle, and the second module is mainly used to obtain time series prediction with a larger receptive field and Characterized by good stability. The scenarios described in the present invention include, but are not limited to, research and applications involving video human gesture recognition. The invention is based on a modularized neural network combination training method, which effectively improves the generalization ability of three-dimensional human body posture recognition.

Among them, the method process in the stage of virtual data generation includes but is not limited to public academic datasets, datasets collected by motion capture systems, etc., as long as there are 3D annotations and camera parameters (that is, as long as there is a 2D/3D projection relationship). .

The implementation of the first and second modules defined in the respective method processes of the model training stage and the unconstrained video inference stage is not limited to the example implementation in the description of the present invention, and any one that is applicable to predicting a three-dimensional human body posture based on single-frame two-dimensional human body detection, And the neural network for predicting the continuous sequence of three-dimensional human posture based on the time series model, etc., can replace the first and second modules referred to in the present invention.

The process of the method in the unconstrained video inference stage is suitable for unconstrained videos, that is, natural conditions acquisition, or video sequences that have undergone transformations including but not limited to scaling, cropping, variable speed, and other color adjustments.

The normalization method used as a special preprocessing stage in the model training stage and the unconstrained video inference stage is applicable to unconstrained videos: even if the relative projection position relationship between the characters and the camera is destroyed, as long as there are detectable two-dimensional human key Point results or detection modules, the method of the present invention is applicable.

Further, the specific process details of each stage in the method of the present invention are as follows:

In the perspective-independent video three-dimensional human posture recognition method proposed by the present invention, the virtual data generation stage further includes: for the existing data set of three-dimensional human posture data, a number of different camera parameters are generated by designing a reasonable random scheme, including determining The external parameters of the camera's position and orientation and the internal parameters that determine the focal length of the camera's projection frame. On the basis of random parameters, the projection relationship is continuously used to generate corresponding 2D human body data under different camera parameters for 3D human body data, and then obtain 2D/3D human body pose data tuples; for video data sets, the video sequence should also be considered The range of motion of the observed human body is based on the motion trajectory to obtain a reasonable internal reference, so that the projection frustum contains the moving point set of the key points of the three-dimensional human body as much as possible.

In the perspective-independent video 3D human gesture recognition method proposed by the present invention, the model training stage further includes: sub-module neural network training. That is, for the first module, either single frame or continuous can be used, but mainly single-frame data tuples are used for viewing angle enhancement training to obtain a model with camera viewing angle generalization ability; for the second module, mainly use The continuous sequence of data tuples is used for temporal model training to obtain a model that can preserve the continuity of motion between frames. Note: Generally, for RGB video input, the first module can be understood as a two-dimensional to three-dimensional regression problem, and the second module is a time-series continuous three-dimensional to time-series continuous three-dimensional regression problem; for RGB-D video input, additional The depth latitude can be added to the first and second modules at the same time.

In the perspective-independent video three-dimensional human body gesture recognition method proposed by the present invention, the unconstrained video reasoning stage further includes: first obtain the two-dimensional detection result through any known two-dimensional human body key point detection method, and then use the method described in the present invention The camera-independent two-dimensional detection normalization method performs data preprocessing, and then sequentially passes through the first and second modules of the present invention for forward reasoning to obtain the corresponding three-dimensional human body pose estimation result of the video sequence.

In the angle-of-view-independent video three-dimensional human body posture recognition method proposed by the present invention, the camera-independent two-dimensional detection normalization method further includes: using the two-dimensional human body key point detection frame (or multiplying by an appropriate coefficient) as a normalization Standard, to normalize the input of key points of the two-dimensional human body. This normalization method has better camera resistance, and has better resistance to the loss and destruction of the projection relationship caused by local zooming and cropping of the picture. sex.

The view-independent video three-dimensional human gesture recognition method proposed by the present invention will be specifically described below in conjunction with specific embodiments.

In the first stage of the method of the present invention, the stage of virtual data generation: first, according to the existing three-dimensional human body posture data set, such as the public academic data set Human3.6M, a random configuration of camera parameters is generated for the three-dimensional human body world coordinates of each section of video , this configuration will set the camera position and rotation angle according to the height and activity range of the observed human body. 0.5 times the Gaussian radius, randomly determine a point as the passing point O in the direction of the camera's optical axis; randomly select the Euclidean distance from the camera to O point with a uniform distribution within 4.0 meters to 6.5 meters, the camera attitude angle is shown in Figure 2, fixed roll(Rotate oncamera's direction vector The angle of rotation in the camera direction of the camera's direction vector is fixed, and the pitch (Rotate on the cross product of the other camera's up and directionvectors) angle is randomly generated between -15 degrees and +15 degrees, randomly Generate yaw (Rotate on camera's up vector camera vertical rotation) angle between 0 degrees and +360 degrees; because the Human3.6M dataset comes with camera internal parameters, internal reference generation can not be done here temporarily. The above-mentioned values are randomly generated every time sampling, and the two-dimensional/three-dimensional human body posture data tuple is obtained by using the projection relationship.

In the second stage of the present invention, the model training stage: for the first module (camera-agnosticregressor), a random single-frame data tuple is used for training, and this implementation uses a two-time iterative regression model with a residual connection based on a deep neural network , according to the key point regression of the 2D human body posture, the key points of the 3D human body posture are obtained, which has the characteristics of camera independence and strong viewing angle generalization; the second module (temporal regressor) is improved based on the hollow convolution model in time series, and a 3D posture is designed The modified network is used to use the hole convolution method to expand the characteristics of the receptive field to increase the spatial continuity of the three-dimensional prediction results in time series, and to compensate for the prediction results of the first module. The loss function used for the two-part supervision of the training process is shown below.

和

分别为第一模块和第二模块各自的权重，

和

分别为第一模块和第二模块各自的损失。In the formula, L is the total loss,

and

are the respective weights of the first module and the second module,

and

are the respective losses of the first module and the second module, respectively.

In the third stage of the present invention, the unconstrained video reasoning stage: for any video collected without constraints, on the basis of the existing two-dimensional detection results, use the multi-module deep neural network obtained by the second stage of the present invention to train and converge , sequentially predicted to obtain the 3D human pose result. The implementation of the first and second modules described in this example, as well as the reasoning process are shown in Figure 3. The single-frame and multi-frame in Figure 3 represent a single and multiple two-dimensional frames, respectively.

The aforementioned camera-independent two-dimensional detection normalization method used in the second and third stages of the present invention: as shown in Figure 4, the principal point and optical point in Figure 4 represent the center of the two-dimensional detection frame before and after projection respectively In general, the projection relationship and the traditional method use the pixel size of the original image frame or the size of the circumscribed square of the original image as the normalization standard. The calculation function is as follows:

It can be seen that it depends on the camera parameters (focal length) and its equivalent form also depends on the size of the original image, lacking transformation resistance such as cropping. The normalization method proposed by the present invention is shown in Figure 5, and the specific calculation function is as follows, which has the characteristics that the two-dimensional detection size can be kept stable and has nothing to do with camera parameters:

表示原二维点坐标，

表示二维检测框的中心坐标，m大于1，本实施例取1.2，w^d,h^d分别为二维检测框的宽度和高度。In the formula, K _{x, y} represent the two-dimensional point coordinates after two-dimensional detection normalization preprocessing,

Indicates the original two-dimensional point coordinates,

Indicates the center coordinates of the two-dimensional detection frame, m is greater than 1, and this embodiment takes 1.2, w ^d , h ^d are the width and height of the two-dimensional detection frame, respectively.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalents within the technical scope disclosed in the present invention. Modifications or replacements shall all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (7)

1. A visual angle-independent video three-dimensional human gesture recognition method, characterized in that, the recognition method comprises: Step 1: Virtual data generation stage: based on any human body pose dataset containing 3D annotations, generate 2D/3D data tuples after synthesizing virtual camera parameters; Step 2: Model training phase: Use the generated 2D/3D data tuples to train the first module of the modular neural network for obtaining a model with camera view generalization capability and the first module for obtaining a model that can protect the continuity of motion between frames. Model the second module of the modular neural network; Step 3: Unconstrained video inference stage: For any unconstrained collected video, the 3D human pose recognition result is obtained by using the multi-module deep neural network trained in step 2 to predict. 2. A view-independent video three-dimensional human pose recognition method according to claim 1, wherein said step 1 specifically comprises: for any human pose data set containing three-dimensional annotations, using a camera viewing angle enhancement module to synthesize Virtual camera parameters, and generate 2D/3D data tuples using projection relations. 3. A view-independent video three-dimensional human gesture recognition method according to claim 2, wherein said camera parameters include external parameters for determining the position and orientation of the camera and internal parameters for determining the focal length and frame size of the camera projection. 4. A kind of viewpoint-independent video three-dimensional human gesture recognition method according to claim 1, characterized in that, the first module in the step 2 uses single-frame data tuples to carry out the training of viewpoint enhancement. 5. A view-independent video three-dimensional human gesture recognition method according to claim 1, characterized in that, the second module in the step 2 uses continuous sequence data tuples to perform time-series model training. 6. A view-independent video three-dimensional human gesture recognition method according to claim 1, characterized in that, in the step 2 and step 3, before inputting the neural network, it also includes performing a camera test on the two-dimensional detection results. The irrelevant two-dimensional detection normalization preprocessing process, its corresponding description formula is:

In the formula, K _{x, y} represent the two-dimensional point coordinates after two-dimensional detection normalization preprocessing,

Indicates the original two-dimensional point coordinates,

Represents the center coordinates of the two-dimensional detection frame, w ^d , h ^d are the width and height of the two-dimensional detection frame, respectively. 7. A kind of angle-of-view-independent video three-dimensional human body gesture recognition method according to claim 1, characterized in that, the video obtained by the unconstrained collection in the step 3 specifically includes natural condition collection, or after zooming, cutting, Variable speed, and other color adjustments to transform video sequences.

Images