CN108446583A - Human bodys' response method based on Attitude estimation - Google Patents

Abstract

The invention discloses a human body behavior recognition method based on gesture estimation, which mainly solves the problem of too slow processing speed in video human body behavior in the prior art. The implementation steps are: 1. Use the Open-pose method to estimate the pose of the human body in the video, and extract the position coordinates of the joint points of the human body in each frame of the video; 2. Calculate the joint points of the human body in two adjacent frames according to the position coordinates of the joint points of the human body in each frame Distance variation matrix; 3. Segment the video and use the distance variation matrix of each video to generate video features; 4. Divide the video in the data set into two parts, the training set and the test set, and train the classifier with the video features of the training set , use the trained classifier to classify the videos in the test set. The invention improves the speed of human body behavior recognition in video, and can be used for intelligent video monitoring, human-computer interaction and video retrieval.

Description

Human Action Recognition Method Based on Pose Estimation

technical field

The invention belongs to the technical field of image processing, and in particular relates to a video human behavior recognition method, which can be used for intelligent video monitoring, human-computer interaction and video retrieval.

Background technique

With the development and application of computer science and artificial intelligence, video analysis technology has risen rapidly and received widespread attention. A core of video analysis is human behavior recognition. The accuracy and speed of behavior recognition will directly affect the results of follow-up work of the video analysis system. Therefore, how to improve the accuracy and rapidity of human behavior recognition in video has become a key issue in the research of video analysis systems.

At present, typical video human behavior recognition methods mainly include: spatio-temporal interest points, dense trajectories, etc. in:

Spatio-temporal points of interest are human behavior recognition by detecting corners in the video and extracting the features of the corners. However, some corners are generated by background noise, which will not only affect the final result, but also slow down the recognition speed.

Dense trajectory is to perform dense sampling on multiple scales for each frame of the video, then track the sampled points to obtain the trajectory, and then extract the characteristics of the trajectory for behavior recognition. However, the calculation complexity of this method is high, and the generated feature dimension is high, which will occupy a large amount of memory, and it is difficult to achieve real-time recognition.

Contents of the invention

The object of the present invention is to solve the problem of poor real-time performance in the above prior art, and propose a human behavior recognition method based on pose estimation, so as to improve the speed of human behavior recognition.

The technical idea of the present invention is: by estimating the posture of the human body in the video, the position of the joint points of the human body is obtained in each frame, and the movement of the human body is analyzed by using the position change of the joint points of the human body, so as to quickly recognize the human body behavior.

According to above-mentioned train of thought, the realization scheme of the present invention comprises as follows:

(1) Extract the position coordinates of the joint points of each frame of the human body in the video:

(1a) Use the Open-pose method to estimate the pose of each frame of the human body in the video, and obtain the human neck, chest, head, right shoulder, left shoulder, right hip, left hip, right elbow, left elbow, right knee, left knee, The position coordinates of the 15 joint points of the right wrist, left wrist, right ankle and left ankle, wherein the coordinates of the kth joint point are expressed as L _k =(x _k ,y _k ), k ranges from 1 to 15;

(1b) Normalize the position coordinates of each joint point;

(1c) Use the normalized 15 joint point position coordinates to form a coordinate matrix P, P=[(x ₁ ,y ₁ ),(x ₂ ,y ₂ ),...,(x _k ,y _k ) ,...,(x ₁₅ ,y ₁₅ )], where (x _k ,y _k ) represents the normalized coordinates of the kth joint point;

(2) Calculate the distance variation matrix of human joint points in two adjacent frames:

(2a) According to the coordinate matrices P _n and P _n-1 of two adjacent frames, calculate the coordinate change matrix of joint point positions in two adjacent frames

(2b) According to the coordinate change matrix of the joint point position Calculate the distance change matrix D of joint points;

(3) Generate video features:

(3a) According to the time length of the video, the video is divided into 4 segments on average, and the distance change matrix D generated by two adjacent frames in each segment of the video is added to obtain the cumulative distance change matrix D _i of each segment, where i ranges from 1 to 4 ;

(3b) Perform L2 normalization on D _i to obtain D _i ' after normalization;

(3c) Concatenate the cumulative distance variation matrix D _i ' as the feature of the entire video: F=[D ₁ ', D ₂ ', D ₃ ', D ₄ '];

(4) Train the classifier to classify the video:

(4a) divide the video of sub-JHMDB data set into training set and test set two parts, input the feature of training set video in the support vector machine and train, obtain the well-trained support vector machine;

(4b) Input the features of the test set video into the trained support vector machine to obtain the classification result.

The present invention has the following advantages:

Since the present invention adopts the Open-pose method to estimate the pose of the human body in the video, it can quickly obtain the joint point position coordinates of each frame of the human body in the video, and at the same time, because the video is segmented, it can obtain the position of the human body in different time periods of the video. The amount of change in the position of the joint points, so as to use the amount of position change to classify the human behavior in the video.

Description of drawings

Fig. 1 is the realization flowchart of the present invention;

Figure 2 is a schematic diagram of the position of human joint points estimated by Open-pose;

Detailed ways

Below with reference to accompanying drawing, technical scheme and effect of the present invention are further described:

With reference to Fig. 1, the implementation steps of the present invention are as follows:

Step 1. Extract the position information of human joint points in each frame of the video.

1.1) Use the Open-pose method to estimate the pose of each frame of the human body in the video, and obtain the human neck, chest, head, right shoulder, left shoulder, right hip, left hip, right elbow, left elbow, right knee, left knee, right The position coordinates of the 15 joint points of the wrist, left wrist, right ankle and left ankle, where the coordinates of the kth joint point are expressed as L _k = (x _k , y _k ), k ranges from 1 to 15, as shown in Figure 2 Show;

1.2) Normalize the position coordinates of each joint point:

Where x, y represent the coordinates before normalization, x', y' represent the coordinates after normalization, W represents the width of each frame of the video, and H represents the height of each frame of the video;

1.3) Use the normalized 15 joint point position coordinates to form a coordinate matrix P, P=[(x ₁ ,y ₁ ),(x ₂ ,y ₂ ),...,(x _k ,y _k ), ...,(x ₁₅ ,y ₁₅ )], where (x _k ,y _k ) represents the normalized coordinates of the kth joint point.

Step 2. Calculate the distance variation matrix of human joint points in two adjacent frames.

2.1) According to the coordinate matrices P _n and P _n-1 of two adjacent frames, calculate the coordinate change matrix of joint point positions in two adjacent frames

Among them, P _n and P _n-1 represent the coordinate matrix of the joint point position in the previous frame and the next frame respectively, and dx and dy represent the coordinate change of the same joint point in two adjacent frames;

2.2) According to the coordinate change matrix of the joint point position Calculate the distance change matrix D of joint points:

where dx _k and dy _k represent The kth element in .

Step 3. Generate video features.

3.1) Divide the video into 4 segments on average according to the time length of the video, add the distance change matrix D generated by two adjacent frames in each video segment, and obtain the cumulative distance change matrix D _i of each segment, where i ranges from 1 to 4.

3.2) Perform L2 normalization on D _i to obtain D _i ' after normalization:

Wherein D _i =[d ₁ ,d ₂ ,...,d _k ,...,d ₁₅ ] is the cumulative distance variation matrix of the i-th video, and d _k represents the kth element in D _i , is the L2 norm of D _i , Indicates the square of the kth element in D _i ;

3.3) Concatenate the cumulative distance variation matrix D _i ' as the feature of the entire video:

F＝[D ₁ ', D ₂ ', D ₃ ', D ₄ '] <5>

Step 4. Train a classifier to classify the video.

4.1) divide the video of sub-JHMDB data set into training set and test set two parts, input the feature of training set video in the support vector machine and train, obtain the well-trained support vector machine;

4.2) Input the features of the test set video into the trained support vector machine to get the classification result.

Effect of the present invention can be further illustrated by following experiments:

1. Experimental conditions.

Experimental environment: The computer adopts Intel(R) Core(TM) i7-7700CPU@3.8Ghz, 16GB memory, GPU is GTX1080, and the software adopts Matlab2014b simulation experiment platform.

Experimental parameters: the support vector machine uses a linear kernel, and the parameter c=8.

2. Experimental content and results.

The experiment is carried out on the sub-JHMDB data set, which contains 12 types of human actions, including a total of 316 video clips, and each video clip contains a human behavior. According to the presetting of the sub-JHMDB data set provider, the videos in the data set are divided into two parts: training set and test set. Utilize the inventive method to process the video in the sub-JHMDB dataset to obtain the video feature, the feature of the training set video is used to train the classifier, then use the trained classifier to classify the test set video, and the correct classification ratio of the test set video is used as final classification result.

The classification result on the sub-JHMDB dataset reaches 43.9%, and the processing speed of the video is 10fps on average.

In summary, it can be concluded that the present invention can realize rapid recognition of human behavior in videos.

Claims (5)

1. a kind of Human bodys' response method based on Attitude estimation, including：

(1) it extracts in video per frame human joint points position coordinates：

(1a) using Open-pose methods to Attitude estimation is carried out in video per frame human body, obtain human body neck, chest, head, Right shoulder, left shoulder, right hips, left buttocks, right hand elbow, left hand elbow, right knee, left knee, right finesse, left finesse, right ankle and a left side The position coordinates of this 15 artis of ankle, wherein the coordinate representation of k-th of artis is L_k=(x_k,y_k), k is from 1 to 15；

The position coordinates of each artis are normalized in (1b)；

(1c) constitutes coordinates matrix P, P=[(x with 15 artis position coordinates after normalization₁,y₁),(x₂,y₂),..., (x_k,y_k),...,(x₁₅,y₁₅)], wherein (x_k,y_k) indicate coordinate after the normalization of k-th artis；

(2) adjacent two frames human joint points distance change moment matrix is calculated：

(2a) is according to the coordinates matrix P of adjacent two frame_nAnd P_n-1, calculate adjacent two frames artis position coordinates and change moment matrix

(2b) changes moment matrix according to artis position coordinatesCalculate artis distance change moment matrix D；

(3) video features are generated：

Video is divided into 4 sections by (3a) according to the time span of video, by the distance that adjacent two frame generates in each section of video Variable quantity matrix D is added, and obtains each section of cumulative distance variable quantity matrix D_i, i is from 1 to 4；

(3b) is to D_iCarry out L2 normalization, the D after being normalized_i'；

(3c) is by cumulative distance variable quantity matrix D_i' the feature that is together in series as entire video：

F=[D₁',D₂',D₃',D₄']；

(4) training grader classifies to video：

The video of sub-JHMDB data sets is divided into training set and test set two parts by (4a), and the feature of training set video is defeated Enter and be trained into support vector machines, obtains trained support vector machines；

(4b) is input to the feature of test set video in trained support vector machines and obtains classification results.

2. according to the method described in claim 1, wherein the position coordinates of each artis are normalized in step (1b), It carries out as follows：

Wherein x, y indicate that the coordinate before normalization, x', y' indicate that the coordinate after normalization, W indicate each frame width of video, H Indicate each frame height of video.

3. according to the method described in claim 1, wherein calculating adjacent two frames artis position coordinates variable quantity in step (2a) MatrixIt is calculated as follows：

Wherein P_nAnd P_n-1The artis location matrix of former frame and a later frame, dx are indicated respectively_kAnd dy_kIndicate k-th of artis Adjacent two frame coordinates variable quantity.

4. according to the method described in claim 1, artis distance change moment matrix D is wherein calculated in step (2b), by as follows Formula calculates：

Wherein dx_kAnd dy_kIt indicatesIn k-th of element.

5. according to the method described in claim 1, wherein to D in step (3b)_iL2 is carried out to normalize to obtain D_i', as follows It calculates：

Wherein D_i=[d₁,d₂,...,d_k,...,d₁₅] it is i-th section of video cumulative distance variation moment matrix, d_kIndicate D_iIn k-th Element,It is D_iL2 norms,Indicate D_iIn k-th of element square.