CN105787439A - Depth image human body joint positioning method based on convolution nerve network - Google Patents

Abstract

The invention discloses a depth image human body joint positioning method based on a convolution nerve network. The method is characterized by comprising a training process and an identification process. The training process comprises the following steps: 1, inputting a training sample; 2, initializing a deep convolution nerve network and its parameters, wherein the parameters comprise a weight and a bias of each layer edge; and 3, by use of a forward algorithm and a backward algorithm, learning the parameters of the constructed convolution nerve network. The identification process comprises the following steps: 4, inputting a test sample; and 5, performing regression on the input test sample by use of the trained convolution nerve network to find positions of human body joints. According to the invention, by use of the deep convolution nerve network and large data, multiple challenges such as shielding, noise and the like can be resisted, and the accuracy is quite high; and at the same time, by means of parallel calculation, the effect of accurately positioning the human body joints in real time can be realized.

Description

A kind of depth image human synovial localization method based on convolutional neural networks

Technical field

The present invention relates to computer vision, pattern recognition and field of human-computer interaction, particularly to a kind of depth image human synovial localization method based on convolutional neural networks.

Background technology

Body pose estimation and motion capture are important research directions of computer vision field.Its application includes home entertaining, man-machine interaction, action recognition, security system, remotely monitoring, intelligent monitoring, even also patient health nursing etc..But in common RGB image or video, carry out human posture estimates it is a very challenging job.Because for color, illumination, the factor of natural environment such as blocking and cannot accomplish robust, add the difference of the too many degree of freedom of human posture and observation angle so that this problem is extremely difficult naturally.

Depth image is the gray-scale map of a kind of two dimension, but different from traditional gray level image, and the gray value reflection of each pixel of depth image is object millimeter distance of distance video camera in real space that this point is corresponding.Compared to traditional colored two dimensional image, depth image has the advantages that not to be subject to the such environmental effects such as illumination, shade, can effectively express the geometry information of object in real world, therefore in the research of computer vision and man-machine interaction and application, there is critical role.Along with popularizing of cheap depth camera, research and application based on depth image have wide market and bright prospect.

Depth image human synovial localization method refers to, in a depth image comprising personage or human body, it is determined that human joint points position.Here human joint points refers to: the skeletal joint of hands, ancon, wrist, shoulder, head, ankle, knee, buttocks et al..Determine that the position of human joint points allows us to parse human skeleton structure, and then simply judge the posture of human body, so that the action of identification people and behavior, this is significant for human-computer interaction amusement and computer vision.

Depth image human synovial location is primarily present following difficult point:

1) depth image has the defect that resolution is low, mechanical noise is big.The feature making hand-designed cannot obtain good effect to position human synovial.

2) location of human synovial is because the angles of video camera is different, video camera coverage extent different from the distance of personage, personage self is different, will reach accurate robust extremely difficult.

3) there is restriction relation between human skeleton joint: during the limb motion of personage, between limbs and limbs, there is the restriction relations such as linkage and braking, and learn and to give expression to this linkage restriction relation extremely difficult.

4) location in human skeleton joint and tracking are difficult to merge.The position of current personage and posture location be both for individual depth image, reason be in that beyond expression of words go out skeletal joint Movement consistency in time domain.

Above-mentioned difficult point makes to realize accurate robust and carries out the target of human synovial location and also have a certain distance, therefore, solves above-mentioned difficult point very necessary.

Summary of the invention

Present invention is primarily targeted at the shortcoming overcoming prior art with not enough, it is provided that a kind of depth image human synovial localization method based on convolutional neural networks.

In order to arrive above-mentioned purpose, the present invention by the following technical solutions:

The present invention compared with prior art, has the advantage that and beneficial effect:

1, the present invention has relied on and has got most of the attention and had the study of the potential technology degree of depth at present, namely the convolutional neural networks of profound level is built, from substantial amounts of training sample (these training samples contain the multiple distance on multiple angles, video camera and people way that video camera is put and the multiple coverage extent of personage self), automatically learn effective feature, and not in the feature relying on people's hand-designed.By learning the validity feature, directly return out the articulare position of human body.

2, the convolutional neural networks of the present invention uses Three dimensional convolution layer to express skeletal joint Movement consistency in time domain；The loss function based on skeleton relational tree is used to express the restriction relations such as the linkage between skeleton joint and braking at top layer.

Accompanying drawing explanation

The flow chart of Fig. 1 present invention；

Fig. 2 convolutional neural networks Organization Chart of the present invention；

Fig. 3 human body skeletal joint point schematic diagram of the present invention.

Detailed description of the invention

Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited to this.

Embodiment

As it is shown in figure 1, the present invention is based on the depth image human synovial localization method of convolutional neural networks, including training process and identification process；

The step of training process is as follows:

1) input training sample；

2) initializing profound convolutional neural networks and parameter thereof, described parameter includes weight and the biasing on every layer of limit；

3) adopt forwards algorithms and backward algorithm, utilize training sample to learn the parameter of convolutional neural networks of structure；

The step of identification process is as follows:

4) input test sample；

5) utilize the convolutional neural networks the trained test sample to input, return out the position of human joint points therein.

Below in conjunction with concrete technical scheme, technical scheme is further elaborated:

1. the framework of convolutional neural networks

The present invention proposes the full convolutional neural networks of a profound level (as shown in Figure 2), and the human posture's articulare in depth image and range image sequence is estimated.This network is composed in series by 9 convolutional layers, is wherein also interspersed with down-sampled layer and normalization layer.To make introductions all round below:

Two-dimensional convolution layer: convolutional layer refers to that the image to input or feature carry out convolution at two-dimensional space, and it can extract the feature that some are important.Assume width and highly respectively w and the h of input picture, three dimensional convolution kernel be sized to w' × h' × m', wherein w', h', m' represent width, height and port number respectively.A characteristic pattern can be obtained after convolution.Wherein be positioned at characteristic pattern (x, y) value of position can be expressed as,

$v_{xys}=\tanh (b+\underset{i=0}{\overset{w^{\′}-1}{\Σ}}\underset{j=0}{\overset{h^{\′}-1}{\Σ}}\underset{k=0}{\overset{m^{\′}-1}{\Σ}}{\mathrm{\ω}}_{ijk}\·p_{(x+i)(y+j)(s+k)})$

Wherein p_{(x+i)(y+j)(s+k)}Represent the pixel value of (x+i, y+j) position, ω in (s+k) frame of input_ijkRepresenting the parameter of convolution kernel, b represents with the biasing relevant to this characteristic pattern.So we can obtain 1 characteristic pattern, each characteristic pattern be sized to (w-w'+1, h-h'+1).Owing to single convolution kernel can only extract a type of feature, therefore we introduce multiple convolution kernel at each layer of convolutional layer and extract multiple different feature.

Down-sampled layer: down-sampled we use max-pooling to operate.This operation refers to and according to certain strategy (choosing maximum), characteristic pattern is carried out down-sampled process.This is a kind of effective procedure being widely used, and it can extract the feature keeping shape and skew invariance.One stack features figure, max-pooling is operated by they are down-sampled, obtains same amount of one group of low resolution characteristic pattern.More, if at a₁×a₂Applying the max-pooling operation of 2 × 2 on the characteristic pattern of size, the maximum on extraction 2 × 2 not overlapping region, we will obtain being sized to a₁/2×a₂The new feature figure of/2.

Correct linear elementary layer (ReLUNonliearityLayer): this layer is to adopt simple non-linear threshold function, input is only allowed the conversion that non-negative signal passes through.Assuming the output representing this layer of g, W represents the weight on this layer of limit, and a represents this layer of input, then we have

G=max (0, W^Ta)

Experiment proves in profound convolutional neural networks, and when using the linear elementary layer of rectification can make to train, the convergence rate of network is than traditional excitation function faster.

Full articulamentum: we with the addition of the full articulamentum of two-layer in a model, it is possible to regard the perceptron model (hidden layer and logistic regression layer) set up on the basis of above two-dimensional convolution layer output as.First the characteristic pattern obtained from M sub-network is connected into a long characteristic vector by us.Namely this vector represents the feature being drawn into from range image sequence.Its every one-dimensional element all connects all nodes to first full articulamentum (hidden layer), and is entirely connected to all of output unit further.Output unit 2K altogether, K represents the number of skeleton node here, and namely the value of output unit is skeleton node two-dimensional coordinate position on depth image.

Normalization layer: normalization layer refers to concentrates the coordinate of artificial mark to be normalized operation to data.Train the CNN network of a detection personage, then use in normalization layer, by the target cutting in depth map out.So can reduce the interference of background, make the last precision to skeleton point detection improve.

2. the hot map generalization in joint

If given data set is { I_n,L_n, n=1 ..., N, N is the sum of data set sample.Wherein I_nRepresent n-th image, L_nRepresent the skeleton point that n-th image is corresponding, L_n={ l_k, k=1 ..., K, K represents total K the skeleton point being marked, and it is 19 that our model arranges K, refers to Fig. 3.l_k=(x_k,y_k), for the position of kth skeleton point.The thermal map assuming kth skeleton point is hp_k, l_kIt is mapped in hp_kOn coordinate lh_k=(xh_k,yh_k) it is expressed as follows:

x_k=stride × xh_k+offset(1.1)

y_k=stride × yh_k+offset(1.2)

Wherein, stride represents that step-length, offset represent side-play amount, extra setup oneDetermine the size of orange little rhombus.hp_kIn each value represent that this value is at I_nThe probability of middle kth skeleton point position, value is [0,1].The algorithm generating thermal map is as follows.

3. the training of model

An i.e. given pictures, the network proposed by the present invention obtains K thermal map of correspondence.It is assumed that this K thermal map lines up fixing order by human body, so can facilitate and compare with real joint thermal map and learn the prediction thermal map corresponding with real joint thermal map.For the size of normalization input picture, adopt the size first determining thermal map then the method calculating the suitable size of input picture here.The size s of thermal map_hp×s_hpBeing determined by experience, it is 50 × 50 that our model arranges it.The then size s of input picture_I×s_IDefine as follows:

s_I=(s_hp-1)×stride+offset×2+1(3.1)

To above formula, owing to, in real joint thermal map training data, the position of human body can be easy near image border, therefore plus the filling being sized to offset around input picture.K the real joint thermal map that input is image and its correspondence of our model, is output as the corresponding prediction thermal map to K skeleton point.Do so not only makes the complexity of model reduce (avoiding training a R CNN into each thermal map), it is also possible to allow the weights of each thermal map share.

Propagated forward:

The R CNN model that each two field picture in data set all defines along us is propagated computing, and ground floor is a normalization layer, different size of input picture can be normalized to unified size, facilitate process during subsequent propagation.Will pass through full convolutional network as shown in Figure 2 subsequently, output be sized to (batch_size, K, S_hp,S_hp), wherein batch_size is the training number of batch training.

Back propagation:

After propagated forward completes, it is thus achieved that export as above.Back propagation then needs the residual error J (ω) first obtaining between thermal map and the real joint thermal map of forward-propagating output, then tries to achieve its gradient for parameter ωAnd adopting the algorithm of stochastic gradient descent to update ω to minimize residual error, loss function J (ω) definition of residual error is as follows.

$\begin{array}{c}J\left(\mathrm{\ω}\right)=\frac{1}{batch\_size}\underset{s=1}{\overset{bacth\_size}{\Σ}}\underset{k=1}{\overset{K}{\Σ}}\left|\right|Y_k^{pred}\left(\mathrm{\ω}\right)-Y_k^{gt}|{|}_F\\ =\frac{1}{batch\_size}\underset{s=1}{\overset{bacth\_size}{\Σ}}\underset{k=1}{\overset{K}{\Σ}}\sqrt{\underset{i=1}{\overset{S_{hp}}{\Σ}}\underset{j=1}{\overset{S_{hp}}{\Σ}}|Y_k^{pred}\left(\mathrm{\ω}\right)\[i,j\]-Y_k^{gt}\[i,j\]{|}^2}\end{array}---\left(3.2\right)$

Wherein | | | |_FFor not Luo Beini this norm of crow, Y^predFor the thermal map of prediction, Y^gtFor real joint thermal map

But the final effect only such error back propagation obtained bad, reason is in that the region of background is far longer than the region of prospect.Therefore we increase a scale factor, namely in certain ratio, the residual error of background are set to 0 when back propagation, it is possible to the ratio making foreground and background is close.Such as, thermal map is sized to 50 × 50, and the little diamond-shaped area of the inside is 5 × 5, then the ratio of foreground and background is then 1:100.Our model arranges the proportional factor r atio that value is 0.012 and acts in background, and when back propagation, the ratio regular meeting of foreground and background is become 1:1.2 from 1:100.

The learning process of model is summarized as algorithm 2:

4. the test of model

A given test image, in the model that input trains, can obtain the thermal map of 19 skeleton points.To each thermal map, find out its maximum response value, be a skeleton point of human body.Finally by formula (1.1) and (1.2), this coordinate transform is returned under original image, the coordinate of 19 human body skeleton points can be obtained.Evaluating standard is as follows:

$\frac{\left|\right|pred\_{\mathrm{coord}}_j^i-gt\_{\mathrm{coord}}_j^i\left|\right|}{\left|\right|pred\_{\mathrm{coord}}_{ls}^i-gt\_{\mathrm{coord}}_{rh}^i\left|\right|}<r\%---\left(4.1\right)$

Wherein, pred_coord is the coordinate predicted, and gt_coord is true coordinate, under be designated as the index of skeleton point.What in denominator, subscript ls (leftshoulder) represented is left shoulder, and what rh (righthip) represented is RG, and namely what whole denominator represented is the length of human posture's trunk.This evaluation and test truly imply that prediction coordinate and true coordinate between distance should less than in this image human posture certain ratio of true torso length, taking r in our model is 20.

Above-described embodiment is the present invention preferably embodiment; but embodiments of the present invention are also not restricted to the described embodiments; the change made under other any spirit without departing from the present invention and principle, modification, replacement, combination, simplification; all should be the substitute mode of equivalence, be included within protection scope of the present invention.

Claims (9)

1. the depth image human synovial localization method based on convolutional neural networks, it is characterised in that include training process and identification process；

The step of training process is as follows:

1) input training sample；

2) initializing profound convolutional neural networks and parameter thereof, described parameter includes weight and the biasing on every layer of limit；

3) adopt forwards algorithms and backward algorithm, utilize training sample to learn the parameter of convolutional neural networks of structure；

The step of identification process is as follows:

4) input test sample；

5) utilize the convolutional neural networks the trained test sample to input, return out the position of human joint points therein.

2. the depth image human synovial localization method based on convolutional neural networks according to claim 1, it is characterized in that, described step 1) in training sample be the set of the original depth image comprising personage caught of the depth camera of free angle and mark thereof.

3. the depth image human synovial localization method based on convolutional neural networks according to claim 1, it is characterized in that, described step 2) in convolutional neural networks have profound level structure, and by convolutional layer, down-sampled layer, correct linear elementary layer, full articulamentum is stacking forms, and it being normalized operation by normalization layer, the top layer of this convolutional neural networks directly exports the position of human joint points.

4. the depth image human synovial localization method based on convolutional neural networks according to claim 3, it is characterised in that described convolutional layer refers to that the image to input or feature carry out convolution at two-dimensional space, extracts important feature；

Described down-sampled layer uses max-pooling operation, and this operation refers to and according to setting strategy, characteristic pattern is carried out down-sampled process, for extracting the feature keeping shape and skew invariance；

The linear elementary layer of described rectification adopts simple non-linear threshold function, and input only allows the conversion that non-negative signal passes through；

In described full articulamentum, first the characteristic pattern obtained from M sub-network is connected into a long characteristic vector, namely this long characteristic vector represents the feature being drawn into from range image sequence, its every one-dimensional element all connects all nodes to first full articulamentum, and entirely it is connected to all of output unit further, output unit 2K altogether, K represents the number of skeleton node here, and namely the value of output unit is skeleton node two-dimensional coordinate position on depth image；

Described normalization layer is that data are concentrated the coordinate of artificial mark be normalized operation.

5. the depth image human synovial localization method based on convolutional neural networks according to claim 1, it is characterised in that step 3) in, also include the generation step of joint thermal map, particularly as follows:

If given data set is { I_n,L_n, n=1 ..., N, N is the sum of data set sample, wherein I_nRepresent n-th image, L_nRepresent the skeleton point that n-th image is corresponding, L_n={ l_k, k=1 ..., K, K represents total K the skeleton point being marked；l_k=(x_k,y_k), for the position of kth skeleton point, it is assumed that the thermal map of kth skeleton point is hp_k, l_kIt is mapped in hp_kOn coordinate lh_k=(xh_k,yh_k) it is expressed as follows:

x_k=stride × xh_k+offset(1.1)

y_k=stride × yh_k+offset(1.2)

Wherein, stride represents that step-length, offset represent side-play amount, and one l of extra setup determines the size of orange little rhombus, hp_kIn each value represent that this value is at I_nThe probability of middle kth skeleton point position, value is [0,1].

6. the depth image human synovial localization method based on convolutional neural networks according to claim 5, it is characterized in that, step 3) in, suppose that K thermal map lines up fixing order by human body when training, the prediction thermal map corresponding with real joint thermal map for being compared by joint thermal map and learning, for the size of normalization input picture, first determine the size of thermal map, then calculate the size that input picture is suitable；The size s of thermal map_hp×s_hpDetermined by experience, then the size s of input picture_I×s_IDefine as follows:

s_I=(s_hp-1)×stride+offset×2+1(3.1)

To above formula, due in real joint thermal map training data, the position of human body can be easy near image border, therefore plus the filling being sized to offset around input picture, K the real joint thermal map that input is image and its correspondence of this model, is output as the corresponding prediction thermal map to K skeleton point.

7. the depth image human synovial localization method based on convolutional neural networks according to claim 6, it is characterised in that forward algorithm particularly as follows:

Each two field picture in data set is all propagated computing along the R CNN model defined, ground floor is a normalization layer, different size of input picture can be normalized to unified size, facilitate process during subsequent propagation, will pass through full convolutional network subsequently, what export is sized to (batch_size, K, S_hp,S_hp), wherein batch_size is the training number of batch training；

Backward algorithm particularly as follows:

Back propagation then needs the residual error J (ω) first obtaining between thermal map and the real joint thermal map of forward-propagating output, then tries to achieve its gradient for parameter ωAnd adopting the algorithm of stochastic gradient descent to update ω to minimize residual error, loss function J (ω) definition of residual error is as follows；

$\begin{array}{c}J\left(\mathrm{\&omega;}\right)=\frac{1}{batch\_size}\underset{s=1}{\overset{batch\_size}{\&Sigma;}}\underset{k=1}{\overset{K}{\&Sigma;}}\left|\right|Y_k^{pred}\left(\mathrm{\&omega;}\right)-Y_k^{gt}|{|}_F\\ =\frac{1}{batch\_size}\underset{s=1}{\overset{batch\_size}{\&Sigma;}}\underset{k=1}{\overset{K}{\&Sigma;}}\sqrt{\underset{i=1}{\overset{S_{hp}}{\&Sigma;}}\underset{j=1}{\overset{S_{hp}}{\&Sigma;}}|Y_k^{pred}\left(\mathrm{\&omega;}\right)\&lsqb;i,j\&rsqb;-Y_k^{gt}\&lsqb;i,j\&rsqb;{|}^2}\end{array}---\left(3.2\right)$

Wherein | | | |_FFor not Luo Beini this norm of crow, Y^predFor the thermal map of prediction, Y^gtFor real joint thermal map.

8. the depth image human synovial localization method based on convolutional neural networks according to claim 1, it is characterised in that described step 4) in, test the original depth image that the depth camera that sample is free angle is caught.

9. the depth image human synovial localization method based on convolutional neural networks according to claim 5, it is characterised in that step 5) in, return out the position of human joint points therein method particularly includes:

A given test image, in the model that input trains, the thermal map of skeleton point can be obtained, to each thermal map, find out its maximum response value, be a skeleton point of human body, finally by formula (1.1) and (1.2), returning under original image by this coordinate transform, can obtain the coordinate of body skeleton point, evaluating standard is as follows:

$\frac{\left|\right|pred\_{\mathrm{coord}}_j^i-gt\_{\mathrm{coord}}_j^i\left|\right|}{\left|\right|pred\_{\mathrm{coord}}_{ls}^i-gt\_{\mathrm{coord}}_{rh}^i\left|\right|}<r\%---\left(4.1\right)$

Wherein, pred_coord is the coordinate predicted, and gt_coord is true coordinate, under be designated as the index of skeleton point, what in denominator, subscript ls represented is left shoulder, and what rh represented is RG, and namely what whole denominator represented is the length of human posture's trunk.