CN105912985A - Human skeleton joint point behavior motion expression method based on energy function - Google Patents

Abstract

The invention relates to a human skeleton joint point behavior motion expression method based on the energy function. According to the method, firstly, the position information of human skeleton joint points is acquired through the video equipment; secondly, the kinetic energy and the potential energy of each human skeleton joint point and the interaction potential energy information of a figure are calculated, and human motion characteristics are expressed quantitatively; human behavior video sequence division is carried out through adjacent frame merging, adjacent frame merging is carried out on the basis of energy similarity of two adjacent frames, energy values of the two adjacent frames are calculated through human motion characteristics for comparison, when the similarity is smaller than the similarity threshold, the two frames of pictures belong to one division group, or the two frames of pictures belong to different motion division segments, and thereby multiple integral-motion-meaning sub motion video sequences are acquired. Through the method, video division accuracy and reliability can be greatly improved, and the method can be further applied to multiple aspects such as human motion identification and key frame extraction.

Description

The behavior act method for expressing of human skeleton articulare based on energy function

Technical field

The present invention relates to a kind of Image Information Processing technology, particularly to a kind of human bone based on energy function The behavior act method for expressing of frame articulare.

Background technology

Along with being widely used of video equipment and 3D camera, the identification of human body behavior act also becomes computer and regards One important subject in feel field.Rightly represent human body behavior act be one thorny and important Work, this has special meaning for monitoring, intelligent robot, man-machine interaction etc., it be accurately identify with Understand basis and the committed step of human body behavior act.At present, behavior representation method master based on human skeleton Have: use the features such as the skeleton framework of band joint angle, single articulare movement locus, speed to be used for representing The information such as the position of human action, motion and track, but these method for expressing are difficult to explain intuitively, and And different people to do same action be also discrepant, therefore articulare movement locus is not quite similar.

Through finding the retrieval of existing document, Ofli et al. delivers at J.Vis.Commun.Image R Sequence of the most informative joints(SMIJ):A new representation for The article of human skeletal action recognition proposes human body perform same action time, The use number of articulare is consistent with order, and uses entropy to describe the information of human action quantitatively, But using this action method for expressing is defective at some specific action, if human body is at time rotational, by Varying less in joint angle, therefore the varying less of entropy, be mistaken for resting state.And from the sense of people Property understanding on, such method for expressing is directly perceived not, the most bad division being directly used in video sequence.

Summary of the invention

The present invention be directed to present action method for expressing some specific action exist fail to judge problem, propose A kind of behavior act method for expressing of human skeleton articulare based on energy function, simply, fixed intuitively Scale is leted others have a look at body action, and is advantageously applied to the division of the action video sequence of people.Not only establish human body The kinetic energy of skeleton joint point, potential energy, and have also contemplated that the reciprocal action of people and thing, use an energy letter Number represents human body behavior act, compensate for the defect using entropy to represent quantity of information size, can describe intuitively The size of the form of the action of people, severe degree and quantity of information.

The technical scheme is that the behavior act table of a kind of human skeleton articulare based on energy function Show method, first pass through video equipment and obtain the positional information of human skeleton articulare；Then human bone is calculated The mutual potential energy information of kinetic energy, potential energy and the personage of each articulare of frame, represents the motion characteristic of people quantitatively； The sequence being merged into pedestrian's body behavior video by consecutive frame is divided, and consecutive frame merges the energy with adjacent two frames Similarity divides, and the energy value being calculated adjacent two frames by the motion characteristic of people is compared, and similarity is little When similarity threshold, this two frames picture belongs to same group of division, otherwise, then the action belonging to different divides Fragment, the final sub-action video sequence obtaining complete action meaning one by one.

The calculating of the kinetic energy of each articulare of described human skeleton: Kincet or Asus Xtion Pro Live is utilized to obtain human skeleton and the position of human skeleton articulation nodes Three-dimensional coordinate, P_i,t(x_i,t,y_i,t,z_i,t) be video equipment gather t human skeleton i-th articulare position, (x_i,t,y_i,t,z_i,t) it is three-dimensional coordinate, the unit of coordinate figure is rice, and by two frame articulare position P_i,tDifference or The mean value computation of Multi Frame Difference goes out movement velocity v of each node_i,t, then it is calculated as follows the dynamic of each articulare Can sum E_K,t, k_iFor the kinetic energy weights of corresponding articulare, F_tIt it is the collection of t a certain frame picture intrinsic articulation point Close,

Wherein: E_K,tRepresent the kinetic energy sum of the whole articulare of human body, v in t frame fragment_i,tRepresent at t The speed of i-th node in frame fragment, Δ t represents the time interval between adjacent two frames, and the unit of coordinate figure is Rice,Representing in the set of the whole node of human body, lower target i is expressed as i-th node, i=1 ..., N, n represent articulare number, and K represents the articulare coefficient of correspondence.

The potential energy of each articulare of described human skeleton calculates:

Wherein P_i,0(x_i,0,y_i,0,z_i,0) represent human skeleton i-th articulare initial position, l_iGesture for corresponding articulare Energy weights, L represents the parameter of vector, is set as 1.

The mutual potential energy of described personage calculates:

$E_{OP,t}=M^T{(P_{j,t}-P_{j,t}^O)}^{-2}$

WhereinThe position that t exists the jth object of interactive relation with human body is gathered for video equipment, P_j,tRepresenting that video equipment gathers t gravity center of human body position, M represents the parameter of vector, is set as 1.

The determination formula of described similarity threshold:

$Sim(t_1,t_2)=\frac{(E_{AP,t1}+E_{OP,t1}+E_{K,t1})-(E_{AP,t2}+E_{OP,t2}+E_{K,t2})}{E_{K,t1}+E_{AP,t1}+E_{OP,t1}}.$

The beneficial effects of the present invention is: the behavior of present invention human skeleton based on energy function articulare is moved Make method for expressing, not only establish the kinetic energy of human skeleton articulare, potential energy, and have also contemplated that people and thing Reciprocal action, use an energy function to represent human body behavior act, compensate for use entropy represent quantity of information The defect of size, can make to describe the size of the form of the action of people, severe degree and quantity of information intuitively Which solve the quantificational expression problem of human action, apply it to the sub-action of video sequence and divide, Ke Yi great Amplitude improves accuracy and the reliability that video divides, and is equally applicable to identification and the key of human action simultaneously The many aspects such as frame extraction.

Accompanying drawing explanation

Fig. 1 is that human skeleton of the present invention is by connecting rod and articulare composition diagram；

Fig. 2 is behavior act method for expressing flow chart of the present invention；

Fig. 3 is k of the present invention_iThe kinetic energy smoothed curve of=10 long actions and the running point detected and key frame figure；

Fig. 4 is that RGB image of the present invention display makes oatmeal behavior schematic diagram；

Fig. 5 is the results contrast figure that the present invention makes oatmeal behavior many seeds action division methods；

Fig. 6 is that the present invention drinks water the energy ordering comparison diagram of action difference articulare；

Fig. 7 is the energy ordering comparison diagram that the present invention wears contact lens action difference articulare.

Detailed description of the invention

This method first passes through video equipment and obtains the positional information of human skeleton articulare, then calculates human body The reciprocal action potential energy information of kinetic energy, potential energy and the personage of each articulare of skeleton, represents that people's is dynamic quantitatively Make feature.Finally this method for expressing is applied in the sub-action division of long video, thus obtains one by one The sub-action video sequence of complete action meaning.

The present embodiment we study Cambridge Microsoft Research initially with above-mentioned method for Microsoft The simple motion of Cambridge-12 (MSRC-12) action data collection is tested experiment, then to Connell Compound action in Dataset-1200 (CAD-120) data set of university, demonstrates our dividing method pair The effectiveness of compound action identification.

The present embodiment comprises the steps:

The first step, obtains the skeleton of human body from Kinect SDK software, and skeleton is to be connected by 19 to close successively The line segment of point and 20 articulation nodes composition, such as hands, cervical region, trunk, left shoulder, left hand elbow, the left palm, Right shoulders etc., human skeleton is by connecting rod and articulare composition diagram as shown in Figure 1.Thus obtain each moment Human skeleton articulation nodes position three-dimensional coordinate (x, y, z) (as shown in Figure 1), and by two frame differences calculate Go out the movement velocity of each node, be then calculated as follows the kinetic energy sum of each articulare:

Wherein: E_K,tRepresent the kinetic energy sum of the whole articulare of human body in t frame fragment,

v_i,tRepresenting the speed of i-th node in t frame fragment, Δ t represented between the time between adjacent two frames Every, F_tRepresent the set of t a certain frame picture intrinsic articulation point,

x_i,t、y_i,t,、z_i,tRepresent the D coordinates value of this moment t each articulare, the list of these coordinate figures respectively Position is rice,

Represent in the set of the whole node of human body, lower target i be expressed as i-th node (i=1 ..., N), n represents articulare number, is 20 here, K represent correspondence articulare coefficient (each articulare be Number is all specific).

Second step, utilize Kincet software or ASUS Xtion Pro Live software obtain human skeleton and The position three-dimensional coordinate of human skeleton articulation nodes (x, y, z) information, and calculate current joint point position and just Hand gesture location deviation E between beginning articulare position_Apt,The i.e. potential energy of human body；Object is positioned human body simultaneously In the range of three dimensions around skeleton, then use object detector to obtain in a group RGB image and visited Object and human body are at the position relationship in each moment, according to formula E_OptRepresent the mutual of human body and object Potential energy.

Described energy equation contains the mutual potential energy of the potential energy of human body and human body and object.In order to Being easy to observe the uniformity of adjacent segmentation point, we create two energy equations and represent human body respectively and connect Touch the potential energy of object, as follows:

$E_{OP,t}=M^T{(P_{j,t}-P_{j,t}^O)}^{-2}$

Wherein: E_AptAnd E_OptRepresent potential energy and the mutual potential energy of personage, the P of t human body respectively_i,0(x_i,0,y_i,0, z_i,0) represent human skeleton i-th articulare initial position.It can be regarded as the nature of a human body by us Posture, as accompanying drawing 1 shows the posture that human body natural stands.T and human body is gathered for video equipment There is the position of the jth object of interactive relation, P in position_j,tRepresent that video equipment gathers t gravity center of human body institute In position, the parameter that L and M is vectorial in representing two formula respectively, l_iPotential energy power for corresponding human joint points Value.E_AptRepresent the hand gesture location deviation between t current joint point position and initial articulare position, E_Opt Represent the mutual potential energy of t human body and object.In test process, when we continue the maximum of merging group Between be limited in 20 seconds within.K=[10 ..., 10]^T, and all L, the element of M vector parameter is disposed as 1。

3rd step, more complicated carries more identification information in order to sub-action similar for energy is collectively incorporated into In complete action sequence, the sequence that we are merged into pedestrian's body behavior video by consecutive frame divides, such as Fig. 2 Shown behavior act method for expressing flow chart.The energy similarity measure of adjacent two frames defines such as formula: when this similarity During less than similarity threshold, (we are the limit of maximum persistent period of the video combination after merging in this embodiment System is within 20 seconds, and similarity threshold is set to 0.85), time i.e. less than this similarity threshold, this two frames picture Belong to same group of division, otherwise, then the action belonging to different divides fragment.It is thus determined that similarity threshold is i.e. It it is the key of action division.Through experiment, we can obtain the determination formula of similarity threshold:

$Sim(t_1,t_2)=\frac{(E_{AP,t1}+E_{OP,t1}+E_{K,t1})-(E_{AP,t2}+E_{OP,t2}+E_{K,t2})}{E_{K,t1}+E_{AP,t1}+E_{OP,t1}}$

Fig. 3 is k_iThe kinetic energy smoothed curve of=10 long actions and the running point detected and key frame figure, Fig. 3 refers to Go out the scope that can merge.In an embodiment, kinetic energy and potential energy performance kinestate are used.In order to obtain Cut-point and key frame, we find Local Minimum and maximum kinetic energy point (2 seconds in given time interval By 6 seconds).These Local Minimum kinetic energy points are exactly the cut-point of sub-piece, and local maxima energy range is used to The similarity utilized when relatively merging adjacent segment.In test process, we are the video group after merging Within the conjunction maximum persistent period is limited in 20 seconds.Parameter be set to K=[10 ..., 10]^T, and all L, M The element of vector parameter is disposed as 1, and similarity threshold is set to 0.85.Table 1 shows Microsoft The Video segmentation precision (meansigma methods) of Research Cambridge-12 (MSRC-12) action data collection, experiment Result shows that our method is that effect is fine.

Table 1

This RGB image display making this behavior of oatmeal contains four sub-actions as can be seen from Figure 4: mobile, Arriving, topple over, place, the results contrast of Fig. 5 many seeds action division methods, partial results derives from ginseng Examine document (H.Koppula, R.Gupta, and A.Saxena, Learning human activities and object affordances from RGB-D videos,The International Journal of Robotics Research, 32 (8): 951 970,2013) Fig. 6,7 be to drink water and wear contact lens The energy ordering comparison diagram of two action difference articulares.Fig. 6, Fig. 7 contrast can find different actions, people The Energy distribution of body articulare is different.

Claims (5)

1. the behavior act method for expressing of a human skeleton articulare based on energy function, it is characterised in that first First pass through video equipment and obtain the positional information of human skeleton articulare；Then each joint of human skeleton is calculated The mutual potential energy information of kinetic energy, potential energy and the personage of point, represents the motion characteristic of people quantitatively；By adjacent Frame is merged into the sequence of pedestrian's body behavior video and divides, and consecutive frame merges to come with the energy similarity measure of adjacent two frames Dividing, the energy value being calculated adjacent two frames by the motion characteristic of people is compared, and similarity is less than similarity During threshold value, this two frames picture belongs to same group of division, otherwise, then the action belonging to different divides fragment, Obtain the sub-action video sequence of complete action meaning one by one eventually.

The behavior act method for expressing of human skeleton articulare based on energy function the most according to claim 1, It is characterized in that, the calculating of the kinetic energy of each articulare of described human skeleton:

Kincet or Asus Xtion Pro Live is utilized to obtain human skeleton and the position of human skeleton articulation nodes Three-dimensional coordinate, P_i,t(x_i,t,y_i,t,z_i,t) be video equipment gather t human skeleton i-th articulare position, (x_i,t,y_i,t,z_i,t) it is three-dimensional coordinate, the unit of coordinate figure is rice, and by two frame articulare position P_i,tDifference or The mean value computation of Multi Frame Difference goes out movement velocity v of each node_i,t, then it is calculated as follows the dynamic of each articulare Can sum E_K,t, k_iFor the kinetic energy weights of corresponding articulare, F_tIt it is the collection of t a certain frame picture intrinsic articulation point Close,

Wherein: E_K,tRepresent the kinetic energy sum of the whole articulare of human body, v in t frame fragment_i,tRepresent at t The speed of i-th node in frame fragment, Δ t represents the time interval between adjacent two frames, and the unit of coordinate figure is Rice,Representing in the set of the whole node of human body, lower target i is expressed as i-th node, i=1 ..., N, n represent articulare number, and K represents the articulare coefficient of correspondence.

The behavior act method for expressing of human skeleton articulare based on energy function the most according to claim 2, It is characterized in that, the potential energy of each articulare of described human skeleton calculates:

Wherein P_i,0(x_i,0,y_i,0,z_i,0) represent human skeleton i-th articulare initial position, l_iGesture for corresponding articulare Energy weights, L represents the parameter of vector, is set as 1.

The behavior act method for expressing of human skeleton articulare based on energy function the most according to claim 3, It is characterized in that, the mutual potential energy of described personage calculates:

$E_{OP,t}=M^T{(P_{j,t}-P_{j,t}^O)}^{-2}$

WhereinThe position that t exists the jth object of interactive relation with human body is gathered for video equipment, P_j,tRepresenting that video equipment gathers t gravity center of human body position, M represents the parameter of vector, is set as 1.

The behavior act method for expressing of human skeleton articulare based on energy function the most according to claim 4, It is characterized in that, the determination formula of described similarity threshold:

$Sim(t_1,t_2)=\frac{(E_{AP,t1}+E_{OP,t1}+E_{K,t1})-(E_{AP,t2}+E_{OP,t2}+E_{K,t2})}{E_{K,t1}+E_{AP,t1}+E_{OP,t1}}.$