CN108304790A - Skeleton motion prediction processing method, device and limb motion prediction processing method - Google Patents

Skeleton motion prediction processing method, device and limb motion prediction processing method Download PDF

Info

Publication number
CN108304790A
CN108304790A CN201810055213.2A CN201810055213A CN108304790A CN 108304790 A CN108304790 A CN 108304790A CN 201810055213 A CN201810055213 A CN 201810055213A CN 108304790 A CN108304790 A CN 108304790A
Authority
CN
China
Prior art keywords
skeleton motion
vector
state vector
skeleton
motion state
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810055213.2A
Other languages
Chinese (zh)
Other versions
CN108304790B (en
Inventor
唐永毅
马林
刘威
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201810055213.2A priority Critical patent/CN108304790B/en
Priority to CN202210143192.6A priority patent/CN114495283A/en
Publication of CN108304790A publication Critical patent/CN108304790A/en
Application granted granted Critical
Publication of CN108304790B publication Critical patent/CN108304790B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

The present invention relates to a kind of skeleton motion prediction processing method, device and limb motion prediction processing method, which includes：Multiple continuous skeleton motion state vectors of history observation are inputted in the machine learning model of pre-training, corresponding skeleton motion feature vector is obtained to carry out feature coding respectively；Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion hidden state vector of the last moment is to carry out motion intention to the skeleton motion feature vector for the last moment to extract to obtain；Obtain the skeleton motion state vector at current time；It is decoded according to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of the last moment, to calculate the skeleton motion state vector of subsequent time.The scheme of the application realizes the motion prediction process for predicting this details to the skeleton motion of target object itself.

Description

Skeleton motion prediction processing method, device and limb motion prediction processing method
Technical field
The present invention relates to field of computer technology, more particularly to a kind of skeleton motion prediction processing method, device and limb Body motion prediction process method.
Background technology
With the rapid development of science and technology, the technology in various relatively forward positions is increasingly paid attention to by everybody.For example, fortune Dynamic prediction, refers to the motion prediction next issuable movement by having generated.
However, traditional motion estimation technique is not mature enough, an Orientation observation is chosen usually from target object Point (for example, central point of target object), is analyzed by the movement of the position to the Orientation observation point, to predict target pair As next position that may be moved to.Obviously, traditional motion estimation technique can only be by target object as a whole into line position The prediction of variation is set, and cannot achieve and the movement details of target object itself is predicted.
Invention content
Based on this, it is necessary to cannot achieve for conventional method and ask what the movement details of target object itself was predicted Topic provides a kind of skeleton motion prediction processing method, device, computer equipment and storage medium, additionally provides a kind of limbs fortune Dynamic prediction processing method, device, computer equipment and storage medium.
A kind of skeleton motion prediction processing method, the method includes：
Obtain the skeleton motion state vector of multiple continuous history；
Condition code coding is carried out to each skeleton motion state vector respectively by machine learning model, generate respectively with The corresponding skeleton motion feature vector of each skeleton motion state vector；
Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion of the last moment is hidden It is to carry out motion intention to the skeleton motion feature vector for the last moment to extract to obtain containing state vector；
Obtain the skeleton motion state vector at current time；
According to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of the last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time.
A kind of skeleton motion prediction processing device, described device include：
Coding module, the skeleton motion state vector for obtaining multiple continuous history；Pass through machine learning model point It is other that condition code coding is carried out to each skeleton motion state vector, it generates corresponding with each skeleton motion state vector respectively Skeleton motion feature vector；
Hidden state vector determining module, for determine current time last moment skeleton motion hidden state to Amount；The skeleton motion hidden state vector of the last moment, be for the last moment to the skeleton motion feature to Amount carries out motion intention and extracts to obtain；
Skeleton motion state vector acquisition module, the skeleton motion state vector for obtaining current time；
Prediction module is decoded, the bone at the skeleton motion hidden state vector sum current time according to the last moment is used for Bone motion state vector is decoded, to calculate the skeleton motion state vector of subsequent time.
A kind of computer equipment, including memory and processor are stored with computer program, the meter in the memory When calculation machine program is executed by processor so that the processor executes following steps：
Obtain the skeleton motion state vector of multiple continuous history；
Condition code coding is carried out to each skeleton motion state vector respectively by machine learning model, generate respectively with The corresponding skeleton motion feature vector of each skeleton motion state vector；
Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion of the last moment is hidden It is to carry out motion intention to the skeleton motion feature vector for the last moment to extract to obtain containing state vector；
Obtain the skeleton motion state vector at current time；
According to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of the last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time.
A kind of storage medium being stored with computer program, when the computer program is executed by processor so that processing Device executes following steps：
Obtain the skeleton motion state vector of multiple continuous history；
Condition code coding is carried out to each skeleton motion state vector respectively by machine learning model, generate respectively with The corresponding skeleton motion feature vector of each skeleton motion state vector；
Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion of the last moment is hidden It is to carry out motion intention to the skeleton motion feature vector for the last moment to extract to obtain containing state vector；
Obtain the skeleton motion state vector at current time；
According to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of the last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time.
Above-mentioned skeleton motion prediction processing method, device, computer equipment and storage medium, to multiple continuous history Skeleton motion state vector carries out feature coding and obtains corresponding skeleton motion feature vector respectively, realizes to each skeleton motion The skeleton motion feature extraction of state vector.Determine the skeleton motion hidden state vector of the last moment at current time；The bone Bone moves hidden state vector, is extracted by carrying out motion intention to obtained skeleton motion feature vector for last moment It arrives, realizes the extraction for carrying out motion intention according to the skeleton motion feature of extraction.It is implicit according to the skeleton motion of last moment State vector and the skeleton motion state vector at current time are decoded, with calculate the skeleton motion state of subsequent time to Amount.It is decoded by the motion intention and current skeleton motion state vector of extraction, realizes the intention letter to extraction The excavation of breath, to calculate the skeleton motion state vector of subsequent time based on decoding excavation intent information.It realizes to mesh Mark the motion prediction process of this details of the prediction of the skeleton motion of object itself.
A kind of limb motion prediction processing method, the method includes：
Obtain the limb motion state of multiple continuous history；
Motion feature extraction is carried out respectively to the limb motion state of each history, obtains the limb with each history The corresponding limb motion feature of body motion state；
Determine the limb motion hidden state of the last moment at current time；The limb motion of institute's last moment implies shape State is to carry out motion intention to the limb motion feature for the last moment to extract to obtain；
Obtain the limb motion state at current time；
According to the limb motion state of the limb motion hidden state of the last moment and current time, lower a period of time is determined The limb motion state at quarter.
A kind of limb motion prediction processing device, which is characterized in that described device includes：
Motion feature extraction module, the limb motion state for obtaining multiple continuous history；To the institute of each history It states limb motion state and carries out Motion feature extraction respectively, obtain limbs fortune corresponding with the limb motion state of each history Dynamic feature；
Hidden state determining module, the limb motion hidden state of the last moment for determining current time；On described The limb motion hidden state at one moment is to carry out motion intention extraction to the limb motion feature for the last moment It obtains；
Limb motion state determining module, the limb motion state for obtaining current time；According to the last moment Limb motion hidden state and current time limb motion state, determine the limb motion state of subsequent time.
A kind of computer equipment, including memory and processor are stored with computer program, the meter in the memory When calculation machine program is executed by processor so that the processor executes following steps：
Obtain the limb motion state of multiple continuous history；
Motion feature extraction is carried out respectively to the limb motion state of each history, obtains the limb with each history The corresponding limb motion feature of body motion state；
Determine the limb motion hidden state of the last moment at current time；The limb motion of the last moment implies shape State is to carry out motion intention to the limb motion feature for the last moment to extract to obtain；
Obtain the limb motion state at current time；
According to the limb motion state of the limb motion hidden state of the last moment and current time, lower a period of time is determined The limb motion state at quarter.
A kind of storage medium being stored with computer program, when the computer program is executed by processor so that processing Device executes following steps：
Obtain the limb motion state of multiple continuous history；
Motion feature extraction is carried out respectively to the limb motion state of each history, obtains the limb with each history The corresponding limb motion feature of body motion state；
Determine the limb motion hidden state of the last moment at current time；The limb motion of the last moment implies shape State is to carry out motion intention to the limb motion feature for the last moment to extract to obtain；
Obtain the limb motion state at current time；
According to the limb motion state of the limb motion hidden state of the last moment and current time, lower a period of time is determined The limb motion state at quarter.
Above-mentioned limb motion prediction processing method, device, computer equipment and storage medium, to multiple continuous history Limb motion state carries out Motion feature extraction and obtains corresponding limb motion feature respectively.Determine the last moment at current time Limb motion hidden state；The limb motion hidden state, by being carried out to obtained limb motion feature for last moment Motion intention is extracted to obtain, and realizes the extraction that motion intention is carried out according to the limb motion feature of extraction.According to last moment Limb motion hidden state and the limb motion state at current time be decoded, to calculate the limb motion shape of subsequent time State.It is decoded by the motion intention of extraction and the limb motion state at current time, realizes the intention letter to extraction The excavation of breath, to calculate the limb motion state of subsequent time based on decoding excavation intent information.It realizes to target pair As the motion prediction process of this more details of the prediction of the limb motion of itself.
Description of the drawings
Fig. 1 is the flow diagram of skeleton motion prediction processing method in one embodiment；
Fig. 2 is the effect diagram of skeleton motion prediction processing method in one embodiment；
Fig. 3 is the processing block schematic illustration of skeleton motion prediction processing method in one embodiment；
Fig. 4 is the principle framework schematic diagram of skeleton motion prediction processing method in one embodiment；
Fig. 5 is the processing procedure schematic diagram for the skeleton motion state vector that subsequent time is calculated in one embodiment；
Fig. 6 is the schematic diagram by behavior label control targe object in one embodiment；
Fig. 7 is the applied environment figure of skeleton motion prediction processing in one embodiment；
Fig. 8 is the flow diagram of skeleton motion prediction processing method in another embodiment；
Fig. 9 is the flow diagram of limb motion prediction processing method in one embodiment；
Figure 10 is the block diagram of skeleton motion prediction processing device in one embodiment；
Figure 11 is the block diagram of skeleton motion prediction processing device in another embodiment；
Figure 12 is the block diagram of limb motion prediction processing device in one embodiment；
Figure 13 is the internal structure schematic diagram of one embodiment Computer equipment.
Specific implementation mode
In order to make the purpose , technical scheme and advantage of the present invention be clearer, with reference to the accompanying drawings and embodiments, right The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and It is not used in the restriction present invention.
Fig. 1 is the flow diagram of skeleton motion prediction processing method in one embodiment.The present embodiment is mainly with the bone Bone motion prediction process method is illustrated applied to computer equipment.Referring to Fig.1, this method specifically comprises the following steps：
S102 obtains the skeleton motion state vector of multiple continuous history.
Wherein, skeleton motion state vector is that the vector of skeleton motion state indicates, i.e., describes bone by the form of vector Bone motion state.The skeleton motion state vector of history, the skeleton motion state for indicating to have generated.It is multiple continuously to go through The skeleton motion state vector of history, for multiple skeleton motion states indicating history, having continuously generated.
In one embodiment, skeleton motion state vector can be by between expression adjacent target skeletal joint point It is rotationally-varying to describe the rotating vector of skeleton motion state.Wherein, target skeletal joint point is preassigned, is generating The skeletal joint point referred to when skeleton motion state vector.It is rotary shaft that rotating vector, which is finger direction, and size is rotation angle Vector.It is appreciated that each vector element is each target skeletal joint point relative to previous targeted bone in skeleton motion state vector The spin data of bone artis.
Specifically, computer equipment can directly acquire the skeleton motion state vector of multiple continuous history.Computer Equipment can also obtain the skeleton motion status image frame of multiple continuous acquisitions, respectively to the bone of multiple continuous acquisitions of acquisition Motion state image frame is parsed, and the skeleton motion state vector of multiple continuous history is obtained.
Implement at one, this method further includes：Obtain the skeleton motion status image frame of multiframe continuous acquisition；For every frame Skeleton motion status image frame identifies multiple target skeletal joint points from skeleton motion status image frame；According to multiple targets Tandem between skeletal joint point obtains latter object skeletal joint point relative to previous target skeletal joint point respectively Spin data；It is spelled by the tandem between respective objects skeletal joint point using each spin data as vector element It connects to obtain skeleton motion state vector corresponding with skeleton motion status image frame.
Wherein, skeleton motion status image frame is the image for the skeleton motion status information for including target object.Targeted bone Bone artis is skeletal joint point that is preassigned, being referred to when generating skeleton motion state vector.It needs to illustrate It is that the target skeletal joint point identified from every frame skeleton motion status image frame is identical.
It is appreciated that target skeletal joint point can be all or part of skeletal joint point of target object.Because of target The skeletal joint point quantity of object is very more, can be selected from whole skeletal joint points when generating skeleton motion state vector Go out bony segment artis as target skeletal joint point.
Specifically, computer equipment can be respectively directed to per frame skeleton motion status image frame, carry out skeleton motion state Analysis, obtains skeleton motion state vector corresponding with each frame skeleton motion status image frame respectively.Computer equipment can root According to the corresponding skeleton motion state vector of the continuous skeleton motion status image frame of multiframe, the bone of multiple continuous history is obtained Motion state vector.
In one embodiment, computer equipment can directly acquire between pre-set multiple target skeletal joint points Tandem, can also be in conjunction with the bone in target object of each target skeletal joint point in skeleton motion status image frame Information determines the tandem between multiple target skeletal joint points.
Computer equipment can obtain latter object bone respectively according to the tandem between multiple target skeletal joint points Spin data of the bone artis relative to previous target skeletal joint point.Wherein, spin data is to latter object skeletal joint The rotationally-varying data being described that point is generated compared to previous target skeletal joint point.In one embodiment, rotation number According to that can be rotating vector, for describing rotation angle and direction of rotation in vector form.
It is appreciated that since spin data is that latter object skeletal joint point is generated relative to previous target skeletal joint point It is rotationally-varying to obtain, so spin data is corresponding with the latter object skeletal joint point.Due to the first target skeletal joint The no previous target skeletal joint point of point, so the spin data of the first target skeletal joint point can be set as default initial values. In one embodiment, which could be provided as zero.
It should be noted that in the present embodiment, " latter " and " previous " is relativeness, such as, it is assumed that targeted bone bone closes Node A is adjacent to target skeletal joint point B and before B, then target skeletal joint point A is then target skeletal joint point B's Previous target skeletal joint point, target skeletal joint point B are then the latter object skeletal joint point of target skeletal joint point A.
Computer equipment can be using each spin data as vector element, by between respective objects skeletal joint point Tandem, splicing obtain skeleton motion state vector corresponding with skeleton motion status image frame.
The step of being generated to skeleton motion state vector of now illustrating illustrates.For example, target skeletal joint point is followed successively by A, B, C, D and E, A are the first targeted bone bone artis, then the spin data 1 corresponding to A is initial default value 0, computer equipment Spin data 2 (spin data 2 then correspond to B), C spin data 3 (rotations compared to B of the B compared to A can be obtained respectively Data 3 then correspond to C), spin datas 4 (spin data 4 then correspond to D) and E spin data compared to D of the D compared to C 5 (spin data 5 then corresponds to E).Computer equipment can be using each spin data of acquisition as vector element, according to corresponding Tandem between target skeletal joint point, splicing obtain skeleton motion state vector x (0, spin data 1, spin data 2, Spin data 3, spin data 4, spin data 5).
S104 carries out condition code coding to each skeleton motion state vector respectively by machine learning model, generates difference Skeleton motion feature vector corresponding with each skeleton motion state vector.
Wherein, machine learning model is the advance model for carrying out machine learning and training.Skeleton motion feature vector, It is that the feature vector that Motion feature extraction obtains is carried out to skeleton motion state vector.It is appreciated that the process of feature coding is To carry out the process of Motion feature extraction.
In one embodiment, it in machine learning model may include recurrent neural networks model (Recurrent Neural Networks, RNN).
The skeleton motion state vector of multiple continuous history of acquisition can be inputted the machine of pre-training by computer equipment In device learning model, to carry out feature coding to the skeleton motion state vector of each history, each skeleton motion state is obtained Vectorial corresponding skeleton motion feature vector.It is appreciated that there are one corresponding for the skeleton motion state vector of each history Historical juncture.Historical juncture is skeleton motion state represented by skeleton motion state vector of the computer equipment to history into At the time of row observation acquisition.
In one embodiment, multiple continuous skeleton motion state vectors that computer equipment can observe history are defeated Enter encoder in the recurrent neural networks model of pre-training and carry out feature coding, with obtain respectively with each skeleton motion shape The corresponding skeleton motion feature vector of state vector.It is appreciated that each corresponding skeleton motion state of skeleton motion feature vector to The historical juncture of amount is corresponding.
For example, multiple continuous skeleton motion state vectors of history observation are { x1, x2, x3..., xT ＇, each skeleton motion State vector corresponds respectively to the historical juncture 1 to T ＇, is encoded, can be obtained to each skeleton motion state vector by encoder To corresponding skeleton motion feature vector { e1, e2, e3..., eT ＇, when each skeleton motion feature vector also corresponds respectively to history Carve 1 to T ＇.
In one embodiment, computer equipment can encode to obtain the skeleton motion observed with history by following formula The corresponding skeleton motion feature vector of state vector：
et'=fe(xt')；(formula 1)
Wherein, t ＇ are the t ＇ historical junctures；xt'For the skeleton motion state vector of t ＇ historical junctures；et'It is The skeleton motion feature vector of t ＇ historical junctures is (i.e. to the skeleton motion state vector x of t ＇ historical juncturesT ＇It is encoded The skeleton motion feature vector obtained afterwards)；feIt is characterized coding function.
In one embodiment, step S104 includes：It, will according to the sequencing of the skeleton motion state vector of each history It is previous to encode obtained skeleton motion feature vector and when secondary skeleton motion state vector to be encoded, input the machine of pre-training Encoder in learning model is encoded, and output obtains compiling when secondary with when secondary skeleton motion state vector to be encoded is corresponding The skeleton motion feature vector that code obtains.
In one embodiment, computer equipment can be encoded to obtain according to following formula skeleton motion feature to Amount：
et'=Weφ(Ueet'-1+be)+Uxxt'+bx；
Wherein, et'To be transported to the bone obtained when time coding that the skeleton motion state vector of t ＇ historical junctures carries out Dynamic feature vector；et'-1The bone obtained for the previous coding of the skeleton motion state vector progress to ＇ -1 historical junctures of t Motion feature vector；xt'It it is the t ＇ historical junctures when skeleton motion state vector time to be encoded；We、Ue、be、UxAnd bx It is trained in advance parameter；φ indicates line rectification function.
S106 determines the skeleton motion hidden state vector of the last moment at current time.
Wherein, current time, at the time of being current progress skeleton motion status predication processing.
It is appreciated that at the time of current time is an opposite variation, i.e., bone is carried out to the subsequent time at current time After motion state vector prediction, then the subsequent time of skeleton motion state vector can will be predicted as when newly current It carves, continues to execute the prediction that step S106~S110 carries out skeleton motion state vector.First current time is in step S102 Multiple continuous skeleton motion state vectors in the last one skeleton motion state vector corresponding historical juncture, i.e., from last Skeleton motion status predication is carried out from one skeleton motion state vector corresponding historical juncture, to predict next lower a period of time The skeleton motion state vector at quarter.
For example, multiple continuous skeleton motion state vectors are { x1, x2, x3..., xT ＇, then first current time is T ＇, It can predict the subsequent time i.e. skeleton motion state vector of T ＇+1 of T ＇.The skeleton motion state vector of T ＇+1 is obtained in prediction Afterwards, by T ＇+1 as new current time, to predict the subsequent time i.e. skeleton motion state vector of T ＇+2 of T ＇+1, class successively It pushes away, constantly carries out the prediction processing of the skeleton motion state vector of future time instance.
Skeleton motion hidden state vector, is to carry out the vector that motion intention is extracted to skeleton motion feature vector, For integrating the skeleton motion state characteristic information of history to represent corresponding sports intention.Motion intention is desirable to carry out certain The plan of kind movement, for embodying the next desired movement carried out.
It is appreciated that since different moments corresponding skeleton motion state may be variant, so motion intention may have Institute's difference, then different moments have corresponding skeleton motion hidden state vector.The bone of the last moment at current time Hidden state vector is moved, is to carry out motion intention to skeleton motion feature vector for last moment to extract to obtain.
In one embodiment, skeleton motion hidden state vector is exported by the hidden layer in recurrent neural networks model.
In one embodiment, for (i.e. the last one corresponding history of skeleton motion state vector of first current time Moment is as current time), the skeleton motion hidden state vector of last moment can be default default value.Implement at one In example, skeleton motion hidden state vector, which presets default value, can be initialized as null vector.
It is appreciated that in other embodiments, for (i.e. the last one skeleton motion state vector pair of first current time The historical juncture answered is as current time), the skeleton motion hidden state vector of last moment can not also be default acquiescence Value.
S108 obtains the skeleton motion state vector at current time.
It is appreciated that first (the i.e. skeleton motion state vector corresponding historical juncture of the last one history at current time As current time) skeleton motion state vector, as history observation the last one skeleton motion state vector.Work as prediction When the corresponding moment of calculated skeleton motion state vector is as current time, the skeleton motion state vector at the current time As predicted calculated skeleton motion state vector.
For example, the skeleton motion state vector of multiple continuous history is { x1, x2, x3..., xT ＇, then first current time Skeleton motion state vector for T ＇, first current time T ＇ is xT ＇, can predict the subsequent time i.e. skeleton motion of T ＇+1 of T ＇ State vector is, it is assumed that obtains x for predictionT ＇+1.By T ＇+1 as new current time, the skeleton motion of current time T ＇+1 The state vector then x to be calculatedT ＇+1。
S110, according to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time.
It is appreciated that since the skeleton motion hidden state vector of last moment is carried out to the skeleton motion feature of history Motion intention is extracted to obtain, and extracts to obtain the bone of last moment so carrying out motion intention based on the skeleton motion feature to history Bone movement hidden state vector and the skeleton motion state vector at current time are decoded, and can predict subsequent time Skeleton motion state vector.
Fig. 2 is the effect diagram of skeleton motion prediction processing method in one embodiment.With reference to Fig. 2, it is in region 202 In the skeleton motion state represented by multiple continuous skeleton motion state vectors that the moment 1 to T ＇ observes, based on going through in 202 The skeleton motion state vector of history observation can predict the skeleton motion state vector of the future time instance after and then moment T ＇, It is the skeleton motion state represented by the skeleton motion state vector of the future time instance of prediction shown in region 204.
Above-mentioned skeleton motion prediction processing method carries out spy respectively to the skeleton motion state vector of multiple continuous history Assemble-publish code obtains corresponding skeleton motion feature vector, realizes and is carried to the skeleton motion feature of each skeleton motion state vector It takes.Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion hidden state vector, by being directed to Last moment carries out motion intention to obtained skeleton motion feature vector and extracts to obtain, and realizes the skeleton motion according to extraction Feature carries out the extraction of motion intention.According to the skeleton motion at the skeleton motion hidden state vector sum current time of last moment State vector is decoded, to calculate the skeleton motion state vector of subsequent time.By the motion intention of extraction and current Skeleton motion state vector be decoded, realize the excavation to the intent information of extraction, to based on decoding excavate be intended to Information calculates the skeleton motion state vector of subsequent time.Realizing the prediction to the skeleton motion of target object itself, this is thin The motion prediction process of section.
In one embodiment, step S106 includes：Obtain the last moment at current time estimates velocity characteristic vector； The motion relevance of each skeleton motion feature vector and last moment estimated between velocity characteristic vector is determined respectively；According to fortune Dynamic correlation determines the weight of each skeleton motion feature vector；Weight and motion relevance positive correlation；By each skeleton motion feature Vector is weighted summation according to corresponding weight respectively, obtain the skeleton motion hidden state of the last moment at current time to Amount.
Wherein, it is vectorial to estimate velocity characteristic, between the skeleton motion state vector for characterizing the adjacent moment estimated Variation.Variation size between the skeleton motion state vector of adjacent moment and estimate velocity characteristic vector magnitude positive correlation.When The last moment at preceding moment estimates velocity characteristic vector, that is, embodies the velocity characteristic of immediate history, current for characterizing Variation between the last moment at moment and upper skeleton motion of upper moment state vector.
In one embodiment, for first current time, computer equipment can be according to the last moment at current time Skeleton motion state vector and the skeleton motion state vector at upper upper moment carry out velocity characteristic analysis, estimate to obtain and estimate speed Spend feature vector.
Skeleton motion feature vector and the motion relevance between velocity characteristic vector is estimated, for describing skeleton motion spy Correlation between the skeleton motion information that the movable information that sign vector is embodied is embodied with velocity characteristic vector.
Specifically, computer equipment can according to each skeleton motion feature vector and last moment estimate velocity characteristic to Motion relevance between amount determines the weight of each skeleton motion feature vector respectively；Weight and motion relevance positive correlation.It can To understand, last moment estimates the velocity characteristic that velocity characteristic vector is immediate history, i.e., according to each historical juncture Motion relevance between the skeleton motion feature vector and the velocity characteristic of immediate history of corresponding history, distribution are respectively gone through The skeleton motion feature vector of the history weight shared when participating in motion intention extraction process, by the skeleton motion feature of each history Vector is weighted summation according to corresponding weight respectively, obtain the skeleton motion hidden state of the last moment at current time to Amount.Wherein, motion relevance is bigger, and weight is bigger, and the influence to motion intention extraction is bigger, conversely, motion relevance is got over Small, weight is smaller, and the influence to motion intention extraction is with regard to smaller.
In one embodiment, determine each skeleton motion feature vector and last moment respectively estimates velocity characteristic vector Between motion relevance include：Each skeleton motion feature vector of velocity characteristic vector sum of estimating of last moment is inputted into machine Attention model in learning model determines the pre- of each skeleton motion feature vector and last moment according to attention model respectively Estimate the motion relevance between velocity characteristic vector.
Wherein, attention model (Attention Model) is each skeleton motion feature vector for being observed from history In, extract the machine learning model that more crucial skeleton motion feature vector is extracted to motion intention.It is appreciated that attention Model can determine the movement of each skeleton motion feature vector and last moment estimated between velocity characteristic vector by analyzing Correlation, to extract to extracting more crucial skeleton motion feature vector to motion intention.
In one embodiment, computer equipment can be determined by following formula last moment estimate velocity characteristic to Motion relevance between amount and each skeleton motion feature vector：
βt'=Wβtanh(Uβνvt-1+Uβeet'+bβ)；(formula 2)
Wherein, t is current time；T-1 is the last moment at current time；vt-1For current time last moment it is pre- Estimate velocity characteristic vector；T ＇ are the t ＇ historical junctures；et'For the skeleton motion feature vector of t ＇ historical junctures；βt'For The skeleton motion feature vector e of t ＇ historical juncturest'Velocity characteristic vector v is estimated with the last moment at current timet-1It Between correlation；Tanh () is hyperbolic tangent function；Wβ、Uβν、UβeAnd bβBe in machine learning model attention model it is pre- The parameter that training obtains.
It should be noted that clear in order to express, at the time of carrying out prediction processing in each embodiment of the application and history Moment distinguish and has indicated, i.e., is added to a slash (i.e. t ＇) for indicating the historical juncture in the upper right corner of t, and carries out at prediction It is not skimmed (i.e. t) in the upper right corner of t addition one then at the time of reason.It is appreciated that it is in the historical juncture to carry out t at the time of prediction processing The last one moment on the basis of it is added at the time of numerical value, that is, assume the historical juncture the last one at moment (i.e. the last one Historical juncture) be T ＇, then carry out prediction processing at the time of then be T ＇+t, the embodiment of the present application in order to be concise in expression, clearly, then exist When expression carries out at the time of prediction is handled, T ＇ are eliminated, are directly indicated with t.Wherein, t is greater than or equal to 0.
It is appreciated that above-mentioned tanh () function can be replaced by other activation primitives, such as sigmoid functions (S types Curvilinear function).
In one embodiment, computer equipment can determine the power of each skeleton motion feature vector by following formula Weight：
Wherein, αt'For the weight of the skeleton motion feature vector of t ＇ historical junctures；βt'For the t ＇ historical junctures The correlation of skeleton motion feature vector and the last moment at current time estimated between velocity characteristic vector；T' is last A historical juncture.
In one embodiment, computer equipment can obtain the bone of the last moment at current time by following formula Move hidden state vector：
Wherein, t-1 is the last moment at current time；ht-1Skeleton motion for the last moment at current time implies shape State vector；et'For the skeleton motion feature vector of t ＇ historical junctures；αt'For the skeleton motion feature of t ＇ historical junctures The weight of vector；T' is the last one historical juncture.
Fig. 3 is the processing block schematic illustration of skeleton motion prediction processing method in one embodiment.Wherein, xt'Indicate t ＇ The skeleton motion state vector of a historical juncture, t ＇ values are 1 to T ＇, thenIndicate the multiple continuous of history observation Skeleton motion state vector；Each skeleton motion state vector input coding device is encoded, it is special to obtain corresponding skeleton motion Levy vector et', similarly, et'In t ＇ values be 1 to T ＇, then by each skeleton motion feature vector et'Input attention model In, by each skeleton motion feature vector et'Velocity characteristic vector v is estimated with the last moment at current timet-1Pass through sigmoid Function layer obtains correlation β between the twot', by βt'Input softmax function layers are mapped, and respective weights α is obtainedt', and By each skeleton motion feature vector et'By respective weights αt'It is weighted summation, output obtains the last moment t- of current time t 1 skeleton motion hidden state vector ht-1.Wherein, t values are 0 to T, it will be understood that T here is appointing more than or equal to 0 Meaning integer value.By the skeleton motion hidden state vector h of last moment t-1t-1And the skeleton motion state of current time t to Amount, input decoder are decoded, and export the skeleton motion state vector of the subsequent time t+1 at current time.It is appreciated that In the prediction process of a new round, the subsequent time in last round of prediction process is new current time, then this is new Current time skeleton motion state vector, the as last round of skeleton motion state vector predicted can participate in new In this wheel prediction process, to continue to calculate the skeleton motion state vector of subsequent time.Wherein, decoder, for pair Coding information is decoded processing.Softmax functions are to map multiple input value so that the numerical value after mapping is added For 1 function.
In above-described embodiment, pass through the bone of the velocity characteristic and the history corresponding to each historical juncture of immediate history Motion relevance between bone motion feature vector, the skeleton motion feature vector for distributing each history are participating in motion intention extraction Shared weight when processing, ensure that the accuracy of weight distribution.Therefore, by the skeleton motion feature vector of each history by correlation Weight is weighted summation, obtains the skeleton motion hidden state vector of last moment, ensure that the accurate of motion intention extraction Property, that is, it ensure that the accuracy of the skeleton motion hidden state vector of the last moment at current time.
In one embodiment, step S110 includes：When the skeleton motion hidden state vector sum of last moment is current Decoder in the skeleton motion state vector input machine learning model at quarter is decoded, and obtain current time estimates speed Feature vector；According to the skeleton motion state vector for estimating velocity characteristic vector sum current time at current time, calculate next The skeleton motion state vector at moment.
Wherein, decoder, for being decoded processing to coding information.It is appreciated that subsequent time here is current The subsequent time at moment.In one embodiment, decoder can be improve high-speed cells (Modified Highway Unit, MHU), for the coding/decoding information of history to be integrated into the processing of encoding and decoding next time, to realize the dress of high speed encoding and decoding It sets.
In one embodiment, computer equipment can be determined according to following formula current time estimate velocity characteristic to Amount：
Wherein, t is current time, and t-1 is the last moment at current time；vtFor current time estimate velocity characteristic to Amount；ht-1For the skeleton motion hidden state vector of last moment；φ indicates line rectification function；Wv、Uvh、bvAnd bvhIt is The offset parameter that pre-training obtains in the machine learning model；For the skeleton motion state vector at current time.
Fig. 4 is the principle framework schematic diagram of skeleton motion prediction processing method in one embodiment.With reference to Fig. 4, often pass through One MHU unit 402, expression have passed through primary decoding prediction processing, and it is identical to decode prediction process every time, thus it is existing according only to First MHU is explained.By the skeleton motion hidden state vector h of last momentt-1It is transported with the bone of current time t Dynamic state vectorIt is decoded in input MHU units 402, what output obtained current time estimates velocity characteristic vector vtAnd Predict the skeleton motion state vector of subsequent time t+1And by vtWith each skeleton motion feature vector et'(wherein, t ＇ take Value is 1 to T ＇) it inputs in attention model 404, correlation is carried out than peer processes, output skeleton motion hidden state vector. It is appreciated that last round of prediction is handled predicted subsequent time t+1 as when the current time t of a following new round, The opposite skeleton motion for being denoted as last moment is implicit again on time dimension for the skeleton motion hidden state vector of last round of output State vector ht-1, the last round of skeleton motion state vector predicted is again opposite on time dimension to be denoted as the current of a new round The skeleton motion state vector of moment tInput next MHU units.
In one embodiment, according to the skeleton motion state for estimating velocity characteristic vector sum current time at current time Vector, the skeleton motion state vector for calculating subsequent time include：The skeleton motion shape with current time is obtained by decoder The corresponding first prediction weight vectors of state vector；According to the first prediction weight vectors, determine current time estimates velocity characteristic Second prediction weight vectors of vector；First prediction weight vectors and second prediction weight vectors and for it is complete one vector；It will work as The skeleton motion state vector at preceding moment and current time estimate velocity characteristic vector, respectively with corresponding first prediction weight Vector sum second is predicted to be added after weight vectors carry out dot product, obtains the skeleton motion state vector of subsequent time.
Wherein, the first prediction weight vectors, the skeleton motion state vector for characterizing current time participate in skeleton motion Shared weight when status predication processing.Second prediction weight vectors, for characterize current time estimate velocity characteristic vector Participate in weight shared when the processing of skeleton motion status predication.A full vector, 1 vector is all for vector element.It needs to illustrate , first prediction weight vectors and second prediction weight vectors and for it is complete one vector.
In one embodiment, computer equipment can be obtained according to following formula the skeleton motion state of subsequent time to Amount：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ is indicated Sigmoid functions；φ indicates line rectification function；For the skeleton motion state vector at current time；vtFor current time Estimate velocity characteristic vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in machine learning model；T+1 is to work as The subsequent time at preceding moment；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol. It is appreciated that 1-ztIn 1 indicate it is complete one vector.
Fig. 5 is the processing procedure schematic diagram for the skeleton motion state vector that subsequent time is calculated in one embodiment.Reference Fig. 5 inputs the skeleton motion hidden state vector h for last moment for each decoding unitt-1With current time Skeleton motion state vectorBy ht-1By line rectification function layer and linear layer handle as a result, withBy linear The results added of layer processing, obtain current time estimates velocity characteristic vector vt(corresponding to above-mentioned formula 5), obtained vtOne Aspect is exported, and the decoding of next decoding unit is participated in, in addition, obtained vtThe solution of current decoding unit can also be participated in Code detailed process is as follows.The skeleton motion state vector at current timeAlso need to by line rectification function layer and Sigmoid function layers are handled, and corresponding first prediction weight vectors z is obtainedt(correspond to above-mentioned formula 6) according to complete one to Amount and zt, obtain and vtCorresponding second prediction weight vectors 1-zt,With corresponding first prediction weight vectors ztCarry out dot product As a result it adds and vtCorresponding second prediction weight vectors 1-ztDot product obtains the subsequent time at current time as a result, exporting Skeleton motion state vector(corresponding to above-mentioned formula 7).
In above-described embodiment, the skeleton motion hidden state vector of last moment is for last moment to skeleton motion Feature vector carries out motion intention and extracts to obtain, so describing motion intention to a certain extent.According to description motion intention The skeleton motion state vector at the skeleton motion hidden state vector sum current time of last moment is decoded, when obtaining current That carves estimates velocity characteristic vector, which estimates velocity characteristic vector, then accurately embody bone The information in movement velocity dimension is moved, so, the bone for estimating velocity characteristic vector sum current time based on current time Bone motion state vector can accurately calculate the skeleton motion state vector of subsequent time.
In one embodiment, this method further includes machine learning model training step, specifically includes following steps：It will be true The sequencing that the skeleton motion state vector generated in fact is generated according to skeleton motion state, is divided into historical sample and pre- test sample This；Machine learning model training is carried out according to the skeleton motion state vector in historical sample, is exported by machine learning model The skeleton motion state vector of model parameter expression；According in the skeleton motion state vector and forecast sample of model parameter expression Skeleton motion state vector build loss function；Model parameter when loss function is minimized is as machine learning model Stable model parameter.
Wherein, the skeleton motion state vector really generated is the skeleton motion state vector actually generated.History Sample is the sample data of the skeleton motion state vector in machine learning model training as history.Forecast sample, be As the sample data for predicting calculated skeleton motion state vector based on historical sample in machine learning model training.
It is appreciated that machine learning model training, is to be inputted in machine learning model based on known sample data, constantly Iteration updates model parameter until the process that model parameter is stablized, machine learning model progress is carried out so historical sample is inputted When model training, output obtains the skeleton motion state vector expressed by the model parameter of machine learning model.
Computer equipment can be transported according to the bone in the skeleton motion state vector and forecast sample that model parameter is expressed Dynamic state vector builds loss function.Wherein, loss function be used for indicate model parameter expression skeleton motion state vector and Difference degree in forecast sample between corresponding skeleton motion state vector.Computer equipment can seek most the loss function of structure Small value, the model parameter that model parameter when loss function is minimized is stablized as machine learning model.
In above-described embodiment, by building loss function, determines the model parameter that machine learning model is stablized, improve machine The accuracy of device learning model, it is more accurate when thereby using machine learning model progress skeleton motion test.
In one embodiment, it is transported according to the bone in the skeleton motion state vector and forecast sample of model parameter expression Dynamic state vector builds loss function：Per bone adjacent two-by-two in the skeleton motion state vector expressed for model parameter Bone motion state vector is spliced, and the first splicing vector is obtained；Splice the transposition of vector according to the first splicing vector sum first The apposition of obtained vector, obtains the first matrix；Skeleton motion state vector adjacent two-by-two in forecast sample is spliced, Obtain the second splicing vector；The apposition for splicing the vector that the transposition of vector obtains according to the second splicing vector sum second, obtains the Two matrixes；According to the mean square deviation of corresponding first matrix and the second matrix, loss function is obtained.
It is appreciated that splicing vector, as carries out the vector that splicing retrieves by vector.For example, by model parameter table The adjacent skeleton motion state vector reachedWithSpliced, obtaining the first splicing vector isFor another example, will Skeleton motion state vector x adjacent two-by-two in forecast sampletAnd xt-1Spliced, it is [x to obtain the second splicing vectort；xt -1]。
In one embodiment, the first matrix and the second matrix can be gram matrix (gram matrix).
In one embodiment, computer equipment can obtain the first matrix and the second matrix according to following formula respectively：
G(xt,xt-1)=[xt；xt-1][xt；xt-1]T；
Wherein, G () indicates gram matrix,Indicate the first matrix；WithIt indicates by model parameter table The adjacent skeleton motion state vector reached；Indicate the first splicing vector；For turning for the first splicing vector The vector set；G(xt,xt-1) indicate the second matrix；xtAnd xt-1Indicate skeleton motion state vector adjacent in forecast sample xtAnd xt-1；[xt；xt-1] indicate the second splicing vector；[xt；xt-1]TThe vector obtained for the transposition of the second splicing vector.
In one embodiment, computer equipment can obtain loss function according to following formula respectively：
Wherein, LgramIndicate loss function；N is the first matrix quantity or the second matrix quantity；Indicate first Matrix；G(xt,xt-1) indicate the second matrix.It is appreciated that the first matrix quantity is consistent with the second matrix quantity.
It should be noted that in the embodiment of the present application, the t in the formula that machine learning model training process is related to and reality It is not related that border carries out t at the time of when skeleton motion prediction is handled.It needs to carry out model training and the model based on pre-training T is distinguished at the time of in skeleton motion prediction processing.
In above-described embodiment, by the skeleton motion state vector expressed for model parameter per bone adjacent two-by-two Motion state vector is spliced, and the first splicing vector is obtained；The transposition for splicing vector according to the first splicing vector sum first obtains The apposition of the vector arrived, obtains the first matrix；Skeleton motion state vector adjacent two-by-two in forecast sample is spliced, is obtained To the second splicing vector；Splice the vectorial apposition that the transposition of vector obtains according to the second splicing vector sum second, obtains second Matrix；According to the mean square deviation of corresponding first matrix and the second matrix, loss function is built.I.e. based on adjacent skeleton character It builds loss function and carries out machine learning model training, to consider the continuous journey between bone in machine learning model training Degree, improves the accuracy of machine learning model.
In one embodiment, this method further includes：It obtains corresponding with the skeleton motion hidden state vector of last moment Behavior label vector；Behavior label vector is by obtaining behavior label coding.In the present embodiment, according to the bone of last moment The skeleton motion state vector at movement hidden state vector sum current time is decoded, to calculate the skeleton motion of subsequent time State vector includes：Behavior label vector is spliced to the skeleton motion hidden state vector of corresponding last moment；According to spelling The skeleton motion state vector at the skeleton motion hidden state vector sum current time of the last moment after connecing is decoded, in terms of Calculate the skeleton motion state vector of subsequent time；The bone fortune that the skeleton motion state vector of calculated subsequent time is characterized Dynamic state matches with behavior label.
Wherein, behavior label vector, i.e., behavior label vector indicate, be to behavior label encoded to Amount.Behavior is the general designation of all actions showed.Behavior label includes the mark for identifying the behaviors such as walk, sit down, being directed toward Label.One behavior label is not limited to identify single behavior, can be used for identifier combination behavior, such as the bind lines of walking+direction For.
In one embodiment, computer equipment can be directed to behavior label and carry out one-hot coding (One-Hot Encoding), behavior label vector is obtained.One-hot coding is a kind of binary coding, and obtained vector element is 1 i.e. 0.Than Such as, it walks being encoded to [0,0,1] of label, sensing label is encoded to [0,1,0].
Specifically, computer equipment can prestore the behavior label vector being arranged for each moment.It is appreciated that such as It is described above, in carrying out skeleton motion prediction process, in output skeleton motion hidden state vector of each current time, meeting As the skeleton motion hidden state for inputting so-called last moment in next skeleton motion prediction processing, so with constantly right The behavior label vector and skeleton motion hidden state that should be arranged are also corresponding.It is appreciated that different skeleton motions implies shape Behavior label vector corresponding to state can be different.
It should be noted that configuration behavior label at the time of computer equipment can transfer just for behavior occurs, other When corresponding behavior label at the time of not having a configuration behavior label, which is former configuration, behavior label and immediate Carve corresponding behavior label.
Computer equipment can obtain behavior label vector corresponding with the skeleton motion hidden state vector of last moment, Behavior label vector is spliced to the skeleton motion hidden state vector of corresponding last moment；According to spliced last moment The skeleton motion state vector at skeleton motion hidden state vector sum current time be decoded, to calculate the bone of subsequent time Bone motion state vector.For example, behavior label is encoded to [0,0,1], the skeleton motion hidden state vector with last moment ht-1Spliced with [0,0,1], the skeleton motion hidden state vector of obtained spliced last moment can be [ht-1, 0,0,1].It should be noted that being not limited to the front and back of stitching position here.
It is appreciated that due to spliced in the skeleton motion hidden state vector of spliced last moment behavior label to Amount, therefore the behavioural information specified by behavior label, institute are carried after splicing in the skeleton motion hidden state vector of last moment With the skeleton motion state vector at the skeleton motion hidden state vector sum current time based on spliced last moment carries out Decoding, the skeleton motion state that the skeleton motion state vector of the subsequent time predicted is characterized largely with behavior mark The specified behavior of label matches.Therefore, by configuration behavior label, it can control and realize specified behavior act.
In one embodiment, this method further includes：According to the calculated skeleton motion shape to match with behavior label State vector, generates the control instruction for target object；Control instruction is used to indicate target object and is transported according to calculated bone Dynamic state vector carries out corresponding sports, the behavior characterized with process performing label.
Specifically, computer equipment is generated according to the calculated skeleton motion state vector to match with behavior label For the control instruction of target object.It is appreciated that control instruction is used to indicate target object according to calculated skeleton motion State vector carries out corresponding sports, the behavior characterized with process performing label.
Computer equipment can export the control instruction for target object to target object, and target object can be according to meter The skeleton motion state vector of calculating carries out corresponding sports, and target object carries out phase according to calculated skeleton motion state vector The process that should be moved, the process for the behavior that as process performing label is characterized.
It is appreciated that the behavior label vector corresponding to different skeleton motion hidden states can be different, so according to The calculated control instruction of skeleton motion state vector output to match from behavior label is different, then can be by different Behavior label, control targe object execute different behavior acts.
Fig. 6 is the schematic diagram by behavior label control targe object in one embodiment.With reference to shown in Fig. 6,602,604 And the skeleton motion state in 606 corresponds respectively to the behavior that sensing label, walking label and sensing label are characterized, i.e., " refers to To-walking-direction " this serial behavior.608, the skeleton motion state in 610 and 612 corresponds respectively to walking label, coordinate The behavior that label and walking label are characterized, i.e. " walking-seat-walking " this serial behavior.
In one embodiment, this method further includes motor behavior anticipation processing step, specifically includes following steps：According to The multiple continuous skeleton motion state vectors being calculated carry out motor behavior anticipation to target object；According to what is prejudged out Motor behavior determines interactive interbehavior logic corresponding to the motor behavior realization prejudged out；According to interbehavior logical AND Target object interacts.
Wherein, motor behavior prejudges, and is to prejudge the following motor behavior to be made.
It is appreciated that due to being that temporally dimension carries out prediction calculating one by one, so the skeleton motion shape being calculated State vector is continuous.
Specifically, computer equipment can be characterized according to the multiple continuous skeleton motion state vectors being calculated Skeleton motion state feature prejudges the target object next motor behavior to be made.
In one embodiment, the motor behavior and interbehavior logic of target object are pre-set in computer equipment Between correspondence.Wherein, interbehavior logic, for interaction corresponding to the realization of the motor behavior of target object.According to pre- Sentence the motor behavior, determines interactive interbehavior logic corresponding to the motor behavior realization prejudged out；According to interbehavior Logical AND target object interacts.For example, the motor behavior of target object is " waving ", corresponding interbehavior logic then may be used Think for realizing the logic of " going forward to move towards target object " this interactive action.
In one embodiment, computer equipment can be robot.Fig. 7 is in one embodiment at skeleton motion prediction The applied environment figure of reason.With reference to Fig. 7, robot 702 can be observed target object 704 (for example, people), collect more A continuous skeleton motion status image frame, then extracted from skeleton motion status image frame corresponding skeleton motion state to Amount, obtains the skeleton motion state vector of multiple continuous history, robot 702 can be by machine learning model respectively to each The skeleton motion state vector of history carries out condition code coding, generates corresponding with the skeleton motion state vector of each history respectively Skeleton motion feature vector, and according to the method provided in each embodiment of the application, prediction is calculated corresponding to target object 704 Subsequent time skeleton motion state vector.In one embodiment, robot 702 can be according to the multiple companies being calculated Continuous skeleton motion state vector carries out motor behavior anticipation to target object 704；According to the motor behavior prejudged out, determine Interactive interbehavior logic corresponding to the motor behavior realization prejudged out；According to interbehavior logical AND target object 704 into Row interaction.
In above-described embodiment, by calculating the skeleton motion state vector of future time instance, to the motor behavior of target object It is prejudged, and determines corresponding interbehavior logic, interacted with target object, improved interactive intelligent and flexible Property, improve human-computer interaction efficiency.
In one embodiment, as shown in figure 8, another embodiment provides a kind of skeleton motion prediction processing Method, this method specifically include following steps：
S802 obtains the skeleton motion state vector of multiple continuous history；According to each history skeleton motion state to The sequencing of amount encodes obtained skeleton motion feature vector and when skeleton motion state vector time to be encoded by previous, The encoder inputted in the machine learning model of pre-training is encoded, and output obtains and works as secondary skeleton motion state to be encoded The corresponding skeleton motion feature vector obtained when time coding of vector.
In one embodiment, this method further includes：Obtain the skeleton motion status image frame of multiframe continuous acquisition；For Per frame skeleton motion status image frame, multiple target skeletal joint points are identified from skeleton motion status image frame；According to multiple Tandem between target skeletal joint point obtains latter object skeletal joint point relative to previous target skeletal joint respectively The spin data of point；Using each spin data as vector element, by front and back suitable between respective objects skeletal joint point Sequence, splicing obtain skeleton motion state vector corresponding with skeleton motion status image frame.
S804, obtain the last moment at current time estimates velocity characteristic vector；Last moment is estimated into speed spy The attention model in each skeleton motion feature vector input machine learning model of vector sum is levied, is distinguished according to attention model true The motion relevance of fixed each skeleton motion feature vector and last moment estimated between velocity characteristic vector.
Wherein, it is vectorial to estimate velocity characteristic, between the skeleton motion state vector for characterizing the adjacent moment estimated Variation.
S806 determines the weight of each skeleton motion feature vector according to motion relevance；By each skeleton motion feature vector It is weighted summation according to corresponding weight respectively, obtains the skeleton motion hidden state vector of the last moment at current time.
Wherein, the weight of each skeleton motion feature vector and corresponding motion relevance positive correlation.
It should be noted that the skeleton motion of the last moment at determination current time described in step S804~806 is hidden Processing step containing state vector can be adapted for non-first current time.It, can directly will be current for first current time The skeleton motion hidden state vector of the last moment at moment is set as initial default value.
In other embodiments, for first current time, current time can also be calculated according to step S804~806 Last moment skeleton motion hidden state vector.It specifically, can be according to the upper a period of time at current time in step S804 The skeleton motion state vector and the skeleton motion state vector at upper upper moment at quarter carry out velocity characteristic analysis, are estimated Velocity characteristic vector, to calculate the skeleton motion hidden state of the last moment at current time according to step S804~806 Vector.
S808 obtains behavior label vector corresponding with the skeleton motion hidden state vector of last moment；By behavior mark The skeleton motion hidden state of the splicing of label vector to corresponding last moment are vectorial.
Wherein, behavior label vector is by obtaining behavior label coding.
S810 obtains the skeleton motion state vector at current time；The skeleton motion of spliced last moment is implied Decoder in state vector and the skeleton motion state vector at current time input machine learning model is decoded, and is worked as The preceding moment estimates velocity characteristic vector.
S812, by decoder obtain it is corresponding with the skeleton motion state vector at current time first prediction weight to Amount；According to the first prediction weight vectors, the second prediction weight vectors for estimating velocity characteristic vector at current time are determined.
Wherein, first prediction weight vectors and second prediction weight vectors and for it is complete one vector.
The skeleton motion state vector at current time and current time are estimated velocity characteristic vector by S814, respectively with Corresponding first prediction weight vectors and the second prediction weight vectors are added after carrying out dot product, obtain the skeleton motion of subsequent time State vector；The skeleton motion state that the skeleton motion state vector of obtained subsequent time is characterized and behavior label phase Match.
Above-mentioned skeleton motion prediction processing method carries out spy respectively to the skeleton motion state vector of multiple continuous history Assemble-publish code obtains corresponding skeleton motion feature vector, realizes and is carried to the skeleton motion feature of each skeleton motion state vector It takes.Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion hidden state vector, by being directed to Last moment carries out motion intention to obtained skeleton motion feature vector and extracts to obtain, and realizes the skeleton motion according to extraction Feature carries out the extraction of motion intention.According to the skeleton motion at the skeleton motion hidden state vector sum current time of last moment State vector is decoded, to calculate the skeleton motion state vector of subsequent time.By the motion intention of extraction and current Skeleton motion state vector be decoded, realize the excavation to the intent information of extraction, to based on decoding excavate be intended to Information calculates the skeleton motion state vector of subsequent time.Realizing the prediction to the skeleton motion of target object itself, this is thin The motion prediction process of section.
As shown in figure 9, in one embodiment, providing a kind of limb motion prediction processing method, this method is specifically wrapped Include following steps：
S902 obtains the limb motion state of multiple continuous history.
Wherein, limb motion state, present state when being limb motion.It is appreciated that limb motion state can be with Including dynamic and static state.The limb motion state of history is the limb motion state generated.
In one embodiment, limb motion state includes skeleton motion state vector.Wherein, skeleton motion state to Amount is that the vector of skeleton motion state indicates, i.e., describes skeleton motion state by the form of vector.It is appreciated that due to limb Body motion state can be indicated by skeleton motion state, and skeleton motion state can pass through skeleton motion state vector It is indicated, therefore, limb motion state can be indicated by multiple continuous skeleton motion state vectors, i.e., limbs are transported Dynamic state may include skeleton motion state vector.
It is appreciated that in other embodiments, limb motion state can also be indicated by limbs curve data, than Such as, the bending degree of limbs curve data characterization is different, it may be said that bright limb motion state is different.
It is appreciated that the skeleton motion state vector of each history can be there are one the corresponding historical juncture.When history It carves, is that the skeleton motion state having occurred and that represented by skeleton motion state vector of the computer equipment to history is observed At the time of acquisition.
S904 carries out Motion feature extraction to the limb motion state of each history, obtains the limbs with each history respectively The corresponding limb motion feature of motion state.
Wherein, limb motion feature, for characterizing feature of the limbs when being moved.It is appreciated that computer equipment Motion feature extraction can be carried out to the limb motion state of each history, obtained corresponding with the limb motion state of each history Limb motion feature.
In one embodiment, limb motion feature may include skeleton motion feature vector.Skeleton motion feature vector, It is that the feature vector that Motion feature extraction obtains is carried out to skeleton motion state vector.
It is appreciated that in other embodiments, limb motion feature can also by limbs curve data from bending journey Degree or bending turning point etc. carry out feature extraction and obtain.
S906 determines the limb motion hidden state of the last moment at current time.
Wherein, current time, at the time of being current progress limb motion prediction processing.Limb motion hidden state, is used for The limb motion characteristic information of comprehensive history is to represent corresponding sports intention.Motion intention is desirable to carry out certain movement Intend, for embodying the next desired movement carried out.The limb motion hidden state of the last moment at current time, is needle To the last moment, a pair limb motion feature corresponding with the limb motion state of history carries out motion intention and extracts to obtain.
In one embodiment, limb motion hidden state may include skeleton motion hidden state vector.Wherein, it is pair Skeleton motion feature vector carries out the vector that motion intention is extracted, the skeleton motion state characteristic information for integrating history To represent corresponding sports intention.
It is appreciated that since limb motion feature may include skeleton motion feature vector, then to limb motion feature Carry out motion intention extraction, can be to skeleton motion feature vector carry out motion intention extraction, then to skeleton motion feature to Amount carries out the skeleton motion hidden state vector that motion intention is extracted and can be used to indicate that limb motion hidden state, i.e. limb Body movement hidden state may include skeleton motion hidden state vector.So, the limb motion of the last moment at current time Hidden state may include carrying out the bone that motion intention is extracted to skeleton motion feature vector for the last moment to transport Dynamic hidden state vector, the skeleton motion feature vector are corresponding with the skeleton motion state vector of history.
S908 obtains the limb motion state at current time.
It is appreciated that first current time (the i.e. corresponding historical juncture conduct of limb motion state of the last one history Current time) limb motion state, as history observation the last one limb motion state.When the calculated limbs of prediction When the corresponding moment of motion state is as current time, the limb motion state at the current time is had calculated that limbs Motion state.
In one embodiment, step S908 includes：Obtain the skeleton motion state vector at current time.
S910 is determined next according to the limb motion state of the limb motion hidden state of last moment and current time The limb motion state at moment.
It is appreciated that since the limb motion hidden state of last moment is moved to the limb motion feature of history It is intended to extraction to obtain, is transported so carrying out motion intention based on the limb motion feature to history and extracting to obtain the limbs of last moment Dynamic hidden state and the limb motion state at current time are decoded, and can predict the limb motion for calculating subsequent time State.
In one embodiment, step S910 includes：It is current according to the skeleton motion hidden state vector sum of last moment The skeleton motion state vector at moment calculates the skeleton motion state vector of subsequent time, to be transported according to calculated bone Dynamic state vector determines the limb motion state of the subsequent time.
It is special to carry out movement respectively to the limb motion state of multiple continuous history for above-mentioned limb motion prediction processing method Sign extraction obtains corresponding limb motion feature.Determine the limb motion hidden state of the last moment at current time；The limbs Hidden state is moved, extracts to obtain by carrying out motion intention to obtained limb motion feature for last moment, realizes root The extraction of motion intention is carried out according to the limb motion feature of extraction.According to the limb motion hidden state of last moment and it is current when The limb motion state at quarter is decoded, to calculate the limb motion state of subsequent time.By the motion intention of extraction and The limb motion state at current time is decoded, and realizes the excavation to the intent information of extraction, to be excavated based on decoding Intent information calculates the limb motion state of subsequent time.Realize the prediction to the limb motion of target object itself this The more motion prediction process of details.
In one embodiment, limb motion state includes skeleton motion state vector；Limb motion feature includes bone Motion feature vector；Limb motion hidden state includes skeleton motion hidden state vector.Step S904 includes：Pass through engineering It practises model and condition code coding is carried out to the skeleton motion state vector of each history respectively, generate the skeleton motion with each history respectively The corresponding skeleton motion feature vector of state vector.Step S910 includes：According to the skeleton motion hidden state of last moment to Amount and the skeleton motion state vector at current time are decoded, and the skeleton motion state vector of subsequent time are calculated, with root The limb motion state of the subsequent time is determined according to calculated skeleton motion state vector.
In one embodiment, limb motion state includes skeleton motion state vector；Limb motion feature includes bone Motion feature vector；Limb motion hidden state includes skeleton motion hidden state vector.Step S906 includes：When obtaining current The last moment at quarter estimates velocity characteristic vector；Estimate velocity characteristic vector, the bone for characterizing the adjacent moment estimated Variation between motion state vector；Determine each skeleton motion feature vector and last moment respectively estimates velocity characteristic vector Between motion relevance；The weight of each skeleton motion feature vector is determined according to motion relevance；Weight and motion relevance Positive correlation；Each skeleton motion feature vector is weighted summation according to corresponding weight respectively, obtains upper the one of current time The skeleton motion hidden state vector at moment.
In one embodiment, determine each skeleton motion feature vector and last moment respectively estimates velocity characteristic vector Between motion relevance include：Each skeleton motion feature vector of velocity characteristic vector sum of estimating of last moment is inputted into machine Attention model in learning model determines the pre- of each skeleton motion feature vector and last moment according to attention model respectively Estimate the motion relevance between velocity characteristic vector.
In one embodiment, according to the skeleton motion at the skeleton motion hidden state vector sum current time of last moment State vector is decoded, and the skeleton motion state vector for calculating subsequent time includes：
The skeleton motion state vector at the skeleton motion hidden state vector sum current time of last moment is inputted into machine Decoder in learning model is decoded, and obtain current time estimates velocity characteristic vector；
According to the skeleton motion state vector for estimating velocity characteristic vector sum current time at current time, lower a period of time is calculated The skeleton motion state vector at quarter.
In one embodiment, by the skeleton motion shape at the skeleton motion hidden state vector sum current time of last moment Decoder in state vector input machine learning model is decoded, and the velocity characteristic vector of estimating for obtaining current time includes：
According to following formula determine current time estimate velocity characteristic vector：
Wherein, t is current time, and t-1 is the last moment at current time；vtFor current time estimate velocity characteristic to Amount；ht-1For the skeleton motion hidden state vector of the last moment at current time；φ indicates line rectification function；Wv、Uvh、bv And bvhIt is the offset parameter that pre-training obtains in machine learning model；For the skeleton motion state vector at current time.
In one embodiment, according to the skeleton motion state for estimating velocity characteristic vector sum current time at current time Vector, the skeleton motion state vector for calculating subsequent time include：The skeleton motion shape with current time is obtained by decoder The corresponding first prediction weight vectors of state vector；According to the first prediction weight vectors, determine current time estimates velocity characteristic Second prediction weight vectors of vector；First prediction weight vectors and second prediction weight vectors and for it is complete one vector；It will work as The skeleton motion state vector at preceding moment and current time estimate velocity characteristic vector, respectively with corresponding first prediction weight Vector sum second is predicted to be added after weight vectors carry out dot product, obtains the skeleton motion state vector of subsequent time.
In one embodiment, by the skeleton motion state vector at current time and current time estimate velocity characteristic to Amount is added after carrying out dot product with corresponding first prediction weight vectors and the second prediction weight vectors respectively, obtains subsequent time Skeleton motion state vector include：
The skeleton motion state vector of subsequent time is calculated according to following formula：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ is indicated Sigmoid functions；φ indicates line rectification function；For the skeleton motion state vector at current time；vtIt is pre- for current time Estimate velocity characteristic vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in machine learning model；T+1 is current The subsequent time at moment；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol.
In one embodiment, this method further includes the training step of machine learning model, specifically includes following steps：It will The sequencing that the skeleton motion state vector really generated is generated according to skeleton motion state, is divided into historical sample and prediction Sample；Machine learning model training is carried out according to the skeleton motion state vector in historical sample, is exported by machine learning model Model parameter expression skeleton motion state vector；According to the skeleton motion state vector and forecast sample of model parameter expression In skeleton motion state vector build loss function；Loss function be used for indicate model parameter expression skeleton motion state to Difference degree in amount and forecast sample between corresponding skeleton motion state vector；Model parameter when loss function is minimized The model parameter stablized as machine learning model.
In one embodiment, it is transported according to the bone in the skeleton motion state vector and forecast sample of model parameter expression Dynamic state vector builds loss function：Per bone adjacent two-by-two in the skeleton motion state vector expressed for model parameter Bone motion state vector is spliced, and the first splicing vector is obtained；Splice the transposition of vector according to the first splicing vector sum first The apposition of obtained vector, obtains the first matrix；Skeleton motion state vector adjacent two-by-two in forecast sample is spliced, Obtain the second splicing vector；The apposition for splicing the vector that the transposition of vector obtains according to the second splicing vector sum second, obtains the Two matrixes；According to the mean square deviation of corresponding first matrix and the second matrix, loss function is obtained.
In one embodiment, which further includes：Obtain the skeleton motion with last moment The corresponding behavior label vector of hidden state vector；Behavior label vector is by obtaining behavior label coding.According to last moment The skeleton motion state vector at skeleton motion hidden state vector sum current time be decoded, to calculate the bone of subsequent time Bone motion state vector includes：Behavior label vector is spliced to the skeleton motion hidden state vector of corresponding last moment； It is solved according to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of spliced last moment Code, to calculate the skeleton motion state vector of subsequent time；The skeleton motion state vector of calculated subsequent time is characterized Skeleton motion state match with behavior label.
In one embodiment, which further includes：According to calculated with behavior label phase Matched skeleton motion state vector generates the control instruction for target object；Control instruction be used to indicate target object by Corresponding sports, the behavior characterized with process performing label are carried out according to calculated skeleton motion state vector.
In one embodiment, which further includes：It is multiple continuous according to what is be calculated Skeleton motion state vector carries out motor behavior anticipation to target object；According to the motor behavior prejudged out, determines and prejudge out Motor behavior realize accordingly interactive interbehavior logic；It is interacted according to interbehavior logical AND target object.
In one embodiment, which further includes：Obtain the bone fortune of multiframe continuous acquisition Dynamic status image frame；For every frame skeleton motion status image frame, multiple targeted bones are identified from skeleton motion status image frame Bone artis；According to the tandem between multiple target skeletal joint points, it is opposite that latter object skeletal joint point is obtained respectively In the spin data of previous target skeletal joint point；Using each spin data as vector element, by respective objects bone Tandem between artis, splicing obtain skeleton motion state vector corresponding with skeleton motion status image frame.
In one embodiment, feature is carried out to the skeleton motion state vector of each history by machine learning model respectively Code coding, generating skeleton motion feature vector corresponding with the skeleton motion state vector of each history respectively includes：According to respectively going through The sequencing of the skeleton motion state vector of history encodes obtained skeleton motion feature vector and when secondary to be encoded by previous The skeleton motion state vector of history, the encoder inputted in the machine learning model of pre-training are encoded, output obtain with When the corresponding skeleton motion feature vector obtained when time coding of the skeleton motion state vector of secondary history to be encoded.
As shown in Figure 10, in one embodiment, a kind of skeleton motion prediction processing device 1000 is provided, the device packet It includes：Coding module 1002, hidden state vector determining module 1004, skeleton motion state vector acquisition module 1006 and decoding Prediction module 1008, wherein：
Coding module 1002, the skeleton motion state vector for obtaining multiple continuous history；Pass through machine learning mould Type carries out condition code coding to each skeleton motion state vector respectively, generate respectively with each skeleton motion state vector Corresponding skeleton motion feature vector.
Hidden state vector determining module 1004, the skeleton motion hidden state of the last moment for determining current time Vector；The skeleton motion hidden state vector of last moment, was moved to skeleton motion feature vector for last moment It is intended to extraction to obtain.
Skeleton motion state vector acquisition module 1006, the skeleton motion state vector for obtaining current time.
Prediction module 1008 is decoded, the bone at the skeleton motion hidden state vector sum current time according to last moment is used for Bone motion state vector is decoded, to calculate the skeleton motion state vector of subsequent time.
In one embodiment, hidden state vector determining module 1004 is additionally operable to obtain the last moment at current time Estimate velocity characteristic vector；Estimate velocity characteristic vector, for characterize the adjacent moment estimated skeleton motion state vector it Between variation；Determine that each skeleton motion feature vector is related to the movement of last moment estimated between velocity characteristic vector respectively Property；The weight of each skeleton motion feature vector is determined according to motion relevance；Weight and motion relevance positive correlation；By each bone Motion feature vector is weighted summation according to corresponding weight respectively, and the skeleton motion for obtaining the last moment at current time is hidden Containing state vector.
In one embodiment, hidden state vector determining module 1004 is additionally operable to last moment estimating velocity characteristic Attention model in each skeleton motion feature vector input machine learning model of vector sum, determines respectively according to attention model The motion relevance of each skeleton motion feature vector and last moment estimated between velocity characteristic vector.
In one embodiment, decoding prediction module 1008 is additionally operable to the skeleton motion hidden state vector of last moment It is decoded with the decoder in the skeleton motion state vector input machine learning model at current time, obtains current time Estimate velocity characteristic vector；According to the skeleton motion state vector for estimating velocity characteristic vector sum current time at current time, Calculate the skeleton motion state vector of subsequent time.
In one embodiment, decoding prediction module 1008 is additionally operable to estimate speed according to what following formula determined current time Spend feature vector：
Wherein, t is current time, and t-1 is the last moment at current time；vtFor current time estimate velocity characteristic to Amount；ht-1For the skeleton motion hidden state vector of last moment；φ indicates line rectification function；Wv、Uvh、bvAnd bvhIt is The offset parameter that pre-training obtains in machine learning model；For the skeleton motion state vector at current time.
In one embodiment, decoding prediction module 1008 is additionally operable to obtain transporting with the bone at current time by decoder The dynamic corresponding first prediction weight vectors of state vector；According to the first prediction weight vectors, determine current time estimates speed Second prediction weight vectors of feature vector；First prediction weight vectors and second prediction weight vectors and for it is complete one vector； The skeleton motion state vector at current time and current time are estimated into velocity characteristic vector, respectively with corresponding first prediction Weight vectors and the second prediction weight vectors are added after carrying out dot product, obtain the skeleton motion state vector of subsequent time.
In one embodiment, decoding prediction module 1008 is additionally operable to that the bone of subsequent time is calculated according to following formula Bone motion state vector：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ is indicated Sigmoid functions；φ indicates line rectification function；For the skeleton motion state vector at current time；vtIt is pre- for current time Estimate velocity characteristic vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in machine learning model；T+1 is current The subsequent time at moment；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol.
As shown in figure 11, in one embodiment, which further includes：
Machine learning model training module 1001, the skeleton motion state vector for will really generate is according to skeleton motion The sequencing that state generates, is divided into historical sample and forecast sample；According to the skeleton motion state vector in historical sample Carry out machine learning model training, the skeleton motion state vector that output is expressed by the model parameter of machine learning model；According to Skeleton motion state vector in the skeleton motion state vector and forecast sample of model parameter expression builds loss function；Loss Function is used to indicate in the skeleton motion state vector that model parameter is expressed and forecast sample between corresponding skeleton motion state vector Difference degree；The model parameter that model parameter when loss function is minimized is stablized as machine learning model.
In one embodiment, machine learning model training module 1001 is additionally operable to the bone fortune for model parameter expression Spliced per skeleton motion state vector adjacent two-by-two in dynamic state vector, obtains the first splicing vector；It is spelled according to first It connects vector sum first and splices the vectorial apposition that the transposition of vector obtains, obtain the first matrix；It will be adjacent two-by-two in forecast sample Skeleton motion state vector spliced, obtain the second splicing vector；Splice vector according to the second splicing vector sum second The apposition for the vector that transposition obtains, obtains the second matrix；According to the mean square deviation of corresponding first matrix and the second matrix, obtain Loss function.
In one embodiment, which further includes：
Behavior label vector acquisition module (not shown), for obtaining the skeleton motion hidden state vector with last moment Corresponding behavior label vector；Behavior label vector is by obtaining behavior label coding；
Decoding prediction module 1008 is additionally operable to splice behavior label vector hidden to the skeleton motion of corresponding last moment Containing state vector；According to the skeleton motion state at the skeleton motion hidden state vector sum current time of spliced last moment Vector is decoded, to calculate the skeleton motion state vector of subsequent time；The skeleton motion state of calculated subsequent time The skeleton motion state that vector is characterized matches with behavior label.
In one embodiment, device 1000 further includes：
Motion-control module (not shown), for according to the skeleton motion state to match with behavior label that predicts to Amount generates the control instruction for target object；Control instruction is used to indicate target object according to the skeleton motion shape predicted State vector carries out corresponding sports, the behavior characterized with process performing label.
In one embodiment, device further includes：
Interactive controlling module (not shown), it is right for multiple continuous skeleton motion state vectors that basis is calculated Target object carries out motor behavior anticipation；According to the motor behavior prejudged out, determination is corresponding to the motor behavior realization prejudged out Interactive interbehavior logic；It is interacted according to interbehavior logical AND target object.
In one embodiment, skeleton motion state vector acquisition module 1006 is additionally operable to obtain the bone of multiframe continuous acquisition Bone motion state image frame；For every frame skeleton motion status image frame, multiple mesh are identified from skeleton motion status image frame Mark skeletal joint point；According to the tandem between multiple target skeletal joint points, latter object skeletal joint point is obtained respectively Spin data relative to previous target skeletal joint point；Using each spin data as vector element, by respective objects Tandem between skeletal joint point, splicing obtain skeleton motion state vector corresponding with skeleton motion status image frame.
In one embodiment, coding module 1002 is additionally operable to suitable according to the priority of the skeleton motion state vector of each history Sequence encodes obtained skeleton motion feature vector and when secondary skeleton motion state vector to be encoded, input pre-training by previous Machine learning model in encoder encoded, output obtain with when secondary skeleton motion state vector to be encoded is corresponding When the secondary skeleton motion feature vector for encoding and obtaining.
As shown in figure 12, in one embodiment, a kind of limb motion prediction processing device 1200, limbs fortune are provided Moving prediction processing device 1200 includes：Motion feature extraction module 1202, hidden state determining module 1204 and limb motion State determining module 1206, wherein：
Motion feature extraction module 1202, the limb motion state for obtaining multiple continuous history；To each history The limb motion state carry out Motion feature extraction respectively, obtain limb corresponding with the limb motion state of each history Body motion feature.
Hidden state determining module 1204, the limb motion hidden state of the last moment for determining current time；Institute The limb motion hidden state for stating last moment is to carry out motion intention to the limb motion feature for the last moment Extraction obtains.
Limb motion state determining module 1206, the limb motion state for obtaining current time；According to described upper one The limb motion hidden state at moment and the limb motion state at current time, determine the limb motion state of subsequent time.
In one embodiment, limb motion state includes skeleton motion state vector；Limb motion feature includes bone Motion feature vector；Limb motion hidden state includes skeleton motion hidden state vector.Motion feature extraction module 1202 is also For by machine learning model respectively to the skeleton motion state vector of each history carry out condition code coding, generate respectively with respectively The corresponding skeleton motion feature vector of skeleton motion state vector of history.Limb motion state determining module 1206 is additionally operable to root It is decoded, calculates down according to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of last moment The skeleton motion state vector at one moment, to determine that the limbs of the subsequent time are transported according to calculated skeleton motion state vector Dynamic state.
In one embodiment, limb motion state includes skeleton motion state vector；Limb motion feature includes bone Motion feature vector；Limb motion hidden state includes skeleton motion hidden state vector.Hidden state determining module 1204 is also For obtain current time last moment estimate velocity characteristic vector；Velocity characteristic vector is estimated, is estimated for characterizing Variation between the skeleton motion state vector of adjacent moment；The pre- of each skeleton motion feature vector and last moment is determined respectively Estimate the motion relevance between velocity characteristic vector；The weight of each skeleton motion feature vector is determined according to motion relevance；Power Weight and motion relevance positive correlation；Each skeleton motion feature vector is weighted summation according to corresponding weight respectively, is obtained The skeleton motion hidden state vector of the last moment at current time.
In one embodiment, hidden state determining module 1204 is additionally operable to last moment estimating velocity characteristic vector The attention model in machine learning model is inputted with each skeleton motion feature vector, determines each bone respectively according to attention model The motion relevance of bone motion feature vector and last moment estimated between velocity characteristic vector.
In one embodiment, limb motion state determining module 1206 is additionally operable to imply the skeleton motion of last moment Decoder in state vector and the skeleton motion state vector at current time input machine learning model is decoded, and is worked as The preceding moment estimates velocity characteristic vector；According to the skeleton motion shape for estimating velocity characteristic vector sum current time at current time State vector, calculates the skeleton motion state vector of subsequent time.
In one embodiment, limb motion state determining module 1206 is additionally operable to determine current time according to following formula Estimate velocity characteristic vector：
Wherein, t is current time, and t-1 is the last moment at current time；vtFor current time estimate velocity characteristic to Amount；ht-1For the skeleton motion hidden state vector of the last moment at current time；φ indicates line rectification function；Wv、Uvh、bv And bvhIt is the offset parameter that pre-training obtains in machine learning model；For the skeleton motion state vector at current time.
In one embodiment, limb motion state determining module 1206 is additionally operable to obtain by decoder and current time Skeleton motion state vector it is corresponding first prediction weight vectors；According to the first prediction weight vectors, current time is determined Estimate the second prediction weight vectors of velocity characteristic vector；First prediction weight vectors and second prediction weight vectors and be complete One vector；The skeleton motion state vector at current time and current time are estimated into velocity characteristic vector, respectively with it is corresponding First prediction weight vectors and the second prediction weight vectors are added after carrying out dot product, obtain the skeleton motion state of subsequent time to Amount.
In one embodiment, limb motion state determining module 1206 is additionally operable to be calculated according to following formula next The skeleton motion state vector at moment：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ is indicated Sigmoid functions；φ indicates line rectification function；For the skeleton motion state vector at current time；vtIt is pre- for current time Estimate velocity characteristic vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in machine learning model；T+1 is current The subsequent time at moment；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol.
In one embodiment, which further includes：
Machine learning model training module (not shown), the skeleton motion state vector for will really generate is according to bone The sequencing that motion state generates, is divided into historical sample and forecast sample；According to the skeleton motion state in historical sample Vector carries out machine learning model training, the skeleton motion state vector that output is expressed by the model parameter of machine learning model； Loss function is built according to the skeleton motion state vector in the skeleton motion state vector and forecast sample of model parameter expression； Loss function be used to indicate model parameter expression skeleton motion state vector and forecast sample in corresponding skeleton motion state to Difference degree between amount；The model parameter that model parameter when loss function is minimized is stablized as machine learning model.
In one embodiment, machine learning model training module is additionally operable to the skeleton motion shape for model parameter expression Spliced per skeleton motion state vector adjacent two-by-two in state vector, obtains the first splicing vector；According to first splice to The apposition for the vector that the transposition of amount and the first splicing vector obtains, obtains the first matrix；By bone adjacent two-by-two in forecast sample Bone motion state vector is spliced, and the second splicing vector is obtained；Splice the transposition of vector according to the second splicing vector sum second The apposition of obtained vector, obtains the second matrix；According to the mean square deviation of corresponding first matrix and the second matrix, lost Function.
In one embodiment, which further includes：
Behavior label vector acquisition module (not shown), for obtaining the skeleton motion hidden state vector with last moment Corresponding behavior label vector；Behavior label vector is by obtaining behavior label coding.
Limb motion state determining module 1206 is additionally operable to splice behavior label vector to the bone of corresponding last moment Bone moves hidden state vector；According to the bone at the skeleton motion hidden state vector sum current time of spliced last moment Motion state vector is decoded, to calculate the skeleton motion state vector of subsequent time；The bone of calculated subsequent time The skeleton motion state that motion state vector is characterized matches with behavior label.
In one embodiment, which further includes：
Motion-control module (not shown), for according to the calculated skeleton motion state to match with behavior label to Amount generates the control instruction for target object；Control instruction is used to indicate target object according to calculated skeleton motion shape State vector carries out corresponding sports, the behavior characterized with process performing label.
In one embodiment, which further includes：
Interactive controlling module (not shown), it is right for multiple continuous skeleton motion state vectors that basis is calculated Target object carries out motor behavior anticipation；According to the motor behavior prejudged out, determination is corresponding to the motor behavior realization prejudged out Interactive interbehavior logic；It is interacted according to interbehavior logical AND target object.
In one embodiment, Motion feature extraction module 1202 is additionally operable to obtain the skeleton motion shape of multiframe continuous acquisition State picture frame；For every frame skeleton motion status image frame, identify that multiple target bones are closed from skeleton motion status image frame Node；According to the tandem between multiple target skeletal joint points, latter object skeletal joint point is obtained respectively relative to preceding The spin data of one target skeletal joint point；Using each spin data as vector element, by respective objects skeletal joint Tandem between point, splicing obtain skeleton motion state vector corresponding with skeleton motion status image frame.
In one embodiment, Motion feature extraction module 1202 is additionally operable to the skeleton motion state vector according to each history Sequencing, by it is previous encode obtained skeleton motion feature vector and when time history to be encoded skeleton motion state to Amount, the encoder inputted in the machine learning model of pre-training are encoded, and output obtains and the bone when secondary history to be encoded The corresponding skeleton motion feature vector obtained when time coding of bone motion state vector.
Figure 13 is the internal structure schematic diagram of one embodiment Computer equipment.Referring to Fig.1 3, which can To be terminal or server.Terminal can be personal computer, mobile terminal, mobile unit or robot, and mobile terminal includes At least one of mobile phone, tablet computer, personal digital assistant or wearable device etc..Server can use independent server The either server cluster of multiple physical servers composition is realized.The computer equipment includes being connected by system bus Processor, memory and network interface.Wherein, memory includes non-volatile memory medium and built-in storage.The computer is set Standby non-volatile memory medium can storage program area and computer program.The computer program is performed, and may make place It manages device and executes a kind of skeleton motion prediction processing method.The processor of the computer equipment is calculated for offer and control ability, Support the operation of entire computer equipment.Computer program can be stored in the built-in storage, the computer program is by processor When execution, processor may make to execute a kind of skeleton motion prediction processing method.The network interface of computer equipment is for carrying out Network communication.
It will be understood by those skilled in the art that structure shown in Figure 13, only with the relevant part of application scheme The block diagram of structure, does not constitute the restriction for the computer equipment being applied thereon to application scheme, and specific computer is set Standby may include either combining certain components than more or fewer components as shown in the figure or being arranged with different components.
In one embodiment, skeleton motion prediction processing device provided by the present application can be implemented as a kind of computer journey The form of sequence, computer program can be run on computer equipment as shown in fig. 13 that, the non-volatile memories of computer equipment Medium can store each program module for forming the skeleton motion prediction processing device, for example, coding module shown in Fig. 10 1002, hidden state vector determining module 1004, skeleton motion state vector acquisition module 1006 and decoding prediction module 1008.The computer program that each program module is formed is for making the computer equipment execute this Shen described in this specification Step that please be in the skeleton motion prediction processing method of each embodiment, for example, computer equipment can be by as shown in Figure 10 Skeleton motion prediction processing device 1000 in coding module 1002 obtain the skeleton motion states of multiple continuous history to Amount；Condition code coding is carried out to each skeleton motion state vector respectively by machine learning model, generates and is transported respectively with each bone The corresponding skeleton motion feature vector of dynamic state vector.Computer equipment can be true by hidden state vector determining module 1004 Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion hidden state of last moment is vectorial, is Motion intention is carried out for last moment to skeleton motion feature vector to extract to obtain.Computer equipment can pass through skeleton motion State vector acquisition module 1006 obtains the skeleton motion state vector at current time, and by decoding 1008 basis of prediction module The skeleton motion state vector at the skeleton motion hidden state vector sum current time of last moment is decoded, next to calculate The skeleton motion state vector at moment.
In one embodiment, limb motion prediction processing device provided by the present application can be implemented as a kind of computer journey The form of sequence, computer program can be run on computer equipment as shown in fig. 13 that, the non-volatile memories of computer equipment Medium can store each program module for forming the limb motion prediction processing device, for example, motion feature shown in Figure 12 carries Modulus block 1202, hidden state determining module 1204 and limb motion state determining module 1206.Each program module institute group At computer program be used for make the computer equipment execute each embodiment of the application described in this specification bone fortune Step in dynamic prediction processing method, for example, computer equipment can predict processing dress by limb motion as shown in figure 12 Set the limb motion state that the Motion feature extraction module 1202 in 1200 obtains multiple continuous history；To the limb of each history Body motion state carries out Motion feature extraction respectively, obtains limb motion feature corresponding with the limb motion state of each history. Computer equipment can determine that the limb motion of the last moment at current time implies shape by hidden state determining module 1204 State；The limb motion hidden state of last moment is to carry out motion intention to limb motion feature for last moment to extract It arrives.Computer equipment can obtain the limb motion state at current time by limb motion state determining module 1206；According to The limb motion hidden state of last moment and the limb motion state at current time, determine the limb motion shape of subsequent time State.
In one embodiment, a kind of computer equipment, including memory and processor are provided, is stored in memory Computer program, when computer program is executed by processor so that processor executes following steps：Obtain multiple continuous history Skeleton motion state vector；Condition code volume is carried out to each skeleton motion state vector respectively by machine learning model Code generates skeleton motion feature vector corresponding with each skeleton motion state vector respectively；Determine upper the one of current time The skeleton motion hidden state vector at moment；The skeleton motion hidden state vector of last moment is for last moment to bone Bone motion feature vector carries out motion intention and extracts to obtain；Obtain the skeleton motion state vector at current time；According to upper a period of time The skeleton motion state vector at the skeleton motion hidden state vector sum current time at quarter is decoded, to calculate subsequent time Skeleton motion state vector.
In one embodiment, determine that the skeleton motion hidden state vector of the last moment at current time includes：It obtains The last moment at current time estimates velocity characteristic vector；Velocity characteristic vector is estimated, for characterizing the adjacent moment estimated Skeleton motion state vector between variation；Determine each skeleton motion feature vector and last moment respectively estimates speed spy Motion relevance between sign vector；The weight of each skeleton motion feature vector is determined according to motion relevance；Weight and movement Correlation positive correlation；Each skeleton motion feature vector is weighted summation according to corresponding weight respectively, obtains current time Last moment skeleton motion hidden state vector.
In one embodiment, determine each skeleton motion feature vector and last moment respectively estimates velocity characteristic vector Between motion relevance include：Each skeleton motion feature vector of velocity characteristic vector sum of estimating of last moment is inputted into machine Attention model in learning model determines the pre- of each skeleton motion feature vector and last moment according to attention model respectively Estimate the motion relevance between velocity characteristic vector.
In one embodiment, according to the skeleton motion at the skeleton motion hidden state vector sum current time of last moment State vector is decoded, and the skeleton motion state vector to calculate subsequent time includes：The skeleton motion of last moment is hidden It is decoded, obtains containing the decoder in state vector and the skeleton motion state vector at current time input machine learning model Current time estimates velocity characteristic vector；According to the skeleton motion for estimating velocity characteristic vector sum current time at current time State vector calculates the skeleton motion state vector of subsequent time.
In one embodiment, by the skeleton motion shape at the skeleton motion hidden state vector sum current time of last moment Decoder in state vector input machine learning model is decoded, and the velocity characteristic vector of estimating for obtaining current time includes： According to following formula determine current time estimate velocity characteristic vector：
Wherein, t is current time, and t-1 is the last moment at current time；vtFor current time estimate velocity characteristic to Amount；ht-1For the skeleton motion hidden state vector of last moment；φ indicates line rectification function；Wv、Uvh、bvAnd bvhIt is The offset parameter that pre-training obtains in machine learning model；For the skeleton motion state vector at current time.
In one embodiment, according to the skeleton motion state for estimating velocity characteristic vector sum current time at current time Vector, the skeleton motion state vector for calculating subsequent time include：The skeleton motion shape with current time is obtained by decoder The corresponding first prediction weight vectors of state vector；According to the first prediction weight vectors, determine current time estimates velocity characteristic Second prediction weight vectors of vector；First prediction weight vectors and second prediction weight vectors and for it is complete one vector；It will work as The skeleton motion state vector at preceding moment and current time estimate velocity characteristic vector, respectively with corresponding first prediction weight Vector sum second is predicted to be added after weight vectors carry out dot product, obtains the skeleton motion state vector of subsequent time.
In one embodiment, by the skeleton motion state vector at current time and current time estimate velocity characteristic to Amount is added after carrying out dot product with corresponding first prediction weight vectors and the second prediction weight vectors respectively, is predicted down The skeleton motion state vector at one moment includes：The skeleton motion state vector of subsequent time is calculated according to following formula：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ is indicated Sigmoid functions；φ indicates line rectification function；For the skeleton motion state vector at current time；vtIt is pre- for current time Estimate velocity characteristic vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in machine learning model；T+1 is current The subsequent time at moment；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol.
In one embodiment, computer program also makes processor execute following steps：The bone really generated is transported The sequencing that dynamic state vector is generated according to skeleton motion state, is divided into historical sample and forecast sample；According to history sample Skeleton motion state vector in this carries out machine learning model training, and output is expressed by the model parameter of machine learning model Skeleton motion state vector；According to the skeleton motion state in the skeleton motion state vector and forecast sample of model parameter expression Vector structure loss function；Loss function is used to indicate phase in the skeleton motion state vector that model parameter is expressed and forecast sample Answer the difference degree between skeleton motion state vector；Model parameter when loss function is minimized is as machine learning model Stable model parameter.
In one embodiment, it is transported according to the bone in the skeleton motion state vector and forecast sample of model parameter expression Dynamic state vector builds loss function：Per bone adjacent two-by-two in the skeleton motion state vector expressed for model parameter Bone motion state vector is spliced, and the first splicing vector is obtained；Splice the transposition of vector according to the first splicing vector sum first The apposition of obtained vector, obtains the first matrix；Skeleton motion state vector adjacent two-by-two in forecast sample is spliced, Obtain the second splicing vector；The apposition for splicing the vector that the transposition of vector obtains according to the second splicing vector sum second, obtains the Two matrixes；According to the mean square deviation of corresponding first matrix and the second matrix, loss function is obtained.
In one embodiment, computer program also makes processor execute following steps：Obtain the bone with last moment Bone moves the corresponding behavior label vector of hidden state vector；Behavior label vector is by obtaining behavior label coding；According to upper The skeleton motion state vector at the skeleton motion hidden state vector sum current time at one moment is decoded, to calculate lower a period of time The skeleton motion state vector at quarter includes：Behavior label vector is spliced to the skeleton motion hidden state of corresponding last moment Vector；According to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of spliced last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time；The skeleton motion state vector institute of calculated subsequent time The skeleton motion state of characterization matches with behavior label.
In one embodiment, computer program also makes processor execute following steps：According to predict and behavior The skeleton motion state vector that label matches generates the control instruction for target object；Control instruction is used to indicate target Object carries out corresponding sports, the behavior characterized with process performing label according to the skeleton motion state vector predicted.
In one embodiment, computer program also makes processor execute following steps：It is multiple according to what is be calculated Continuous skeleton motion state vector carries out motor behavior anticipation to target object；According to the motor behavior prejudged out, determine with The motor behavior prejudged out realizes accordingly interactive interbehavior logic；It is handed over according to interbehavior logical AND target object Mutually.
In one embodiment, computer program also makes processor execute following steps：Obtain multiframe continuous acquisition Skeleton motion status image frame；For every frame skeleton motion status image frame, identified from skeleton motion status image frame multiple Target skeletal joint point；According to the tandem between multiple target skeletal joint points, latter object skeletal joint is obtained respectively Spin data of the point relative to previous target skeletal joint point；Using each spin data as vector element, by corresponding mesh Mark skeletal joint point between tandem, splicing obtain skeleton motion state corresponding with skeleton motion status image frame to Amount.
In one embodiment, condition code volume is carried out to each skeleton motion state vector by machine learning model respectively Code, generating skeleton motion feature vector corresponding with each skeleton motion state vector respectively includes：It is transported according to the bone of each history The sequencing of dynamic state vector encodes obtained skeleton motion feature vector and when secondary skeleton motion shape to be encoded by previous State vector, the encoder inputted in the machine learning model of pre-training are encoded, and output obtains and works as secondary bone to be encoded The corresponding skeleton motion feature vector obtained when time coding of motion state vector.
In one embodiment, a kind of storage medium being stored with computer program is provided, computer program is handled When device executes so that processor executes following steps：Obtain the skeleton motion state vector of multiple continuous history；Pass through machine Learning model carries out condition code coding to each skeleton motion state vector respectively, generate respectively with each skeleton motion shape The corresponding skeleton motion feature vector of state vector；Determine the skeleton motion hidden state vector of the last moment at current time；On The skeleton motion hidden state vector at one moment, is to carry out motion intention extraction to skeleton motion feature vector for last moment It obtains；Obtain the skeleton motion state vector at current time；It is current according to the skeleton motion hidden state vector sum of last moment The skeleton motion state vector at moment is decoded, to calculate the skeleton motion state vector of subsequent time.
In one embodiment, determine that the skeleton motion hidden state vector of the last moment at current time includes：It obtains The last moment at current time estimates velocity characteristic vector；Velocity characteristic vector is estimated, for characterizing the adjacent moment estimated Skeleton motion state vector between variation；Determine each skeleton motion feature vector and last moment respectively estimates speed spy Motion relevance between sign vector；The weight of each skeleton motion feature vector is determined according to motion relevance；Weight and movement Correlation positive correlation；Each skeleton motion feature vector is weighted summation according to corresponding weight respectively, obtains current time Last moment skeleton motion hidden state vector.
In one embodiment, determine each skeleton motion feature vector and last moment respectively estimates velocity characteristic vector Between motion relevance include：Each skeleton motion feature vector of velocity characteristic vector sum of estimating of last moment is inputted into machine Attention model in learning model determines the pre- of each skeleton motion feature vector and last moment according to attention model respectively Estimate the motion relevance between velocity characteristic vector.
In one embodiment, according to the skeleton motion at the skeleton motion hidden state vector sum current time of last moment State vector is decoded, and the skeleton motion state vector to calculate subsequent time includes：The skeleton motion of last moment is hidden It is decoded, obtains containing the decoder in state vector and the skeleton motion state vector at current time input machine learning model Current time estimates velocity characteristic vector；According to the skeleton motion for estimating velocity characteristic vector sum current time at current time State vector calculates the skeleton motion state vector of subsequent time.
In one embodiment, by the skeleton motion shape at the skeleton motion hidden state vector sum current time of last moment Decoder in state vector input machine learning model is decoded, and the velocity characteristic vector of estimating for obtaining current time includes： According to following formula determine current time estimate velocity characteristic vector：
Wherein, t is current time, and t-1 is the last moment at current time；vtFor current time estimate velocity characteristic to Amount；ht-1For the skeleton motion hidden state vector of last moment；φ indicates line rectification function；Wv、Uvh、bvAnd bvhIt is The offset parameter that pre-training obtains in machine learning model；For the skeleton motion state vector at current time.
In one embodiment, according to the skeleton motion state for estimating velocity characteristic vector sum current time at current time Vector, the skeleton motion state vector for calculating subsequent time include：The skeleton motion shape with current time is obtained by decoder The corresponding first prediction weight vectors of state vector；According to the first prediction weight vectors, determine current time estimates velocity characteristic Second prediction weight vectors of vector；First prediction weight vectors and second prediction weight vectors and for it is complete one vector；It will work as The skeleton motion state vector at preceding moment and current time estimate velocity characteristic vector, respectively with corresponding first prediction weight Vector sum second is predicted to be added after weight vectors carry out dot product, obtains the skeleton motion state vector of subsequent time.
In one embodiment, by the skeleton motion state vector at current time and current time estimate velocity characteristic to Amount is added after carrying out dot product with corresponding first prediction weight vectors and the second prediction weight vectors respectively, is predicted down The skeleton motion state vector at one moment includes：The skeleton motion state vector of subsequent time is calculated according to following formula：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ is indicated Sigmoid functions；φ indicates line rectification function；For the skeleton motion state vector at current time；vtIt is pre- for current time Estimate velocity characteristic vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in machine learning model；T+1 is current The subsequent time at moment；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol.
In one embodiment, computer program also makes processor execute following steps：The bone really generated is transported The sequencing that dynamic state vector is generated according to skeleton motion state, is divided into historical sample and forecast sample；According to history sample Skeleton motion state vector in this carries out machine learning model training, and output is expressed by the model parameter of machine learning model Skeleton motion state vector；According to the skeleton motion state in the skeleton motion state vector and forecast sample of model parameter expression Vector structure loss function；Loss function is used to indicate phase in the skeleton motion state vector that model parameter is expressed and forecast sample Answer the difference degree between skeleton motion state vector；Model parameter when loss function is minimized is as machine learning model Stable model parameter.
In one embodiment, it is transported according to the bone in the skeleton motion state vector and forecast sample of model parameter expression Dynamic state vector builds loss function：Per bone adjacent two-by-two in the skeleton motion state vector expressed for model parameter Bone motion state vector is spliced, and the first splicing vector is obtained；Splice the transposition of vector according to the first splicing vector sum first The apposition of obtained vector, obtains the first matrix；Skeleton motion state vector adjacent two-by-two in forecast sample is spliced, Obtain the second splicing vector；The apposition for splicing the vector that the transposition of vector obtains according to the second splicing vector sum second, obtains the Two matrixes；According to the mean square deviation of corresponding first matrix and the second matrix, loss function is obtained.
In one embodiment, computer program also makes processor execute following steps：Obtain the bone with last moment Bone moves the corresponding behavior label vector of hidden state vector；Behavior label vector is by obtaining behavior label coding；According to upper The skeleton motion state vector at the skeleton motion hidden state vector sum current time at one moment is decoded, to calculate lower a period of time The skeleton motion state vector at quarter includes：Behavior label vector is spliced to the skeleton motion hidden state of corresponding last moment Vector；According to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of spliced last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time；The skeleton motion state vector institute of calculated subsequent time The skeleton motion state of characterization matches with behavior label.
In one embodiment, computer program also makes processor execute following steps：According to predict and behavior The skeleton motion state vector that label matches generates the control instruction for target object；Control instruction is used to indicate target Object carries out corresponding sports, the behavior characterized with process performing label according to the skeleton motion state vector predicted.
In one embodiment, computer program also makes processor execute following steps：It is multiple according to what is be calculated Continuous skeleton motion state vector carries out motor behavior anticipation to target object；According to the motor behavior prejudged out, determine with The motor behavior prejudged out realizes accordingly interactive interbehavior logic；It is handed over according to interbehavior logical AND target object Mutually.
In one embodiment, computer program also makes processor execute following steps：Obtain multiframe continuous acquisition Skeleton motion status image frame；For every frame skeleton motion status image frame, identified from skeleton motion status image frame multiple Target skeletal joint point；According to the tandem between multiple target skeletal joint points, latter object skeletal joint is obtained respectively Spin data of the point relative to previous target skeletal joint point；Using each spin data as vector element, by corresponding mesh Mark skeletal joint point between tandem, splicing obtain skeleton motion state corresponding with skeleton motion status image frame to Amount.
In one embodiment, condition code volume is carried out to each skeleton motion state vector by machine learning model respectively Code, generating skeleton motion feature vector corresponding with each skeleton motion state vector respectively includes：It is transported according to the bone of each history The sequencing of dynamic state vector encodes obtained skeleton motion feature vector and when secondary skeleton motion shape to be encoded by previous State vector, the encoder inputted in the machine learning model of pre-training are encoded, and output obtains and works as secondary bone to be encoded The corresponding skeleton motion feature vector obtained when time coding of motion state vector.
It should be understood that although each step in each embodiment of the application is not necessarily to be indicated according to step numbers Sequence execute successively.Unless expressly stating otherwise herein, there is no stringent sequences to limit for the execution of these steps, these Step can execute in other order.Moreover, in each embodiment at least part step may include multiple sub-steps or Multiple stages, these sub-steps or stage are not necessarily to execute completion in synchronization, but can be at different times Execute, these sub-steps either the stage execution sequence be also not necessarily carry out successively but can with other steps or its At least part in the sub-step of its step either stage executes in turn or alternately.
One of ordinary skill in the art will appreciate that realizing all or part of flow in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the program can be stored in a non-volatile computer and can be read In storage medium, the program is when being executed, it may include such as the flow of the embodiment of above-mentioned each method.Wherein, provided herein Each embodiment used in any reference to memory, storage, database or other media, may each comprise non-volatile And/or volatile memory.Nonvolatile memory may include that read-only memory (ROM), programming ROM (PROM), electricity can be compiled Journey ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) directly RAM (RDRAM), straight Connect memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
Each technical characteristic of above example can be combined arbitrarily, to keep description succinct, not to above-described embodiment In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance Shield is all considered to be the range of this specification record.
Only several embodiments of the present invention are expressed for above example, the description thereof is more specific and detailed, but can not Therefore it is construed as limiting the scope of the patent.It should be pointed out that for those of ordinary skill in the art, Under the premise of not departing from present inventive concept, various modifications and improvements can be made, these are all within the scope of protection of the present invention. Therefore, the protection domain of patent of the present invention should be determined by the appended claims.

Claims (15)

1. a kind of skeleton motion prediction processing method, the method includes：
Obtain the skeleton motion state vector of multiple continuous history；
Condition code coding is carried out to each skeleton motion state vector respectively by machine learning model, generate respectively with each institute State the corresponding skeleton motion feature vector of skeleton motion state vector；
Determine the skeleton motion hidden state vector of the last moment at current time；The skeleton motion of the last moment implies shape State vector is to carry out motion intention to the skeleton motion feature vector for the last moment to extract to obtain；
Obtain the skeleton motion state vector at current time；
It is solved according to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of the last moment Code, to calculate the skeleton motion state vector of subsequent time.
2. according to the method described in claim 1, it is characterized in that, the skeleton motion of the last moment at the determining current time Hidden state vector includes：
Obtain the last moment at current time estimates velocity characteristic vector；The velocity characteristic of estimating is vectorial, pre- for characterizing Variation between the skeleton motion state vector for the adjacent moment estimated；
The movement of each skeleton motion feature vector and the last moment estimated between velocity characteristic vector is determined respectively Correlation；
The weight of each skeleton motion feature vector is determined according to the motion relevance；The weight and motion relevance are just It is related；
Each skeleton motion feature vector is weighted summation according to the corresponding weight respectively, obtains current time The skeleton motion hidden state vector of last moment.
3. according to the method described in claim 2, it is characterized in that, it is described respectively determine each skeleton motion feature vector with The motion relevance of the last moment estimated between velocity characteristic vector includes：
Each skeleton motion feature vector of velocity characteristic vector sum of estimating of the last moment is inputted into the machine learning Attention model in model determines the pre- of each skeleton motion feature vector and last moment respectively according to the attention model Estimate the motion relevance between velocity characteristic vector.
4. according to the method described in claim 1, it is characterized in that, the skeleton motion according to the last moment implies shape The skeleton motion state vector at state vector sum current time is decoded, to calculate the skeleton motion state vector packet of subsequent time It includes：
Described in skeleton motion state vector input by the skeleton motion hidden state vector sum current time of the last moment Decoder in machine learning model is decoded, and obtain current time estimates velocity characteristic vector；
According to the skeleton motion state vector for estimating velocity characteristic vector sum current time at current time, subsequent time is calculated Skeleton motion state vector.
5. according to the method described in claim 4, it is characterized in that, described estimate velocity characteristic vector sum according to current time The skeleton motion state vector at current time, the skeleton motion state vector for calculating subsequent time include：
The first prediction weight vectors corresponding with the skeleton motion state vector at current time are obtained by the decoder；
According to it is described first predict weight vectors, determine current time estimate velocity characteristic vector second prediction weight to Amount；It is described first prediction weight vectors and it is described second prediction weight vectors and for it is complete one vector；
The skeleton motion state vector at current time and current time are estimated into velocity characteristic vector, respectively with corresponding first Prediction weight vectors and the second prediction weight vectors are added after carrying out dot product, obtain the skeleton motion state vector of subsequent time.
6. according to the method described in claim 5, it is characterized in that, described by the skeleton motion state vector at current time and work as The preceding moment estimates velocity characteristic vector, is carried out a little with corresponding first prediction weight vectors and the second prediction weight vectors respectively It is added after multiplying, the skeleton motion state vector for obtaining subsequent time includes：
The skeleton motion state vector of subsequent time is calculated according to following formula：
Wherein, t is current time；ztFor the first prediction weight vectors；1-ztFor the second prediction weight vectors；σ indicates Sigmoid Function；φ indicates line rectification function；For the skeleton motion state vector at current time；vtSpeed is estimated for current time Feature vector；Wz、Uzx、bzAnd bzxIt is the offset parameter that pre-training obtains in the machine learning model；When t+1 is current The subsequent time at quarter；For the skeleton motion state vector of the subsequent time at current time；⊙ is vector dot symbol.
7. according to the method described in claim 1, it is characterized in that, further including：
The sequencing that the skeleton motion state vector really generated is generated according to skeleton motion state, is divided into historical sample And forecast sample；
Machine learning model training is carried out according to the skeleton motion state vector in historical sample, is exported by machine learning model The skeleton motion state vector of model parameter expression；
According to the skeleton motion state vector structure in the skeleton motion state vector and the forecast sample of model parameter expression Loss function；The loss function is used to indicate the skeleton motion state vector of the model parameter expression and the forecast sample In difference degree between corresponding skeleton motion state vector；
The model parameter that model parameter when loss function is minimized is stablized as machine learning model.
8. the method according to the description of claim 7 is characterized in that it is described according to model parameter expression skeleton motion state to Skeleton motion state vector in amount and the forecast sample builds loss function：
Spliced per skeleton motion state vector adjacent two-by-two in the skeleton motion state vector of model parameter expression, Obtain the first splicing vector；
Splice the vectorial apposition that the transposition of the first splicing vector described in vector sum obtains according to described first, obtains the first square Battle array；
Skeleton motion state vector adjacent two-by-two in forecast sample is spliced, the second splicing vector is obtained；
Splice the vectorial apposition that the transposition of the second splicing vector described in vector sum obtains according to described second, obtains the second square Battle array；
According to the mean square deviation of corresponding first matrix and second matrix, loss function is obtained.
9. method according to any one of claim 1 to 8, which is characterized in that further include：
Obtain behavior label vector corresponding with the skeleton motion hidden state vector of the last moment；The behavior label to Amount is by obtaining behavior label coding；
The skeleton motion state vector at the skeleton motion hidden state vector sum current time according to the last moment into Row decoding, the skeleton motion state vector to calculate subsequent time include：
The behavior label vector is spliced to the skeleton motion hidden state vector of corresponding last moment；
According to the skeleton motion state vector at the skeleton motion hidden state vector sum current time of spliced last moment into Row decoding, to calculate the skeleton motion state vector of subsequent time；The skeleton motion state of the calculated subsequent time to The characterized skeleton motion state of amount matches with the behavior label.
10. according to the method described in claim 9, it is characterized in that, further including：
According to the calculated skeleton motion state vector to match with the behavior label, the control for target object is generated Instruction；The control instruction is used to indicate the target object and is carried out accordingly according to the calculated skeleton motion state vector Movement, to execute the behavior that the behavior label is characterized.
11. method according to any one of claim 1 to 8, which is characterized in that further include：
According to the multiple continuous skeleton motion state vectors being calculated, motor behavior anticipation is carried out to target object；
According to the motor behavior prejudged out, determine that interactive interbehavior corresponding to the motor behavior realization prejudged out is patrolled Volume；
It is interacted according to target object described in the interbehavior logical AND.
12. method according to any one of claim 1 to 8, which is characterized in that further include：
Obtain the skeleton motion status image frame of multiframe continuous acquisition；
For skeleton motion status image frame described in every frame, multiple target bones are identified from the skeleton motion status image frame Artis；
According to the tandem between the multiple target skeletal joint point, obtain respectively latter object skeletal joint point relative to The spin data of previous target skeletal joint point；
It is spelled by the tandem between respective objects skeletal joint point using each spin data as vector element It connects to obtain skeleton motion state vector corresponding with the skeleton motion status image frame.
13. method according to any one of claim 1 to 8, which is characterized in that described to be distinguished by machine learning model Condition code coding is carried out to each skeleton motion state vector, is generated corresponding with each skeleton motion state vector respectively Skeleton motion feature vector includes：
According to the sequencing of the skeleton motion state vector of each history, by it is previous encode obtained skeleton motion feature to Amount with described when secondary skeleton motion state vector to be encoded, compiled by the encoder inputted in the machine learning model of pre-training Code, output obtain and the skeleton motion feature worked as time coding when secondary skeleton motion state vector to be encoded is corresponding and obtained Vector.
14. a kind of limb motion prediction processing method, the method includes：
Obtain the limb motion state of multiple continuous history；
Motion feature extraction is carried out respectively to the limb motion state of each history, obtains transporting with the limbs of each history The corresponding limb motion feature of dynamic state；
Determine the limb motion hidden state of the last moment at current time；The limb motion hidden state of institute's last moment is Motion intention is carried out for the last moment to the limb motion feature to extract to obtain；
Obtain the limb motion state at current time；
According to the limb motion state of the limb motion hidden state of the last moment and current time, subsequent time is determined Limb motion state.
15. a kind of skeleton motion prediction processing device, which is characterized in that described device includes：
Coding module, multiple continuous skeleton motion state vectors for observing history input the machine learning mould of pre-training In type, corresponding skeleton motion feature vector is obtained to carry out feature coding respectively；
Hidden state vector determining module, the skeleton motion hidden state vector of the last moment for determining current time；Institute The skeleton motion hidden state vector for stating last moment, was carried out to the skeleton motion feature vector for the last moment Motion intention is extracted to obtain；
Skeleton motion state vector acquisition module, the skeleton motion state vector for obtaining current time；
Prediction module is decoded, the bone for the skeleton motion hidden state vector sum current time according to the last moment is transported Dynamic state vector is decoded, to predict the skeleton motion state vector of subsequent time.
CN201810055213.2A 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method Active CN108304790B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201810055213.2A CN108304790B (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method
CN202210143192.6A CN114495283A (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810055213.2A CN108304790B (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method

Related Child Applications (1)

Application Number Title Priority Date Filing Date
CN202210143192.6A Division CN114495283A (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method

Publications (2)

Publication Number Publication Date
CN108304790A true CN108304790A (en) 2018-07-20
CN108304790B CN108304790B (en) 2022-03-18

Family

ID=62865970

Family Applications (2)

Application Number Title Priority Date Filing Date
CN202210143192.6A Pending CN114495283A (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method
CN201810055213.2A Active CN108304790B (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN202210143192.6A Pending CN114495283A (en) 2018-01-19 2018-01-19 Skeletal motion prediction processing method and device and limb motion prediction processing method

Country Status (1)

Country Link
CN (2) CN114495283A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110310350A (en) * 2019-06-24 2019-10-08 清华大学 Action prediction generation method and device based on animation
CN110827226A (en) * 2019-11-13 2020-02-21 北京迈格威科技有限公司 Skeleton point smoothing method and device and electronic equipment
WO2021012348A1 (en) * 2019-07-23 2021-01-28 深圳大学 Method for generating object attribute recognition model, storage medium and electronic device
CN116453220A (en) * 2023-04-19 2023-07-18 北京百度网讯科技有限公司 Target object posture determining method, training device and electronic equipment
CN117238026A (en) * 2023-07-10 2023-12-15 中国矿业大学 Gesture reconstruction interactive behavior understanding method based on skeleton and image features

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101216885A (en) * 2008-01-04 2008-07-09 中山大学 Passerby face detection and tracing algorithm based on video
US20090052740A1 (en) * 2007-08-24 2009-02-26 Kabushiki Kaisha Toshiba Moving object detecting device and mobile robot
CN103310188A (en) * 2012-03-06 2013-09-18 三星电子株式会社 Method and apparatus for pose recognition
CN103500456A (en) * 2013-10-22 2014-01-08 北京大学 Object tracking method and equipment based on dynamic Bayes model network
US20180018508A1 (en) * 2015-01-29 2018-01-18 Unifai Holdings Limited Computer vision systems

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090052740A1 (en) * 2007-08-24 2009-02-26 Kabushiki Kaisha Toshiba Moving object detecting device and mobile robot
CN101216885A (en) * 2008-01-04 2008-07-09 中山大学 Passerby face detection and tracing algorithm based on video
CN103310188A (en) * 2012-03-06 2013-09-18 三星电子株式会社 Method and apparatus for pose recognition
CN103500456A (en) * 2013-10-22 2014-01-08 北京大学 Object tracking method and equipment based on dynamic Bayes model network
US20180018508A1 (en) * 2015-01-29 2018-01-18 Unifai Holdings Limited Computer vision systems

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110310350A (en) * 2019-06-24 2019-10-08 清华大学 Action prediction generation method and device based on animation
WO2021012348A1 (en) * 2019-07-23 2021-01-28 深圳大学 Method for generating object attribute recognition model, storage medium and electronic device
CN110827226A (en) * 2019-11-13 2020-02-21 北京迈格威科技有限公司 Skeleton point smoothing method and device and electronic equipment
CN110827226B (en) * 2019-11-13 2022-09-27 北京迈格威科技有限公司 Skeleton point smoothing method and device and electronic equipment
CN116453220A (en) * 2023-04-19 2023-07-18 北京百度网讯科技有限公司 Target object posture determining method, training device and electronic equipment
CN116453220B (en) * 2023-04-19 2024-05-10 北京百度网讯科技有限公司 Target object posture determining method, training device and electronic equipment
CN117238026A (en) * 2023-07-10 2023-12-15 中国矿业大学 Gesture reconstruction interactive behavior understanding method based on skeleton and image features
CN117238026B (en) * 2023-07-10 2024-03-08 中国矿业大学 Gesture reconstruction interactive behavior understanding method based on skeleton and image features

Also Published As

Publication number Publication date
CN114495283A (en) 2022-05-13
CN108304790B (en) 2022-03-18

Similar Documents

Publication Publication Date Title
CN108304790A (en) Skeleton motion prediction processing method, device and limb motion prediction processing method
Villegas et al. Hierarchical long-term video prediction without supervision
Ghosh et al. Steer: Simple temporal regularization for neural ode
CN112001914B (en) Depth image complement method and device
CN109271933B (en) Method for estimating three-dimensional human body posture based on video stream
Zhao et al. Self-supervised pretraining for RGB-D salient object detection
CN109464803A (en) Virtual objects controlled, model training method, device, storage medium and equipment
Shuai et al. Adaptive multi-view and temporal fusing transformer for 3d human pose estimation
CN114663496B (en) Monocular vision odometer method based on Kalman pose estimation network
CN111461437B (en) Data-driven crowd motion simulation method based on generation of countermeasure network
CN114663509B (en) Self-supervision monocular vision odometer method guided by key point thermodynamic diagram
Xu et al. Vitpose++: Vision transformer for generic body pose estimation
Hu et al. Forward-reverse adaptive graph convolutional networks for skeleton-based action recognition
Vendrow et al. Somoformer: Multi-person pose forecasting with transformers
CN114170353B (en) Multi-condition control dance generation method and system based on neural network
Su et al. Narrowing the coordinate-frame gap in behavior prediction models: Distillation for efficient and accurate scene-centric motion forecasting
CN114723784A (en) Pedestrian motion trajectory prediction method based on domain adaptation technology
CN113128446A (en) Human body posture estimation method based on belief map enhanced network
Kiasari et al. Human action generation with generative adversarial networks
Wezeman et al. Distance-based classifier on the Quantum Inspire
Tang et al. A deep map transfer learning method for face recognition in an unrestricted smart city environment
Yuan et al. Jointpose: Jointly optimizing evolutionary data augmentation and prediction neural network for 3D human pose estimation
CN115544307A (en) Directed graph data feature extraction and expression method and system based on incidence matrix
CN112017265A (en) Virtual human motion simulation method based on graph neural network
CN113222100A (en) Training method and device of neural network model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant