CN114187393A - ResNet-LSTM-based movement data redirection method - Google Patents

ResNet-LSTM-based movement data redirection method Download PDF

Info

Publication number
CN114187393A
CN114187393A CN202111512627.1A CN202111512627A CN114187393A CN 114187393 A CN114187393 A CN 114187393A CN 202111512627 A CN202111512627 A CN 202111512627A CN 114187393 A CN114187393 A CN 114187393A
Authority
CN
China
Prior art keywords
lstm
resnet
network
neural network
sample set
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111512627.1A
Other languages
Chinese (zh)
Inventor
杨晶
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sichuan Qiruike Technology Co Ltd
Original Assignee
Sichuan Qiruike Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sichuan Qiruike Technology Co Ltd filed Critical Sichuan Qiruike Technology Co Ltd
Priority to CN202111512627.1A priority Critical patent/CN114187393A/en
Publication of CN114187393A publication Critical patent/CN114187393A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The invention discloses a ResNet-LSTM-based movement data redirection method, which integrates the advantages of high prediction precision of ResNet and the advantage of LSTM in favor of processing sequence data by connecting a ResNet-based convolution neural network and an LSTM-based circulation neural network, thereby improving the accuracy and smooth naturalness of movement data output by the network. The invention maps the motion data of the source role into the motion data of the target role with different topological structures, thereby effectively promoting the multiplexing of the motion data.

Description

ResNet-LSTM-based movement data redirection method
Technical Field
The invention relates to the technical field of computer three-dimensional animation, in particular to a ResNet-LSTM-based motion data redirection method.
Background
Human skeletal motion data is widely used in the field of character animation, and the driving of three-dimensional models by skeletal motion data is the main mode of character animation. The collection of the bone motion data mainly depends on motion capture equipment on the market, and although the bone motion data captured by the motion capture equipment is high in precision, natural and real, the captured data is limited; the skeleton topological structure and the length proportion of the character in the movie animation are various, and the requirement on motion data is unlimited. Therefore, redirecting existing source character skeletal motion data to a target character with a different skeletal structure or skeletal length scale is a solution to this problem.
At present, various researches on motion reorientation exist, but the problems of huge calculation amount, poor visual quality and the like exist:
the core of the conventional numerical analysis method represented by IK is to apply various constraints on motion values to realize redirection. Common constraint types include spatio-temporal constraints, kinematic constraints, kinetic constraints. But because of many constraints and complicated calculation, the redirection time is long, and the movement lacks physical reality.
Holden utilizes inverse kinematics to establish network constraints, uses a convolutional neural network to establish a connection between motion control constraints and hidden motion characteristics, and generates three-dimensional motion of a role by a two-dimensional foot joint track.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides a method for realizing the operation redirection based on ResNet-LSTM, and solves the problems of large calculation amount and poor visual quality in the prior art.
The invention realizes the purpose through the following technical scheme:
a ResNet-LSTM-based motion data redirection method comprises the following steps:
step 1, acquiring a training sample set and a test sample set; the training sample set and the testing sample set respectively comprise role movement segments with different skeleton topological structures of a source and a target, and the movement types respectively comprise walking and running;
step 2, training a ResNet-LSTM network by using the training sample set with the minimization of a total loss function as an optimization target to obtain the trained ResNet-LSTM network;
and 3, acquiring the motion data of the source role to be redirected by using the test data set, and inputting the test data into the trained ResNet-LSTM network to obtain the redirected motion data.
Further, in the step 1, selecting motion segments with motion types of walking and running from a CMU database, and redirecting the motion segments of walking and running to two roles with different skeleton topological structures, namely a source role and a target role, by using an IK method; taking 60% of the source role and the target role as a training sample set, and taking 40% of the source role and the target role as a testing sample set; the training sample set and the testing sample set respectively comprise two role movement segments with different skeleton topological structures, namely a source role and a target role, and the movement types comprise walking and running.
Further, the step 2 specifically comprises:
constructing a ResNet-LSTM network; the ResNet-LSTM network comprises a ResNet-based convolutional neural network and an LSTM-based cyclic neural network;
firstly, inputting the motion data of a source role in a training sample set into the convolutional neural network based on ResNet to obtain the output data of the convolutional neural network; secondly, inputting the output data of the convolutional neural network into the LSTM-based cyclic neural network to obtain the output data of the cyclic neural network;
calculating the mean square error and the skeleton length error of the current ResNet network according to the output data of the cyclic neural network and the motion data of the target role in the training sample set;
according to the formula: loss ═ λmLossm(Y)+λbLossb(Y) updating the total Loss function, where Loss denotes the total Loss function, λm、λbRespectively representing a mean square error weight parameter, a smoothness error weight parameter, Lossm(Y)、Losss(Y) represents a mean square error and a smoothness error, respectively;
and dynamically adjusting weight parameters and a learning rate through a back propagation algorithm of errors and an adaptive moment estimation optimization algorithm, and obtaining a trained ResNet-LSTM network by taking the minimization of a total error loss function as a target.
Further, in the step 3, the loss function of the mean square error constraint is
Figure BDA0003398729370000031
Figure BDA0003398729370000032
Wherein f is the total frame number of the training data, YLSTMRepresenting the output of a ResNet-LSTM based network, X being the corresponding target character motion data;
the smoothness constraint penalty function is:
Figure BDA0003398729370000033
repeating the boundary elements of the output of the ResNet-LSTM based network to Y'LSTM：Y′LSTM(1，k)＝Y′LSTM(2，k)＝YLSTM(i，k)，Y′LSTM(f+1，k)＝Y′LSTM(f+2，k)＝YLSTM(f，k)，
Figure BDA0003398729370000034
Figure BDA0003398729370000035
K is more than or equal to 1 and less than or equal to 177, and i is more than or equal to 2 and less than or equal to f-1. The smoothing matrix O is:
Figure BDA0003398729370000036
further, the ResNet-LSTM network comprises a convolution neural network based on ResNet and a circulation neural network based on LSTM;
the convolutional neural network based on ResNet comprises four layers of fully-connected networks and a ResNet network; the LSTM-based recurrent neural network comprises two layers of fully-connected networks and an LSTM long-time memory unit.
The invention has the beneficial effects that:
the invention maps the motion data of the source role into the motion data of the target role with different topological structures, thereby effectively promoting the multiplexing of the motion data.
According to the invention, the ResNet-based convolutional neural network and the LSTM-based cyclic neural network are connected, so that the high prediction precision of ResNet and the advantage of LSTM that the LSTM is beneficial to processing sequence data are integrated, and the accuracy and smooth naturalness of the motion data output by the network are improved.
The invention solves the mapping relation of heterogeneous skeleton model data by adopting mean square error constraint and skeleton length error constraint, and can improve the consistency of network output data and target role motion data.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the embodiments or the drawings needed to be practical in the prior art description, and obviously, the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a method provided by an embodiment of the present invention;
fig. 2 is a block diagram of a ResNet-LSTM network according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the examples given herein without any inventive step, are within the scope of the present invention.
In any embodiment, as shown in fig. 1-2, a method for redirecting motion data based on ResNet-LSTM of the present invention includes:
101, obtaining a training sample set, wherein the training sample set and the testing sample set both include two role movement segments with different skeletal topological structures of a source and a target, and the movement types both include walking and running, and specifically include:
selecting motion segments with the motion types of walking and running from a CMU database, and redirecting the motion segments of walking and running to two roles with different skeletal topologies, namely a source role and a target role by using an IK method; taking 60% of the source role and the target role as a training sample set, and taking 40% of the source role and the target role as a testing sample set; the training sample set and the testing sample set respectively comprise two role movement segments with different skeleton topological structures, namely a source role and a target role, and the movement types comprise walking and running;
step 102, training a ResNet-LSTM network by using the training sample set with the total loss function minimization as an optimization target, and obtaining the trained ResNet-LSTM network, which specifically comprises:
constructing a ResNet-LSTM network; the ResNet-LSTM network comprises a ResNet-based convolutional neural network and an LSTM-based cyclic neural network;
training the ResNet-LSTM network according to the training sample set to obtain a trained ResNet-LSTM network model, which specifically comprises the following steps:
training the ResNet-LSTM network according to the training sample set: firstly, inputting the motion data of a source role in a training sample set into the convolutional neural network based on ResNet to obtain the output data of the convolutional neural network; secondly, inputting the output data of the convolutional neural network into the LSTM-based cyclic neural network to obtain the output data of the cyclic neural network;
calculating the mean square error and the skeleton length error of the current ResNet network according to the output data of the cyclic neural network and the motion data of the target role in the training sample set;
according to the formula: loss ═ λmLossm(Y)+λbLossb(Y) updating the total loss function. Where Loss denotes the total Loss function, λm、λbRespectively representing a mean square error weight parameter, a smoothness error weight parameter, Lossm(Y)、Losss(Y) represents a mean square error and a smoothness error, respectively;
and dynamically adjusting weight parameters and a learning rate through a back propagation algorithm of errors and an adaptive moment estimation optimization algorithm, and obtaining a trained ResNet-LSTM network by taking the minimization of a total error loss function as a target.
And 103, acquiring the source role movement data to be redirected by using the test data set, and inputting the test data into the trained ResNet-LSTM network to obtain the redirected movement data.
In this embodiment, the loss function of the mean square error constraint is
Figure BDA0003398729370000051
Wherein f is the total frame number of the training data, YLSTMRepresenting the output of a ResNet-LSTM based network, X is the corresponding target character movement data.
The smoothness constraint penalty function is:
Figure BDA0003398729370000052
repeating the boundary elements of the output of the ResNet-LSTM based network to Y'LSTM：Y′LSTM(1，k)＝YLSTM(2，k)＝YLSTM(1，k)，YLSTM(f+1，k)＝YLSTM(f+2，k)＝YLSTM(f，k)，YLSTM(i，k)＝YLSTM(i-1，k)K is more than or equal to 1 and less than or equal to 177, and i is more than or equal to 2 and less than or equal to f-1. The smoothing matrix O is:
Figure BDA0003398729370000061
fig. 2 is a structure diagram of a ResNet-LSTM network according to an embodiment of the present invention, and the following describes the network structure according to fig. 2:
the ResNet-LSTM network comprises a ResNet-based convolutional neural network and an LSTM-based cyclic neural network;
the convolutional neural network based on ResNet is composed of four layers of fully-connected networks and a ResNet network; the LSTM-based recurrent neural network consists of two layers of fully-connected networks and an LSTM long-time memory unit.
In the flow of the ResNet-LSTM-based movement redirection method provided by the embodiment of the invention, the movement data X input of the source role is based onResNet convolution neural network to obtain ResNet convolution neural network output YResNet＝W4×(W3×(ResNet(W2×(W1×X+b1)+b2))+b3)+b4；YResNetInputting the LSTM-based recurrent neural network to obtain the LSTM-based recurrent neural network output YLSTM＝W6×(LSTM(W5×YResNet+b5))+b6，YLSTMAlso the output of the ResNet-LSTM network. Wherein WiWeight of the full connection layer FCi, biIndicating the bias of fully connected layer FCi.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims. It should be noted that the various technical features described in the above embodiments can be combined in any suitable manner without contradiction, and the invention is not described in any way for the possible combinations in order to avoid unnecessary repetition. In addition, any combination of the various embodiments of the present invention is also possible, and the same should be considered as the disclosure of the present invention as long as it does not depart from the spirit of the present invention.

Claims (5)

1. A ResNet-LSTM-based motion data redirection method is characterized by comprising the following steps:
step 1, acquiring a training sample set and a test sample set; the training sample set and the testing sample set respectively comprise role movement segments with different skeleton topological structures of a source and a target, and the movement types respectively comprise walking and running;
step 2, training a ResNet-LSTM network by using the training sample set with the minimization of a total loss function as an optimization target to obtain the trained ResNet-LSTM network;
and 3, acquiring the motion data of the source role to be redirected by using the test data set, and inputting the test data into the trained ResNet-LSTM network to obtain the redirected motion data.
2. The ResNet-LSTM-based movement data redirection method according to claim 1, wherein in step 1, the movement segments with the movement types of walking and running are selected from CMU database, and the walking and running movement segments are redirected to two roles with different bone topologies, namely source role and target role, by using IK method; taking 60% of the source role and the target role as a training sample set, and taking 40% of the source role and the target role as a testing sample set; the training sample set and the testing sample set respectively comprise two role movement segments with different skeleton topological structures, namely a source role and a target role, and the movement types comprise walking and running.
3. The method for redirecting motion data based on ResNet-LSTM according to claim 1, wherein said step 2 specifically comprises:
constructing a ResNet-LSTM network; the ResNet-LSTM network comprises a ResNet-based convolutional neural network and an LSTM-based cyclic neural network;
firstly, inputting the motion data of a source role in a training sample set into the convolutional neural network based on ResNet to obtain the output data of the convolutional neural network; secondly, inputting the output data of the convolutional neural network into the LSTM-based cyclic neural network to obtain the output data of the cyclic neural network;
calculating the mean square error and the skeleton length error of the current ResNet network according to the output data of the cyclic neural network and the motion data of the target role in the training sample set;
according to the formula: loss ═ λmLossm(Y)+λbLossb(Y) updating the total Loss function, where Loss denotes the total Loss function, λm、λbRespectively representing mean square error weight parameter and smoothness error weight parameter，Lossm(Y)、Losss(Y) represents a mean square error and a smoothness error, respectively;
and dynamically adjusting weight parameters and a learning rate through a back propagation algorithm of errors and an adaptive moment estimation optimization algorithm, and obtaining a trained ResNet-LSTM network by taking the minimization of a total error loss function as a target.
4. The method of claim 1, wherein in step 3, the loss function of the mean square error constraint is
Figure FDA0003398729360000021
Wherein f is the total frame number of the training data, YLSTMRepresenting the output of a ResNet-LSTM based network, X being the corresponding target character motion data;
the smoothness constraint penalty function is:
Figure FDA0003398729360000022
repeating the boundary elements of the output of the ResNet-LSTM based network to Y'LSTM：Y′LSTM(1，k)＝Y′LSTM(2，k)＝YLSTM(1，k)，Y′LSTM(f+1，k)＝Y′LSTM(f+2，k)＝YLSTM(f，k)，Y′LSTM(i，k)＝YLSTM(i-1，k)K is more than or equal to 1 and less than or equal to 177, and i is more than or equal to 2 and less than or equal to f-1. The smoothing matrix O is:
Figure FDA0003398729360000023
5. the ResNet-LSTM based motor data redirection method of claim 1, wherein the ResNet-LSTM network comprises a ResNet based convolutional neural network, an LSTM based cyclic neural network;
the convolutional neural network based on ResNet comprises four layers of fully-connected networks and a ResNet network; the LSTM-based recurrent neural network comprises two layers of fully-connected networks and an LSTM long-time memory unit.
CN202111512627.1A 2021-12-08 2021-12-08 ResNet-LSTM-based movement data redirection method Pending CN114187393A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111512627.1A CN114187393A (en) 2021-12-08 2021-12-08 ResNet-LSTM-based movement data redirection method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111512627.1A CN114187393A (en) 2021-12-08 2021-12-08 ResNet-LSTM-based movement data redirection method

Publications (1)

Publication Number Publication Date
CN114187393A true CN114187393A (en) 2022-03-15

Family

ID=80543290

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111512627.1A Pending CN114187393A (en) 2021-12-08 2021-12-08 ResNet-LSTM-based movement data redirection method

Country Status (1)

Country Link
CN (1) CN114187393A (en)

Similar Documents

Publication Publication Date Title
Kähler et al. Geometry-based muscle modeling for facial animation
KR101796190B1 (en) Apparatus and method for generating digital clone
WO2017031718A1 (en) Modeling method of deformation motions of elastic object
WO2022143178A1 (en) Motion retargeting method and apparatus, electronic device, and storage medium
CN112819971B (en) Method, device, equipment and medium for generating virtual image
CN106600000A (en) Method and system for human-robot motion data mapping
CN111223168B (en) Target object control method, device, storage medium and computer equipment
CN109446952A (en) A kind of piano measure of supervision, device, computer equipment and storage medium
CN113642379B (en) Human body posture prediction method and system based on attention mechanism fusion multi-flow diagram
CN111144553A (en) Image description method based on space-time memory attention
CN105354800A (en) Image structure-based particle swarm optimization non-convex compressed sensing image reconstruction method
CN108520513A (en) A kind of threedimensional model local deformation component extraction method and system
KR20230004837A (en) Generative nonlinear human shape model
CN114663496A (en) Monocular vision odometer method based on Kalman pose estimation network
KR20150075909A (en) Method and apparatus for editing 3D character motion
CN111899317A (en) Movement data redirection method based on neural network
CN114550292A (en) High-physical-reality human body motion capture method based on neural motion control
CN115223201A (en) Monocular sequence image-based three-dimensional human body joint point estimation method, system and device
CN114170353A (en) Multi-condition control dance generation method and system based on neural network
Chen Research on college physical education model based on virtual crowd simulation and digital media
Romeo et al. Muscle Simulation with Extended Position Based Dynamics.
CN114187393A (en) ResNet-LSTM-based movement data redirection method
CN110264551A (en) A kind of motion retargeting method and system
CN112435321A (en) Leap Motion hand skeleton Motion data optimization method
CN111507920B (en) Bone motion data enhancement method and system based on Kinect

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination