CN113012036B - Human motion style migration method and system based on generative flow model - Google Patents
Human motion style migration method and system based on generative flow model Download PDFInfo
- Publication number
- CN113012036B CN113012036B CN202110246980.3A CN202110246980A CN113012036B CN 113012036 B CN113012036 B CN 113012036B CN 202110246980 A CN202110246980 A CN 202110246980A CN 113012036 B CN113012036 B CN 113012036B
- Authority
- CN
- China
- Prior art keywords
- motion
- style
- model
- style migration
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000033001 locomotion Effects 0.000 title claims abstract description 320
- 230000005012 migration Effects 0.000 title claims abstract description 117
- 238000013508 migration Methods 0.000 title claims abstract description 117
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000012549 training Methods 0.000 claims abstract description 83
- 238000010606 normalization Methods 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 21
- 238000000605 extraction Methods 0.000 claims abstract description 18
- 238000012360 testing method Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 11
- 230000002441 reversible effect Effects 0.000 claims description 10
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 9
- 230000008878 coupling Effects 0.000 claims description 9
- 238000010168 coupling process Methods 0.000 claims description 9
- 238000005859 coupling reaction Methods 0.000 claims description 9
- 230000008707 rearrangement Effects 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 8
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000007476 Maximum Likelihood Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 abstract description 3
- 230000002194 synthesizing effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a human motion style migration method and a human motion style migration system based on a generative flow model, wherein the method comprises the following steps: acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data; inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence; and inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human motion style migration model is obtained by training a generation flow model through a sample motion content sequence. The method extracts the hidden codes through the generated stream model, thereby efficiently and accurately extracting the motion style characteristics, keeping the motion content unchanged while transferring the motion style, and synthesizing more accurate human motion.
Description
Technical Field
The invention relates to the technical field of computer vision and artificial intelligence, in particular to a human motion style migration method and a human motion style migration system based on a generative flow model.
Background
The existing human body motion style migration method needs paired motion data, wherein paired motion data means that the motion data belong to the same motion content but the motion styles are different, or the method does not depend on the paired motion data but needs a deep learning method to perform supervised training to learn a motion migration model.
The existing method is based on paired exercise data, and often needs complex data preprocessing, for example, human exercise data is staged and each stage is registered respectively; or the movement style migration is limited to paired movement data; or a supervised training deep learning model is needed, namely, in the training process, each training data needs to be labeled, so that the difficulty of data preprocessing and training is increased, and the application range is limited. The existing motion style feature extraction method usually adopts artificial definition or tries to decouple motion style and motion content, so that the motion style extraction is inaccurate. In addition, the movement style migration result is uniquely determined not to be editable, and movement style migration between movements of different movement contents may cause failure of movement style migration.
Therefore, there is a need for a method and a system for human motion style migration based on a flow model to solve the above problems.
Disclosure of Invention
The invention provides a human motion style migration method and system based on a generative flow model, aiming at the problems of high difficulty and low accuracy of human motion style migration training in the prior art.
The invention provides a human motion style migration method based on a generative flow model, which comprises the following steps:
acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data;
inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence;
and inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human motion style migration model is obtained by training a streaming model from a sample motion content sequence.
According to the human motion style migration method based on the generated flow model, the trained human motion style migration model is obtained by training the following steps:
extracting each human body motion data in the sample motion content sequence to obtain a control signal corresponding to each human body motion data, wherein the control signal is represented by the forward direction of a human body root node, the lateral direction of the human body root node and the rotation angular velocity of the human body root node relative to a vertical ground axis;
normalizing each joint coordinate in each human body motion data to obtain normalized motion data corresponding to each joint coordinate;
constructing a training sample set according to the control signal and the normalized motion data;
and inputting the training sample set into a generated flow model for layer-by-layer training to obtain a trained human style migration model.
According to the human motion style migration method based on the generated flow model, the generated flow model is composed of a plurality of layers of flow basic modules, and the flow basic modules are composed of an activation normalization layer, a rearrangement layer, a division-combination operation layer, a reversible attention mechanism converter and an affine transformation coupling layer.
According to the human motion style migration method based on the generative flow model provided by the invention, the training sample set is input into the generative flow model to be trained layer by layer, so that a trained human style migration model is obtained, and the method comprises the following steps:
inputting the sample data in the training sample set into an activation standardization layer to obtain first input information, wherein each characteristic channel in the first input information has a mean value of 0 and a unit variance;
inputting the first input information into a rearrangement layer, and increasing the change of characteristic dimensionality to obtain second input information;
dividing the second input information through a dividing-combining operation layer to obtain a first isometric segment and a second isometric segment;
inputting the first equal-length segment into a reversible attention mechanism converter to obtain a converted first equal-length segment;
obtaining an offset parameter and a scaling parameter of an affine transformation coupling layer according to the transformed first equal-length segment, and performing offset and scaling processing on the second equal-length segment and the condition input information according to the offset parameter and the scaling parameter to obtain a third equal-length segment;
and combining the first equal-length segment and the third equal-length segment through the division-combination operation layer, inputting output information obtained by the combined operation into a next flow basic module for training, and after all the flow basic modules finish the training of the current round, taking the output information of the last layer of flow basic module as the input information of the first layer of flow basic module for training again until preset training conditions are met to obtain the trained human style migration model.
According to the human motion style migration method based on the generative flow model provided by the invention, the method further comprises the following steps:
and training and optimizing the parameters of the generated flow model based on the maximum likelihood estimation so as to obtain a trained human style migration model.
The invention also provides a human motion style migration system based on the generative flow model, which comprises the following components:
the device comprises a motion content acquisition unit, a motion content processing unit and a motion content processing unit, wherein the motion content acquisition unit is used for acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data;
the hidden code extraction unit is used for inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence;
and the style migration unit is used for inputting the hidden codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human style migration model is obtained by training a streaming model from a sample motion content sequence.
According to the human motion style migration system based on the generative flow model provided by the invention, the system further comprises:
the control signal extraction unit is used for extracting each human body motion data in the sample motion content sequence to obtain a control signal corresponding to each human body motion data, wherein the control signals are forward representation of a human body root node, lateral representation of the human body root node and representation of a rotation angular velocity of the human body root node relative to a vertical ground axis;
the preprocessing unit is used for carrying out normalization processing on each joint coordinate in each human body motion data to obtain normalized motion data corresponding to each joint coordinate;
the sample set construction unit is used for constructing a training sample set according to the control signal and the normalized motion data;
and the training unit is used for inputting the training sample set into a generation flow model to carry out layer-by-layer training so as to obtain a trained human style migration model.
The invention further provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the human motion style migration method based on the generated flow model.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method for human movement style migration based on a generative flow model as set forth in any of the above.
The human motion style migration method and the human motion style migration system based on the generated stream model, which are provided by the invention, aim at the limitation of the prior art that motion styles are migrated on different motion content data, and extract the hidden codes as the specified motion styles through the generated stream model, so that the motion style characteristics are efficiently and accurately extracted, the motion contents are kept unchanged while the motion styles are migrated, and more accurate human motion can be synthesized.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a human motion style migration method based on a generated flow model according to the present invention;
FIG. 2 is a schematic diagram illustrating a comparison of the movement style migration provided by the present invention;
FIG. 3 is a schematic structural diagram of a basic block module provided in the present invention;
FIG. 4 is a schematic structural diagram of a human motion style migration system based on a generated flow model according to the present invention;
fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow chart of a human motion style migration method based on a generated flow model, as shown in fig. 1, the present invention provides a human motion style migration method based on a generated flow model, which includes:
In the present invention, the motion content data is generally represented by a sequence expressed as time-dependent, each frame in the sequence representing one pose of the moving character, and the character pose is generally represented by joint angles or joint positions of the character as parameters. The motion sequence to be migrated is a motion content sequence, and the motion sequence with the target motion style is a preset motion style sequence.
And 102, inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence.
And 103, inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human motion style migration model is obtained by training a streaming model through a sample motion content sequence.
In the invention, when the human motion style is transferred, a motion sequence to be transferred and a motion sequence with a target motion style are required, a preset motion style sequence is input into a trained human motion style model to extract a hidden code corresponding to the motion style, and the hidden code corresponding to the motion style is transferred onto the motion sequence to be transferred by inputting the hidden code and motion content input data into the model, so that the motion sequence with the transferred style has the preset motion style. Fig. 2 is a schematic diagram comparing the movement style migration provided by the present invention, and as shown in fig. 2, the existing method migrates the input running movement style to the kicking movement content, and the movement content cannot be kept unchanged, whereas the method of the present invention successfully migrates the running movement style to the kicking movement content, and keeps the movement content unchanged while migrating the movement style.
Aiming at the limitation of the prior art that the movement style is migrated on different movement content data, the method for migrating the movement style of the human body based on the generated flow model extracts the hidden codes as the specified movement style, thereby efficiently and accurately extracting the characteristics of the movement style, keeping the movement content unchanged while migrating the movement style and synthesizing more accurate human body movement.
On the basis of the above embodiment, the trained human style migration model is obtained by training through the following steps:
extracting each human body movement data in the sample movement content sequence to obtain a control signal corresponding to each human body movement data, wherein the control signal is represented by the forward direction of a human body root node, the lateral direction of the human body root node and the rotation angular velocity of the human body root node relative to a vertical ground axis;
in the invention, training data and test data are obtained according to a sample motion content sequence, and a control signal of each human motion data in the sample motion content sequence is respectively extracted.
And carrying out normalization processing on each joint coordinate in each human body motion data to obtain normalized motion data corresponding to each joint coordinate.
And constructing a training sample set according to the control signal and the normalized motion data.
In the invention, the coordinates of each joint in all the human motion data are normalized, and then the normalized motion data and the control signal form the input information for generating the flow model. Specifically, the processing steps of the sample motion content sequence are as follows:
step 201, down-sampling the data of the sample motion content sequence to 30 frames per second, dividing all motion data into motion segments, each motion segment having 32 frames, and each motion segment having 8 frames of cross-overlapped segments. 1518 movement segments are provided, each frame of data of each movement segment comprises 21 joint point positions, and each joint point position is represented by a 3D Cartesian coordinate;
step 202, extracting control signals of each motion segment obtained in step 201, wherein the control signals are represented by the forward direction, the lateral direction and the rotation angular velocity relative to the vertical ground axis of the human body root node, and each motion segment and the control signals jointly form new motion segment data;
step 203, dividing the new motion segment data into training data and test data, wherein in the invention, the training data are 1406 motion segments, and the test data are 112 motion segments;
and 204, performing normalization processing on the motion segments in the training data obtained in the above step, specifically, calculating a mean value mu and a standard deviation sigma of each joint coordinate in all the training data, and then subtracting mu from the motion segment of each training data and dividing the result by sigma to obtain corresponding normalized motion data. In addition, based on the same method, the motion segments in the test data are normalized.
Step 205, respectively extracting the joint position of the previous 10 frames of data, the previous 10 frames of motion segment, the previous 10 frames of control signals and the current frame of control signals from the motion segments of the normalized motion training data and the normalized test data, and respectively obtaining condition input information for training and condition input information for testing.
And inputting the training sample set into a generated flow model for layer-by-layer training to obtain a trained human style migration model.
In the invention, after training of the human body style migration model is completed, the model is tested through test set data, motion data of a preset style in a section of test data is input into the model, and a hidden code corresponding to the motion data is extracted, as shown in the following formula:
wherein, fKFor mathematical definition of the flow basic block, fKHas a parameter of thetaK(for simplicity, f is shown)KF of (1) is omitted).
Then, the obtained hidden code of the preset motion style and a section of test data are extracted to obtain motion content input data (normalized motion data and control signals) to form input information, the input information is input into a human body style migration model, and a section of motion data of the preset motion style and the content is generated, so that the human body motion style migration is realized.
On the basis of the above embodiment, the generated flow model is composed of a plurality of layers of flow basic blocks, and the flow basic blocks are composed of an activation normalization layer, a rearrangement layer, a division-combination operation layer, a reversible attention mechanism transformer, and an affine transformation coupling layer.
Fig. 3 is a schematic structural diagram of a basic block module provided by the present invention, and can refer to fig. 3, in the present invention, training data is input into a Flow basic module (Step of Flow, abbreviated as SoF) for generating a Flow model, so that the generated Flow model learns motion data distribution. As shown in fig. 3, each SoF includes an Activation Normalization Layer (ANL), a rearrangement Layer (PL), a Split-and-combine operation Layer (SC), an Inverse Attention Transformer (IAT), and an Affine Coupling Layer (ACL).
On the basis of the above embodiment, the inputting the training sample set into a generated flow model for layer-by-layer training to obtain a trained human style migration model includes:
and inputting the sample data in the training sample set into an activation standardization layer to obtain first input information, wherein each characteristic channel in the first input information has a mean value of 0 and a unit variance.
In the present invention, as shown in fig. 3, sample data in a training sample set is input to an activation normalization layer, so that each feature channel of input information has a mean value of 0 and a unit variance, thereby obtaining first input information.
And inputting the first input information into a rearrangement layer, and increasing the change of the characteristic dimension to obtain second input information.
In the invention, the flow basic module for generating the flow model is a reversible differentiable nonlinear function, so that the generated flow model has stronger expression capability and can better fit the real motion data distribution, and meanwhile, the reversible transformation in the flow model is generated, so that the hidden codes of different human body motions are easy to extract and are easy to apply to different motion contents, thereby extracting diversified motion styles and even including the motion style which does not appear in training data. Specifically, after the training data is input to the stream basic module, first input information is input to the rearrangement layer, and the feature dimension change of the first input information is increased, so that second input information is obtained.
And performing division operation on the second input information through a division-combination operation layer to obtain a first equal-length segment and a second equal-length segment.
In the invention, the second input information is input into the division-combination operation layer to carry out division operation, and the division operation divides the characteristics of the second input information into equal-length segments, namely a first equal-length segment a 'and a second equal-length segment a'.
And inputting the first equal-length segment into a reversible attention mechanism converter to obtain a converted first equal-length segment.
Obtaining an offset parameter and a scaling parameter of an affine transformation coupling layer according to the transformed first isometric segment, and performing offset and scaling processing on the second isometric segment and the condition input information according to the offset parameter and the scaling parameter to obtain a third isometric segment;
and combining the first equal-length segment and the third equal-length segment through the division-combination operation layer, inputting output information obtained by the combination operation into a next flow basic module for training, and after all the flow basic modules complete the training of the current round, training again by taking the output information of the last layer of flow basic module as the input information of the first layer of flow basic module until preset training conditions are met to obtain a trained human style migration model.
In the present invention, referring to fig. 3, the first equal-length segment a' is inputted to the reversible attention mechanism converter to obtain the converted first equal-length segment
Then according to the obtained transformed first equal-length segment
Extracting an offset parameter t and a scaling parameter s of the affine transformation coupling layer, and applying the offset parameter t and the scaling parameter s to the second isometric segment a 'and the conditional input information to obtain a third isometric segment b', wherein the formula is as follows:
b″＝(a″+t)⊙s；
further, the third equal-length segment b ″ and the first equal-length segment a' are combined by dividing-combining layers.
And finally, inputting the output obtained by the SoF module into the next SoF module, wherein the whole generated flow model consists of 16 layers of iterative SoF modules, training all the SOF modules as a round of training, returning the output of the last layer to the first layer of SoF module for iterative training again, and obtaining the trained model after the preset training times are met.
On the basis of the above embodiment, the method further comprises:
and training and optimizing the parameters of the generated flow model based on the maximum likelihood estimation so as to obtain a trained human style migration model.
In the present invention, the generation flow model is trained to learn the human motion data X ═ X1,x2,…,xNThe invention learns the model parameters of data distribution by optimizing maximum marginal likelihood (marginal likelihood), and the formula is as follows:
the generated flow model is obtained by optimizing the maximum likelihood probability of motion data distribution, so that the synthesized motion is more accurate, the problems of footstep sliding and the like are solved, and various reasonable motions meeting the generation conditions can be generated.
Fig. 4 is a schematic structural diagram of a human motion style migration system based on a generated stream model, and as shown in fig. 4, the present invention provides a human motion style migration system based on a generated stream model, which includes a motion content obtaining unit 401, a hidden code extracting unit 402, and a format migration unit 403, where the motion content obtaining unit 401 is configured to obtain a preset motion style sequence and a motion content sequence, and perform control signal extraction and normalization processing on the motion content sequence to obtain motion content input data; the hidden code extracting unit 402 is configured to input the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence; the style migration unit 403 is configured to input the implicit code and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, where the trained human style migration model is obtained by training a generative flow model from a sample motion content sequence.
The human motion style migration system based on the generated stream model provided by the invention is used for extracting the hidden codes as the specified motion style by the generated stream model aiming at the limitation of the prior art that the motion style is migrated on different motion content data, thereby efficiently and accurately extracting the motion style characteristics, keeping the motion content unchanged while migrating the motion style and synthesizing more accurate human motion.
On the basis of the above embodiment, the system further includes a control signal extraction unit, a preprocessing unit, a sample set construction unit and a training unit, wherein the control signal extraction unit is configured to extract each human motion data in the sample motion content sequence to obtain a control signal corresponding to each human motion data, and the control signal is a forward representation of a human root node, a lateral representation of the human root node, and a rotation angular velocity representation of the human root node relative to a vertical ground axis; the preprocessing unit is used for carrying out normalization processing on each joint coordinate in each human body motion data to obtain normalized motion data corresponding to each joint coordinate; the sample set construction unit is used for constructing a training sample set according to the control signal and the normalized motion data; and the training unit is used for inputting the training sample set into a generation flow model to carry out layer-by-layer training to obtain a trained human style migration model.
The system provided by the present invention is used for executing the above method embodiments, and for the specific processes and details, reference is made to the above embodiments, which are not described herein again.
Fig. 5 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 5, the electronic device may include: a processor (processor)501, a communication interface (communications interface)502, a memory (memory)503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 are all communicated with each other through the communication bus 504. The processor 501 may invoke logic instructions in the memory 503 to perform a human movement style migration method based on the generated flow model, the method comprising: acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data; inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence; and inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human motion style migration model is obtained by training a generation flow model through a sample motion content sequence.
In addition, the logic instructions in the memory 503 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In another aspect, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer being capable of executing the human motion style migration method based on the generated flow model provided by the above methods, the method including: acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data; inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence; and inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human motion style migration model is obtained by training a streaming model from a sample motion content sequence.
In still another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented by a processor to perform the human motion style migration method based on the generated flow model provided in the foregoing embodiments, where the method includes: acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data; inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence; and inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human motion style migration model is obtained by training a generation flow model through a sample motion content sequence.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A human motion style migration method based on a generative flow model is characterized by comprising the following steps:
acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data;
inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence;
inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, wherein the trained human style migration model is obtained by training a generation flow model through a sample motion content sequence;
the generated flow model is composed of a plurality of layers of flow basic modules, and the flow basic modules are composed of an activation normalization layer, a rearrangement layer, a division-combination operation layer, a reversible attention mechanism converter and an affine transformation coupling layer;
after the training of the trained human body style migration model is completed, testing the trained human body style migration model through test set data, inputting motion data of a preset style in a section of test data into the trained human body style migration model, and extracting a hidden code corresponding to the motion data of the preset style, wherein the hidden code formula is as follows:
wherein,
for the mathematical definition of the flow basis module,
has the parameters of
。
2. The method for human movement style migration based on generation flow model according to claim 1, wherein the trained human movement style migration model is obtained by training through the following steps:
extracting each human body motion data in the sample motion content sequence to obtain a control signal corresponding to each human body motion data, wherein the control signal is represented by the forward direction of a human body root node, the lateral direction of the human body root node and the rotation angular velocity of the human body root node relative to a vertical ground axis;
normalizing each joint coordinate in each human body motion data to obtain normalized motion data corresponding to each joint coordinate;
constructing a training sample set according to the control signal and the normalized motion data;
and inputting the training sample set into a generated flow model for layer-by-layer training to obtain a trained human style migration model.
3. The method for human motion style migration based on generative flow model according to claim 2, wherein the step of inputting the training sample set into the generative flow model for layer-by-layer training to obtain the trained human motion style migration model comprises:
inputting the sample data in the training sample set into an activation normalization layer to obtain first input information, wherein each characteristic channel in the first input information has a mean value of 0 and a unit variance;
inputting the first input information into a rearrangement layer, and increasing the change of characteristic dimensionality to obtain second input information;
dividing the second input information through a dividing-combining operation layer to obtain a first isometric segment and a second isometric segment;
inputting the first isometric segment into a reversible attention mechanism converter to obtain a converted first isometric segment;
obtaining an offset parameter and a scaling parameter of an affine transformation coupling layer according to the transformed first isometric segment, and performing offset and scaling processing on the second isometric segment and the condition input information according to the offset parameter and the scaling parameter to obtain a third isometric segment;
and combining the first equal-length segment and the third equal-length segment through the division-combination operation layer, inputting output information obtained by the combination operation into a next flow basic module for training, and after all the flow basic modules complete the training of the current round, training again by taking the output information of the last layer of flow basic module as the input information of the first layer of flow basic module until preset training conditions are met to obtain a trained human style migration model.
4. The method for human movement style migration based on generative flow model as claimed in claim 2, wherein the method further comprises:
and training and optimizing the parameters of the generated flow model based on the maximum likelihood estimation so as to obtain a trained human style migration model.
5. A human motion style migration system based on a generative flow model, comprising:
the motion content acquisition unit is used for acquiring a preset motion style sequence and a motion content sequence, and performing control signal extraction and normalization processing on the motion content sequence to obtain motion content input data;
the hidden code extraction unit is used for inputting the preset motion style sequence into a trained human motion style migration model to obtain a hidden code corresponding to the preset motion style sequence;
the style migration unit is used for inputting the implicit codes and the motion content input data into a trained human motion style migration model to obtain a motion content sequence after style migration, and the trained human style migration model is obtained by training a generation flow model through a sample motion content sequence;
the generated flow model is composed of a plurality of layers of flow basic modules, and the flow basic modules are composed of an activation normalization layer, a rearrangement layer, a division-combination operation layer, a reversible attention mechanism converter and an affine transformation coupling layer;
after the training of the trained human body style migration model is completed, testing the trained human body style migration model through test set data, inputting motion data of a preset style in a section of test data into the trained human body style migration model, and extracting a hidden code corresponding to the motion data of the preset style, wherein the hidden code formula is as follows:
wherein,
for the mathematical definition of the flow basic block,
has the parameters of
。
6. The generative flow model-based human movement style migration system according to claim 5, wherein the system further comprises:
the control signal extraction unit is used for extracting each human body motion data in the sample motion content sequence to obtain a control signal corresponding to each human body motion data, wherein the control signal is represented by the forward direction of a human body root node, the lateral direction of the human body root node and the rotation angular velocity of the human body root node relative to a vertical ground axis;
the preprocessing unit is used for carrying out normalization processing on each joint coordinate in each human body motion data to obtain normalized motion data corresponding to each joint coordinate;
the sample set construction unit is used for constructing a training sample set according to the control signal and the normalized motion data;
and the training unit is used for inputting the training sample set into a generation flow model to carry out layer-by-layer training so as to obtain a trained human style migration model.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the computer program implements the steps of the method for human movement style migration based on a generative flow model according to any one of claims 1 to 4.
8. A non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the steps of the human movement style migration method based on a generative flow model as set forth in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110246980.3A CN113012036B (en) | 2021-03-05 | 2021-03-05 | Human motion style migration method and system based on generative flow model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110246980.3A CN113012036B (en) | 2021-03-05 | 2021-03-05 | Human motion style migration method and system based on generative flow model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113012036A CN113012036A (en) | 2021-06-22 |
| CN113012036B true CN113012036B (en) | 2022-07-15 |
Family
ID=76407316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110246980.3A Active CN113012036B (en) | 2021-03-05 | 2021-03-05 | Human motion style migration method and system based on generative flow model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113012036B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108537776A (en) * | 2018-03-12 | 2018-09-14 | 维沃移动通信有限公司 | A kind of image Style Transfer model generating method and mobile terminal |
| CN110533752A (en) * | 2019-07-23 | 2019-12-03 | 深圳大学 | A kind of generation method, storage medium and the electronic equipment of human action edit model |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116823593A (en) * | 2016-10-21 | 2023-09-29 | 谷歌有限责任公司 | Stylized input image |
-
2021
- 2021-03-05 CN CN202110246980.3A patent/CN113012036B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108537776A (en) * | 2018-03-12 | 2018-09-14 | 维沃移动通信有限公司 | A kind of image Style Transfer model generating method and mobile terminal |
| CN110533752A (en) * | 2019-07-23 | 2019-12-03 | 深圳大学 | A kind of generation method, storage medium and the electronic equipment of human action edit model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113012036A (en) | 2021-06-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020207281A1 (en) | Method for training posture recognition model, and image recognition method and apparatus | |
| CN110675475B (en) | Face model generation method, device, equipment and storage medium | |
| CN109063584B (en) | Facial feature point positioning method, device, equipment and medium based on cascade regression | |
| CN111028319B (en) | Three-dimensional non-photorealistic expression generation method based on facial motion unit | |
| CN112116090A (en) | Neural network structure searching method and device, computer equipment and storage medium | |
| CN109508686B (en) | Human behavior recognition method based on hierarchical feature subspace learning | |
| CN112488067B (en) | Face pose estimation method and device, electronic equipment and storage medium | |
| CN111914878A (en) | Feature point tracking training and tracking method and device, electronic equipment and storage medium | |
| CN112258557B (en) | Visual tracking method based on space attention feature aggregation | |
| CN113192175A (en) | Model training method and device, computer equipment and readable storage medium | |
| CN113570685A (en) | Image processing method and device, electronic device and storage medium | |
| CN113298152A (en) | Model training method and device, terminal equipment and computer readable storage medium | |
| CN115958606A (en) | Mechanical arm obstacle avoidance track planning method and device and electronic equipment | |
| CN114550292A (en) | High-physical-reality human body motion capture method based on neural motion control | |
| CN113012036B (en) | Human motion style migration method and system based on generative flow model | |
| CN116092120B (en) | Image-based action determining method and device, electronic equipment and storage medium | |
| CN116485962A (en) | Animation generation method and system based on contrast learning | |
| WO2022262337A1 (en) | Video labeling method and apparatus, computing device, and computer-readable storage medium | |
| CN116362265A (en) | Text translation method, device, equipment and storage medium | |
| CN114565964B (en) | Emotion recognition model generation method, emotion recognition device, emotion recognition medium and emotion recognition equipment | |
| US12051135B2 (en) | System and method for a precise semantic segmentation | |
| CN115620082A (en) | Model training method, head posture estimation method, electronic device, and storage medium | |
| CN115393934A (en) | Face key point detection method and system based on knowledge distillation and multitask learning | |
| CN114187487A (en) | Processing method, device, equipment and medium for large-scale point cloud data | |
| Lagneaux et al. | An Automatic Highly Dynamical Digital Twin Design with YOLOv8 for hydrodynamic studies on living animals |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |