CN111104964B - Method, equipment and computer storage medium for matching music with action - Google Patents
Method, equipment and computer storage medium for matching music with action Download PDFInfo
- Publication number
- CN111104964B CN111104964B CN201911158848.6A CN201911158848A CN111104964B CN 111104964 B CN111104964 B CN 111104964B CN 201911158848 A CN201911158848 A CN 201911158848A CN 111104964 B CN111104964 B CN 111104964B
- Authority
- CN
- China
- Prior art keywords
- action
- music
- distance
- piece
- pieces
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
- G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
- G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
- G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
- G10H2210/031—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal
- G10H2210/076—Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal for extraction of timing, tempo; Beat detection
Abstract
The disclosure discloses a method, equipment and a computer storage medium for matching music and actions, and belongs to the field of music dancing. The method comprises the following steps: acquiring a plurality of music dance fragments and corresponding rhythm characteristics; determining the distance between the music piece and the action piece; determining a music fragment and an action fragment with the largest distance and the smallest distance; taking the feature sequence of the action segment, the feature sequence of the music segment and the distance between the music segment and the action segment as training samples, training to obtain a matching distance model of the music segment and the action segment, wherein the output of the model is the matching distance between the music segment and the action segment; acquiring an action transition distance; taking the sum of the matching distance and the action transition distance as the distance between the music piece and the action piece; obtaining music to be matched; and determining a plurality of target action fragments with the smallest total distance with the music to be matched and matching the target action fragments with the music to be matched. The method and the device solve the problem that the matching degree of the action segment and the music to be matched is poor in the related art.
Description
Technical Field
The present disclosure relates to the field of music dance, and in particular, to a method, apparatus, and computer storage medium for matching music and actions.
Background
Dance actions matching a given music performance have found widespread use in music dance games and other fields.
In a method for matching music and actions in the related art, firstly, music to be matched and an action segment library are obtained, wherein the action segment library can comprise a plurality of action segments, and then a plurality of action segments are randomly selected from the action segment library to be matched with the music to be matched.
However, in the above method, the action segments are randomly selected from the action segment library, and the matching degree of the action segments and the music to be matched is poor.
Disclosure of Invention
The embodiment of the disclosure provides a music and action matching method, which can solve the problem of poor matching degree of action fragments and music to be matched in the related technology. The technical scheme is as follows:
according to a first aspect of the present disclosure, there is provided a music and action matching method, the music and action matching method including:
acquiring a plurality of manually-coded music dance segments, wherein each music dance segment comprises a music segment and a corresponding action segment;
acquiring a plurality of rhythm characteristics corresponding to the music dancing pieces;
determining the Euclidean distance of any two rhythm features in the plurality of rhythm features as the distance between a music piece corresponding to a first rhythm feature and an action piece corresponding to a second rhythm feature in the any two rhythm features;
determining n music pieces and action pieces with the largest distance and m music pieces and action pieces with the smallest distance in the music pieces and action pieces corresponding to the rhythm features, wherein m and n are integers larger than 0;
acquiring the characteristic sequences of the n pieces of music and the action pieces and the characteristic sequences of the m pieces of music and the action pieces;
training to obtain a music piece and action piece matching distance model by taking the feature sequences of the n music pieces and the action pieces, the feature sequences of the m music pieces and the action pieces, the distances between the n music pieces and the action pieces and the distances between the m music pieces and the action pieces as training samples, wherein the output of the music piece and action piece matching distance model is the matching distance between the music pieces and the action pieces;
acquiring an action transition distance formula, wherein the action transition distance formula is used for outputting an action transition distance;
taking the sum of the matching distance between the music piece and the action transition distance as the distance between the music piece and the action piece;
obtaining music to be matched, wherein the music to be matched comprises a plurality of pieces of music to be matched;
determining a plurality of target action fragments with minimum total distances to the plurality of music fragments to be matched in an action fragment library, wherein the action fragment library comprises a plurality of action fragments;
and matching the target action fragments with the music to be matched.
Optionally, the obtaining a plurality of rhythm features corresponding to the plurality of music dance segments includes:
determining the rhythm feature according to a rhythm feature formula, the rhythm feature formula comprising:
z(M)＝h z (f motion (M))＝[z 1 ，z 2 ，…，z zdim ] T ；
wherein M is any action segment, z (M) is the rhythm feature corresponding to M, and h is z For feature mapping, the zdim is the dimension of the z (M), the f motion (M)＝[f anim (M，t 1 )，f anim (M，t 2 )，…，f anim (M，t N )]The characteristic sequence of M is in a matrix form, t is any moment of M, N is the sampling number of M, and f anim (M，t)＝[p(M，t)，q 1 (M，t)，q 2 (M，t)，…，q r (M，t)] T The characteristic of M at the moment t in matrix form, wherein p (M, t) is a root nodeAnd q (M, t) is rotation information of a joint of a character, and r is a serial number of the joint.
Optionally, the determining, according to the determining, that the euclidean distance between any two rhythm features in the plurality of rhythm features is the distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature in the any two rhythm features includes:
determining the distance between a music piece corresponding to a first rhythm feature and an action piece corresponding to a second rhythm feature in the two rhythm features according to a first distance formula, wherein the first distance formula comprises:
D match (A i ，M j )＝D motion (M i ，M j )＝||z(M i )-z(M j )||；
wherein the A i For the music piece corresponding to the first playing feature, the M j For the action segment corresponding to the second rhythm feature, the D match For the distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature in the arbitrary two rhythm features, the M is i For the action segment corresponding to the first playing feature, the D motion For the distance between the motion segment corresponding to the first rhythm feature and the motion segment corresponding to the second rhythm feature in the arbitrary two rhythm features, z (M i ) For the first nodal feature, the z (M j ) Is the second cadence characteristic.
Optionally, the obtaining the feature sequences of the n pieces of music and the action pieces, and the feature sequences of the m pieces of music and the action pieces include:
according to f motion (M)＝[f anim (M，t 1 )，f anim (M，t 2 )，…，f anim (M，t N )]Determining a characteristic sequence of the action segment in a matrix form;
according to f audio (A)＝[f mfcc (A，t 1 )，f mfcc (A，t 2 )，…，f mfcc (A，t N )]Determining a characteristic sequence of the music piece in a matrix form;
wherein A is any music piece, t is any time of M and A, N is the sampling number of M and A, and f mfcc (a, t) is a mel-frequency cepstrum coefficient of said a at said any time t.
Optionally, the training to obtain a matching distance model of the music piece and the action piece by using the feature sequences of the n music pieces and the action piece, the feature sequences of the m music pieces and the action piece, the distances between the n music pieces and the action piece, and the distances between the m music pieces and the action piece as training samples includes:
acquiring training data, wherein the training data comprises characteristic sequences of the n music pieces and action pieces, characteristic sequences of the m music pieces and action pieces, distances between the n music pieces and the action pieces and distances between the m music pieces and the action pieces;
training the initial neural network model according to the training data to obtain feature mapping about music fragments and feature mapping about action fragments;
obtaining a matching distance model of the music piece and the action piece according to the feature mapping of the music piece and the feature mapping of the action piece, wherein the matching distance model of the music piece and the action piece comprises the following steps:
D match (A i ，M j )＝h match (f audio (A i )，f motion (M j ))；
h match (f audio (A i )，f motion (M j ))＝||h audio (f audio (A i ))-h motion (f motion (M j ))||；
wherein the h is audio For feature mapping with respect to a piece of music, the h motion For feature mapping with respect to action segments, the h match Mapping of matching distance for musical piece to action piece。
Optionally, the action transition distance formula includes:
cost(M i ，M j )＝max{speed(f trans (M i ，M j ))}；
f trans (M i ，M j )＝blend(f from (M i )，f to (M j ))；
f from (M)＝[f anim (M，t N-s )，f anim (M，t N-s+1 )，…，f anim (M，t N+s-1 )，f anim (M，t N+s )]；
f to (M)＝[f anim (M，t -s )，f anim (M，t -s+1 )，…，f anim (M，t s-1 )，f anim (M，t s )]；
wherein the D is trans For the motion transition distance, the cost (M i ，M j ) To take the maximum value in all joint speeds as the M i Transition to the M j At the cost of θ is a first threshold, speed is the joint velocity, and f trans For transitional actions, the blend is an action mixing algorithm, and f from To the M i The time of the last frame is taken as the center, the length of the half beat is taken as a window, and the M is taken as the center i Intercepting the action of half beat, said f to To the M j A first frame time is taken as a center, a half beat length is taken as a window, and the M is taken as a center j And intercepting the action of half beat, wherein s is the radius of the window.
Optionally, the step of taking the sum of the matching distance between the music piece and the action transition distance as the distance between the music piece and the action piece includes:
determining the distance between the music piece and the action piece according to a distance formula between the music piece and the action piece, wherein the distance formula between the music piece and the action piece comprises the following steps:
wherein D is the distance between the music piece and the action piece, and M is the sum of the distance between the music piece and the action piece x1 ，M x2 ，…，M xn And the action fragments are a plurality of the action fragments in the action fragment library.
Optionally, the determining a plurality of target action segments with the smallest total distance from the plurality of music segments to be matched in the action segment library includes:
and determining the target action fragments with the smallest total distance with the music fragments to be matched in the action fragment library through a dynamic programming algorithm.
In another aspect, a music and action matching device is provided, the music and action matching device including a processor and a memory, the memory storing at least one instruction, at least one program, a code set or a set of instructions, the at least one instruction, the at least one program, the code set or the set of instructions being loaded and executed by the processor to implement the music and action matching method according to the first aspect.
In yet another aspect, a computer storage medium is provided, where at least one instruction, at least one program, a set of codes, or a set of instructions is stored in the computer storage medium, where the at least one instruction, the at least one program, the set of codes, or the set of instructions are loaded and executed by a processor to implement the method for matching music and actions according to the first aspect.
The technical scheme provided by the embodiment of the disclosure has the beneficial effects that at least:
the method comprises the steps of taking a characteristic sequence of a music piece, a characteristic sequence of an action piece and a distance between the music piece and the action piece as training samples, training to obtain a music piece and action piece matching distance model, obtaining an action transition distance by using the output of the model as the matching distance between the music piece and the action piece, obtaining to-be-matched music comprising a plurality of to-be-matched music pieces by taking the sum of the matching distance and the action transition distance as the distance between the music piece and the action piece, determining a plurality of target action pieces with the smallest total distance with the to-be-matched music pieces, and matching the plurality of target action pieces with the to-be-matched music, wherein the matching degree of the action pieces with the to-be-matched music is higher. The method solves the problem of poor matching degree of the action segment and the music to be matched in the related technology.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 is a schematic diagram of an implementation environment of a music and action matching method provided by an embodiment of the present disclosure;
FIG. 2 is a flow chart of a method for matching music to actions provided by an embodiment of the present disclosure;
FIG. 3 is a flow chart of another method of matching music to actions provided by an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a matching device for music and actions according to an embodiment of the present disclosure.
Specific embodiments of the present disclosure have been shown by way of the above drawings and will be described in more detail below. These drawings and the written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the disclosed concepts to those skilled in the art by reference to specific embodiments.
Detailed Description
For the purposes of clarity, technical solutions and advantages of the present disclosure, the following further details the embodiments of the present disclosure with reference to the accompanying drawings.
In a current matching method of music and actions, firstly, music to be matched and an action segment library are obtained, wherein the action segment library can comprise a plurality of action segments, and then a plurality of action segments are randomly selected from the action segment library to be matched with the music to be matched.
However, according to the music and action matching method, action fragments are randomly selected from the action fragment library, and the matching degree of the action fragments and the music to be matched is poor.
The embodiment of the disclosure provides a music and action matching method, equipment and a computer storage medium.
Fig. 1 is a schematic diagram of an implementation environment of a music and action matching method according to an embodiment of the present disclosure, where the implementation environment may include a server 11 and a terminal 12.
The server 11 may be a server or a cluster of servers.
The terminal 12 may be a mobile phone, a tablet computer, a notebook computer, an intelligent wearable device, or other terminals. The terminal 12 may be connected to the server by wire or wirelessly (fig. 1 shows the case of a connection made wirelessly).
Fig. 2 is a flowchart of a method for matching music and actions according to an embodiment of the present disclosure. The matching method of the music and the actions can be applied to the server of the implementation environment. The matching method of the music and the action can comprise the following steps:
step 201, a plurality of manually-encoded music dance segments are obtained, wherein each music dance segment comprises a music segment and a corresponding action segment.
Step 202, obtaining a plurality of rhythm features corresponding to a plurality of music dance segments.
In step 203, the euclidean distance between any two rhythm features in the plurality of rhythm features is determined as the distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature in the any two rhythm features.
Step 204, determining n pieces of music and motion segments with the largest distance and m pieces of music and motion segments with the smallest distance from the pieces of music and motion segments corresponding to the rhythmic features, where m and n are integers greater than 0.
Step 205, obtain the feature sequences of n pieces of music and action pieces, and the feature sequences of m pieces of music and action pieces.
In step 206, the matching distance model between the music piece and the action piece is obtained by training with the feature sequences of the n music pieces and the action piece, the feature sequences of the m music pieces and the action piece, the distances between the n music pieces and the action piece, and the distances between the m music pieces and the action piece as training samples, and the output of the matching distance model between the music piece and the action piece is the matching distance between the music piece and the action piece.
Step 207, obtaining an action transition distance formula, wherein the action transition distance formula is used for outputting the action transition distance.
Step 208, taking the sum of the matching distance between the music piece and the action transition distance as the distance between the music piece and the action piece.
In step 209, music to be matched is obtained, where the music to be matched includes a plurality of pieces of music to be matched.
Step 210, determining a plurality of target action segments with minimum total distance from the plurality of music segments to be matched in the action segment library, wherein the action segment library comprises a plurality of action segments.
Step 211, matching the plurality of target action segments with the music to be matched.
In summary, the embodiment of the disclosure provides a method for matching music and actions, which uses a feature sequence of a music piece, a feature sequence of an action piece, and distances between the music piece and the action piece as training samples to train and obtain a music piece and action piece matching distance model, wherein the output of the model is a matching distance between the music piece and the action piece, an action transition distance is obtained, the sum of the matching distance and the action transition distance is used as a distance between the music piece and the action piece, to obtain to-be-matched music including a plurality of to-be-matched music pieces, a plurality of target action pieces with the smallest total distance to the to-be-matched music pieces are determined, the plurality of target action pieces are matched with the to-be-matched music, and the matching degree of the action pieces and the to-be-matched music is higher. The method solves the problem of poor matching degree of the action segment and the music to be matched in the related technology.
Fig. 3 is a flowchart of another music and action matching method according to an embodiment of the present disclosure, where the music and action matching method may be applied to the server of the above-described implementation environment. The music and action matching method provided by the embodiment of the disclosure can be applied to music dance games, and the music dance games can be realized by means of skeleton animation. As can be seen with reference to fig. 3, the method for matching music with actions may include:
step 301, a plurality of manually-encoded music dance segments are obtained, wherein each music dance segment comprises a music segment and a corresponding action segment.
And the person dances given music to obtain a plurality of manually-dance music pieces. The music dance can be split into a plurality of pieces according to a certain step distance or length, so that a plurality of music dance pieces are obtained. Each music piece corresponds to an action piece.
For example, music may be danced by a person wearing the motion capture device, by acquiring motion captured by the motion capture device, and processing the acquired motion to obtain a music dance. The music dance can be split according to a section of step distance, a section of time length and a section of step distance, and the two sections of time length are used for obtaining a plurality of music dance fragments.
Step 302, determining the rhythm feature according to the rhythm feature formula.
The rhythm characteristic formula comprises:
z(M)＝h z (f motion (M))＝[z 1 ，z 2 ，…，z zdim ] T ；
f motion (M)＝[f anim (M，t 1 )，f anim (M，t 2 )，…，f anim (M，t N )]；
f anim (M，t)＝[p(M，t)，q 1 (M，t)，q 2 (M，t)，…，q r (M，t)] T ；
wherein M is any action segment, z (M) is rhythm characteristic corresponding to M, h z For feature mapping, zdim is the dimension of z (M), illustratively zdim may take 128.f (f) motion The characteristic sequence of M is in a matrix form, t is any moment of M, N is the sampling number of M, the sampling number can be 8 sampling points per beat, 1 bar is 4 beats, and the sampling number of the action of one bar can be 32.f (f) anim The characteristic of the matrix form M at the moment t is that p (M, t) is the three-dimensional space position of the root node, q (M, t) is the rotation information of the joint of the character, and r is the serial number of the joint.
f motion The motion information comprises the gesture information of the motion segment in a certain time, namely the motion information of the motion segment formed by the information of a plurality of joints at a certain moment, and z (M) can reflect the rhythm characteristics of the motion segment M, so that the motion segment can be associated with the music segment. Feature map h z The learning can be performed by an unsupervised training method. The unsupervised training may include a self-encoder. Feature map h z May include: encoding the source features into feature spaces with different dimensions, restoring the compressed features to the source features by a decoder, minimizing the difference between the restored features and the source features, and obtaining the final intermediate encoding features which are the better feature mapping h z 。
Illustratively, the joints of the character are shown in table 1.
TABLE 1
| Sequence number | Joint | Name of the name |
| 1 | Bip01 | Root node |
| 2 | Bip01 Neck | Neck (B) |
| 3 | Bip01 Spine | Spinal column |
| 4 | Bip01 L Thigh | Left thigh |
| 5 | Bip01 R Thigh | Right thigh |
| 6 | Bip01 L Calf | Left calf |
| 7 | Bip01 R Calf | Right lower leg |
| 8 | Bip01 L UpperArm | Left upper arm |
| 9 | Bip01 R UpperArm | Right upper arm |
| 10 | Bip01 L Forearm | Left forearm |
| 11 | Bip01 R Forearm | Right forearm |
| 12 | Bip01 L Hand | Left hand |
| 13 | Bip01 R Hand | Right hand |
| 14 | Bip01 L Foot | Left foot |
| 15 | Bip01 R Foot | Right foot |
Exemplary, f motion May comprise a two-dimensional matrix. First f is carried out motion Inputting a 2-layer 3*3 convolutional neural network to obtain a group of local features f local Then f loca1 Inputting a 2-layer 3*3 convolutional neural network and a full-connection layer to obtain global features f global . Last f global And inputting a full connection layer to obtain a rhythm characteristic z. And then can be connected with f global (M i ) And f local (M i ) As positive samples, f is connected by means of negative sampling global (M i ) And randomly selected f local (M j ) As a negative sample, the positive and negative samples are distinguished by training the arbiter, optimizing the cadence signature z. The arbiter may comprise a 3-layer fully connected layer neural network. Rhythm feature z can distinguish different action piecesThe segment varies in cadence over time.
Step 303, determining the distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature in any two rhythm features according to the first distance formula.
The first distance formula includes:
D match (A i ，M j )＝D motion (M i ，M j )＝||z(M i )-z(M j )||；
wherein A is i For the music piece corresponding to the first playing feature, M j For the action segment corresponding to the second rhythm feature, D match M is the distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature in any two rhythm features i For the action segment corresponding to the first playing characteristic, D motion For the distance between the motion segment corresponding to the first rhythm feature and the motion segment corresponding to the second rhythm feature in any two rhythm features, z (M i ) For the first characteristic of playing, z (M j ) Is a second cadence characteristic.
According to the Euclidean distance between the first rhythm feature and the second rhythm feature, the distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature can be determined. The euclidean distance, i.e. euclidean metric, is the actual distance between two points in two and three dimensions.
For example, the Euclidean distance may be used to calculate the difference between the first and second rhythm features, i.e. the distance D between the action segment corresponding to the first rhythm feature and the action segment corresponding to the second rhythm feature motion Further obtaining the distance D between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature match 。
Step 304, determining n pieces of music and action pieces with the largest distance and m pieces of music and action pieces with the smallest distance from the pieces of music and action pieces corresponding to the rhythm features. m and n are integers greater than 0.
For the followingY-section manually-danced music dance segment can obtain Y 2 Distance between music piece and action piece, Y 2 The distance may have a problem of more data, so that the n pieces of music and action pieces with the largest distance and the m pieces of music and action pieces with the smallest distance among the pieces of music and action pieces corresponding to the rhythm features can be determined according to the first distance formula, so that the data can be reduced and the data can be more representative.
Step 305, obtaining the feature sequences of n pieces of music and action pieces, and the feature sequences of m pieces of music and action pieces.
According to f motion (M)＝[f anim (M，t 1 )，f anim (M，t 2 )，…，f anim (M，t N )]Determining a characteristic sequence of the action segment in a matrix form;
according to f audio (A)＝[f mfcc (A，t 1 )，f mfcc (A，t 2 )，…，f mfcc (A，t N )]Determining a characteristic sequence of the music piece in a matrix form;
wherein A is any music piece, t is any time of M and A, N is the sampling number of M and A, f mfcc (A, t) is the Mel frequency cepstrum coefficient of A at any time t.
Mel-frequency cepstral coefficients (Mel-Frequency Cepstral Coefficients, MFCC) are coefficients that constitute a Mel-frequency cepstral, whose frequency band more closely approximates the human auditory system.
Step 306, training data is acquired.
The training data comprises characteristic sequences of n pieces of music and action pieces, characteristic sequences of m pieces of music and action pieces, distances of n pieces of music and action pieces and distances of m pieces of music and action pieces.
The distance between the music piece and the action piece can comprise n distances farthest from the music piece and m distances nearest to the music piece, so that training data can be more representative.
Step 307, training the initial neural network model according to the training data to obtain feature maps about the music pieces and feature maps about the action pieces.
The initial neural network model may include, among other things, a 3-layer Long Short-term memory network (Long Short-TermMemory, LSTM).
Step 308, obtaining a matching distance model of the music piece and the action piece according to the feature mapping of the music piece and the feature mapping of the action piece.
The music piece and action piece matching distance model comprises:
D match (A i ，M j )＝h match (f audio (A i )，f motion (M j ))；
h match (f audio (A i )，f motion (M j ))＝||h audio (f audio (A i ))-h motion (f motion (M j ))||；
wherein h is audio For feature mapping with respect to musical pieces, h motion Mapping h for features about action segments match Mapping the matching distance of the music piece and the action piece.
In the embodiment of the disclosure, the Euclidean distance can be used for calculating the distance between the feature map about the music piece and the feature map about the action piece, so as to obtain the matching distance between the music piece and the action piece.
Step 309, determining the action transition distance according to the action transition distance formula.
The action transition distance formula comprises:
cost(M i ，M j )＝max{speed(f trans (M i ，M j ))}；
f trans (M i ，M j )＝blend(f from (M i )，f to (M j ))；
f from (M)＝[f anim (M，t N-s )，f anim (M，t N-s+1 )，…，f anim (M，t N+s-1 )，f anim (M，t N+s )]；
f to (M)＝[f anim (M，t -s )，f anim (M，t -s+1 )，…，f anim (M，t s-1 )，f anim (M，t s )]；
wherein D is trans To move the transition distance, cost (M i ，M j ) To take the maximum value as M in all joint speeds i Transition to M j At the cost of θ is a first threshold, speed is joint velocity, f trans For transitional motion, blend is a motion mixing algorithm, f from To be M i The time of the last frame is taken as the center, the length of the half beat is taken as a window, and the time of the last frame is taken as the center, from M i Intercepting half beat action, f to To be M j The moment of the first frame is taken as the center, the length of the half beat is taken as a window, and the time of the first frame is taken as the center, and the time of the second frame is taken as the window j The action of half beat is intercepted, s is the radius of the window.
The action blending algorithm blend may synthesize two action segments into one action segment.
In the embodiment of the disclosure, the number of samples may be 8 samples per beat, and the radius s of the window is 4.
Step 310, determining the distance between the music piece and the action piece according to the distance formula between the music piece and the action piece.
The distance formula of the music piece and the action piece comprises:
wherein D is the distance between the music piece and the action piece, { M x1 ，M x2 ，...，M xn And the action fragments in the action fragment library.
Matching the music piece with the action piece by a distance D match Transition distance D from action segment trans The distance D between the music piece and the action piece can be obtained by adding.
Step 311, obtain the music to be matched.
The music to be matched is music which is not manually danced, and the music to be matched comprises a plurality of music pieces to be matched. The music to be matched can be selected by a user through a terminal, or can be selected by an operator of the server, or can be selected by the server directly in a music library comprising a plurality of music to be matched.
In step 312, a plurality of target action segments with the smallest total distance from the plurality of music segments to be matched in the action segment library are determined by a dynamic programming algorithm.
The action fragment library comprises a plurality of action fragments. The dynamic programming algorithm is a method for solving the optimization problem, and can be used for determining a plurality of target action fragments with the smallest total distance from a plurality of music fragments to be matched in the action fragment library.
Step 313, matching the plurality of target action segments with the music to be matched.
And matching the target action fragments with the smallest total distance with the music fragments to be matched in the action fragment library with the music to be matched. After the matching is completed, connecting a plurality of target action fragments to obtain dance actions matched with the music to be matched.
According to the music and action matching method, a server can dance a large amount of music to be matched, and matching time of the music and the action is shortened.
In summary, the present disclosure provides a method for matching music and actions, where a training sample is a feature sequence of a music piece, a feature sequence of an action piece, and a distance between a music piece and an action piece, and a matching distance model of a music piece and an action piece is obtained by training, an output of the model is a matching distance between a music piece and an action piece, an action transition distance is obtained, a sum of the matching distance and the action transition distance is used as a distance between a music piece and an action piece, to-be-matched music including a plurality of to-be-matched music pieces is obtained, a plurality of target action pieces with the smallest total distance to the to-be-matched music pieces are determined, and a matching degree between the action pieces and the to-be-matched music is high. The method solves the problem of poor matching degree of the action segment and the music to be matched in the related technology.
In one exemplary embodiment, music is danced by a person wearing the motion capture device, and music dance is obtained by acquiring motion captured by the motion capture device and processing the acquired motion. According to the one-section step distance, one-section time length and one-section step distance, the two-section time length splits the music dance to obtain a plurality of music dance fragments, and the server acquires the plurality of music dance fragments. Determining rhythm characteristics according to the rhythm characteristic formula, determining Euclidean distance between a music piece corresponding to a first rhythm characteristic and an action piece corresponding to a second rhythm characteristic in any two rhythm characteristics according to the first distance formula, determining n music pieces and action pieces with the largest distance and m music pieces and action pieces with the smallest distance (m and n are integers larger than 0) in the music pieces and the action pieces corresponding to the rhythm characteristics, and obtaining feature sequences of the n music pieces and the action pieces and feature sequences of the m music pieces and the action pieces.
The method comprises the steps of obtaining feature sequences of n pieces of music and action pieces, feature sequences of m pieces of music and action pieces, distances of n pieces of music and action pieces and distances of m pieces of music and action pieces as training data. Training the initial neural network model according to the training data to obtain feature mapping about the music piece and feature mapping about the action piece, and obtaining a matching distance model of the music piece and the action piece according to the feature mapping about the music piece and the feature mapping about the action piece, wherein the output of the model is the matching distance of the music piece and the action piece. And determining the action transition distance according to the action transition distance formula. And determining the distance between the music piece and the action piece according to the output of the matching distance model of the music piece and the action piece.
And obtaining music to be matched, wherein the music to be matched comprises a plurality of music pieces to be matched. And determining a plurality of target action fragments with the smallest total distance with the plurality of music fragments to be matched in the action fragment library through a dynamic programming algorithm. And matching the plurality of target action fragments with the music to be matched.
Referring to fig. 4, a schematic structural diagram of a music and action matching device 400 according to an embodiment of the disclosure is shown, where the music and action matching device 400 may be a server. By way of example, as shown in fig. 4, the apparatus 400 includes a Central Processing Unit (CPU) 401, a system memory 404 including a Random Access Memory (RAM) 402 and a Read Only Memory (ROM) 403, and a system bus 405 connecting the system memory 404 and the central processing unit 401. Apparatus 400 also includes a basic input/output system (I/O system) 406 to facilitate the transfer of information between various devices within the computer, and a mass storage device 407 for storing an operating system 413, application programs 414, and other program modules 415.
The basic input/output system 406 includes a display 408 for displaying information and an input device 409, such as a mouse, keyboard, etc., for user input of information. Wherein both the display 408 and the input device 409 are coupled to the central processing unit 401 via an input output controller 410 coupled to the system bus 405. The basic input/output system 406 may also include an input/output controller 410 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input/output controller 410 also provides output to a display screen, a printer, or other type of output device.
The mass storage device 407 is connected to the central processing unit 401 through a mass storage controller (not shown) connected to the system bus 405. The mass storage device 407 and its associated computer-readable medium provide non-volatile storage for the apparatus 400. That is, mass storage device 407 may include a computer-readable medium (not shown) such as a hard disk or CD-ROM drive.
Computer readable storage media may include computer storage media and communication media without loss of generality. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the ones described above. The system memory 404 and mass storage device 407 described above may be collectively referred to as memory.
The apparatus 400 may also operate via a network, such as the internet, connected to a remote computer on the network, in accordance with various embodiments of the present disclosure. I.e., the apparatus 400 may be connected to the network 412 through a network interface unit 411 coupled to the system bus 405, or alternatively, the network interface unit 411 may be used to connect to other types of networks or remote computer systems (not shown).
The memory further includes one or more programs, one or more programs stored in the memory and configured to be executed by the CPU to implement the methods provided by the embodiments of the present disclosure.
The embodiment of the application also provides a computer storage medium, in which at least one instruction, at least one section of program, a code set or an instruction set is stored, and the at least one instruction, the at least one section of program, the code set or the instruction set is loaded and executed by a processor to implement the matching method of music and actions as provided in the above method embodiment.
The foregoing is merely an alternative embodiment of the present disclosure, and is not intended to limit the present disclosure, any modification, equivalent replacement, improvement, etc. that comes within the spirit and principles of the present disclosure are included in the scope of the present disclosure.
Claims (6)
1. A method of matching music to actions, the method comprising:
acquiring a plurality of manually-coded music dance segments, wherein each music dance segment comprises a music segment and a corresponding action segment;
determining a plurality of rhythm characteristics corresponding to the plurality of music dance segments according to a rhythm characteristic formula, wherein the rhythm characteristic formula comprises:
z(M)＝h z (f motion (M))＝[z 1 ，z 2 ，…，z zdim ] T ；
f motion (M)＝[f anim (M，t 1 )，f anim (M，t 2 )，…，f anim (M，t N )]；
f anim (M，t)＝[p(M，t)，q 1 (M，t)，q 2 (M，t)，…，q r (M，t)] T ；
wherein M is any action segment, z (M) is the rhythm feature corresponding to M, and h is z For feature mapping, the zdim is the dimension of the z (M), the f motion The characteristic sequence of M is in a matrix form, t is any moment of M, N is the sampling number of M, and f anim The characteristic of the M at the moment t in a matrix form is that p (M, t) is the three-dimensional space position of a root node, q (M, t) is the rotation information of the joint of the character, and r is the serial number of the joint;
determining the Euclidean distance of any two rhythm features in the plurality of rhythm features as the distance between a music piece corresponding to a first rhythm feature and an action piece corresponding to a second rhythm feature in the any two rhythm features;
determining n music pieces and action pieces with the largest distance and m music pieces and action pieces with the smallest distance in the music pieces and action pieces corresponding to the rhythm features, wherein m and n are integers larger than 0;
acquiring the characteristic sequences of the n pieces of music and the action pieces and the characteristic sequences of the m pieces of music and the action pieces;
acquiring training data, wherein the training data comprises characteristic sequences of the n music pieces and action pieces, characteristic sequences of the m music pieces and action pieces, distances between the n music pieces and the action pieces and distances between the m music pieces and the action pieces;
training the initial neural network model according to the training data to obtain feature mapping about music fragments and feature mapping about action fragments;
obtaining a matching distance model of the music piece and the action piece according to the feature mapping of the music piece and the feature mapping of the action piece, wherein the matching distance model of the music piece and the action piece comprises the following steps:
D match (A i ，M j )＝h match (f audio (A i )，f motion (M j ))；
h match (f audio (A i )，f motion (M j ))＝||h audio (f audio (A i ))-h motion (f motion (M j ))||；
wherein the A i For the music piece corresponding to the first playing feature, the M j For the action segment corresponding to the second rhythm feature, the D match The distance between the music piece corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature in the arbitrary two rhythm features is h audio For feature mapping with respect to a piece of music, the h motion For feature mapping with respect to action segments, the h match Mapping the matching distance between the music piece and the action piece, wherein the output of the matching distance model between the music piece and the action piece is the matching distance between the music piece and the action piece;
acquiring an action transition distance formula, wherein the action transition distance formula is used for outputting an action transition distance;
taking the sum of the matching distance between the music piece and the action transition distance as the distance between the music piece and the action piece;
obtaining music to be matched, wherein the music to be matched comprises a plurality of pieces of music to be matched;
determining a plurality of target action fragments with minimum total distances to the plurality of music fragments to be matched in an action fragment library, wherein the action fragment library comprises a plurality of action fragments;
matching the target action fragments with the music to be matched;
the obtaining the feature sequences of the n pieces of music and the action pieces, and the feature sequences of the m pieces of music and the action pieces, includes:
according to f motion (M)＝[f anim (M，t 1 )，f anim (M，t 2 )，…，f anim (M，t N )]Determining a characteristic sequence of the action segment in a matrix form;
according to f audio (A)＝[f mfcc (A，t 1 )，f mfcc (A，t 2 )，…，f mfcc (A，t N )]Determining a characteristic sequence of the music piece in a matrix form;
wherein A is any music piece, t is any time of M and A, N is the sampling number of M and A, and f mfcc (a, t) is the mel-frequency cepstral coefficient of said a at said any time t;
the action transition distance formula comprises:
cost(M i ，M j )＝max{speed(f trans (M i ，M j ))}；
f trans (M i ，M j )＝blend(f from (M i )，f to (M j ))；
f from (M)＝[f anim (M，t N-s )，f anim (M，t N-s+1 )，…，f anim (M，t N+s-1 )，f anim (M，t N+s )]；
f to (M)＝[f anim (M，t -s )，f anim (M，t -s+1 )，…，f anim (M，t s-1 )，f anim (M，t s )]；
wherein the M i For the action segment corresponding to the first playing feature, the D trans For the motion transition distance, the cost (M i ，M j ) To take the maximum value in all joint speeds as the M i Transition to the M j At the cost of θ is a first threshold, speed is the joint velocity, and f trans For transitional actions, the blend is an action mixing algorithm, and f from To the M i The time of the last frame is taken as the center, the length of the half beat is taken as a window, and the M is taken as the center i Intercepting the action of half beat, said f to To the M j A first frame time is taken as a center, a half beat length is taken as a window, and the M is taken as a center j And intercepting the action of half beat, wherein s is the radius of the window.
2. The method according to claim 1, wherein determining the euclidean distance between any two of the plurality of rhythm features as the distance between the piece of music corresponding to the first rhythm feature and the action piece corresponding to the second rhythm feature includes:
determining the distance between a music piece corresponding to a first rhythm feature and an action piece corresponding to a second rhythm feature in the two rhythm features according to a first distance formula, wherein the first distance formula comprises:
D match (A i ，M j )＝D motion (M i ，M j )＝||z(M i )-z(M j )||；
wherein the D is motion For the distance between the motion segment corresponding to the first rhythm feature and the motion segment corresponding to the second rhythm feature in the arbitrary two rhythm features, z (M i ) For the first nodal feature, the z (M j ) Is the second cadence characteristic.
3. The method of claim 1, wherein said summing the matching distance of the musical piece to the action piece and the action transition distance as the distance of the musical piece to the action piece comprises:
determining the distance between the music piece and the action piece according to a distance formula between the music piece and the action piece, wherein the distance formula between the music piece and the action piece comprises the following steps:
wherein D is the distance between the music piece and the action piece, and M is the sum of the distance between the music piece and the action piece x1 ，M x2 ，...，M xn And the action fragments are a plurality of the action fragments in the action fragment library.
4. The method of claim 1, wherein determining a plurality of target action segments in the action segment library that have a smallest total distance from the plurality of music segments to be matched comprises:
and determining the target action fragments with the smallest total distance with the music fragments to be matched in the action fragment library through a dynamic programming algorithm.
5. A music and action matching device, characterized in that it comprises a processor and a memory, in which a computer program is stored, which computer program is loaded and executed by the processor to implement the music and action matching method according to any one of claims 1 to 4.
6. A computer storage medium having a computer program stored therein, the computer program being loaded and executed by a processor to implement the method of matching music and actions of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911158848.6A CN111104964B (en) | 2019-11-22 | 2019-11-22 | Method, equipment and computer storage medium for matching music with action |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911158848.6A CN111104964B (en) | 2019-11-22 | 2019-11-22 | Method, equipment and computer storage medium for matching music with action |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111104964A CN111104964A (en) | 2020-05-05 |
| CN111104964B true CN111104964B (en) | 2023-10-17 |
Family
ID=70421341
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911158848.6A Active CN111104964B (en) | 2019-11-22 | 2019-11-22 | Method, equipment and computer storage medium for matching music with action |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111104964B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116830158A (en) * | 2020-09-30 | 2023-09-29 | 斯纳普公司 | Music reaction animation of human character |
| CN112989071B (en) * | 2020-12-14 | 2022-11-04 | 北京航空航天大学 | Music selection method based on human body dance emotion |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005231012A (en) * | 2004-02-23 | 2005-09-02 | Sony Corp | Robot device and its control method |
| AU2006292461A1 (en) * | 2005-09-16 | 2007-03-29 | Flixor, Inc. | Personalizing a video |
| JP2007292847A (en) * | 2006-04-21 | 2007-11-08 | Victor Co Of Japan Ltd | Musical piece editing/reproducing device |
| TW200746040A (en) * | 2005-12-19 | 2007-12-16 | David John Lumsden | Digital music composition device, composition software and method of use |
| CN101615302A (en) * | 2009-07-30 | 2009-12-30 | 浙江大学 | The dance movement generation method that music data drives based on machine learning |
| WO2013086534A1 (en) * | 2011-12-08 | 2013-06-13 | Neurodar, Llc | Apparatus, system, and method for therapy based speech enhancement and brain reconfiguration |
| CN106292424A (en) * | 2016-08-09 | 2017-01-04 | 北京光年无限科技有限公司 | Music data processing method and device for anthropomorphic robot |
| CN106598062A (en) * | 2016-07-29 | 2017-04-26 | 深圳曼塔智能科技有限公司 | Flight motion control method and device for unmanned plane |
| CN106875930A (en) * | 2017-02-09 | 2017-06-20 | 深圳市韵阳科技有限公司 | Lamp light control method and system based on song sound accompaniment and microphone voice real-time detection |
| CN107193551A (en) * | 2017-04-19 | 2017-09-22 | 北京永航科技有限公司 | A kind of method and apparatus for generating picture frame |
| CN108202334A (en) * | 2018-03-22 | 2018-06-26 | 东华大学 | A kind of Dancing Robot that can identify music beat and style |
| CN108369799A (en) * | 2015-09-29 | 2018-08-03 | 安泊音乐有限公司 | Using machine, system and the process of the automatic music synthesis and generation of the music experience descriptor based on linguistics and/or based on graphic icons |
| CN108527376A (en) * | 2018-02-27 | 2018-09-14 | 深圳狗尾草智能科技有限公司 | Control method, apparatus, equipment and the medium of robot dance movement |
| CN108733508A (en) * | 2017-04-17 | 2018-11-02 | 伊姆西Ip控股有限责任公司 | Method and system for controlling data backup |
| CN108744542A (en) * | 2018-06-08 | 2018-11-06 | 武汉蛋玩科技有限公司 | A kind of Robot dancing movements design method and a kind of robot |
| CN108877838A (en) * | 2018-07-17 | 2018-11-23 | 黑盒子科技（北京）有限公司 | Music special efficacy matching process and device |
| CN109189979A (en) * | 2018-08-13 | 2019-01-11 | 腾讯科技（深圳）有限公司 | Music recommended method, calculates equipment and storage medium at device |
| CN109176541A (en) * | 2018-09-06 | 2019-01-11 | 南京阿凡达机器人科技有限公司 | A kind of method, equipment and storage medium realizing robot and dancing |
| CN109522959A (en) * | 2018-11-19 | 2019-03-26 | 哈尔滨理工大学 | A kind of music score identification classification and play control method |
| CN109833608A (en) * | 2018-12-29 | 2019-06-04 | 南京华捷艾米软件科技有限公司 | A kind of auxiliary method and system of dance movement religion based on 3D body-sensing camera |
| CN109948796A (en) * | 2019-03-13 | 2019-06-28 | 腾讯科技（深圳）有限公司 | Self-encoding encoder learning method, device, computer equipment and storage medium |
| CN110324728A (en) * | 2019-06-28 | 2019-10-11 | 浙江传媒学院 | The competitive sports whole audience based on deeply study looks back short video generation method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010165169A (en) * | 2009-01-15 | 2010-07-29 | Kddi Corp | Rhythm matching parallel processing apparatus in music synchronization system of motion capture data and computer program thereof |
| KR101729195B1 (en) * | 2014-10-16 | 2017-04-21 | 한국전자통신연구원 | System and Method for Searching Choreography Database based on Motion Inquiry |
| US9672800B2 (en) * | 2015-09-30 | 2017-06-06 | Apple Inc. | Automatic composer |
| JP2017093803A (en) * | 2015-11-24 | 2017-06-01 | 富士通株式会社 | Evaluation program, evaluation method and evaluation device |
-
2019
- 2019-11-22 CN CN201911158848.6A patent/CN111104964B/en active Active
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005231012A (en) * | 2004-02-23 | 2005-09-02 | Sony Corp | Robot device and its control method |
| AU2006292461A1 (en) * | 2005-09-16 | 2007-03-29 | Flixor, Inc. | Personalizing a video |
| TW200746040A (en) * | 2005-12-19 | 2007-12-16 | David John Lumsden | Digital music composition device, composition software and method of use |
| JP2007292847A (en) * | 2006-04-21 | 2007-11-08 | Victor Co Of Japan Ltd | Musical piece editing/reproducing device |
| CN101615302A (en) * | 2009-07-30 | 2009-12-30 | 浙江大学 | The dance movement generation method that music data drives based on machine learning |
| WO2013086534A1 (en) * | 2011-12-08 | 2013-06-13 | Neurodar, Llc | Apparatus, system, and method for therapy based speech enhancement and brain reconfiguration |
| CN108369799A (en) * | 2015-09-29 | 2018-08-03 | 安泊音乐有限公司 | Using machine, system and the process of the automatic music synthesis and generation of the music experience descriptor based on linguistics and/or based on graphic icons |
| CN106598062A (en) * | 2016-07-29 | 2017-04-26 | 深圳曼塔智能科技有限公司 | Flight motion control method and device for unmanned plane |
| CN106292424A (en) * | 2016-08-09 | 2017-01-04 | 北京光年无限科技有限公司 | Music data processing method and device for anthropomorphic robot |
| CN106875930A (en) * | 2017-02-09 | 2017-06-20 | 深圳市韵阳科技有限公司 | Lamp light control method and system based on song sound accompaniment and microphone voice real-time detection |
| CN108733508A (en) * | 2017-04-17 | 2018-11-02 | 伊姆西Ip控股有限责任公司 | Method and system for controlling data backup |
| CN107193551A (en) * | 2017-04-19 | 2017-09-22 | 北京永航科技有限公司 | A kind of method and apparatus for generating picture frame |
| CN108527376A (en) * | 2018-02-27 | 2018-09-14 | 深圳狗尾草智能科技有限公司 | Control method, apparatus, equipment and the medium of robot dance movement |
| CN108202334A (en) * | 2018-03-22 | 2018-06-26 | 东华大学 | A kind of Dancing Robot that can identify music beat and style |
| CN108744542A (en) * | 2018-06-08 | 2018-11-06 | 武汉蛋玩科技有限公司 | A kind of Robot dancing movements design method and a kind of robot |
| CN108877838A (en) * | 2018-07-17 | 2018-11-23 | 黑盒子科技（北京）有限公司 | Music special efficacy matching process and device |
| CN109189979A (en) * | 2018-08-13 | 2019-01-11 | 腾讯科技（深圳）有限公司 | Music recommended method, calculates equipment and storage medium at device |
| CN109176541A (en) * | 2018-09-06 | 2019-01-11 | 南京阿凡达机器人科技有限公司 | A kind of method, equipment and storage medium realizing robot and dancing |
| CN109522959A (en) * | 2018-11-19 | 2019-03-26 | 哈尔滨理工大学 | A kind of music score identification classification and play control method |
| CN109833608A (en) * | 2018-12-29 | 2019-06-04 | 南京华捷艾米软件科技有限公司 | A kind of auxiliary method and system of dance movement religion based on 3D body-sensing camera |
| CN109948796A (en) * | 2019-03-13 | 2019-06-28 | 腾讯科技（深圳）有限公司 | Self-encoding encoder learning method, device, computer equipment and storage medium |
| CN110324728A (en) * | 2019-06-28 | 2019-10-11 | 浙江传媒学院 | The competitive sports whole audience based on deeply study looks back short video generation method |
Non-Patent Citations (2)
| Title |
|---|
| 方丹芳 ; 李学明 ; 柳杨 ; 李荣锋 ; .基于过渡帧插值的音乐驱动舞蹈动作合成.复旦学报(自然科学版).2018,(03),全文. * |
| 樊儒昆 ; 傅晶 ; 程司雷 ; 张翔 ; 耿卫东 ; .动作与音乐的节奏特征匹配模型.计算机辅助设计与图形学学报.2010,(06),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111104964A (en) | 2020-05-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108615055B (en) | Similarity calculation method and device and computer readable storage medium | |
| US20210029305A1 (en) | Method and apparatus for adding a video special effect, terminal device and storage medium | |
| CN109462776B (en) | Video special effect adding method and device, terminal equipment and storage medium | |
| CN112333179B (en) | Live broadcast method, device and equipment of virtual video and readable storage medium | |
| CA2843343C (en) | Systems and methods of detecting body movements using globally generated multi-dimensional gesture data | |
| Laraba et al. | Dance performance evaluation using hidden Markov models | |
| US20230260326A1 (en) | Dance segment recognition method, dance segment recognition apparatus, and storage medium | |
| CN111104964B (en) | Method, equipment and computer storage medium for matching music with action | |
| CN109308437B (en) | Motion recognition error correction method, electronic device, and storage medium | |
| WO2023273628A1 (en) | Video loop recognition method and apparatus, computer device, and storage medium | |
| CN110298220B (en) | Action video live broadcast method, system, electronic equipment and storage medium | |
| JP2021140780A (en) | Computer-executed method and device for map creation, electronic apparatus, storage medium, and computer program | |
| CN110516749A (en) | Model training method, method for processing video frequency, device, medium and calculating equipment | |
| CN113870395A (en) | Animation video generation method, device, equipment and storage medium | |
| CN111784776A (en) | Visual positioning method and device, computer readable medium and electronic equipment | |
| CN115691544A (en) | Training of virtual image mouth shape driving model and driving method, device and equipment thereof | |
| CN113569753A (en) | Action comparison method and device in video, storage medium and electronic equipment | |
| CN114708660A (en) | Tennis action scoring method, system and equipment based on average sequence law finding | |
| CN108182227B (en) | Accompanying audio recommendation method and device and computer-readable storage medium | |
| CN111353347B (en) | Action recognition error correction method, electronic device, and storage medium | |
| WO2023061229A1 (en) | Video generation method and device | |
| CN114554111A (en) | Video generation method and device, storage medium and electronic equipment | |
| CN111782858B (en) | Music matching method and device | |
| CN110781820B (en) | Game character action generating method, game character action generating device, computer device and storage medium | |
| Ofli et al. | Multi-modal analysis of dance performances for music-driven choreography synthesis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |