KR20040008378A - Synchronization Method of Rhythmic Motion with Music based on Beat Analysis - Google Patents

Abstract

PURPOSE: A motion rhythm analysis method, a motion transferring method in consideration of continuity, and a method of synchronizing a motion with music based on bit analysis using the methods are provided to effectively produce animation with music. CONSTITUTION: Example motions are analyzed to obtain bits corresponding to a moment when a sudden direction change is periodically generated in the motion of the human body as candidate bits. A bit of the most predominant periodic moment among the candidate bits is obtained as a reference bit. Actual motion bits are acquired from representatives of candidate bits distributed around the reference bit.

Description

Rhythm analysis method of rhythm, motion transition method considering continuity and method of synchronizing rhythm and music using beat analysis {Synchronization Method of Rhythmic Motion with Music based on Beat Analysis}

The present invention relates to a rhythm analysis method of rhythm, an operation transition method considering continuity, and a method of synchronizing rhythm with music based on beat analysis using the same, and more specifically, to obtain a motion beat automatically by analyzing a given rhythm. The present invention relates to a method for generating a new rhythm synchronized with a given piece of music.

In computer animation, film and television propaganda, virtual character movements are used in combination with sound to impress.

In particular, music is widely used because it can effectively convey various emotions through its unique rhythm and melody.

Since music has rhythms and melodies, unlike normal sound effects, it can't be arbitrarily modified to synchronize with the action.

Therefore, character movements are often produced after the music track is recorded first.

In general, rhythmic movements such as dance or march, ie rhythms, are formed by moving to background music and have the same rhythm as background music.

Meanwhile, as realistic motion clips are widely used, studies have been conducted to create new motions using captured motion clips.

Research related to the present invention can be broadly divided into motion editing, motion transition modeling, sound rendering, and mouth synchronization.

First, a number of studies have been proposed for the editing of motions.

Bruderlin and Willianms [A. Bruderlin and L. Williams. Motion Siganl Processing. Computer Graphics (Proceedings of SIGGRAPH '95) , 29: 97-104, August 1995.] introduced signal processing techniques and transformed them into desired forms while preserving the details of their operation.

Witkin and Popovic (A. Witkin and Z. Popovic. Motion Warping. Computer Graphics (Proceedings of SIGGRAPH '95) , 29: 105-108, August 1995.) proposed a motion warping technique for the same purpose.

Rose et al. (C. Rose, B. Guenter, B. Bodenheimer, and MF Cohen.Efficient Generation of Motion Transitions using Spacetime Constraints.Computer Graphics (Proceedings of SIGGRAPH '96) , 30: 147-154, August 1996). Conditions were used to create a smooth transition between actions.

Gleicher (M. Gleicher.Motion Editing with Spacetime Constraints.In Proceedings of Symp.Interactive 3D Graphics , pages 139-148, 1997.) (M. Gleicher.Retargetting Motion to New Characters.Computer Graphics (Proceedings of SIGGRAPH '98) , 32: 33-42, July 1998.) solved the motion reapplied problem as a simplified spatiotemporal problem with kinematic constraints.

To efficiently solve the problem of motion reapply, Lee and Shin (J. Lee and SYShin.A Hierarchical Approach to Interactive Motion Editing for Human-like Figures.Computer Graphics (Proceedings of SIGGRAPH '99) , 33: 395-408, August 1999.) and Korein and Badler (J. Kor Korin and NI Badler. Techniques for Generating the Goal-directed Motion of Articulated Structures.IEEE Computer Graphics and Applications , pages 71) -81, November 1982.) proposed an efficient inverse kinematics solution that introduces the elbow circle.

Shin et al. (HJ Shin, J. Lee, M. Gleicher, and SY Shin.Computer Puppetry: An Importance-Based Approach.ACM Transactions On Graphics , 20, 2001.) are online behaviors for real-time performance animation. An application technique is proposed.

For the motion transition modeling, the Hidden Markov Model (HMM) has been used for expressing human motion and generating realistic motion from it.

Second, in motion transition modeling, HMM (Hidden Markov Model) has been used for expressing human motion and generating realistic motion from it.

Galata et al. (A. Galata, N. Johnson, and D. Hogg.Learning Variable-Length Markov Models of Behavior.Computer Vision and Image Understanding , 81) describe behavior as an extended Markov model with long memory dependencies.

Brand and Herzmann (M. Brand and A. Hertzmann. Style Machines.Computer Graphics (Proceedings of SIGGRAPH 2000) , 34: 183-192, July 2000.), SHMM (Stylistic HMM), where HMM is a parameter whose parameters depend on style variables. After separating the action into actions and styles, we create new actions according to the given style variables.

Tanco and Hilton used the Markov model to represent motion with a group of similar postures.

Given a sketch of the motion, the state path in the Markov model is determined, and a new motion is created by getting the most appropriate motion from each state.

Lee et al. [J. Lee, J. Chai, PSA Reitsma, JK Hodgins, and NS Pollard. Interactive Control of Avatars Animated with Human Motion Data. Computer Graphics (Proceedings of SIGGRAPH 2002) (to appear) , 36, July 2002.) used a two-layered structure to control virtual characters.

The lower layer represents the transition between each frame of the example motion data, and the upper layer represents the bundle of similar frames and the transition between them.

Third, there have been a number of studies for creating soundtracks that are synchronized to a given action for the sound rendering.

Takala and Han (T. Takala and J. Hahn.Sound Rendering.Computer Graphics (Proceedings of SIGGRAPH '92) , 26: 211-220, 1992.) used the term sound rendering for the first time. By creating a sound scene, we synchronized the sound with the image.

O'Brien et al. (JF O'Brien, PR Cook, and G. Essl.Synthesizing Sounds From Physically Based Motion.Computer Graphics (Proceedings of SIGGRAPH2001) , 35: 529-536, August 2001.) A technique for simulating sound effects by calculating the transition of acoustic pressure waves is proposed.

van den Doel et al. (K. van den Doel, PG Kry, and DK Phai.FOLEYAUTOMATIC: Physically-Based Sound Effects for Interactive Simulation and Animation.Computer Graphics (Proceedings of SIGGRAPH 2001) , 35: 537-544, August 2001.) Automatically generated contact sounds between objects using the physical parameters obtained by the rigid body simulation.

Fourth, about the mouth synchronization Due to the large proportion of face movements in the animation of humanoid characters, considerable research has been conducted on the synchronization of mouth movements and voices.

Bregler et al. (C. Bregler, M. Covell, and M. Slaney.Video Rewrite: driving visual speech with audio.Computer Graphics (Proceedings of SIGGRAPH '97) , 31: 353-360, August 1997.) The mouth movement was recombined to generate a new mouth movement most suitable for the input voice.

Brand (M. Brand. Voice Puppetry. Computer Graphics (Proceedings of SIGGRAPH '99) , 33: 21-28, August 1999.) automatically acquires a control model for facial movement from the example video, A new animation was created by manipulating the control model with the input voice.

Cassel et al. (J. Cassell, HH Vilhjlmsson, and T. Bickmore.BEAT: the Behavior Expression Animation Toolkit.Computer Graphics (Proceedings of SIGGRAPH 2001) , 35: 477-486, August 2001.) We proposed an animation tool that can synchronize motion.

However, conventional motion-related technologies focus mainly on satisfying space-time constraints related to kinematics and dynamics, and do not place much emphasis on rhythm.

In addition, conventional studies for synchronizing a given motion and sound mainly focus on sound effect generation or mouth movement.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problem, and proposes a technique for generating a new rhythm synchronized to a given music using example motion clips having the object of the present invention in advance.

Another object of the present invention, the rhythm can be expressed as a sequence of unit operations corresponding to the rhythm pattern of the background music, each rhythm pattern is composed of bits of a certain time unit, in the beat of the background music It is known that a sudden change of direction occurs, and based on the observation result, the concept of an operation bit corresponding to a beat of music is proposed.

It is still another object of the present invention to divide an operation into bits by analyzing the bits of the example operations and to construct a motion transition graph representing a linkable relationship between the unit motions, and given a new music, the music given from the motion transition graph To create a new rhythm that is synchronized to.

Another object of the present invention is to provide a means by which a choreographer can efficiently design a new movement, which can be effectively used in the combination of a character's motion and music.

1 is an overall structural diagram of a synchronization method according to the present invention.

2 illustrates a reference bit estimation step from a single operation signal according to the present invention.

3 is a diagram illustrating a reference bit estimating step from a multiple operation signal according to the present invention.

4 is a diagram illustrating a distribution of candidate bits according to the present invention.

5 is an operation transition graph according to the present invention.

6 is a diagram illustrating determination of a transition possibility between unit operations.

7 is a graph of the accuracy of the bit analysis according to the present invention.

8 is a comparison of synchronized and generally unsynchronized operations in accordance with the present invention.

9 is a diagram of an example couple operation synthesis according to the present invention.

10 is a diagram showing that various operations can be obtained from various operations from the same path on the operation transition graph according to the present invention.

Hereinafter, the configuration and operation of the present invention will be described with reference to the accompanying drawings.

Referring first to the technical principle of the present invention, the present invention consists of a motion analysis step (S100) and operation synthesis step (S200) as shown in FIG.

In the operation analysis step S100, the example operations are analyzed (S110) to obtain a transition between these operation bits and the unit operations constituting them (S120).

Based on the observation that the sudden change of direction occurs periodically in operation, the moment of change of direction is found and the operation bit is estimated from them.

The fact that a sudden change of direction at a moment does not necessarily mean that the moment corresponds to an operation bit can be regarded as a candidate for the operation bit.

Using the fact that the operation bits are periodic, the reference bits that can be expected as operation bits are obtained by finding the dominant periodic pattern from the candidate bits.

An estimate of the operation bit is obtained by selecting representatives from the candidate bits based on each reference bit.

Based on the operation bits, the example operations are divided into unit operations according to the length of a given rhythm pattern.

For an effective combination of unit operations, an edge transition is a unit motion and a node constructs a motion transition graph indicating transitions of unit operations.

The motion transfer model analyzes the arrangement of the unit actions shown in the example action, so that the more frequent the connection of the unit actions is, the more likely this connection is.

When the input music is given in the operation synthesis step (S200), a new rhythm is generated from the motion transition graph corresponding thereto (S210).

By traversing the motion transition graph based on the possibility of transition between unit motions obtained from the example motions, a path corresponding to a new motion can be obtained in real time.

In addition, by finding a path having the most optimal probability on the motion transition graph, an operation most suitable for the input music can be obtained.

A new motion can be generated by connecting the unit motions corresponding to each edge of the path on the graph.

In order to synchronize the operation with the input music, a time warping technique is applied to match the operation bits with the bits of the music.

The generated operation adjusts the degree of change of direction in the operation bit using timing control (S220).

Finally, the generated motion is reapplied to various characters (S230).

The above steps will be described in more detail below.

1. Operation bit analysis

Human body motion data consists of a bundle of motion signals, each motion signal being represented by a sequence of values representing the position or direction of each joint.

Due to the hierarchical structure of the human body, there is a time difference in movement of each joint in motion, which may cause bits to appear slightly different for each joint.

To get a bit out of motion, we need to offset the difference between these joints.

In the present invention, first, candidate bits are obtained for each operation signal, the most prevalent reference bits are obtained, and the actual operation bits are estimated using the statistical technique.

1-1 candidate bit extraction.

According to observations in the field of human psychology and biomechanics (MR Jones and M. Boltz. Dynamic Attending and Responses to Time.Psychological Review , 96 (3): 459-491, 1989.) Happens and these form the rhythm of movement.

Therefore, these periodic moments can be regarded as operation bits.

The moment of rapid change of direction has been used to acquire important characteristics of motion (J. Gomes and L. Velho. From Fourier Analysis to Wavelets. In Course Notes # 5 (SIGGRAPH '99) , 1999.).

Here, the zero-crossing of the second derivative of the motion signal is considered the moment when the motion and environment interact.

By obtaining the angular acceleration for each motion signal, we can find the moment when the second derivative crosses zero.

In actual operation, the direction change may occur severely regardless of the bit, and thus, the moment when the second derivative passes zero can be regarded as a candidate of the operation bit.

The distribution of candidate bits includes both the actual motion bits and the time difference between the joints and the outliers that are not related to the bits.

Thus, the most prevalent periodic moments must be obtained throughout them.

1-2. Reference bit estimation

Reference bit estimation is estimated by dividing into a single joint and multiple joints.

First, the most prevalent periodic moments are obtained from the distribution of candidate bits obtained from one motion signal in a single joint.

The problem of estimating the most dominant period of the candidate bits can be expressed as a problem of obtaining the period of pulse sequence.

In the pulse sequence period estimation problem, the generation time of each pulse is modeled as in Equation 1 below.

here Is the cycle you want to find, Is a uniformly distributed random phase value, Is a positive integer, Has 0 as the average White Gaussian noise in the range.

Let be a list of candidate bits. here In Equation 1 Corresponds to

However, as mentioned above, if the second derivative passes zero, the moment is not a bit, May be a simple outer layer.

This is a period in Equation 1 It has nothing to do with.

Without losing generality, Assume that is sorted in ascending order.

The problem of estimating the most dominant period of Sine wave with maximum in We can transform it into the problem of finding the most predominant frequency of.

To make the sine wave simple from One period is entered into the interval between, that is to be represented by equation (2).

here Is Is an integer belonging to to be. Is not strictly a sine wave, but there is no big problem because the difference from the sine wave is so small that it can be regarded as simple noise.

Uniformly based on the Nyquist sampling theorem Values Extract (but, ).

Then, using the periodogram method of Equation 3 You can analyze the spectrum.

The power spectrum density of is represented by a clear impulse signal (J. Gomes and L. Velho. From Fourier Analysis to Wavelets. In Course Notes # 5 (SIGGRAPH '99) , 1999.)

therefore, Reference bits by finding the highest peak of Can be estimated.

As shown in FIG. 2, a reference bit for a single operation signal estimates the reference bit by constructing a sinusoid from the candidate bits and obtaining a weighted dominant frequency.

The periodic estimation technique for a single joint is extended for simple application to multiple motion signals due to the linearity of the Fourier transform.

Second, the multiple joints obtain the most prevalent period from the candidate bits obtained from the multiple motion signals by power spectrum analysis as in the single motion signal.

In the case of analyzing two operating signals, for each operating signal, a sine wave is formed from bit candidates, Sine waves Superpose to obtain one signal.

And the signals combined using the periodogram Power spectral density Obtain then, Frequency to maximize The most dominant period across multiple motion signals by obtaining Can be obtained.

Phase value in Equation 1 Can be obtained from Equation 4 below.

Reference bits are estimated period And phase It can be obtained from equation (5) from.

here, Is Th reference bit, It represents the moment expected by the first actual operation bit.

As shown in FIG. 3, the reference bits for the multiple joints form a sine wave for each operation signal, and the reference bits are formed by finding the most predominant frequency in the signal obtained by the sum of these signals.

The actual operation bit is estimated based on the candidate bits and the reference bits obtained by the method described above.

1-3 Operation Bit Estimation

Ideally, the reference bits and the actual operating bits should match exactly, but in reality the bit spacing is not exactly uniform and there is a slight phase difference between the candidate bits of each joint.

Therefore, the representative of candidate bits distributed around each reference bit can be viewed as actual bits.

When projecting the candidate bits for all joints at once, the group of candidate bits is distributed around the actual operation bits as shown in FIG. 4, and the outer layer is also widely distributed.

Some operations may not include abrupt changes in direction over time over several bits.

Motion bit end Consider the case in which the bit candidates are observed within the low range.

section Their representatives from the bit candidates within Is obtained using Huber's M-estimator (PJ Huber. Robust Statistics . John Wiley & Sons, New York, 1981.).

That is, satisfying the following equation (6) Obtain

here Is defined by Equation 7 as a function of Huber.

Window function Is defined by Equation 8.

Estimate Is calculated using a downhill simplex minimization algorithm (WH Press, SA Teukolsky, WT Vetterling, and BP Flannery. Numerical Recipes in C-The Art of Scientific Computing . Cambridge University Press, second edition, 1999.) Get the enemy.

At this time, reference bit Is used as the initial value of the minimization algorithm.

Beat candidate Reference bit if not distributed in To Motion bit Specify with.

The virtual operation bit thus defined does not appear in actual operation, but is necessary for synchronization.

2. Motion Transition Graph

A rhythm to a background music can be expressed as a sequence of unit actions.

The present invention proposes an operation transition graph indicating the connection possibility between unit operations obtained in the example operations.

For a piece of music given as an input, the motion transition graph is constructed independently by grouping together those with the same rhythm pattern among the example motions to create a new motion with only unit motions having the same rhythm pattern as the song.

Since all unit operations with the same rhythm pattern have the same number of bits, all edges of the transition graph have the same number of operation bits.

We automatically learn the transition model between them by analyzing the order in which the unit actions appear in the example actions.

The operation transition graph step consists of constructing an operation transition graph and acquiring a unit operation transition model.

2-1. Graph of Operation Transition

Unit actions are obtained by uniformly cutting the example actions to the length of the rhythm pattern.

At this time, the rhythm pattern of the motion is determined according to the type of background music.

For example, a rhythm pattern to a waltz song has a rhythm pattern of three beats, so that every unit motion consists of three beats.

Thus, the waltz dance unit motions can be obtained by breaking the example motions every three beats.

One example operation may be represented by a directional linear graph in which the edge is a unit motion and the node is a posture corresponding to the start or end of the unit motion.

Any path in the linear graph is part of the example operation as an array of unit operations.

Two unit motions can be linked together if the last posture and speed of one unit motion are sufficiently similar to the first attitude and speed of the other unit motion.

Therefore, other connections than the sequential connection between the unit operations shown in the linear graph are possible.

Possible transitions between unit motions can be obtained by classifying nodes on a linear graph into groups of nodes with similar attitudes and speeds, and merging nodes belonging to the same group into one.

In order to classify the nodes on the graph into groups with similar posture and velocity, we first establish a measure to measure the similarity between the two nodes.

The measure of similarity is defined as the sum of the positional and velocity differences between the joints represented by the two nodes.

In this case, it is assumed that all the example motions are normalized to fit a uniform character in advance, and the defined similarity measure is expressed by the following equation (9).

here Is the number of joints, Wow Are each node and Of the posture corresponding to The position of the second joint.

Also, Wow Are each node and Of the posture corresponding to Is the velocity of the first joint.

and Is a constant representing the weight of the position difference and the speed difference.

Nodes are classified into groups of nodes having similar attitudes and speeds using the similarity measure defined in Equation (9).

At this time, since the number of classified groups is not known, a deduction clustering technique (S. Chiu. Fuzzy Model Identification Based on Cluster Estimation. Journal of Intelligent & Fuzzy Systems , 2 (3), September 1994.) is used.

The subtractive clustering method is a fast single pass algorithm that estimates the number of groups and the representative value of each group from the data based on the influence parameter of each group. Representing values that exert a significant influence on the most surrounding data, provided that they can be obtained.

By merging nodes belonging to the same population into one, the linear graphs representing example operations can be transformed into a single transition graph representing all possible transitions of unit operations.

Since any path on the transition graph corresponds to a sequence of unit actions, it can be converted into one rhythm.

FIG. 5 is a diagram showing an overall view of the transition graph structure. Each example motion is represented by a linear graph in which edges are unit motions and nodes represent the start or end poses of each motion. By merging nodes with similar posture and speed into one, they are transformed into a transition graph.

2-2. Acquisition of unit motion transition model

New movements can be obtained from any path of the transition graph.

However, the randomly chosen path does not always reflect the desired rhythm because it does not reflect the continuity in choreography.

Therefore, there is a need to give higher connection possibilities to transitions between more naturally connected unit operations.

The transition model between them is learned from the sequence of unit actions shown in the example action.

Example actions Suppose is given

Without losing generality, Is a list of unit actions Can be expressed as

From the list of unit actions Is the next unit It is easy to see that there is a high probability that the connections will be made naturally because they are successive operations in the and the actual operation.

Analyzing the example behaviors yields more natural connections in addition to these sequential connections.

In rhythm, unit actions are repeated in a similar fashion (SC Minton. Choreography: A Basic Approach Using Improvisation . Human Kinetics, 2nd edition, 1997.).

In particular, in the case of a standardized operation such as Waltz, this is even more so since a new operation is constructed from predefined unit operations.

Similar unit operations may be regarded as variations of the same operation.

Thus unit operations Groups of variants of the same behavior Can be classified as

A natural transition between groups of actions can then be obtained from the example action, which corresponds to a transition between various unit actions.

For example, ego Speaking of All unit actions belonging to It can be naturally connected to all unit operations belonging to.

because, this Can be connected naturally with All actions belonging to, and This is because all operations belonging to are similar enough to each other.

Example operations in FIG. 6 From the list of unit operations Wow Are each , You can see that it can be connected naturally with.

By grouping similar movements (in the drawing Wow ), Behavior group All unit actions belonging to the action group or It can be seen that it can be naturally connected to all the unit operations belonging to.

therefore I More diverse transitions are obtained, such as

In this way, the possibility of transition between unit operations can be determined.

Unit actions in the first unit action classification Is a group of variations of the same behavior A measure of similarity between two given unit operations is needed to classify as.

According to the literature on the copyright protection of choreographed works, the similarities between actions are often measured by comparing postures at their respective steps (J. Van Camp. Copyright of Choreographic Works. 1994-1995 Entertainment, Publishing and the Arts Handbook , pages 59-92, 1994.).

Based on this, the similarity between the movements is defined as the weighted sum of the posture and the speed difference in the bit as shown in Equation 10 below.

here and Each action and Is the number of operation bits and Is and of The posture at the first bit.

Using this scale, similar operations can be grouped into a group by subtraction clustering.

Second, in order to learn the transition model between groups of similar motions in transition model acquisition, the following three random variables are defined.

here Time Is a state variable indicating on which edge on the graph the transition is Is a virtual edge connectable from all the other edges to indicate the termination state.

Edge Edge Is the probability that a transition to Wow Each edge Indicates that the unit operation corresponding to is used as an initial or end operation.

Since the transition graph is constructed from example operations, each example operation is represented by one path of the transition graph.

For example, example behavior Is unit operation Virtual edge on the list of It may be indicated by a path ending with.

Using the training data as paths corresponding to the example operations, Baum-Welch's (expectation-modificaion) technique (X. Li and M. Parizeau.Training Hidden Markov Models with Multiple Observations-A Combinatorial Method.IEEE Transactions on Pattern Analysis and Machine Intelligence , 22 (4): 371-377, April 2000.) (LR Rabiner.A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition.Proceedings of the IEEE , 77 (2): 257-286, February 1989 Random variables that make up the transition model , , Can be obtained as in Equation 12.

Since the random variables are determined only by the transition of the motion that actually appeared in the example motion, only the transition between the unit motions with continuity in the choreography is allowed.

Start or end probability Wow Instead of automatically estimating by Baum-Welch's technique, the user can specify the start and end actions by user.

3. Operation Synthesis

By using the motion transition graphs configured in the motion analysis step S100, a new rhythm synchronized with a given piece of music is generated.

We first get a list of unit moves from the transition graph, synchronize them to the given music, and then reapply them to the given character.

3-1 Create Action

New movements can be obtained by finding a path with the same length as a given piece on the graph of motion transitions.

At this time, the movement transition graph is selected from among the movements having the same rhythm pattern as a given piece of music.

The rhythm pattern of the input music can be specified by the user or by HMM-based music recognition technology (W. Chai and B. Vercoe. Folk Music Classification Using Hidden Markov Models.In International Conference on Artificial Intelligence , June 2001.) have.

The present invention proposes two path selection techniques for motion generation.

One selects a random path based on a transition probability for real-time animation, and the other selects a path that maximizes continuity in choreography between unit motions.

First, in real-time motion generation, the motion transition graph is traversed on the fly for real-time animation.

A new action is obtained by concatenating the unit actions corresponding to the edge on the selected path.

Graph traversal is the starting probability of unit motion Starts at a nonzero edge.

The unit motion corresponding to one edge is taken out and then transitioned to another edge based on the transition probability at that edge.

At this time, since each edge of the motion transition graph is a group of similar unit motions, one of them is arbitrarily selected.

Thus, when traversing the graph on the fly (A. Schodl, R. Szeliski fact that such Schodl able to reach a dead end with no edge is connected to the other edge, DH Salesin, and I. Essa. Video Textures. Computer Graphics (Proceedings of SIGGRAPH 2000), 34 :.. 489-498, July 2000.) and Lee et al. (J. Lee, J. Chai, PSA Reitsma, JK Hodgins, and NS Pollard Interactive Control of Avatars Animated with Human Motion Data Computer Pointed out by Graphics (Proceedings of SIGGRAPH 2002) (to appear) , 36, July 2002.), he proposed a way to avoid dead ends.

However, the former technique does not completely block the way to the dead edge, and the latter technique is the largest of the strongly connected components in the motion transition graph. If you choose just that, you'll be throwing away much of the example behavior.

In the present invention, dead edges are removed from the graph by sequentially deleting edges that have no connection to other edges in the graph.

This process may render some of the example operations unusable, but by adjusting the appropriate clustering parameters as described above, only a negligible portion of the dead edge is removed.

In the second optimal motion generation, the optimal motion can be obtained by selecting a path where the accumulation of transition probabilities between edges on the path becomes the best.

Length Path The cumulative probability of can be obtained by multiplying the transition probability between the edges, the initial probability and the end probability as shown in Equation 13.

here to be.

Optimal path Can be obtained using Viterbi's dynamic planning algorithm (LR Rabiner.A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition.Proceedings of the IEEE , 77 (2): 257-286, February 1989.).

Variable as shown in Equation 14 Let's define it.

here ego Is the number of edges in the transition graph. In other words, Time in Path ending with The highest probability at.

Equation 14 may be represented by Equation 15 by induction.

only to be.

Through the following initial condition (Equation 16) and end condition (Equation 17), the path having the optimal probability and the cumulative probability value of the path can be obtained.

The optimal path with the optimal probability can be easily obtained by storing the state of the previous step from the previous step in the dynamic programming.

3-2. Synchronize and Reapply Behavior

The bits of the generated generated operation are made from example operations with various tempo, so the bits are not constant and may be different from the bits of the input music.

A time warping technique (C. Rose, B. Bodenheimer, and M. Cohen. Verbs and Adverbs: Multidimensional Motion Interpolation) that has motion bits and music bits as key time to synchronize the generated motion with the input music. IEEE Computer Graphics and Applications , 18 (5): 32-40, 1998.).

Since the motion data are discrete values extracted from the bundle of successive motion signals, first create a B-spline curve passing through them.

The B-spline curve is resampled so that the interval between operating bits is equal to the interval between music bits.

The change in direction in the bits of the generated action can be more strongly or weakly adjusted.

Direct changes in the velocity of the joints change their trajectories and may not meet kinematic conditions.

In the present invention, the dual interpolation technique (D. Chi, M. Costa, L. Zhao, and N. Badler.The EMOTE Model for Effort and Shape.Computer Graphics (Proceedings of SIGGRAPH 2000) can change the speed while maintaining the original trajectory. ) , 34: 173-182, July 2000.), (S. Steketee and N. Badler. Parametric Keyframe Interpolation Incorporating Kinetic Adjustment and Phasing Control.Computer Graphics (Proceedings of SIGGRAPH '85) , 19: 252-262, July 1985 Use timing control.

Timing control can emphasize or mitigate the rhythm of motion.

Finally, using real-time motion reapplying techniques (HJ Shin, J. Lee, M. Gleicher, and SY Shin.Computer Puppetry: An Importance-Based Approach.ACMTransactions On Graphics , 20, 2001.) Applies to

The real-time motion reapplying technology dynamically generates the motion for the desired character by dynamically analyzing the importance of the end of the body reflecting how much to consider the original posture and the posture satisfying the constraints.

4. Experimental results according to the present invention

The present invention was implemented in C ++ using Microsoft DirectX to run on Microsoft Windows XP, and the experiment was performed on a PC equipped with Intel's Pentium4 2.2GHz and 1GB of memory.

The human character model used has 43 degrees of freedom (pelvic position: 3, pelvic orientation: 3, spine: 3, limbs: 7, neck and head: 3).

The motion data used was captured at 30 frames per second, and MIDI data was used to easily obtain music beats as input music.

4-1. Motion bit analysis

In order to measure the accuracy of the motion bit analysis, the first ideal motion and the second actual dance motion are used.

First, in order to obtain an operation that knows exactly the operation bit in an ideal operation, the walking operation, which is a commercial periodic operation, is repeatedly connected.

The length of the operation used is 900 frames and the bit interval is 15 frames.

In order to experiment with the motion of non-uniform bit periods, the walking motions were connected with varying lengths within 15% of the original length.

By knowing the degree of deformation, the correct operation bits are also known.

7 is a graph showing the difference between the bits obtained by the operation bit estimation method according to the present invention and the actual bits for the two operations. The horizontal axis represents the index of the operation bits, and the cell axis represents the error in the unit of frame. Indicates.

The error was less than 1 frame overall. The mean and standard deviation were all zero for fully periodic operation and 0.3233 and 0.2999 frames for nonperiodic operation, respectively.

Secondly, unlike the ideal movement in the actual dance movement, the motion beat of the actual rhythm such as dance is difficult to obtain simply by hand.

For the experiment, it's hard to get it by hand based on the beat of the background music captured with the dance movement.

For the experiment, we obtain the correct beat by hand based on the beat of the background music captured with the dance movement.

The operation used is 714 frames and the bit interval is 12 frames.

In the lower graph of FIG. 7, the difference between the estimated operation bits and the operation bits obtained by hand is illustrated.

The mean and standard deviation of the errors were 0.3702 and 0.3727 frames, respectively.

4-2 Operation Synthesis

In motion synthesis, three experimental results are shown.

The first motion transition compares the motion created from the graph with the motion created from the same path but not synchronized with the background music, and the second motion transition graph is shown as an example of couple motion (couple dance). The graph shows the creation of various movements that can be applied to a number of characters.

In the first synchronized vs. non-synchronized motion, the focus is on the difference between motion synchronized with background music and motion not.

For the experiment, we used freestyle dance motion data of 12.05 frames of bit interval in length of 2320 frames.

We obtained 48 unit operations from the example operation, and constructed a motion transition graph with 38 edges at 7 nodes.

The character on the left of FIG. 8 represents an operation generated in the graph of motion transition but not synchronized, and the character on the right represents an operation of synchronizing it with background music.

Both movements are the result of instantaneous tempo matching a certain input piece.

In this example, it can be seen that the bit of the background music is the 611th frame, and the character on the left does not match the bit of the music.

In the second couple dance, the motion transition graph according to the present invention can be easily extended to use the couple motion.

Since the couple operation has the same rhythm, an operation bit is obtained by using both operation signals of the operation data at once.

Each edge of the configured motion transition graph is a unitary motion, all grouped together.

For the experiment, motion transition graphs for waltz, rumba, and jive dance were constructed independently and motion transition graphs with the same rhythm as the input music were selected and used.

9 is a result of motion generation when a waltz song is input.

In the third group, a group group of character groups is generated as shown in FIG. 10.

The central main character and the surrounding auxiliary characters show various transformations from the optimal path selected in the transition chart.

The motions generated to maintain the formation between the characters are warped so that the distance between the characters in each bit is constant.

Nine street dance data with a length of about 1300 frames were used for the experiment.

From the example operations, we obtained 201 unit operations, from which we constructed a motion transition graph with 18 nodes and 110 edges.

More than half of the edges on the graph have two to six deformation operations.

10 shows that various operations can be obtained from various operations from the same path on the operation transition graph.

As described above, the present invention is a new technique for automatically analyzing the movement of a humanoid virtual character and synchronizing it with a piece of music, which can be used very effectively in the production of an animation using the movement of the character and an audio track together.

Also, the motion synthesis technique according to the present invention can be used to automatically generate motions of supporting characters.

In addition, by designating key operations and automatically generating intermediate operations, it can be usefully used when a choreographer wants to envision a new choreography by using the unit movements previously possessed.

Claims (20)

A candidate bit extraction step of analyzing the example motions of the movement and obtaining the bits of the moments in which the rapid direction change occurs periodically in the human body movement as the candidate bits for each motion bit; A reference bit estimating step of obtaining a reference bit with the bit of the most frequent periodic moment among the candidate bits; An operation bit estimating step of obtaining an actual operation bit as a representative of candidate bits distributed around each reference bit, Rhythm analysis method including rhythm. The method according to claim 1, wherein in the candidate bit extraction step, A rhythm analysis method of rhythm, characterized by extracting each acceleration signal for each operation signal and extracting the bit at the moment when the second derivative passes 0 as a candidate bit. The method according to claim 1, wherein in the reference bit estimation step, In the case of a single joint, a rhythm analysis method comprising constituting a sinusoid from candidate bits of each operation signal and estimating a reference bit by obtaining a frequency most prevalent in the sinusoid. The method according to claim 3, wherein the reference bit is expressed by equation (18) Rhythm analysis method of rhythm, characterized by estimating by finding the highest point. here Is a sine wave, Silver Number, Is the generation time of the pulse. The method according to claim 1, wherein in the reference bit estimation step, The rhythm analysis method of rhythmic motion, characterized in that for forming a sinusoid curve from the candidate bits of each operation signal, and to obtain the most predominant frequency from the signal obtained by the sum of the sinusoids. The method according to claim 1, wherein the reference bit, Phase value estimated in Equation 19 Rhythm analysis method of the rhythm, characterized in that obtained by the equation (20). here Is the period obtained by finding the frequency that maximizes the power spectral density of the combined signal. The method according to claim 1, wherein in the operation bit estimation step, remind Motion bit Is in the interval, and is obtained by Equation 21, Reference bit if not in interval Rhythm analysis method of the rhythm characterized in that the designation as the operation bit. here The Huber function, Is a window function. Example behaviors of the movements are shown by connecting the directional linear graphs whose edges are the unit motions and the nodes corresponding to the start or end of the unit motions. Graph of the motion transition graph And a unit operation transition model acquiring step of providing a higher connection possibility to transition between unit operations shown in the example operation that is more naturally connected in the operation transition graph. The method of claim 8, wherein the similarity measure between nodes is a sum of a joint position and a speed difference between postures represented by two nodes. 10. The method of claim 9, wherein the similarity measure is represented by Equation 22. here Is the number of joints, Wow Is a node and Of the posture corresponding to Location of the joint, Wow Is and Of the posture corresponding to Velocity of the first joint, and Is a constant representing the weight of the position difference and the speed difference. The method according to claim 8, wherein the unit operation transition model acquisition step, A unit motion classification step of classifying the unit motions into a group of variations of the same motion using a measure of similarity; Transformation method considering continuity characterized in that it consists of a step of obtaining a transition model of the transition model is obtained by using the paths corresponding to the example motions as training data and the transition of the motion is determined by the probability variables. . 12. The method of claim 11, wherein the similarity is a sum of weights of a difference between a posture and a speed in a bit as shown in Equation 23. here and Each action and Is the number of operation bits and Is and of The posture at the first bit. The method of claim 8, wherein the random variable, Edge Different edges Probability of a transition to And edge Indicates that the unit operation corresponding to With edge Indicates the possibility that the corresponding unit action will be used as the end action. Operation transition method considering the continuity, characterized in that consisting of. The method of claim 13, wherein the random variable is calculated by Equation 24. The method according to claim 13 or 14, wherein the starting probability Or termination probability Continuity transfer method considering the continuity characterized in that it is determined by the user specifying the start operation or end operation. A motion analysis step of analyzing motions of the example motions to find a moment when the direction change is severe, estimating motion bits from these motions, and constructing a motion transition graph by dividing the example motions into unit motions based on the motion bits; A motion synthesizing step of generating a new song corresponding to the input music from the motion transition graph Beat analysis based on rhythm and music synchronization method. The method of claim 16, wherein the operation synthesis step, An operation generation step of generating new motions by sequentially connecting unit motions corresponding to each edge of the path on the motion transition graph; An operation synchronization step of matching the bits of the input music with the bits of the operation and adjusting the degree of direction change with respect to the generated operation; Re-applying the generated motion to the various characters, The method of synchronizing the movement and music based on beat analysis, characterized in that the configuration. The method of claim 17, wherein the new operation in the operation generation step, A real-time motion generation method of traversing the motion transition graph based on the possibility of transition between unit motions obtained from example motions to obtain a path corresponding to a new motion in real time, and finding a path having an optimal probability on the motion transition graph A method of synchronizing rhythm and music with beat analysis, which is generated by one of the optimal motion generation methods. 19. The method of claim 18, wherein in the optimal motion generation, A cumulative probability of accumulating probabilities is obtained by Equation 25, and using the cumulative probabilities, the highest probability is obtained by Equation 26. here, , ego, Is the number of edges in the transition graph, Time in Path ending with The highest probability at. The method of claim 17, wherein in the operation synchronization step, Create a B-spline curve that passes through successive motion signals, and re-extract the B-spline curve so that the interval between motion bits is equal to the interval between music bits to match the beats. Synchronization method.