Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T13/00—Animation
  • G06T13/20—3D [Three Dimensional] animation
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T13/00—Animation
  • G06T13/20—3D [Three Dimensional] animation
  • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2213/00—Indexing scheme for animation
  • G06T2213/04—Animation description language

Definitions

• the invention relates to computer graphics generally and, more particularly, to analysis of motion and generation of animated computer graphics.
• Motions are generated according to some optimality criteria (such as minimizing joint torques, or following some desired joint trajectories), and taking into account gravity, contact with the floor, moments of inertia of body parts, etc.
• some optimality criteria such as minimizing joint torques, or following some desired joint trajectories
• gravity contact with the floor
• moments of inertia of body parts etc.
• one approach assumes a functional form for joint torque inputs and solves a Lagrangian formulation of the equations of motion with constraints.
• SUBSTITUTE SHEET (RULE 25) produced motions which to human observers appear too “smooth” or “linear” or “flat.” This unrealistic result occurs because human motion is not only a function of the physics of motion, but also a function of the control input from our brains to our bodies. In fact, for typical everyday motions (where velocities are far from maximum), the control input is probably more important than the physics. One can act old, sad, or nervous, all with the same body, just by changing the control inputs.
• Kmematics-based approaches (historically preceding dynamical approaches) have attempted to generate realistic motion by manipulating joint coordinates and body positions directly. This avoids the complicated and computationally costly need for dynamical simulation.
• An example of this type of approach is the computer ammation technique of key- framing, where an artist defines the body pose and position at certain key times (frames of ammation), and the computer produces additional frames between the key-frames to generate the entire sequence. This technique is very labor intensive and requires the trained hand of an artist. Another drawback of this technique is that it cannot be used interactively to adapt ammation in response to user actions or to other runtime conditions.
• kinematics-based approach is the real-time robotics technique of inverse kinematics. While the robotic technique of inverse kinematics does provide a method of automatically altering the joint coordinates of a body in real-time (interactively) in response to user input, it is not satisfactory because the resulting motion is typically unrealistic and robotic-lookmg Thus, kinematics-based approaches have not so far yielded automatic generation of realistic animation.
• the present invention provides a technique for acquiring motion samples, labeling motion samples with labels based on a plurality of parameters, using the motion samples and the labels to learn a function that maps labels to motions generally, and using the function to synthesize arbitrary motions.
• the synthesized motions may be portrayed through computer graphic images to provide realistic animation.
• the invention overcomes the inability of prior dynamics-based modeling techniques to fully incorporate all relevant physical phenomena and the increasing computational complexity that arises in those dynamical approaches that attempt to do so.
• the invention also avoids the labor-intensive efforts required by the prior kinematics-based techniques and provides a capability for responding to high-level control inputs to interactively adapt the animation in a manner not provided by the prior kinematics-based techniques.
• the present invention allows the modeling of labeled motion samples in a manner that can accommodate the synthesis of motion of arbitrary location, speed, and style.
• the modeling can provide subtle details of the motion through the use of probabilistic sub- modeling incorporated into the modeling process.
• Motion samples may be labeled according to any relevant parameters.
• the number of parameters may be increased to provide more variations of possible motion or reduced to increase the efficiency of the process.
• Labels may be used to differentiate between different styles to yield different models, or different styles of a motion may be consolidated into a single baseline model with the labels used to embellish the baseline model.
• the invention allows automation of the animation of synthetic characters by generating the animation based on a general description of the motion desired along with a specification of any embellishments desired.
• FIG. 1 is a block diagram illustrating a system for acquiring motion samples and generating animated images.
• FIG. 2 is a flow diagram illustrating a method for automatic generation of animated images.
• FIG. 3 is a flow diagram illustrating a method for acquiring motion samples.
• FIG. 4 is a flow diagram illustrating a method for labeling motion samples.
• FIG. 5 is a flow diagram illustrating a method for learning functions to map labels to motions.
• FIG. 6 is a flow diagram illustrating a method for synthesizing arbitrary motions using learned functions.
• FIG. 7 is a flow diagram illustrating a process for using a behavioral model in accordance with one embodiment of the invention.
• FIG. 8 is a flow diagram illustrating a process for generating animation in response to an input from a user input device or a sensor.
• FIG. 9 is a flow diagram illustrating a process using a baseline motion component and a residual motion component according to one embodiment of the invention.
• FIG. 10 is a flow diagram illustrating a process including scaling and reconfiguration according to one embodiment of the invention.
• FIG. 11 is a flow diagram illustrating a plurality of embodiments of the invention.
• FIG. 12 is a flow diagram illustrating a process using a learned function to generate a desired motion.
• the present invention provides a technique for acquiring motion samples, labeling motion samples with labels based on a plurality of parameters, using the motion samples and the labels to learn a function that maps labels to motions generally, and using the function to synthesize arbitrary motions.
• the synthesized motions may be portrayed through computer graphic images to provide realistic animation.
• Information adequate to describe motion of a subject can often be expressed in terms of the motion of a finite number of points.
• a human body in motion can be accurately described in a very efficient way with reference to the motion of a dozen or so points on the body.
• it is easy to infer either by psychophysical perception or by graphical rendering techniques, motion of other parts of the human body. From the motion of these points, it is possible to distinguish human gaits from other non-biological motions, to identify gender, and even to determine the subject's emotional state.
• FIG. 2 is a flow diagram illustrating a method for automatic generation of animated images.
• the process begins in step 201.
• step 202 motion samples are acquired.
• step 203 motion samples are labeled.
• step 204 a function is learned to map labels to motions.
• step 205 arbitrary motions are synthesized using the learned function.
• step 206 To acquire information to describe the motion of an subject, several techniques may be employed. For example, motion samples may be provided by the drawings of an artist, such as a sequences of animation drawings entered on a graphics tablet.
• motion samples include two-dimensional optical systems (such as those with one camera), three-dimensional optical systems (such as those with multiple cameras), three-dimensional magnetic systems (such as those utilizing multiple orthogonal magnetic fields), mechanical (exo-skeleton) capture systems (such as those worn by a human or part of a live-action puppet), two- or three-dimensional animations provided by an artist, or reconstruction of two- or three-dimensional motion from video footage or live camera action (for example, using computer vision techniques).
• two-dimensional optical systems such as those with one camera
• three-dimensional optical systems such as those with multiple cameras
• three-dimensional magnetic systems such as those utilizing multiple orthogonal magnetic fields
• mechanical (exo-skeleton) capture systems such as those worn by a human or part of a live-action puppet
• two- or three-dimensional animations provided by an artist
• reconstruction of two- or three-dimensional motion from video footage or live camera action for example, using computer vision techniques.
• FIG. 1 is a block diagram illustrating a system for acquiring motion samples and generating animated images.
• the system comprises one or more cameras 102 for observing the motion of a subject 101 and for providing a sequence of frames of images of the subject 101.
• One camera 102 may be used to observe the motion of subject 101 in two dimensions, while two or more cameras may be used to observe three-dimensional motion.
• the one or more cameras 102 may be coupled to a signal processor 103, such as a digital signal processing (DSP) subsystem.
• DSP digital signal processing
• the one or more cameras 102 and signal processor 103 may be optionally coupled to one or more recording devices 107, such as a videotape recorder or video storage subsystem, to store the sequence of frames of images of the subject 101 for later processing.
• the signal processor 103 processes the sequence of frames of images of the subject 101 to extract information regarding the position of the points 108 for which analysis is desired.
• the signal processor 103 identifies each point 108 in each frame according to its location on the subject 101. For points 108 that are obscured from view in certain frames, their locations are identified by interpolation between adjacent frames.
• CPU 104 segments individual motions and re-samples the sequence of frames to normalize its length in numbers of frames. These steps may be assisted by user input, for example user input provided through user input device 106.
• CPU 104 processes the sequence of frames to provide a matrix where each column of the matrix represents a sample motion and contains the stacked coordinates of all of the points 108 for all of each motion sample's frames.
• the matrix may be of size Nc • b ⁇ Nr by N M , where Nc equals the number of dimensions (e.g., two-dimensional or three-dimensional), b equals the number of points 108 by which a subject 101 is represented, Nr equals the number of re-sampled frames in each motion, and N M equals the number of motion samples.
• the CPU 104 is used to perform labeling, learning, and synthesizing steps. These steps may be controlled through information received from a user input device 106. Information for the user input device 106 may be used to control the operation of signal processor 103, as well. Another step that may be performed by CPU 104, in the processing of three-dimensional information, is the reconstruction of three-dimensional coordinates based on the sequences of frames of images of the subject 101 provided by cameras 102. Synthetic motions generated using the CPU 104 may be displayed on display device 105.
• signal processor 103 may be combined with the functionality of CPU 104 and provided by CPU 104 without the need for a separate signal processor 103.
• one or more cameras 102 and/or one or more recording devices 107 may be coupled to CPU 104.
• one or more recording devices 107 may be coupled to CPU 104 to record information that may have come from the one or more cameras 102, that may have been processed by signal processor 103, or that may have been processed, at least to some degree, by CPU 104.
• FIG. 3 is a flow diagram illustrating a method for acquiring motion samples.
• the process begins in step 301.
• markers are applied to a subject to indicate points of interest.
• the subject is illuminated to maximize contrast of the subject, and particularly the markers, against the background.
• fluorescent spheres, such as ping-pong balls are used as markers, ultraviolet lighting may be used to illuminate the markers and to distinguish them from the background.
• step 303 the subject moves, and motion samples are observed and/or recorded.
• step 304 points of structural features are identified. Since the structural relationship of the structural features is, to at least some extent, known, identification of points of structural features implies knowledge of their structural relationship. This knowledge allows reconstruction of obscured points of structural features in step 305.
• steps 302 through 305 may be omitted, and the motion samples provided by the other techniques may be processed according to steps 306 through 309.
• step 306 individual motions are segmented.
• the continuum of motion of a variety of individual acts is divided in time into individual motions, each representative of an individual act, for example a single reach or a single step.
• the segmentation may be at a low enough level to yield a diverse repertoire of motions and to allow fine-grained control over the generation of animated motion. Alternatively, the segmentation may be at a higher level to reduce the number of individual motions needed to provide a composite motion.
• step 307 the individual motions are re-sampled.
• This re-sampling allows the motion samples to be normalized to a common length or number of frames.
• Subsequent processing of the motion samples may be simplified if the motion samples are known to have a common length or number of frames.
• unnecessary temporal resolution may be reduced, thereby making storage and processing of the motion samples more efficient.
• the technique of dynamic time warping, or other automatic or semi-automatic procedures is used to align in time critical events during the motion. Use of such techniques
• a matrix is constructed of motion sample datasets.
• the matrix contains information representative of the coordinates of the point of structural features for re-sampled frame of each instance of each motion sample. There may be multiple instances of a motion sample if multiple attempts were made at performing the same motion and gathering a motion sample.
• the process ends.
• motion sample datasets may be obtained from sources other than a motion capture system.
• motion samples may be provided based on an artist's animations, which may be generated using a key frame technique described above. The artist may provide information in two or three dimensions. If an artist's animations are used, steps 302 to 306 may be omitted and replaced by the step of providing the artist's animations.
• the motion samples are assigned labels.
• the labels parameterize the motions according to high-level descriptions, such as direction of motion, speed of motion, mood, style, and age of the subject.
• the set of motion samples acquired are reaching motions having a certain destination reach position.
• knowing the destination reach position is sufficient to specify completely the motion, and thus the labels for this dataset may be two- dimensional vectors defining the destination reach position on the image plane. If the dataset were to comprise walking cycles in which the subject varied his gait, then the motion samples would be labeled according to the speed of the motion and to the location of the feet on the floor.
• motion samples may be labeled as to the general type of motion (e.g., walking, running, pointing, etc.), the direction and speed of the motion, the style and mood of the subject, and the age and gender of the subject.
• the subjects may also be identified according to attributes such as size, body proportions, agility, etc.
• motion samples from a subject having certain attributes may be scaled or otherwise modified to substitute for motion samples of subjects having different attributes. For example, motion samples from a tall subject may be re-scaled to substitute for motion samples of a short subject.
• FIG. 4 is a flow diagram illustrating a method for labeling motion samples.
• the process begins in step 401.
• a first label is assigned to motion samples having a first value of a first parameter.
• the first parameter might characterize height, and the first value might be "tall.” Other possible values for the height parameter might include “short,” “6 foot, 1 inch,” “170 cm,” etc.
• a second label is assigned to motion samples having a second value of a second parameter.
• the second parameter might characterize mood, and the first value might be "happy.” Other possible values might include “sad,” “agitated,” “angry,” “tired,” etc.
• the second parameter might characterize "happiness,” and a numeric value might be assigned depending on the level of happiness.
• step 404 a third label is assigned to motion samples having a third value for a third parameter. This process continues with the optional assignment of additional labels.
• step 405 an nth label is assigned to motion samples having an nth value of an nth parameter. The process ends in step 406.
• different labels may be used to represent different aspects of a desired motion.
• goal labels may be applied to the data set based on a goal parameter which describes the goal of the subject in performing an action captured by the motion samples.
• a style label may be applied to the data set based on a style parameter descriptive of a style of subject expressed in the data set.
• Different goal labels and style labels may be applied to each data sample of the data set.
• a state label may be applied to a data sample to indicate a state of the dynamic positional information represented by the data sample.
• the state label may indicate the state of the dynamic positional information at the beginning and/or end of the data sample.
• the state label may be used to coordinate transition from one motion to another.
• a mathematical function may be learned to map the labels to the motions.
• Label-motion pairs that include I' and m', where 1' is the i th label associated with the i th motion vector m', can be used as examples to derive a function / which maps labels into motions.
• the function space over which the best function is selected is such that the fc component of the estimated motion vector m k is given by
• ⁇ vv,* ⁇ are weights to be determined and the N functions j ( ⁇ ) are basis functions over the input space.
• the best function /* in other words the function that most closely maps the labels to the motions, may be determined using various mathematical techniques. For example, different types of basis functions may be applied to yield a function that accurately maps labels to motions. Examples of two types of basis functions that may be used include polynomial basis functions and radial basis functions.
• the best function /* in the function space is found by finding the best set of coefficients ⁇ w* j k ⁇ as w k )
• ⁇ ⁇ and ⁇ define the center and spread of the/ ⁇ function.
• the optimal coefficients may be found using an orthogonal least squares minimization procedure, for example such a procedure provided by a MATLAB neural networks toolbox routine. This routine not only finds the ⁇ vv,* ⁇ coefficients, but also determines the optimal ⁇ ⁇ for the given number of radial basis functions desired and a user-specified ⁇ ,.
• the modeling process serves to effectively generalize the data samples to characterize the desired motion over the available motion space.
• the desired motion may be derived from the learned model by choosing the appropriate values for the input parameters (labels) and applying it to the learned model.
• FIG. 5 is a flow diagram illustrating a method for learning functions to map labels to motions.
• the process begins m step 501.
• step 502 the dataset is analyzed to yield local sub-models. If a global model of a motion is not as accurate as desired across all regions of the motion space, local sub-models may be created to model a particular region of the motion space. The local sub-models allow increasing the accuracy of the global model in the regions where the global model suffers the most inaccuracy.
• step 503 local sub-models are combined with the global model and with each other to yield an overall model of a motion.
• step 504 variations in dynamic position information are characterized according to a probabilistic sub-model. If various instances of a motion sample contain variations between each other, those variations can be averaged or otherwise abstracted away to yield a model of motion represented by the motion samples. However, such averaging or abstraction may reduce the information that characterizes the variability of the motion. If this variability is meaningful to human perception, for example for motions that are repeated or for poses or gestures, the realism of synthesized motions may be reduced. For example, if a human character is intended to stand still, a perfectly still rendering may appear lifeless. By including subtle detail, such as shifting of body weight and gesturing of hands, greater realism is provided. Gesturing of hands may be coordinated with audio information, such as speech, to increase realism.
• a probabilistic model may be created based on information not otherwise incorporated into the overall model for a motion.
• the probabilistic model introduces stochastic motion adjustments as similar as possible to the variation of the instances of a motion sample from the overall model of the motion.
• a Gaussian probability density function could be used, with appropriate parameters, to model the variability of the motion samples and to provide a random variation in animation generated using a model that includes the probabilistic model.
• a mixture of several Gaussian probability density functions can be used to model more complex variability.
• step 505 the probabilistic model is incorporated into the overall model to yield a complete model of a motion.
• a synthesized motion created according to the complete model of the motion not only follows the common aspects of various instances of a motion sample, but includes variations based on the aspects of the various instances of the motion sample that are not common among them.
• step 506 the process ends.
• FIG. 6 is a flow diagram illustrating a method for synthesizing arbitrary motions using learned functions. The process begins step 601. In step 602, virtual markers are defined.
• the desired motion of the virtual markers is designated. This designation may include specification of the desired starting location, the desired path, the desired ending location, and any modifications of the motion that may be desired (e.g., particular types of characters or styles of motion).
• Step 603 may be practiced according to a process illustrated in FIG. 12.
• FIG. 12 is a flow diagram illustrating a process using a learned function to generate a desired motion.
• the process begins in step 1201.
• a desired learned function is chosen.
• labels representing desired values of parameters for the desired motion of the virtual markers are selected.
• the desired motion of virtual markers are computed using the labels as inputs to the desired learned function.
• the process ends.
• step 604 joint coordinates are calculated based on desired motion of virtual markers.
• the joint coordinates specify the desired locations of the points of structural features over time during the desired motion.
• the desired motion is a composite motion that includes multiple actions, for example walking and waving
• the walking motion and the waving motion may be processed individually through steps 602 through 604.
• the calculated joint coordinates for the walking and waving motions may be merged into the composite motion in optional step 607. From optional step 607, the process continues in step 605. Alternatively, if optional step 607 is omitted, the process can proceed from step 604 to step 605.
• step 608 the learned functions are used to designate joint coordinates directly. This can be achieved by using learned functions that output joint coordinates given the input of labels. Such functions can be learned by converting motion samples to joint coordinate data and by using a learning process to determine functions with joint coordinate outputs. As described above, if the desired motion is a composite motion that includes multiple actions, for example walking and waving, the walking motion and the waving motion may be processed individually through step 608. The calculated joint coordinates for the walking and waving motions may be merged into the composite motion in optional step 607. From
• step 607 the process continues in step 605. Alternatively, if optional step 607 is omitted, the process can proceed from step 608 to step 605.
• step 605 the joint coordinates are used to render a synthetic character.
• the rendering process can involve displaying an entire texture-mapped surface of a synthetic character as an alternative to just the points of structural features.
• the synthetic character may be based on the subject used to generate the original motion samples or may be any arbitrarily created character.
• the process ends in step 606.
• the invention may be practiced to generate animation by applying a single model of motion to a motion plan
• the invention may also be used to generate complex animation by applying multiple models of multiple motions to the motion plan.
• the multiple motions may be performed simultaneously, sequentially, or a combination of the two. If the multiple motions are performed simultaneously, the joint coordinates for the multiple motions may be calculated separately and divided among parts of a body to be animated.
• the joint coordinates for the walking and waving motions may be computed and divided so that the joint coordinates for the arm and hand are used to animate the arm and hand, while the remaining joint coordinates are used to animate the remaining parts of the body.
• the walking motion causes the location of the body to be translated in space over time
• the coordinates of the arm and hand can be translated (and if appropriate rotated) in space over time to match the walking motion and keep the arm and hand together with the remaining parts of the body as the complex motion continues.
• a complex motion plan may be divided into a plurality of basic motion plans.
• a corresponding model is applied to each of the basic motion plans to generate a plurality of animation components.
• the animation components are combined to yield animation according to the complex motion plan.
• the animation process may be performed according to a script, a random process, a behavioral model, or some combination of these.
• the script may be any preprogrammed approach to providing a motion plan.
• the random process may be any probabilistic approach to control one or more features of the motion plan. For example, the random process may vary the manner in which animation is provided according to the motion plan or may vary the elements of the motion plan itself.
• the behavioral model may provide agents that have various behavioral characteristics whereby a character animated according to an agent may respond to similar circumstances or conditions in a similar manner.
• the behavioral characteristics may develop over time to provide a level of uniformity or predictability to the behavioral model.
• the agents may be used for interacting with users over a network and/or in computer games.
• the network may comprise a single communication connection or a plurality of communication connections.
• the generation of animation by applying the model to the motion plan may be performed in response to an input received from a user input device.
• the user input device may be a device actively controlled by a user, such as a keyboard, mouse or other pointing device, joystick, etc., or may be a more device for obtaining user input more passively, such as a camera, microphone, motion detector, etc.
• a character in an interactive computer game may be animated in response to an input received from the user input device.
• a synthetic character may go into a sleeping mode and be awakened in response to an input received from a user input device.
• the motion plan for generating animation may be adapted on the basis of historical user interaction. User interaction can lead to a motion plan having increasing predictability and uniformity. Alternatively, variation and unpredictability can be incorporated into the motion plan so that motion plan will differ even with similar user interaction. Variation may be randomly implemented by incorporating a random variation component into the motion plan. The random variation component may be combined with a basic component to yield motion that does not look exactly the same each time it is performed. A random variation component may also be incorporated into a model used to generate animation.
• FIG. 7 is a flow diagram illustrating a process for using a behavioral model in accordance with one embodiment of the invention. The process begins in step 701. In step 702, a behavioral model is defined. In step 703, the behavioral model is modified to adjust the predictability or unpredictability of the behavioral model. In step 704, the behavioral model interacts with user. The interaction with the user can be used to modify the behavioral model when the process returns to step 702.
• Animation may also be generated based on an input received from a sensor.
• the sensor may be of any type capable of detecting a particular phenomenon.
• a camera may detect a static or dynamic image, or a microphone may detect sound and provide audio information.
• the invention may be used to animate a character to imitate the motion of a user observed using the sensor. The motion of the user is observed and captured. The captured motion is used to control a motion plan for animating the character.
• Animation of a character may be performed based on a determination of the position of body parts of the user.
• the position of the body parts of the user are used to control the motion plan for animating the character.
• the ammation of the character may be performed based on modeling of the actions of the user.
• the motion of the user may be captured, labeled, and modeled.
• the model derived from the user's motion is then used to control the motion plan for animating the character.
• the input received from a sensor may be used to determine an activity of a subject, and ammation may be generated or other action may be taken based on the activity.
• This technique may be used while a computer coupled to the sensor is in a quiescent mode, for example when the computer is displaying a screen saver.
• the detection of the activity may be used to cause the computer to alter the screen saver display or to switch the computer to an active mode.
• the invention may be practiced while the computer is in the active mode.
• Animation information may be extracted from audio information.
• the audio information may be obtained in any desired manner, for example, from a microphone, through a communication circuit, or from an audio recording. If, for example, the audio information is musical, the music can be analyzed to obtain animation information. For example, the rhythm may be extracted from the music and used to control or modify the animation.
• Animation information extracted from audio information may be used to modify the performance of a specified animation command. For example, if a command is given to cause an animated character to walk, the cadence of the walking may be modified according to the tempo of music. Speech may also be analyzed to obtain animation information. For example, facial movements or hand gestures may be animated according to speech patterns.
• Animation information may be extracted from motion information.
• an example of human motion may be used to provide animation information.
• the animation information may reflect the velocity of the example of human motion.
• animation information may be extracted from non-human motion.
• FIG. 8 is a flow diagram illustrating a process for generating animation in response to an input from a user input device or a sensor. The process begins in step 801. In step 802, an input is obtained from a user input device or a sensor. In step 803, animation is generated in response to the input form the user input device or sensor. In step 804, the process ends.
• the creation of a model based on a data set of data samples may include the learning of a style based on a residual motion component with respect to a baseline motion component.
• the generation of animation by applying the model to a motion plan may include combining a residual motion animation component to a baseline motion animation component.
• FIG. 9 is a flow diagram illustrating a process using a baseline motion component and a residual motion component according to one embodiment of the invention.
• the process begins in step 901.
• a baseline model is learned based on a baseline motion component.
• a residual model is learned based on a residual motion component.
• animation is generated using the baseline motion component and the residual motion component.
• the process ends in step 905.
• Scaling and reconfiguration may be applied to the generation of animation of a synthetic character. While information regarding a plurality of points denoting a structural relationship of a subject may be obtained during a motion capture step, the reference points corresponding to a structural relationship of a synthetic character for which animation is being generated may differ from the plurality of points.
• the scaling and reconfiguration may be used to generate animation of a synthetic character that differs in appearance from the subject on which the motion capture step was performed. Adjusting the scale of the plurality of points may be used to determine reference points for generating animation of synthetic characters that are larger or smaller than the subject used for the motion capture step.
• the angular relationship of the plurality of points may be adjusted to reconfigure the shape of the synthetic character.
• FIG. 10 is a flow diagram illustrating a process including scaling and reconfiguration according to one embodiment of the invention.
• the process begins in step 1001.
• a model is learned.
• information representing a structural relationship is scaled and reconfigured.
• animation is generated according to the scaled and reconfigured information.
• the process ends.
• scaling and/or reconfiguration may be applied before or during the process of learning a model.
• steps 1002 and 1003 may be interchanged to allow the scaling and reconfiguration to be incorporated into the model. This approach may be useful for reducing the complexity of the motion synthesis process.
• the invention is useful for generating images, such as images to be displayed with computer, video, or motion picture equipment, the invention may also be used to control the motion of apparatus.
• Such apparatus may include mechanical devices, for example, robots, puppets, or mechanically animated figures or toys. Even apparatus such as industrial machinery could be controlled by observing the motion of a user. All of the variations of the invention described with respect to the generation of animation may be applied to controlling the motion of apparatus.
• the invention may also be used to compare a model to a plurality of previously obtained models to allow recognition of a particular action. For example, a previously obtained model similar to a given model may be identified and used to identify the subject. Subtle differences between different subjects can be characterized during the modeling process. By storing or indexing models based on actions of subjects, a subject can be identified by modeling other instances of action by the subject and comparing the model against the stored models or by looking up the appropriate stored model using the index.
• the goal of the subject can also be estimated based on the data set and the model. For example, if the model relates to a reaching motion, and the data set provides initial information about a particular instance of reaching, the final location of that instance of reaching can be estimated by extrapolating from the initial information according to the model.
• a kinesiological condition of the subject can be assessed based on a model of action by that subject.
• an injury may affect the manner in which a subject walks. If the action of an injured subject is observed, labeled, and modeled, the model can be compared to models representing different types of injuries to determine the nature of the subject's injury. While the model of the subject's action can be compared against a plurality of models, alternatively the plurality of models can be indexed according to certain attributes. Similar attributes of the model of the subject's action can be determined and used to identify the most similar model from among the plurality of models without the need to perform an exhaustive comparison.
• quality of movement of a subject can be assessed based on the model.
• the movement of an athlete can compared against a model of instances of that athlete's best performance for a given action, against a model of a similar action performed by an exemplary athlete, such as a world record holder, or against a dynamic model optimized
• SUBSTITUTE SHEET (RULE 25) according to the physical constraints on the action.
• the quality of movement can be quantitatively expressed by determining a value that represents how close the model of the movement of the athlete is to the target model that the athlete seeks to emulate or to the optimized dynamic model.
• FIG. 1 1 is a flow diagram illustrating a plurality of processes according to a plurality of embodiments of the invention.
• the process begins in step 1 101.
• a plurality of models are obtained. This plurality of models may form a library of models and may include models of instances of motion performed by the same or different subjects.
• step 1 103 another model is obtained.
• step 1 104 which follows step 1 103, the model obtained in step 1 103 is compared to the plurality of models obtained in step 1102.
• step 1105 which follows step 1 103, the model obtained in step 1 103 is compared to the plurality of models obtained in step 1 102 to identify a type of motion.
• initial information is applied based on the type of motion determined in step 1 105.
• the initial information may include information representing the position of a subject at the beginning of an action and information representing the sequence of changes of position as the action begins.
• a goal of the subject is estimated, for example by extrapolating the initial information based on the type of motion determined in step 1105.
• step 1 108 which follows step 1103 the model is compared to the plurality of models to assess a physiological condition.
• step 1 109 which follows step 1103, the model is compared to the plurality of models or to a dynamic model based on physical constraints to assess a quality of movement of the subject.
• the description that follows serves as an example of how the invention may be applied.
• the example relates to synthesis of reaching motions, such as the type of motion that a person might make with that person's hand to pick apples from a tree or to place letters from pigeonhole mailboxes.
• motion samples of the subject reaching at different angles and over different distances can be acquired.
• a subject who is standing, facing a single camera may perform reaching motions with one hand to several positions (e.g., 15 positions) spread out at different angles (e.g., approximately "one hour" apart on an imaginary clock surrounding the subject vertically).
• the reaching motions may include some of a longer distance (e.g., reaching on a bigger clock with the arm fully extended) and some of a shorter distance (e.g., reaching on a smaller clock with the arm closer to the body).
• Multiple samples may be acquired for each reaching motion to average out errors that may occur in particular attempts at a reaching motion. For example, if three samples of each "clock size" and reach location are obtained, the total number of samples obtained is equal to the number of samples times the number of reach positions times the number of "clock sizes"
• the acquired samples are used to construct a matrix (e.g., a 28 • 120 by 90 matrix)
• the labels for the dataset are the (x,y) coordinates of the hand- tip for the reaching motions.
• the accuracy of the function in reconstruction of the dataset can be analyzed.
• One mathematical approach to analyzing the accuracy of the function is to measure the percent root-mean-square (RMS) error on a frame-by-frame basis. This error is defined as
• m is the vector mean of all the motion samples. This error measurement has the property of being 0 if the approximating function is exact, and 1 if it is as inaccurate as the mean vector of all of the data, independent of the number of samples used.
• Another approach to measuring the accuracy of the learned function is to perform a two-alternative forced-choice experiment with a number of observers. For each sampled motion, an observer is presented with a display of the motion based on the original dataset obtained from that original motion sample. The observer is also presented with a display of motion reconstructed according to the learned function (e.g. cubic interpolation).
• the learned function e.g. cubic interpolation
• SUBSTITUTE SHEET (RULE 25) displays of motion are presented in random order. The observer then attempts to identify which display of motion appeared more realistic. If the original and reconstructed motions were completely indistinguishable, observers should be performing the discrimination task at the chance level (50% correct responses). Experimental results show that embodiments of the invention are indeed capable of producing realistic animation.
• Another example that illustrates how the invention may be applied involves synthesis of drawing motions. Such motions involve starting at a starting location, moving from the starting location to an ending location, and stopping at the ending location.
• a subject is positioned in view of a camera and performs drawing motions as if he were drawing on a blackboard. It is possible to simplify the learning task by requiring the subject to start at the same starting location for each drawing motion. This obviates the need to provide a label to designate the starting location. Alternatively, different starting locations may be used.
• the motion samples may be re-sampled to normalize their lengths.
• Labels may be applied to the motion samples to designate the starting location, the ending location, and the shape of the stroke, among other parameters.
• a user can input the label by drawing it on the screen.
• an internal representation of the label may be used.
• the trajectory of the stroke is represented as a cubic spline in the x zn ⁇ y coordinates as given by
• each motion sample has a sequence of many (e.g., 60) hand-tip coordinates through time. It is possible to use a least squares method to find the coefficients ⁇ auß b, ⁇ of the spline inte ⁇ olation. However, such an interpolation is not guaranteed to pass exactly through the
• SUBSTITUTE SHEET (RULE 25) starting and ending locations, thereby diminishing the realism of the motion.
• a constrained optimization routine may be used.
• the aforementioned reaching and drawing motions may be merged to form a compound motion that involves reaching from a neutral pose to a starting location, drawing from the starting location to the ending location, and reaching back to the neutral pose.
• combining the two learned motions in some cases may be achieved by matching the hand position from the end of one motion to the beginning of the next, other cases may require more attention to transition from one motion to the next.
• the hand position represents only one point of many points monitored in acquiring motion samples, other monitored points may be incongruent between the different motions. Such incongruence can lead to discontinuous jumps during the transition from one motion to the next.
• the pose discontinuity can be spread out by adding the vector ⁇ to the /' frame.

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=23723285

Family Applications (1)

Country Status (3)

Cited By (3)

Families Citing this family (77)

Family Cites Families (3)

• 1999
  • 1999-11-04 US US09/434,210 patent/US6552729B1/en not_active Expired - Lifetime
• 2000
  • 2000-02-25 AU AU33784/00A patent/AU3378400A/en not_active Abandoned
  • 2000-02-25 WO PCT/US2000/004833 patent/WO2001033508A1/en active Application Filing

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events