WO2023022606A1 - Systèmes et procédés pour l'animation informatique d'un personnage artificiel à l'aide de poses faciales à partir d'un acteur vivant - Google Patents

Systèmes et procédés pour l'animation informatique d'un personnage artificiel à l'aide de poses faciales à partir d'un acteur vivant Download PDF

Info

Publication number
WO2023022606A1
WO2023022606A1 PCT/NZ2021/050172 NZ2021050172W WO2023022606A1 WO 2023022606 A1 WO2023022606 A1 WO 2023022606A1 NZ 2021050172 W NZ2021050172 W NZ 2021050172W WO 2023022606 A1 WO2023022606 A1 WO 2023022606A1
Authority
WO
WIPO (PCT)
Prior art keywords
time
animation
facial
animated
animation control
Prior art date
Application number
PCT/NZ2021/050172
Other languages
English (en)
Inventor
Wan-Duo Kurt Ma
Muhammad Ghifary
Original Assignee
Weta Digital Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/403,727 external-priority patent/US11410370B1/en
Application filed by Weta Digital Limited filed Critical Weta Digital Limited
Publication of WO2023022606A1 publication Critical patent/WO2023022606A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure generally relates to tools for generating computer-generated imagery.
  • the disclosure relates more particularly to apparatus and techniques for computer animation of an artificial character using facial poses from a live actor.
  • the computer-generated imagery may include computer-animated characters that are based on live actors.
  • a feature film creator may want to generate a computer- animated character having facial expressions, movements, behaviors, etc. of a live actor, human or otherwise. It may be possible to have an animator specify, in detail, a surface of the live actor’s body, but that can be difficult when dealing with facial expressions and movements of the live actor, as there are many variables.
  • Some existing animation systems may recreate, in detail, a skin surface of the computer-animated character that closely resembles a live actor.
  • muscle movements performed by the live actor is decomposed and analyzed, which often entails knowledge of an anatomical model of the physical body of the live actor.
  • the anatomical model may include the shape, mass, curvature, and/or other parameters that describe the structure of the muscle layer and the skull of the live actor.
  • the anatomical model can usually include a large number of variables. For example, there are more than 40 skeletal facial muscles that are controlled by seven nerves in a human face. Therefore, building a mathematical topology depicting the three-dimensional structure and movements of the facial muscles can often entail a high degree of computational complexity with compromised system efficiency.
  • the disclosure relates to a computer-implemented method for generating a first data structure usable for representing an animated facial pose applicable in an animation system to an artificial character.
  • the method may comprise, receiving, via a plurality of markers placed on a face of a human actor, data relating to one or more facial poses performed by the human actor; transforming, by a deep learning network, control values corresponding to a set of controllable elements that are distributed on an animated character face based on the data relating to the one or more facial poses performed by the human actor, wherein changes of the control values cause a pose change from a first animated facial pose to a second animated facial pose on the animated character face; obtaining a plurality of animation control curves over a period of time corresponding to the set of controllable elements; jointly selecting, across the plurality of animation control curves over the period of time, a set of snapshots corresponding to a plurality of time points, wherein each snapshot represents a plurality of salient data points on the plurality of animation control curves at the respective
  • a system for generating a first data structure usable for representing an animated facial pose applicable in an animation system to an artificial character may comprise: a data interface receiving, via a plurality of markers placed on a face of a human actor, data relating to one or more facial poses performed by the human actor; a memory storing a deep learning network; and a processor executing processor-executed instructions to perform operations comprising: transforming, by the deep learning network, control values corresponding to a set of controllable elements that are distributed on an animated character face based on the data relating to the one or more facial poses performed by the human actor, wherein changes of the control values cause a pose change from a first animated facial pose to a second animated facial pose on the animated character face; obtaining a plurality of animation control curves over a period of time corresponding to the set of controllable elements; jointly selecting, across the plurality of animation control curves over the period of time, a set of snapshots corresponding to a plurality of time points, wherein each snapshot represents a pluralit
  • At least one computer-readable medium may carry a plurality of processor-executable instructions for generating a first data structure usable for representing an animated facial pose applicable in an animation system to an artificial character.
  • the instructions may be executed by a processor to perform operations comprising: receiving, via a plurality of markers placed on a face of a human actor, data relating to one or more facial poses performed by the human actor; transforming, by a deep learning network, control values corresponding to a set of controllable elements that are distributed on an animated character face based on the data relating to the one or more facial poses performed by the human actor, wherein changes of the control values cause a pose change from a first animated facial pose to a second animated facial pose on the animated character face; obtaining a plurality of animation control curves over a period of time corresponding to the set of controllable elements; jointly selecting, across the plurality of animation control curves over the period of time, a set of snapshots corresponding to a plurality of time points, wherein each snapshot represents a plurality
  • FIG. 1A illustrates an animation pipeline that may be used to render animated content showing animation of a character based on a machine learning model that is trained from scans of a live actor.
  • FIG. IB illustrates an alternative animation pipeline 100b that may be used to render animated content showing animation of a character based a rig that is generated based on geometric characteristics of facial poses.
  • FIG. 2A illustrates an example neural network system in which scan results are provided to a muscle simulator and a neural network and an anatomical model to the muscle simulator.
  • FIG. 2B provides an example block diagram illustrating an alternative deep machine learning system in which marker position data is learnt to be transformed to control elements in a mesh applicable to the character face of an artificial character.
  • FIG. 3A illustrates an example of a data structure that may represent a muscle model.
  • each muscle may be defined by a bone attachment point, a skin attachment point, and a muscle volume.
  • FIG. 3B illustrates an example of a data structure that may represent the position data as show in FIG. 2B.
  • FIG. 4 illustrates inputs and outputs of an animation creation system.
  • FIG. 5 provides an example diagram illustrating an example work flow of generating a mesh usable for representing an animated facial pose applicable in an animation system to an artificial character.
  • FIG. 6 provides an example diagram illustrating an example work flow of generating an animated facial pose on a character face of an artificial character using a mesh usable for representing the animated facial pose applicable in an animation system to the artificial character.
  • FIGS. 7A-7C provide example plots illustrating selecting salient points from animation control curves.
  • FIG. 8 illustrates an example visual content generation system as may be used to generate imagery in the form of still images and/or video sequences of images.
  • FIG. 9 is a block diagram illustrating an example computer system upon which computer systems of the systems illustrated in FIGS. 1-12 may be implemented.
  • Video applications nowadays may adopt computer-animated technology to create simulated characters, human or non-human, to appear a video.
  • the film industry has been using computer animation to generate characters that is often physically difficult or even impossible to be played by human actors.
  • the physical appearance of such computer- animated characters may be designed and controlled by an animator, via configuring timevarying parameters to simulate the muscle, joint and bone structures and movements of a living creature, human or non-human.
  • the computer-animated character may be created to emulate the persona of a real living creature.
  • an animator may refer to a human artist, filmmaker, photography image creator, or the like, who seeks to generate one or more images (such as a video sequence forming an animation) based on animator input and other data available to the animator.
  • the animator may be an automated or partially automated process.
  • Animator inputs may include specifications of values for positions of movable elements. For example, an articulated character’s movement may be specified by values of each available joint in the character.
  • a mesh may refer to a surface representing a facial pose of a human actor.
  • mesh deformations may be derived for different facial poses performed by the human actor.
  • a blendshape may refer to a set of off-set vectors.
  • a blendshape may comprise a set of control values representing position data of a set of markers distributed on different locations of a face of a human actor. Such position data, or position coordinates, may represent the off-set control values, to be added on the basic shape (e.g., a neutral facial pose) to represent a particular facial pose.
  • a rig may refer to a representation of data that corresponds to elements of a character, the allowed movements, etc. One such rig is a facial rig.
  • An animator may be provided with a user interface of an animation creation system that allows the animator to input values for various movable elements of the facial rig.
  • Some movable elements may be a jaw and a collection of muscles.
  • the animation creation system can generate a pose of the facial rig. For example, when variables corresponding to an amount of contraction for the muscles on either side of the mouth are set to values that represent maximum contraction of those muscles, the animation creation system would output a pose with a face having a widened mouth.
  • animation creation system can output positions of elements, thicknesses of elements, etc., which may be provided as input to a rendering system.
  • a state of a facial rig corresponding to a particular expression, movement, or placement of elements of the facial rig so as to convey an expression or positioning of facial elements may be represented in computer memory as a data structure such as a strain vector.
  • a strain vector may have components representing jaw position, eye positions, and strain values for each muscle in the facial rig that can be specified by the strain vector.
  • a particular expression of a live actor can be represented by a strain vector and that strain vector can be used to move or position elements of a facial rig - of that live actor, of a fanciful character, etc. - for generating computer-generated imagery.
  • the strain value components are one per muscle each having a value representing a present strain value for its corresponding muscle.
  • a strain value may have a fixed value for a muscle in a neutral position for that muscle and a range of values covering contractions relative to the neutral position relaxations relative to the neutral position.
  • a neutral position value for a strain is zero
  • a strain value for a muscle contracted relative to the neutral position is a negative number
  • a strain value for a muscle relaxed relative to the neutral position is a positive number.
  • the strain value in that example may correspond to a length assigned to the muscle in the corresponding position.
  • a facial rig may comprise a large number of muscles
  • manually and individually setting each muscle’s strain value in the strain vector can be a tedious process and it can be hard to manually match the strain vector component values to a desired state or expression.
  • An animator can generate animation of a face of a character making an expression, perhaps talking according to certain speech, and moving around by inputting, or otherwise specifying or determining, a set of strains, wherein a strain is a metric of a muscle that can be moved.
  • a strain of a muscle is represented as a numerical value where 0.0 corresponds to the muscle in a rest or default position, a positive number corresponds to muscle contraction and a negative number corresponds to muscle relaxation.
  • the numerical value for a strain, SM, of a muscle, M may be as expressed in Equation 1.
  • Expressions of an actor are determined for example, by scanning the actor’s face to capture scanned data, which can identify how points on the surface of the actor’s face move in three dimensions. A number of expressions can be scanned. While it is generally known which muscles are present in a face of a well-studies species, such as humans, there can be variability in where the muscles are attached, which are activated, where both ends are attached, their thicknesses, and what range of strains are possible. For example, a person with a “jowly” face would have different values for those than a person with a non-jowly face.
  • An animation creation system and animators who use the animation creation system may want a model for a specific live actor, including their skull shape, muscle-to-skull attachment positions, muscle thicknesses, etc., to construct a facial rig that can be animated, but that may not be possible to determine directly from the live actor.
  • the shape blending system may be treated as a rig that the animator can control by specifying blend shape weights, but for faces, this can be hard. Some results may be less than desirable as a human can create an almost infinite number of facial expressions and not all of those can be captured for blending.
  • An Al system can generate a trained manifold based on inputs from the scan results, dynamic muscle activations, and an anatomical model.
  • a physics-based muscle simulator may be used to generate dynamic muscle activations that are then consumed by the Al to produce a trained manifold.
  • the anatomical model may include the muscle topology of the specific live actor, e.g., shape, length, position, distribution, mass, and/or the like, of the facial muscles, and the skull surface of the specific live actor, e g., the shape, dimension, curvature parameters, and/or the like of the skull.
  • a difficulty with animating a face using muscle topology model is that there are a large number of facial muscles and specifying a strain for each can be tedious, especially where many scenes need to be created. For example, there are more than 40 skeletal facial muscles that are controlled by seven nerves in a human face. To build a mathematical topology of the facial muscle structure, track its movement and mapping the facial muscle movements to that of a character face of an artificial character, would only induce computational complexity exponentially.
  • An alternative approach of animating a character face of an artificial character is based on facial poses performed by a live actor. Instead of tracking facial muscle movements of a complex three-dimensional facial muscle topology, geometric characteristics of the facial surface corresponding to each facial pose performed the live actor may be learnt by a machine learning system, which in turn build a blendshape of an array of controllable elements applicable on a character face of an artificial character.
  • an array of markers may be mounted on the face of the live actor, which capture, instead of facial scan of muscle strains, position parameters capture the geometric characteristics of the array of markers as corresponding to a particular facial pose.
  • the machine learning system may in turn translate the position parameters into control values indicating the positions of the controllable elements on the character face of the artificial character.
  • the constructed blendshape of controllable elements may be used to animate a particular facial pose on the character face using the machine learning system generated control values.
  • the machine learning system may be trained with position data of the finite number of markers, and each position may be represented by a set of three-dimensional coordinates, e.g., (X, Y, Z), of a respective marker.
  • X, Y, Z three-dimensional coordinates
  • the machine learning systems may be trained on the features of the markers.
  • the marker feature may include any of the pair-wise distance, the specific angle of the markers. For instance, when the eyes are closed, a specific angle between the upper and lower eye lids decreases, thus this specific angle is used as a marker feature to describe the facial poses relating to closing the eyes.
  • the chin marker when the jaw is opening, the chin marker may have a large distance between the nose marker compared to that in a neutral facial pose. Thus, the distance between the chin marker and the nose marker may be used as a feature relating to facial poses that entails jaw opening.
  • the training data may include a set of marker features of the markers for the human actor, annotated with the corresponding facial pose associated with each set of the specific marker features.
  • the animation provided from the animator is configured to create synthetic training marker.
  • a key challenge in solving facial animation is the ambiguity, e.g., it is generally difficult to estimate whether the actor is showing his/her teeth just by looking at the markers without the footages.
  • the exact expression from the actor cannot be inferred directly so as the solvers.
  • the solver can hardly determine whether the lips are either closed or open unless if more information other than the markers is available.
  • a naive solver would mislead the final solve by only considering the optimization between the blendshape and the performance marker target.
  • the animator may avoid this problem by specifying a specific blendshape animation to guide the solver to avoid this ambiguity effectively.
  • each actor typically has his or her specific way to act the performance (e.g., when some actors are talking, they tend to close the lip a lot, but some actors don't).
  • Those "specific animation” may be captured as a (blendshape) training animation from the perspective that the animator uses for training the facial solver. This guides the facial solver to bias towards to the specific actor's performance.
  • control values for animating a facial pose may usually take the form of an animation function of a time-varying value of the control value.
  • the animation function may take a form of the time-varying position parameter of the controllable elements.
  • the animation function may take a form of the time-varying muscle strain values of the plurality of muscles corresponding to the character face.
  • Salient points may be sampled from the animation control curve to result in a set of salient data points that represent the animation control curve and serve as the control input for applying to the control elements.
  • the salient points may be manually selected by an animator via a user interface, where the animator may determine the number of salient points to choose and where the salient points on the curve are chosen.
  • the salient points may be automatically chosen by a system based on approximation of a cluster of adjacent data points on a segment of the animation curve. The automatically computed salient points may be further fine-tuned by the animator via the user interface. By using a reduced set of sampled salient points as an input to the animation system, computational efficiency can be further enhanced.
  • topology refers to properties of a geometric object that are preserved under deformations, such as stretching, twisting, crumpling and bending.
  • a facial muscle topology includes a set of data depicting the positions, shapes, and strains of facial muscles.
  • a skull surface refers to geometric properties of the outside layer of an object.
  • a skull surface includes parameters that describe the geometric properties of the outer layer of the skull, such as the curve, shape, and/or the like.
  • FIG. 1A illustrates an animation pipeline 100 that may be used to render animated content showing animation of a character based on a model and a rig that is generated from scans of a live actor.
  • a live actor 102 (“Actor A”) may be outfitted with fiducials 104 and have their face, expressions and/or body scanned by a scanner 106. The scanner 106 would then output or store results of scanning to a scan results store 108.
  • the fiducials 104 allow for the scan data that results from scanning to include indications of how specific points on the surface of the face of the live actor 102 move given particular expressions.
  • the scan data could also indicate the surface manifold in 3D space that corresponds to the surface of the live actor’s face. While it may be expected that the skull of the live actor 102 is a constant shape and changes only by translations and rotations (and jaw movement), it is not expected that the surface manifold would be constant, jaw movements, air pressure in the mouth, muscle movements, and other movable parts move and interact. Instead, different movements and facial expressions result in different thicknesses, wrinkles, etc. of the actor’s face.
  • the output from the scanner 106 may be stored as scan results 108, which may include a skin surface representation, muscle parameters, joint parameters, strain parameters, and/or the like.
  • the scan results 108 may also be used to generate one or more data bundles of scan results over a data bundle time period T.
  • each data bundle records a respective time-varying vector representing changes of the skin surface, muscle parameters, joint parameters, strain parameters, and/or the like over the data bundle time period T. Further details of the data bundles over a bundle time period are discussed in commonly-owned and co-pending U S Provisional Application Nos. 63/076,856 and 63/076,858, both filed on September 10, which are both hereby expressly incorporated by reference herein in their entirety.
  • each human actor has more or less the same facial muscles, and more or less the same bone structure in the skull.
  • An anatomical model dataset 112 may be provided that represents muscles, where they connect, what other typical facial elements are present (eyes, eyelids, nose, lips, philtrum, etc.) and other features likely common to most human faces.
  • the anatomical model dataset 112 may also provide a generic skull structure dataset that represents each piece of bone in the skull, the respective position parameters, the respective shape parameters, the respective connector parameter connecting to a neighboring piece, and/or the like.
  • the anatomical model 112 may represent a customized muscle and/or skull dataset that is specific to Actor A, or alternatively, a generic muscle and/or skull dataset which is pre-stored in the database of anatomical database.
  • This anatomical model 112 can be provided to a muscle simulator 110 that performs physics-based muscle simulation and provides a dynamic muscle activations dataset 113 for Actor A.
  • the anatomical model 112 may provide a generic muscle model of muscle and skull that can be used as a reference point, together with the scan results 108 of the live actor A to provide, by the muscle generation module 114, a specific muscle topology that is (1) customized to Actor A, and (2) has a simplified approximation structure for machine learning.
  • the scan results 108 can also be fed to a skin surface generation module 115, which may in turn extract skin surface data from the scan to the machine learning model 118.
  • the generated muscle strains from the muscle strain generation module 114, together with skin representation from module 115, joint representation, and/or the like over a bundle time period, may be input to the machine learning model 118.
  • the machine learning model 118 generates a predicted skin surface representation (e.g., the visible facial expression such as “smile,” “frown,” etc.).
  • the machine learning model 118 can learn a transformation between parameters of the muscles, strains, joints, and/or the like and the skin surface representation of actor A through a training dataset in the form of data bundles representing scan results 108 from the actor A. Further details of the training process for machine learning model 118 may be found in FIG. 2A.
  • the machine learning system 118 establishes a more accurate (compared to what was traditionally done using generic anatomical data) transformative relationship that is specific to Actor A, between Actor A’s skin representation and the underlying muscle/joint movement parameters of Actor A.
  • the underlying muscle/joint movement parameters may include the positions of the muscles that are moved, strain values, direction of movements, attachment points to the joint, and/or the like., all of which are specific to Actor A.
  • the machine learning model 118 may be trained to infer the shape of the live actor’s skull, volume of muscles, range of motion, etc., to build a manifold of possible movements for the actor.
  • the machine learning model 118 may output a manifold to be stored in manifold storage 116.
  • the manifold may represent the range of plausible expressions.
  • the manifold may represent a set of solutions or constraints in a high-dimension space corresponding to a strain vector.
  • the machine learning model 118 may be trained to determine an action locus for the movement or expression adjustment (e.g., from “smile” to “grin”) and a range of action (e g., widened month, showing of teeth, changed upward angle of the mouth, etc.) made by the actor A.
  • the machine learning model 118 may then determine a subset of the muscle strain vector applicable to the range of action, e.g., which muscles are used, and what are the corresponding strains.
  • the machine learning model 118 may determine the manifold that limits changes to the data bundle to changes in the subset of the muscle strain vector.
  • the manifold model 116 may limit the changes to the strain vectors in the data bundle to a subset of muscle strain vectors relating to muscles that widen the month and show teeth, and the corresponding strains that change the upward angle of the mouth.
  • the manifold model 116 also limits the search of updated vector values for muscle vectors or strain vectors to a manifold of allowed values for an updated cache of data vectors when the movement or expression adjustment (e.g., from “smile” to “grin”) takes place.
  • the manifold model 116 of allowed values correspond to known feasible expressions of the live actor.
  • an animator 112 could generate meshes that correspond to facial expressions of the live actor for whom the muscle model was derived.
  • a mesh may be stored in a mesh deformation store 124.
  • the animation creation system 120 could be used by the animator 122 to generate a facial surface of an expression that was not specifically made by Actor A, but would be near what it would be if Actor A had tried that expression.
  • the animation creation system 120 may constrain an animator’s inputs by projecting them onto the manifold, which would have an effect of transforming animator inputs that are not corresponding to a plausible expression into a strain vector that does correspond to a plausible expression.
  • the animator’s inputs may be represented in memory as a strain vector, having components corresponding to some facial muscles, as well as other animation variables that may not be related to muscles or that are more easily represented directly, such as jaw movement, eye movement, and the like.
  • a strain vector may be represented by an array of values, wherein a value in the array corresponds to a vector component that is a value of strain in a particular dimension, representing strain on one muscle perhaps.
  • inputs may include an input strain vector, indicative a strain values for some or all of the muscles in the muscle model, and values for the other animation variables, such as a scalar value for a jaw angle, two 2D vectors corresponding to rotations of the eyes, etc.
  • the animation creation system 120 can determine the volumes occupied by the muscles, and thus the surface of the character’s skin, and output a mesh manifold of the character’s skin, possibly depicting an expression corresponding to the input strain vector.
  • an animator can specify a facial expression in the domain of muscle semantics, which can simplify an animation process compared to limiting the animator to making combinations of recorded expressions as blends of the scanned facial shapes.
  • a length of a muscle is determined from its strain value and its rest length. Allowed strain values may be constrained by the manifold so that strain values remain within plausible boundaries.
  • an Al process can determine a likely strain vector that, when input to an animation generation system, would result in an expression largely matching the scanned expression.
  • the animation generation system can provide those as the domain in which the animator would modify expressions. After training an Al system using dynamic scans of an actor’s face as the ground truth for training, the muscle model can be derived that would allow for the simulation of other expressions that were not captured.
  • an animation process may simulate facial expressions through the use of a unique combination of hi -resolution scans of a human face, simulated muscles, facial control vectors, and constraints to generate unlimited facial expressions.
  • an Al system is employed to receive facial control vectors generated from a series of muscle strain inputs and process those vectors relative to a facial expression manifold configured to constrain facial expressions of the simulation to plausible expressions. Simulation need not be limited to simulating facial expressions that correspond to a real-world physical action, but more generally may be the generation of facial expressions informed by expressions made and recorded.
  • Separate Al systems may be used to train and derive the muscle model and to train and derive the manifold. In order to hit a target expression (and corresponding skin shape), the muscle model may be differentiable.
  • An Al system may include a variational autoencoder (VAE).
  • the Al uses muscle control vectors, instead of blend shape weights or other approaches, and can then specify strains on those muscle control vectors, which would in turn specify lengths of contractions of the muscles in a simulator.
  • Each muscle scan be represented by a curve, which may have a length that is a function of the strain.
  • a muscle vector may comprise strains that affect a mesh representing the skin of a character.
  • the muscles may include a rest length and attachment point, and together represent a muscle geometry.
  • a Tenderer 126 can process the facial surface, perhaps mapping it to a character model from a character model store 128, such as a non-human character played by Actor A, to form animated output that may be stored in animated output store 130.
  • FIG. IB illustrates an alternative animation pipeline 100b that may be used to render animated content showing animation of a character based a rig that is generated based on geometric characteristics of facial poses.
  • the markers 104 placed on the face of the live actor 102 may be used to collect position data of the markers 104.
  • the markers 104 include any combination of a physical marker that is attached on the face of the human actor, a painted marker on the face of the human actor, a virtual marker that is projected on to the face of the human actor.
  • the markers 104 may take a form of an array of dotted pattern placed on the face of the human actor.
  • the position parameters of the markers 104 are stored at the position data store 138.
  • the live actor 102 evolves from “smile” to “neutral,” the time-varying position parameters of the markers 104 during this time period are also stored at the position data store 138.
  • the geometric parameters 134 from the position data 138 are then input to the machine learning system 139 to transform into a blendshape of geometric parameters 126 representing position data on the animated character face.
  • the blendshape of geometric parameters may take a form of a set of position offset vectors, which may be added (optionally scaled by an animated weight factor) to the skin of a basic (neutral) pose.
  • the machine learning system 139 e.g., a deep learning network, or other types of error correcting models, is configured to learn a mapping relationship between the position of the markers 104 on the face of the live actor 102 and the control values and/or positions of the controllable elements for the character face.
  • an error correcting system may be used to compute an error metric associated with the output from module 139. The error metric is then minimized to update the model parameters.
  • the blendshape for the character face 126 may be used to input to an animation creation system 120, which generates the control elements store 134 to create a manifold for animation.
  • the rest of the animation process may be similar to that described in FIG. 1 A.
  • the animation system 100b is conducted based on geometric characteristics of actor face without generating a mesh of a facial rig to reduce computational complexity.
  • FIG. 2A illustrates an example neural network system 200 in which scan results are provided to a muscle simulator 202 and a neural network 204 and an anatomical model to the muscle simulator 202.
  • An output of the muscle simulator 202 is dynamic muscle activations for Actor A, which in turn are provided to neural network 204.
  • Neural network 204 then outputs a manifold to manifold storage 208.
  • FIG. 2B provides an example block diagram illustrating an alternative deep machine learning system 139 in which marker position data is learnt to be transformed to control elements in a mesh applicable to the character face of an artificial character.
  • Position data e.g., marker positions 21 la-n may be input to the deep machine learning system 139.
  • the marker position data 211a may take a form of the geometric coordinates of the position of the respective marker, e g., (X, Y, Z).
  • the machine learning system 139 may further receive characteristic data of the character face 212 (e.g., a set of sampled grids on the skin surface of the character face, etc.) and generate a blendshape of control elements 230 applicable for a character face of an artificial character.
  • the blendshape 230 is comprised of a set of controllable elements 22 la-n, each corresponding to a controllable value.
  • the controllable value may dictate the position of the corresponding controllable element on the character face, and thus changes to the controllable values may cause a deformation of the character face, resulting in a corresponding facial pose.
  • the machine learning system 139 may be trained to transform the marker positions 21 la-n to a set of controllable element values 22 la-n in the blendshape 230 to cause the mesh 230 to generate a desired facial pose on the character face.
  • the machine learning system 139 is configured to learn a mapping relationship between the position of the markers on the face of the live actor and the control values and/or positions of the controllable elements in the mesh for the character face.
  • the machine learning system 139 may receive a training dataset of position data of the markers placed on the face of the live actor, and a set of corresponding ground-truth control element values that generate the same facial pose on the character face as that performed by the live actor.
  • the position data in the training dataset may include marker features, such as a pairwise distance, a specific angle of the marker, and/or the like.
  • marker features such as a pairwise distance, a specific angle of the marker, and/or the like.
  • an example marker feature may include the angle between the upper and lower eye lids (which may indicate whether or how much the eyes are “closed” or “open”).
  • an example marker feature may include a distance between a chin marker and a nose marker compared to the baseline distance in the neural facial pose, which may indicate whether the jaw or how much the haw is “closed” or “open.”
  • the training dataset may includes a time series of time-varying positions of the markers, e.g., from those corresponding to “neutral” to “smile,” and on to another facial pose, and/or the like.
  • the deep learning system 139 may then generate predicted control element values in response to the input position data of the markers, and compare the predicted control element values with the ground-truth control element values to compute a training objective.
  • the training objective is minimized to update the machine learning system 139.
  • the training objective may further incorporate information relating to characteristics specific to an actor.
  • the deep learning system 139 may predict a preferred expression characteristic in addition to the control values transformed from the marker positions.
  • the predicted preferred expression characteristic and the actual ground-truth expression may be compared to learn the preference of marker position inferred expression, e.g., even for the same marker positions around the mouth contour, one actor may prefer to have the lips closed, and another actor may prefer to have the lips open.
  • the set of control elements 221a-n may be distributed at different areas on the character face.
  • the deep learning system 139 may generate control values progressively.
  • the deep learning system 130 may generate control values for a first set of control elements at the eye region of the character face, and then generate control values for a second set of control elements at the jaw region of the character face that has the eye area solved with the first set of control values, and so on.
  • the progressive solving process of the character face may allow progressive adjustment of the control values region by region and thus improve accuracy of the facial solving process.
  • FIG. 3A illustrates an example of a data structure that may represent a muscle model.
  • each muscle may be defined by a bone attachment point, a skin attachment point, and a muscle volume.
  • the volume of the muscle may change shape, and the distance between the bone attachment point and the skin attachment point may change, thus creating expressions. Additional elements may be included in a control vector that are for other animation variables.
  • FIG. 3B illustrates an example of a data structure that may represent the position data 21 la-n as show in FIG. 2B.
  • the control values 350 may each be associated with a controllable element index, and the 3D position coordinates.
  • the control values 350 may be progressively assigned to the controllable elements per different subregions on the face, e.g., eye area, jaw area, cheek area, and/or the like.
  • FIG. 4 illustrates inputs and outputs of an animation creation system 402.
  • Inputs may include an input strain vector 404, indicative a strain values for some or all of the muscles in the muscle model, and values for the other animation variables, such as a scalar value for a jaw angle, two 2D vectors corresponding to rotations of the eyes, etc.
  • the animation creation system 402 can determine the volumes occupied by the muscles, and thus the surface of the character’s skin, and output a mesh manifold of the character’s skin, possibly depicting an expression corresponding to the input strain vector 404.
  • FIG. 5 provides an example diagram illustrating an example workflow of generating a blendshape usable for representing an animated facial pose applicable in an animation system to an artificial character.
  • a machine learning system 139 may receive a set of geometric parameters corresponding to positions 501 of a plurality of markers placed on a face of a human actor. The positions 501 of the plurality of markers represents a facial pose performed by the human actor, e g., “smile,” “neutral,” “angry,” and/or the like.
  • the machine learning system 139 may also receive character face characteristics data 502 that depict the facial features of the character face.
  • the machine learning system 139 has been trained to learn a transformation from geometric parameters labelled with a certain facial pose performed by the human actor to a corresponding animated facial pose of the artificial character.
  • the system 139 may then construct a blendshape 503 comprising a set of control elements applicable to a character face of the artificial character based, at least in part, on a deep learning network input representing the set of geometric parameters from the positions 501.
  • the blendshape 503 includes a set of controllable elements that are mapped from the plurality of markers on the face of the human actor. Position changes of one or more control elements may cause a change from a first animated facial pose to a second animated facial pose on the character face.
  • the three-dimensional blendshape 503 of the facial rig is constructed by determining, via the deep learning network 139, a plurality of positions of the set of the controllable elements on the character face.
  • the controllable elements include one or more control handles that is editable by an animation artist.
  • the blendshape 503 may be applied to the character face to generate an animated facial pose. Specifically, at process 511, a first subset of controllable elements corresponding to a first part of the facial rig is applied to the character face. For example, the control activation values corresponding to eye area controllable elements are generated by the machine learning system 130, and applied to the mesh.
  • a second subset of controllable elements corresponding to a second part of the facial rig are applied to a first resulting animated facial pose of the character face from applying the first subset of controllable elements. For example, after “solving” the eye area based on the control elements at process 511 (e.g., to generate and apply control values of eye area to the set of controllable elements), control activation values corresponding to jaw area controllable elements are generated by the machine learning system 130, and applied to the blendshape that has already been populated with eye area controllable values.
  • a second resulting animated facial pose of the character face from applying the second subset of controllable elements may be fine-tuned with a user input adjusting the first subset of controllable elements or the second subset of controllable elements.
  • a model editor user interface 515 may be used to receive a user input 518 of modification that adjusts the generated animation, and provide the adjusted animated facial pose via a display 520.
  • the received user input may include a modification to the set of controllable elements, e.g., a change to a position of at least one controllable elements.
  • the fine-tuning may be performed in real time and may be stored at the fine-tuned animated facial poses 518.
  • FIG. 6 provides an example diagram illustrating an example workflow of generating an animated facial pose on a character face of an artificial character using salient points on the animation control curves to improve computational efficiency.
  • the animated character face may be generated via a set of animation control curves of time-varying control values.
  • the blendshape includes a set of controllable elements whose value changes cause a change in an animated facial pose on the character face of the artificial character.
  • the animated facial pose on the character face may be controlled by the blendshape of control values representing positions on the animated character face transformed from geometric parameters of markers on a human face representing one or more facial poses performed by the human actor, similar to the process as shown in FIGS. IB and 2B. Each set of the positions of the plurality of markers represents a respective facial pose performed by the human actor.
  • the mesh can be generated from the data relating to one or more facial poses performed by the human actor includes a plurality of facial scans obtained from the face of the human actor, similar to the processes shown in FIGS. 1A and 2A.
  • Each facial scan includes a set of muscle strain values and a corresponding set of skin surface values that correspond to a respective facial pose.
  • a plurality of animation control curves are obtained over a period of time corresponding to the plurality of controllable elements contained in the mesh.
  • Each animation control curve from the plurality of animation control curves takes a form of a time series of muscle strain values evolving over the time period.
  • the animation control curve may take a form of a time series of muscle strain values evolving over the time period (corresponding to the animation system shown in FIG. 1 A), or a time series of geometric parameter depicting a time-varying position of a respective controllable element over the time period (corresponding to the animation system shown in FIG. IB).
  • a set of “snapshots” corresponding to a plurality of time points may be jointly selected, across the plurality of animation control curves over the period of time.
  • Each snapshot represents a plurality of salient data points on the plurality of animation control curves at a particular time point. In this way, a set of salient data points may be jointly selected from the plurality of animation control curves.
  • the snapshots of salient data points may be selected from a user input indicating the density of the salient data points on each animation control curve.
  • the set of snapshots of selected respective set of salient data points may be selected according to a user input indicating one or more particular data points on a particular animation control curve via a user interface.
  • the snapshots of salient data points may be automatically selected via an algorithm or computational procedure.
  • the salient data points may be computed by sampling a discrete-time series of data points from the respective animation control curve, and computing a salient point on the respective animation control curve to approximate a cluster of adjacent data points from the discrete-time series of data points.
  • the salient point is computed as a data point on the respective animation control curve corresponding to an average time instance among the cluster of adjacent data points.
  • the salient point is computed as a data point at which a first order derivative of the respective animation control curve changes a sign among a time range spanned by the cluster of adjacent data points.
  • a machine learning system may be adopted to jointly select the snapshots of salient data points from the plurality of animation control curves. For example, the machine learning system may be trained to minimize a difference (e.g., a root mean square difference, etc.) between an animated facial pose controlled by original animation control curves and an animated facial pose controlled by sampled salient data points. In this way, the machine learning system may learn to select the salient data points to approximate the performance of the original animation control curves.
  • a difference e.g., a root mean square difference, etc.
  • the selected sets of salient data points are applied as time-varying control values for the set of controllable elements over the period of time.
  • one or more animated facial poses of the character face of the artificial character are generated from the application of the selected sets of snapshots of salient data points.
  • FIGS. 7A-7C provide example plots illustrating selecting salient points from animation control curves.
  • various salient points can be selected.
  • the salient points may be evenly sampled from the curve 705, e.g., see 708a-c.
  • the salient points may be selected by a user input at 710a, 710b, 710c.
  • the salient point may be computed, e.g., salient point 710a is computed as an approximation of the cluster of points 708a, 708b and 708c.
  • salient points may be selected by prioritizing data points at a “steep” segment on the animation curve, e.g., 710c, 712, 710d, as these data points may represent control values for more “extreme” variants in facial poses.
  • a user interface may be presented to an animator for select or fine-tune the salient points 720, generate an animation 722 and/or adjust the animation 724.
  • a count of salient points may be decided for sampling, e.g., the more “turns” the curve may have, indicating more drastic change in the animated facial poses, more salient points may be needed.
  • An algorithm may then be applied to choose the salient points by prioritizing selecting data points that depict the “turning points” on the animation control curve (e.g., when the first order derivative changes sign, etc.), and may select relatively fewer data points at a flatter segment on the curve.
  • the selection of the salient point may be optimized by minimizing a mean square error between the original animation curve and an extrapolated curve connecting the selected salient points.
  • a process to automatically compute or select salient points may include: computing at a plurality of sample points, the first/second order derivative of the animation curve, and selecting a subset of salient points at which the first order derivative changes signs, or the second order derivative is close to zero. The system may then compute a mean square error between the animation curve and the selected set of salient points. The process may be repeated for a different subset of salient points to minimize the mean square error and thus fine-tune the selection of salient points.
  • FIG. 7C illustrates an example of jointly selecting snapshots of salient data points across different animation control curves.
  • the set of control elements may be controlled by a set of control animation curves represented by 721, 722, 723 (or additional animation curves).
  • the selected snapshots 725, 726 and 272 at various time points may contain various salient data points represented by the star markers at the intersections of the snapshots of each animation control curve 721, 722 and 723.
  • the salient data points may be used to replace the original control curves 721, 722 and 723.
  • the visual content generation system 800 of FIG. 8 can be is configured to generate and process muscle models, facial rigs, and animator user interfaces described in relation to FIGS. 1-11 and may be implemented by software executing on one or more computer systems (e.g., each like a computer system 1300 illustrated in FIG. 13).
  • FIG. 8 illustrates the example visual content generation system 800 as may be used to generate imagery in the form of still images and/or video sequences of images.
  • the visual content generation system 800 may generate imagery of live action scenes, computer generated scenes, or a combination thereof.
  • users are provided with tools that allow them to specify, at high levels and low levels where necessary, what is to go into that imagery.
  • a user may be an animation artist and may use the visual content generation system 800 to capture interaction between two human actors performing live on a sound stage and replace one of the human actors with a computergenerated anthropomorphic non-human being that behaves in ways that mimic the replaced human actor’s movements and mannerisms, and then add in a third computer-generated character and background scene elements that are computer-generated, all in order to tell a desired story or generate desired imagery.
  • Still images that are output by the visual content generation system 800 may be represented in computer memory as pixel arrays, such as a two-dimensional array of pixel color values, each associated with a pixel having a position in a two-dimensional image array.
  • Pixel color values may be represented by three or more (or fewer) color values per pixel, such as a red value, a green value, and a blue value (e.g., in RGB format).
  • Dimensions of such a two-dimensional array of pixel color values may correspond to a preferred and/or standard display scheme, such as 1920 pixel columns by 880 pixel rows.
  • Images may or may not be stored in a compressed format, but either way, a desired image may be represented as a two- dimensional array of pixel color values.
  • images are represented by a pair of stereo images for three-dimensional presentations and in other variations, some or all of an image output may represent three-dimensional imagery instead of just two-dimensional views.
  • a stored video sequence may include a plurality of images such as the still images described above, but where each image of the plurality of images has a place in a timing sequence and the stored video sequence is arranged so that when each image is displayed in order, at a time indicated by the timing sequence, the display presents what appears to be moving and/or changing imagery.
  • each image of the plurality of images is a video frame having a specified frame number that corresponds to an amount of time that would elapse from when a video sequence begins playing until that specified frame is displayed.
  • a frame rate may be used to describe how many frames of the stored video sequence are displayed per unit time.
  • Example video sequences may include 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other frame rates. Frames are interlaced or otherwise presented for display, but for the purpose of clarity of description, in some examples, it is assumed that a video frame has one specified display time and it should be understood that other variations are possible.
  • One method of creating a video sequence is to simply use a video camera to record a live action scene, i.e., events that physically occur and can be recorded by a video camera.
  • the events being recorded can be events to be interpreted as viewed (such as seeing two human actors talk to each other) and/or can include events to be interpreted differently due to clever camera operations (such as moving actors about a stage to make one appear larger than the other despite the actors actually being of similar build, or using miniature objects with other miniature objects so as to be interpreted as a scene containing life-sized objects).
  • Creating video sequences for story-telling or other purposes often calls for scenes that cannot be created with live actors, such as a talking tree, an anthropomorphic object, space battles, and the like. Such video sequences may be generated computationally rather than capturing light from live scenes. In some instances, an entirety of a video sequence may be generated computationally, as in the case of a computer-animated feature film. In some video sequences, it is desirable to have some computer-generated imagery and some live action, perhaps with some careful merging of the two.
  • a rendering engine performs ray tracing wherein a pixel color value is determined by computing which objects lie along a ray traced in the scene space from the camera viewpoint through a point or portion of the camera view plane that corresponds to that pixel.
  • a camera view plane may be represented as a rectangle having a position in the scene space that is divided into a grid corresponding to the pixels of the ultimate image to be generated, and if a ray defined by the camera viewpoint in the scene space and a given pixel in that grid first intersects a solid, opaque, blue object, that given pixel is assigned the color blue.
  • determining pixel colors - and thereby generating imagery - can be more complicated, as there are lighting issues, reflections, interpolations, and other considerations.
  • a live action capture system 802 captures a live scene that plays out on a stage 804.
  • the live action capture system 802 is described herein in greater detail, but may include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
  • cameras 806(1) and 806(2) capture the scene, while in some systems, there may be other sensor(s) 808 that capture information from the live scene (e.g., infrared cameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).
  • sensor(s) 808 that capture information from the live scene
  • On the stage 804 there may be human actors, animal actors, inanimate objects, background objects, and possibly an object such as a green screen 810 that is designed to be captured in a live scene recording in such a way that it is easily overlaid with computergenerated imagery.
  • the stage 804 may also contain objects that serve as fiducials, such as fiducials 812(l)-(3), that may be used post-capture to determine where an object was during capture.
  • a live action scene may be illuminated by one or more lights, such as an overhead light 814.
  • the live action capture system 802 may output live action footage to a live action footage storage 820.
  • a live action processing system 822 may process live action footage to generate data about that live action footage and store that data into a live action metadata storage 824.
  • the live action processing system 822 may include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
  • the live action processing system 822 may process live action footage to determine boundaries of objects in a frame or multiple frames, determine locations of objects in a live action scene, where a camera was relative to some action, distances between moving objects and fiducials, etc Where elements are sensored or detected, the metadata may include location, color, and intensity of the overhead light 814, as that may be useful in postprocessing to match computer-generated lighting on objects that are computer-generated and overlaid on the live action footage.
  • the live action processing system 822 may operate autonomously, perhaps based on predetermined program instructions, to generate and output the live action metadata upon receiving and inputting the live action footage.
  • the live action footage can be camera-captured data as well as data from other sensors.
  • An animation creation system 830 is another part of the visual content generation system 800.
  • the animation creation system 830 may include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
  • the animation creation system 830 may be used by animation artists, managers, and others to specify details, perhaps programmatically and/or interactively, of imagery to be generated.
  • the animation creation system 830 may generate and output data representing objects (e g., a horse, a human, a ball, a teapot, a cloud, a light source, a texture, etc.) to an object storage 834, generate and output data representing a scene into a scene description storage 836, and/or generate and output data representing animation sequences to an animation sequence storage 838.
  • objects e g., a horse, a human, a ball, a teapot, a cloud, a light source, a texture, etc.
  • Scene data may indicate locations of objects and other visual elements, values of their parameters, lighting, camera location, camera view plane, and other details that a rendering engine 850 may use to render CGI imagery.
  • scene data may include the locations of several articulated characters, background objects, lighting, etc. specified in a two-dimensional space, three-dimensional space, or other dimensional space (such as a 2.5- dimensional space, three-quarter dimensions, pseudo-3D spaces, etc.) along with locations of a camera viewpoint and view place from which to render imagery.
  • scene data may indicate that there is to be a red, fuzzy, talking dog in the right half of a video and a stationary tree in the left half of the video, all illuminated by a bright point light source that is above and behind the camera viewpoint.
  • the camera viewpoint is not explicit, but can be determined from a viewing frustum.
  • the frustum would be a truncated pyramid.
  • Other shapes for a rendered view are possible and the camera view plane could be different for different shapes.
  • the animation creation system 830 may be interactive, allowing a user to read in animation sequences, scene descriptions, object details, etc. and edit those, possibly returning them to storage to update or replace existing data.
  • an operator may read in objects from object storage into a baking processor that would transform those objects into simpler forms and return those to the object storage 834 as new or different objects.
  • an operator may read in an object that has dozens of specified parameters (movable joints, color options, textures, etc.), select some values for those parameters and then save a baked object that is a simplified object with now fixed values for those parameters.
  • data from the data store 832 may be used to drive object presentation. For example, if an artist is creating an animation of a spaceship passing over the surface of the Earth, instead of manually drawing or specifying a coastline, the artist may specify that the animation creation system 830 is to read data from the data store 832 in a file containing coordinates of Earth coastlines and generate background elements of a scene using that coastline data.
  • Animation sequence data may be in the form of time series of data for control points of an object that has attributes that are controllable.
  • an object may be a humanoid character with limbs and joints that are movable in manners similar to typical human movements.
  • An artist can specify an animation sequence at a high level, such as “the left hand moves from location (XI, Yl, Zl) to (X2, Y2, Z2) over time T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5 degrees per frame”) or even at a very high level (e.g., “character A should move, consistent with the laws of physics that are given for this scene, from point Pl to point P2 along a specified path”).
  • Animation sequences in an animated scene may be specified by what happens in a live action scene.
  • An animation driver generator 844 may read in live action metadata, such as data representing movements and positions of body parts of a live actor during a live action scene, and generate corresponding animation parameters to be stored in the animation sequence storage 838 for use in animating a CGI object. This can be useful where a live action scene of a human actor is captured while wearing mo-cap fiducials (e.g., high-contrast markers outside actor clothing, high-visibility paint on actor skin, face, etc.) and the movement of those fiducials is determined by the live action processing system 822.
  • the animation driver generator 844 may convert that movement data into specifications of how joints of an articulated CGI character are to move over time.
  • a rendering engine 850 can read in animation sequences, scene descriptions, and object details, as well as rendering engine control inputs, such as a resolution selection and a set of rendering parameters. Resolution selection may be useful for an operator to control a trade-off between speed of rendering and clarity of detail, as speed may be more important than clarity for a movie maker to test a particular interaction or direction, while clarity may be more important that speed for a movie maker to generate data that will be used for final prints of feature films to be distributed.
  • the rendering engine 850 may include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
  • the visual content generation system 800 can also include a merging system 860 that merges live footage with animated content.
  • the live footage may be obtained and input by reading from the live action footage storage 820 to obtain live action footage, by reading from the live action metadata storage 824 to obtain details such as presumed segmentation in captured images segmenting objects in a live action scene from their background (perhaps aided by the fact that the green screen 810 was part of the live action scene), and by obtaining CGI imagery from the rendering engine 850.
  • a merging system 860 may also read data from a rulesets for merging/combining storage 862.
  • a very simple example of a rule in a ruleset may be “obtain a full image including a two-dimensional pixel array from live footage, obtain a full image including a two-dimensional pixel array from the rendering engine 850, and output an image where each pixel is a corresponding pixel from the rendering engine 850 when the corresponding pixel in the live footage is a specific color of green, otherwise output a pixel value from the corresponding pixel in the live footage.”
  • the merging system 860 may include computer processing capabilities, image processing capabilities, one or more processors, program code storage for storing program instructions executable by the one or more processors, as well as user input devices and user output devices, not all of which are shown.
  • the merging system 860 may operate autonomously, following programming instructions, or may have a user interface or programmatic interface over which an operator can control a merging process.
  • An operator can specify parameter values to use in a merging process and/or may specify specific tweaks to be made to an output of the merging system 860, such as modifying boundaries of segmented objects, inserting blurs to smooth out imperfections, or adding other effects.
  • the merging system 860 can output an image to be stored in a static image storage 870 and/or a sequence of images in the form of video to be stored in an animated/ combined video storage 872.
  • the visual content generation system 800 can be used to generate video that combines live action with computer-generated animation using various components and tools, some of which are described in more detail herein. While the visual content generation system 800 may be useful for such combinations, with suitable settings, it can be used for outputting entirely live action footage or entirely CGI sequences.
  • the code may also be provided and/or carried by a transitory computer readable medium, e.g., a transmission medium such as in the form of a signal transmitted over a network.
  • the techniques described herein are implemented by one or generalized computing systems programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination.
  • Special-purpose computing devices may be used, such as desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
  • FIG. 9 is a block diagram that illustrates a computer system 900 upon which the computer systems described herein and/or the visual content generation system 800 (see FIG. 8) may be implemented.
  • the computer system 900 includes a bus 902 or other communication mechanism for communicating information, and a processor 904 coupled with the bus 902 for processing information.
  • the processor 904 may be, for example, a general purpose microprocessor.
  • the computer system 900 also includes a main memory 906, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 902 for storing information and instructions to be executed by the processor 904.
  • the main memory 906 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 904.
  • Such instructions when stored in non-transitory storage media accessible to the processor 904, render the computer system 900 into a special-purpose machine that is customized to perform the operations specified in the instructions.
  • the computer system 900 further includes a read only memory (ROM) 908 or other static storage device coupled to the bus 902 for storing static information and instructions for the processor 904.
  • ROM read only memory
  • a storage device 910 such as a magnetic disk or optical disk, is provided and coupled to the bus 902 for storing information and instructions.
  • the computer system 900 may be coupled via the bus 902 to a display 912, such as a computer monitor, for displaying information to a computer user.
  • a display 912 such as a computer monitor
  • An input device 914 is coupled to the bus 902 for communicating information and command selections to the processor 904.
  • a cursor control 916 is Another type of user input device
  • cursor control 916 such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor 904 and for controlling cursor movement on the display 912.
  • This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
  • the computer system 900 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs the computer system 900 to be a special-purpose machine.
  • the techniques herein are performed by the computer system 900 in response to the processor 904 executing one or more sequences of one or more instructions contained in the main memory 906. Such instructions may be read into the main memory 906 from another storage medium, such as the storage device 910. Execution of the sequences of instructions contained in the main memory 906 causes the processor 904 to perform the process steps described herein.
  • hard-wired circuitry may be used in place of or in combination with software instructions.
  • Non-volatile media includes, for example, optical or magnetic disks, such as the storage device 910.
  • Volatile media includes dynamic memory, such as the main memory 906.
  • Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
  • Storage media is distinct from but may be used in conjunction with transmission media.
  • Transmission media participates in transferring information between storage media.
  • transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that include the bus 902.
  • Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
  • Any type of medium that can carry computer/processor implementable instructions can be termed a carrier medium and this encompasses a storage medium and a transient medium, such as a transmission medium or signal.
  • Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 904 for execution.
  • the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer.
  • the remote computer can load the instructions into its dynamic memory and send the instructions over a network connection.
  • a modem or network interface local to the computer system 900 can receive the data.
  • the bus 902 carries the data to the main memory 906, from which the processor 904 retrieves and executes the instructions.
  • the instructions received by the main memory 906 may optionally be stored on the storage device 910 either before or after execution by the processor 904.
  • the computer system 900 also includes a communication interface 918 coupled to the bus 902.
  • the communication interface 918 provides a two-way data communication coupling to a network link 920 that is connected to a local network 922.
  • the communication interface 918 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line.
  • ISDN integrated services digital network
  • Wireless links may also be implemented.
  • the communication interface 918 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.
  • the network link 920 typically provides data communication through one or more networks to other data devices.
  • the network link 920 may provide a connection through the local network 922 to a host computer 924 or to data equipment operated by an Internet Service Provider (ISP) 926.
  • the ISP 926 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 928.
  • the local network 922 and Internet 928 both use electrical, electromagnetic, or optical signals that carry digital data streams.
  • the signals through the various networks and the signals on the network link 920 and through the communication interface 918, which carry the digital data to and from the computer system 900, are example forms of transmission media.
  • the computer system 900 can send messages and receive data, including program code, through the network(s), the network link 920, and communication interface 918.
  • a server 930 may transmit a requested code for an application program through the Internet 928, ISP 926, local network 922, and communication interface 18.
  • the received code may be executed by the processor 904 as it is received, and/or stored in the storage device 910, or other non-volatile storage for later execution.
  • Processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof.
  • the code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors.
  • the code may also be carried by any computer-readable carrier medium, such as a storage medium or a transient medium or signal, e.g. a signal transmitted over a communications network.
  • the conjunctive phrases “at least one of A, B, and C” and “at least one of A, B and C” refer to any of the following sets: ⁇ A ⁇ , ⁇ B ⁇ , ⁇ C ⁇ , ⁇ A, B ⁇ , ⁇ A, C ⁇ , ⁇ B, C ⁇ , ⁇ A, B, C ⁇ .
  • conjunctive language is not generally intended to imply that certain embodiments require at least one of A, at least one of B and at least one of C each to be present.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Des modes de réalisation de la présente invention concernent une approche d'animation d'une visage de personnage d'un personnage artificiel sur la base de poses faciales réalisées par un acteur vivant. Des caractéristiques géométriques de la surface faciale correspondant à chaque pose faciale effectuée par l'acteur en direct peuvent être apprises par un système d'apprentissage automatique, qui à son tour construit un maillage d'un appareil facial d'un réseau d'éléments pouvant être commandés applicables sur un visage de personnage d'un personnage artificiel.
PCT/NZ2021/050172 2021-08-16 2021-10-11 Systèmes et procédés pour l'animation informatique d'un personnage artificiel à l'aide de poses faciales à partir d'un acteur vivant WO2023022606A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US202163233611P 2021-08-16 2021-08-16
US17/403,727 US11410370B1 (en) 2021-08-16 2021-08-16 Systems and methods for computer animation of an artificial character using facial poses from a live actor
US17/403,730 2021-08-16
US63/233,611 2021-08-16
US17/403,730 US11587278B1 (en) 2021-08-16 2021-08-16 Systems and methods for computer animation of an artificial character using facial poses from a live actor
US17/403,727 2021-08-16

Publications (1)

Publication Number Publication Date
WO2023022606A1 true WO2023022606A1 (fr) 2023-02-23

Family

ID=78463873

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/NZ2021/050173 WO2023022607A1 (fr) 2021-08-16 2021-10-11 Systèmes et procédés pour l'animation informatique d'un personnage artificiel à l'aide de poses faciales d'un acteur réel
PCT/NZ2021/050172 WO2023022606A1 (fr) 2021-08-16 2021-10-11 Systèmes et procédés pour l'animation informatique d'un personnage artificiel à l'aide de poses faciales à partir d'un acteur vivant

Family Applications Before (1)

Application Number Title Priority Date Filing Date
PCT/NZ2021/050173 WO2023022607A1 (fr) 2021-08-16 2021-10-11 Systèmes et procédés pour l'animation informatique d'un personnage artificiel à l'aide de poses faciales d'un acteur réel

Country Status (1)

Country Link
WO (2) WO2023022607A1 (fr)

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
ABDRASHITOV RINAT RINAT@DGP TORONTO EDU ET AL: "Interactive Exploration and Refinement of Facial Expression using Manifold Learning", PROCEEDINGS OF THE 33RD ANNUAL ACM SYMPOSIUM ON USER INTERFACE SOFTWARE AND TECHNOLOGY, ACMPUB27, NEW YORK, NY, USA, 20 October 2020 (2020-10-20), pages 778 - 790, XP058480930, ISBN: 978-1-4503-7514-6, DOI: 10.1145/3379337.3415877 *
BAILEY STEPHEN W ET AL: "Fast and deep facial deformations", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 39, no. 4, 8 July 2020 (2020-07-08), pages 94:1 - 94:15, XP058458592, ISSN: 0730-0301, DOI: 10.1145/3386569.3392397 *
RICHARD ROBERTS ET AL: "Optimal and interactive keyframe selection for motion capture", 20181204; 1077952576 - 1077952576, 4 December 2018 (2018-12-04), pages 1 - 4, XP058422926, ISBN: 978-1-4503-6062-3, DOI: 10.1145/3283254.3283256 *
RUHLAND KERSTIN ET AL: "Animation of Natural Virtual Characters Data-Driven Approach to Synthesizing Facial Animation Using Motion Capture", 1 January 2017 (2017-01-01), pages 30 - 41, XP055919841, Retrieved from the Internet <URL:https://ieeexplore.ieee.org/document/8013502> [retrieved on 20220510] *
WOODWARD ALEXANDER ET AL: "A low cost framework for real-time marker based 3-D human expression modeling", JOURNAL OF APPLIED RESEARCH AND TECHNOLOGY, vol. 15, no. 1, 1 February 2017 (2017-02-01), pages 61 - 77, XP055920273, ISSN: 1665-6423, Retrieved from the Internet <URL:https://www.sciencedirect.com/science/article/pii/S1665642317300081/pdfft?md5=4cd09710a75caad966d5a52c7919db29&pid=1-s2.0-S1665642317300081-main.pdf> [retrieved on 20220510], DOI: 10.1016/j.jart.2017.01.002 *

Also Published As

Publication number Publication date
WO2023022607A1 (fr) 2023-02-23

Similar Documents

Publication Publication Date Title
US11393150B2 (en) Generating an animation rig for use in animating a computer-generated character based on facial scans of an actor and a muscle model
US11158104B1 (en) Systems and methods for building a pseudo-muscle topology of a live actor in computer animation
US11354774B2 (en) Facial model mapping with a neural network trained on varying levels of detail of facial scans
US20220076409A1 (en) Systems and Methods for Building a Skin-to-Muscle Transformation in Computer Animation
EP3980975B1 (fr) Procédé de déduction de microdétails sur une animation de la peau
US11587278B1 (en) Systems and methods for computer animation of an artificial character using facial poses from a live actor
US20230154094A1 (en) Systems and Methods for Computer Animation of an Artificial Character Using Facial Poses From a Live Actor
WO2023022606A1 (fr) Systèmes et procédés pour l&#39;animation informatique d&#39;un personnage artificiel à l&#39;aide de poses faciales à partir d&#39;un acteur vivant
US11341702B2 (en) Systems and methods for data bundles in computer animation
US11875504B2 (en) Systems and methods for building a muscle-to-skin transformation in computer animation
US11715247B1 (en) Generating a facial rig for use in animating a computer-generated character based on facial scans and muscle models of multiple live actors
US20240169635A1 (en) Systems and Methods for Anatomically-Driven 3D Facial Animation
US11341703B2 (en) Methods and systems for generating an animation control rig
CA3192467A1 (fr) Systemes et procedes pour construire une transformation muscle-peau en animation par ordinateur

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21801248

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE