EP4631015A1

EP4631015A1 - Controllable face video synthesis

Info

Publication number: EP4631015A1
Application number: EP24708013.8A
Authority: EP
Inventors: Kangning LIU; Xuhui JIA; Yu-Chuan Su; Ruijin Cang
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2023-01-17
Filing date: 2024-01-17
Publication date: 2025-10-15
Also published as: WO2024155733A1

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output video frame from a system input. In one example, one of the systems include an encoder subsystem, a three-dimensional (3D) face modeling subsystem, a feature warping subsystem, and an image generation subsystem.

Description

CONTROLLABLE FACE VIDEO SYNTHESIS

CROSS REFERENCE TO RELATED APPLICATIONS

[001] This application is a non-provisional of and claims priority to U.S. Provisional Patent Application No. 63/439,511, filed on January 17, 2023, the entire contents of which are hereby incorporated by reference.

BACKGROUND

|002| This specification relates to video processing using neural networks.

[003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

[004] This specification describes how a video synthesis system implemented as computer programs on one or more computers in one or more locations can generate an output video frame from a system input. Throughout this specification, the output video frame is a video frame that includes a human face, however the video synthesis system can also be configured to generate output video frames, or more generally, output images, that show any of a variety' of types of objects other than human faces, including landmarks, landscape or location features, vehicles, tools, food, clothing, devices, animals, to name just a few examples.

[005] In some implementations, the output video frame is one of a temporal sequence of output video frames that each depict a first human face having a particular motion relative to a prior output video frame. In these implementations, the system input can include a source video frame which depicts the first human face and a temporal sequence of driving video frames that each depict a second human face having the particular motion relative to a prior driving video frame. In some of these implementations, the second human face can be substantively the same as the first human face while in others of these implementations, the two human faces can have different identifies than each other, i.e., belong to different humans. [006] The sequence of driving video frames is referred to as a temporal sequence because the driving video frames are arranged according to the order in which the corresponding video data was captured, e g., by a camera. The second human face that occurs in the driving video frames has a continual motion starting from a beginning frame to an end frame in the temporal sequence. The driving video frames may provide a target motion for the first human face in the source frame to follow. The driving video frames may be considered to “drive” the video synthesis process by providing the target motion.

[007] In some implementations, the system input includes a driving video frame which depicts a human face having an initial expression and target expression data, and the output video frame is an image that depicts the human face having a target expression specified by the target expression data.

[008] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[009] The video synthesis system as described in this specification can achieve results that outperform the state of the art in accurately generating a temporal sequence of output video frames that each depict a first human face having a particular motion relative to a prior video frame. To generate the output video frames, the video synthesis system processes a source video frame depicting the human face and, in some implementations, one or more driving video frames that depict a second human face having the particular motion. By leveraging prior knowledge of the human faces including expression feature and 3D geometry information extracted from the given video frames into the generation pipeline, the described video synthesis system can generate highly detailed output video frames that depict the first human face with improved semantic consistency and identity preservation. Compared with existing systems, the described video synthesis system is capable of better transferring face expressions across different identities, even in cases where there exists a significant pose difference in respective human faces depicted in source and driving video frames.

[0010] Some implementations of the video synthesis system described in this specification address common issues that arise in synchronous video communication under low data bandwidth conditions. In particular, the video synthesis system can be used to resolve delay and/or video quality degradation that could be introduced by network bandwidth limitations. Because the system can continuously generate each output video frame in a temporal sequence of output video frames from a relatively small set of parameters extracted from a corresponding driving video frame, which is more suitable for transmission from one person to another in different locations even with network bandwidth limitations, the system can play that predicted output video frame while the corresponding driving video frame is still in transit from its origin. Thus, to reduce bandwidth requirements, the system can transmit a small set of parameters extracted from the driving video frame, which can be used at the receiving end to generate and display a predicted output video frame rather than transmitting the driving video frame itself for display on the receiving end.

[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 shows an example video synthesis system.

[0013] FIG. 2 shows an example architecture of a generative neural network.

[0014] FIG. 3 is a flowchart of an example process for generating an output video frame. [0015] FIG. 4 is a flow diagram of sub-steps of one of the steps of the process of FIG. 3. [0016] FIG. 5 shows an example communication environment that includes a video synthesis system.

[0017] FIG. 6 is a flowchart of an example process for generating an output video frame in a communication environment.

|0018| FIG. 7 is a flowchart of an example process for generating an output video frame which depicts a human face having a target expression.

[0019] FIG. 8 is a flowchart of an example process for training a video synthesis system. [0020] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0021] FIG. 1 shows an example video synthesis system 100. The video synthesis system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0022] The video synthesis system 100 includes an encoder subsystem 120, a three- dimensional (3D) face modeling subsystem 130, a feature w arping subsystem 140, an image generation subsystem 150. and, optionally, an expression translation module 122. [0023] The encoder subsystem 120 receives a video frame that depicts a human face, and processes the video frame to generate appearance parameters, expression parameters, and pose parameters of the human face depicted in the video frame. For example, the video frame can be a source video frame 102. The source video frame 102 depicts a first human face. As another example, the video frame can be a driving video frame 104. The driving video frame 104 depicts a second human face.

[0024] In some cases, the second human face can be substantively the same as the first human face while in other cases, the two human faces can have different identifies than each other, i.e., belong to different humans.

[0025] In some cases, the driving video frame 104 is one of a temporal sequence of driving video frames included in a driving video that depicts the second human face having a continual motion across the temporal sequence of driving video frames, where each driving video frame (except for the first driving video frame in the temporal sequence) depicts the second human face having a particular motion relative to a prior driving video frame.

[0026] The appearance parameters can characterize the elements and/or portions of a human face depicted in the video frame. The elements and/or portions can include eyes, nose, mouth,jaw, ears, hair, eyebrows, mustache, beard, to name just a few examples. For example, the appearance parameters can characterize, for each element of the human face, a position, size, shape, or a combination thereof of the element. Generally, the elements of a human face are identity-specific, i.e., the appearance parameters generated for two different human faces will generally have different values than each other.

[0027] The expression parameters can characterize the facial expression of the human face depicted in the video frame. The facial expressions can include normal, sad, angry, happy, surprised, puzzled, wink, squint, smile, and frowning, to name just a few examples. For example, the expression parameters can include parameters that classify the human face into a particular one of these example facial expressions, and can include additional parameters that characterize associated information about the particular facial expression, e.g., that characterize a degree or a time duration of the particular facial expression.

[0028] Unlike the appearance parameters, the facial expressions are generally identityagnostic, i.e., the expression parameters generated for two different human faces may share some values in common when they both have the same or similar facial expression.

[0029] The pose parameters can specify a pose of the human face depicted in the video frame. Generally, the pose of the human face is an orientation, e.g., pitch, yaw, and roll, and, in some implementations, a position, e.g., an (x, y, z) position, of the human face in the video frame.

[0030] The exact numbers of the appearance parameters, the expression parameters, and the pose parameters to be generated by the encoder subsystem 120 for each video frame will depend on the actual configuration of the encoder subsystem 120, but typically, because of the complexity in modeling human facial appearance and expression, there will be a larger number of appearance parameters and a larger number of expression parameters than the pose parameters. For example, some implementations of the encoder subsystem 120 can generate at least 100 appearance parameters, at least 50 expression parameters, and no more than 6 pose parameters for a given video frame, i.e., the source video frame 102 or the driving video frame 104.

[0031] For convenience, the appearance parameters, the expression parameters, and the pose parameters generated by the encoder subsystem 120 for the source video frame 102 will be referred to below as the “source appearance parameters.” “source expression parameters, and “source pose parameters,” respectively. Analogously, the appearance parameters, the expression parameters, and the pose parameters generated by the encoder subsystem 120 for the driving video frame 104 will be referred to below as the “driving appearance parameters,” “driving expression parameters, and “driving pose parameters,” respectively.

[0032] In some implementations, to generate these parameters from the video frame, the encoder subsystem 120 includes one or more neural networks. Each neural network can be implemented with any appropriate neural network architecture that enables it to perform its described functions. Depending on the architecture, these parameters may be generated deterministically, e.g. by an output of the neural network, or stochastically e.g. where the output of the neural network parameterizes a distribution from which a parameter is sampled. [0033] In some implementations, the encoder subsystem 120 includes a first appearance feature extractor neural netw ork that is configured to generate a first subset of the appearance parameters. The first subset of appearance parameters can be in the format of a (sparse) appearance parameters vector (e.g., as a 50, 100, or 200 dimensional vector of appearance parameters). The encoder subsystem 120 also includes a second appearance feature extractor neural network that is configured to generate a second subset of the appearance parameters. The second subset of appearance parameters can be in the format of a (dense) appearance feature map (e.g., as a tensor of appearance parameters having the dimension of channel x height x width x depth). [0034] In these implementations, each appearance feature extractor neural network can be configured as a respective convolutional neural network that includes one or more downsampling blocks followed by one or more convolutional layers followed by one or more residual blocks which output the source appearance parameters.

[0035] In some implementations, the encoder subsystem 120 includes an encoder neural network that includes a plurality of convolutional layers followed by a fully connected layer which outputs the expression parameters and the pose parameters. As a particular example of this, the encoder neural network can have a detailed expression capture and animation (DECA) architecture, described in Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from in-the-wild images. ACM Transactions on Graphics (ToG), Proc. SIGGRAPH, 40(4):88: l-88: 13. Aug. 2021. It will be appreciated that any appropriate feature extraction method may be used to determine the appearance, expression and/or pose parameters.

[0036] The expression translation module 122 is an optional component of the video synthesis system 100. When included, the expression translation module 122 obtain data defining a target expression 105 and modify the driving expression parameters that have been generated by the encoder subsystem 120 such that the modified driving expression parameters will characterize the target expression 105 (rather than the original facial expression of the second human face depicted in the driving video frame 104). The modified driving expression parameters will then be provided in place of the driving expression parameters to the 3D face modeling subsystem 130 (or to another component of the video synthesis system 100).

[0037] For example, the expression translation module 122 can use the techniques described in Foivos Paraperas Papantoniou, Panagiotis P Filntisis. Petros Maragos, and Anastasios Roussos. Neural emotion director: Speech-preserving semantic control of facial expressions in “in-the-wild” videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18781-18790, 2022, to modify the driving expression parameters to generate the modified driving expression parameters.

[0038] The 3D face modeling subsystem 130 receives an input that includes the source appearance parameters, source expression parameters, and source pose parameters, and processes the input to generate an output that includes a plurality of 3D source frame key points for the first human face depicted in the source video frame 102.

[0039] The 3D face modeling subsystem 130 also receives another input that includes the driving appearance parameters, driving expression parameters, and driving pose parameters, and processes the other input to generate another output that includes a plurality of 3D source frame keypoints for the second human face depicted in the driving video frame 104.

[0040] Each keypoint can have a 3D position, e.g., an (x, y, z) position, and can correspond to a respective point on a human face. For example, the keypoints can correspond to various points on, or around, one or more elements of the human face, e.g., the eyes, nose, and mouth on the human face.

[0041] To generate the keypoints, the 3D face modeling subsystem 130 includes a predetermined face model. The predetermined face model can be, for example, a statistical model, a rule-based model, or a machine learning model, that is generated, e.g., trained, based on a collection of face images, e.g., 3D face scans. Generating keypoints using a predetermined face model may be considered to be a “supervised⁷’ process and can lead to the generation of more salient keypoints compared to prior art techniques that use unsupervised processes for keypoint generation. As such, fewer keypoints may be required which in turn can lead to higher uality video at a lower bitrates. In some implementations. 16 key points may be used. The predetermined face model can be used to generate a 3D mesh representation of first human face in accordance with the source appearance parameters, the source expression parameters, and the source pose parameters. Analogously, the predetermined face model can be used to generate a 3D mesh representation of the second human face in accordance with the driving appearance parameters, the driving expression parameters, and the driving pose parameters.

[0042] Generally, a 3D mesh of a human face provides the 3D geometry information for the human face that is previously unavailable in a video frame that is received by the video synthesis system 100. The 3D mesh representation includes a predetermined number of mesh vertices. Each mesh vertex is a 3D vertex that has coordinates that define its location in a 3D coordinate system, e.g., x, y, and z coordinates in a Cartesian system. From these 3D mesh vertices, a plurality of 3D keypoints, i.e., a plurality of 3D source frame keypoints or a plurality of 3D driving frame keypoints, can then be determined, e.g.. by selecting a subset of the mesh vertices from the 3D mesh representation as the 3D keypoints.

[0043] In some implementations, to generate the plurality of 3D frame keypoints, the 3D face modeling subsystem 130 process only the first subset of the source appearance parameters, i.e., by processing the (sparse) appearance parameters vector, in addition to the expression parameters and the pose parameters.

[0044] Specifically, the 3D face modeling subsystem 130 can process (the first subset of) the source appearance parameters, the source expression parameters, and the source pose parameters to generate a plurality of 3D source frame keypoints for the first human face depicted in the source video frame. The 3D face modeling subsystem can also process (the first subset of) the driving appearance parameters, the driving expression parameters, and the driving pose parameters to generate a plurality of 3D driving frame keypoints.

[0045] In some implementations, to generate the 3D driving frame keypoints for a human face that share the appearance of the first human face depicted in the source video frame 102 but have motion and expression similar to the second human face depicted in the driving video frame 104, the video synthesis system uses the source appearance parameters instead of the driving appearance parameters (although the system still uses the driving expression parameters and the driving pose parameters), such that the driving appearance parameters that are processed by the 3D face modeling subsystem 130 when generating the plurality of 3D driving frame keypoints are the same as the source appearance parameters processed by the 3D face modeling subsystem 130 when generating the plurality of 3D source frame keypoints. By doing so the 3D face modeling subsystem 130 alleviates the problem that an appearance of the second human face may nevertheless leak from the driving video frame to the output video frame because the driving appearance parameters contain identity-specific information of the second human face.

[0046] The feature warping subsystem 140 generates warped source appearance parameters. To do so. the feature warping subsystem 140 includes a motion field estimator 142 that receives an input that includes the source appearance parameters, the plurality of 3D source frame keypoints, and the plurality of 3D driving frame keypoints, and processes the input to generate a motion field, i.e., a scene flow estimation, that characterizes motion of pixels in the source image from the source video frame to the driving video frame.

[0047] In some implementations, the motion field estimator 142 can generate the scene flow estimation based on computing the flows induced by each 3D keypoint using the zeroth order approximation technique, described in Aliaksandr Siarohin, Stephane Lathuiliere, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in Neural Information Processing Systems, 32, 2019.

[0048] The feature warping subsystem 140 can then determine the warped source appearance parameters by warping the source appearance parameters in accordance with the scene flow estimation. Generally, the source appearance parameters are warped in a way that the warped source appearance parameters more accurately characterize the 3D geometry of the second human face depicted in the driving video frame 104. [0049] In some implementations, the feature warping subsystem 140 can do so by warping only the second subset of the source appearance parameters, i.e., by warping the (dense) appearance feature map generated from the source video frame 102.

[0050] In some implementations, because the flows induced by a 3D keypoint are more reliable within a threshold distance from the 3D keypoint, the motion field estimator 142 determines a 3D flow composition mask indicating which flow field is more reliable in each 3D location based on the source appearance parameters, and applies the 3D flow composition mask to generate the scene flow estimation. This is described in more detail below with reference to FIG. 3.

[0051] The image generation subsystem 150 receives an input that includes the warped source appearance parameters and the driving expression parameters, and processes the input to generate an output video frame 106. The output video frame 106 depicts the first human face that is depicted in the source video frame 102.

[0052] In some cases, the output video frame 106 is one of a temporal sequence of output video frames included in an output video that depicts the first human face having substantively the same continual motion as the second human face in the driving video across the temporal sequence of output video frames. That is, the output video frame 106 depicts the first human face having a particular motion relative to a prior output video frame, where the particular motion is substantively the same as the particular motion of the second human face in the driving video frame 104 relative to a prior driving video frame. In some cases where the expression translation module 122 is included, the output video frame 106 depicts the first human face having the target expression.

[0053] To generate the output video frame 106, the image generation subsystem 150 includes a generator neural network 152. The generator neural network 152 can be implemented with any appropriate neural network architecture that enables it to generate the output video frame, e.g., pixel-by -pixel in an auto-regressive manner, across a reverse diffusion process, and so on, based on the warped source appearance parameters and the driving expression parameters.

[0054] For example, the generator neural network 152 can be implemented as a generative neural network that can include any appropriate types of neural network layers (e.g., recurrent layers, attention layers, convolutional layers, and so forth) in any appropriate number (e.g., 5 layers, or 10 layers, or 100 layers) and connected in any appropriate configuration (e g., as a directed graph of layers). [0055] FIG. 2 shows example operations performed by a generator neural network 252. In particular, in the example of FIG. 2, the generator neural network 252 includes one or more spatially-adaptive normalization (‘’SPADE generator”) blocks 260 that fuse the warped source appearance parameters and the driving expression parameters to generate the output video frame 106.

[0056] The generator neural network 252 receives the driving expression parameters in the form a ID expression feature vector. The generator neural network 252 also receives the warped source appearance parameters in the form of a warped appearance feature map. The generator neural network 252 expands the driving expression parameters along a predetermined dimension, e.g., along the spatial dimension, to generate a 2D expression feature vector match the shape of the warped appearance feature map.

[0057] The generator neural network 252 then combined, e g., concatenates, the 2D expression feature vector and the warped appearance feature map to generate an intermediate conditional feature vector.

[0058] The generator neural network 252 processes the intermediate conditional feature vector using the one or more SPADE generator blocks 260 to generate an output video frame. Although the SPADE generator block is illustrated in FIG. 2, there can be any number of SPADE generator blocks, e.g., two, four, eight, or more SPADE generator blocks. In the case of multiple SPADE generator blocks, they can be stacked, i.e., arranged in a sequence with the output of any block except the last being an input to another of the blocks. The generator neural network 252 uses the output of the last SPADE generator block to generate the output video frame, e.g., by processing the output using one or more output layers.

[0059] The spatially-adaptive normalization block 260 processes a combination of the warped source appearance parameters and the driving expression parameters using a subset of the convolutional (“conv”) layers included in the block to generate scaling modulation parameters y, and process the combination using another subset of the convolutional (“conv”) layers included in the block to generate bias modulation parameters (3.

[0060] The spatially-adaptive normalization block 260 processes the warped source appearance parameters using a normalization layer, e.g., a batch normalization (“batch norm”) layer as illustrated in FIG. 2 and another subset of the convolutional (“conv”) layers included in the block to generate normalized warped source appearance parameters.

[0061] The spatially-adaptive normalization block 260 generates a block output based on (i) determining an elementwise product between the scaling modulation parameters y and the normalized warped source appearance parameters and (ii) determining an elementwise summation between the bias modulation parameters (3 and the elementwise product. Further details regarding SPADE generator blocks may be found in Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.

[0062] FIG. 3 is a flowchart of an example process 300 for generating an output video frame. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video synthesis system, e.g., the video synthesis system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0063] The system processes a source video frame using an encoder subsystem to generate source appearance parameters, source expression parameters, and source pose parameters (step 302). The source video frame depicts a first human face. The encoder subsystem can use one or more neural networks to generate these parameters.

[0064] The system processes a driving video frame using the encoder subsystem to generate driving appearance parameters, driving expression parameters, and driving pose parameters (step 304). The driving video frame can be one of a temporal sequence of driving video frames that each depict a second human face having a particular motion relative to a prior driving video frame. Like in step 302, the encoder subsystem can use the one or more neural networks to generate these parameters.

[0065] The system processes the source appearance parameters, the source expression parameters, and the source pose parameters using a three-dimensional (3D) face modeling subsy stem to generate a plurality of 3D source frame key points for the first human face depicted in the source video frame (step 306).

[0066] The 3D face modeling subsystem can use a predetermined face model to generate a 3D mesh representation of first human face in accordance with the source appearance parameters, the source expression parameters, and the source pose parameters. In implementations the predetermined face model can be a statistical face model. The 3D mesh representation includes a predetermined number of mesh vertices. From these mesh vertices, a plurality of 3D source frame keypoints can then be determined, e.g., by selecting a subset of the mesh vertices from the 3D mesh representation as the 3D source frame keypoints.

[0067] The system processes the driving appearance parameters, the driving expression parameters, and the driving pose parameters using the 3D face modeling subsystem to generate a plurality of 3D driving frame keypoints for the second human face depicted in the driving video frame (step 308). Like in step 306, the 3D face modeling subsystem can use the predetermined face model to generate the plurality of 3D driving frame key points.

[0068] The system processes the source appearance parameters, the plurality of 3D source frame keypoints, and the plurality of 3D driving frame keypoints using a feature warping subsystem to generate a scene flow estimation between the source video frame and the driving video frame (step 310). Step 310 is described in more detail below with reference to FIG. 4, which is a flow diagram of sub-steps 402-406 of step 310.

[0069] The system uses the feature warping subsystem to compute a respective estimated flow from a location (e.g., a pixel location) in the source video frame to a corresponding location in the driving video frame (step 402). For example, the estimated flow can be represented in the form of one or more 2D vectors pointing from the location in the source video frame to the corresponding location in the driving video frame. The locations in the source video frame can include locations corresponding to the 3D source frame keypoints, and the locations in the driving video frame can include locations corresponding to the 3D driving frame keypoints.

[0070] The system uses the feature warping subsystem to generate a 3D flow composition mask based on the respective estimated flows and the source appearance parameters (step 404). The 3D flow composition mask can have the same horizontal and vertical dimension as the source (or driving) video frame and includes values that indicate a reliability of each respective estimated flows.

[0071] For example, the 3D flow composition mask can assign a first value indicating a higher reliability⁷ to an estimated flow between a location that is within a threshold distance from a 3D source frame keypoint and a corresponding location that is within the threshold distance from a corresponding 3D driving frame keypoint. In contrast, the 3D flow composition mask can assign a second value indicating a lower reliability⁷ to an estimated flow between a location that is outside of the threshold distance from any 3D source frame keypoint and a corresponding location that is outside of the threshold distance from any 3D driving frame keypoint.

[0072] The system uses the feature warping subsystem to generate the scene flow estimation based on the 3D flow composition mask and the respective estimated flows (step 406). In implementations the respective estimated flows that have been indicated as having a relatively lower reliability can be masked, i.e., discarded, when generating the scene flow estimation. [0073] The system determines warped source appearance parameters by using the feature warping subsystem to warp the source appearance parameters in accordance with the scene flow estimation (step 312). Generally, the source appearance parameters are warped in a way that the warped source appearance parameters more accurately characterize the 3D geometry' of the second human face depicted in the driving video frame to match the motion of the second human face in the driving video frames.

[0074] The system processes the warped source appearance parameters and the driving expression parameters using an image generation subsystem to generate an output video frame (step 314). The image generation subsystem can use a generator neural network to generate the output video frame. In implementations the generator neural network can have a generative neural network architecture that include one or more spatially-adaptive normalization blocks as described above.

[0075] The system can repeatedly perform the process 300 to generate a plurality' of output video frames. By repeatedly performing the process 300, the system can generate a sequence of output video frames. For example, the output video frame depicts the first human face having a particular motion relative to a prior output video frame, where the particular motion is substantively the same as the particular motion of the second human face in the driving video frame relative to a prior driving video frame.

[0076] That is. the process 300 can be performed as part of predicting an output video frame for a system input for which the desired output video frame, i.e., the output video frame that should be generated by the system for the system input, is not known.

[0077] The process 300 can also be performed as part of processing inputs derived from a set of training data, i.e., inputs derived from a set of inputs for which the output video frame that should be generated by the system is known, in order to train the trainable components of the video synthesis system to determine trained values for the parameters of these trainable components. An example way of training the video synthesis system will be described further below with reference to FIG. 8.

[0078] FIG. 5 shows an example communication environment 500 that includes a video synthesis system. The communication environment 500 includes two or more user devices, including a first user device 510 and a second user device 520, and a network 530 that connects the two or more user devices. For example, the network 530 can include telephone circuits, coaxial cables, fiber optic cables, and Wi-Fi, satellite, or cellular network technology. The network 530 can be a local area network (LAN) that connects user devices in a limited area like a residence, school, laboratory, or office building. LANs include Ethernet and Wi-Fi networks. The network 530 can also be a larger network, including a wide area network (WAN) or the Internet.

[0079] Each user device is capable of communicating with other devices on the network 530. As will explained below, this can involve sending and receiving live data 535 over the network 530. Each user device includes an input device, e.g., a camera that captures video data and a display device that displays video data. Example user devices include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive live data 535 over the network 530.

[0080] Communication software, e.g.. a user application 512. installed on each of the user devices 510, 520 can cause the user devices 510, 520 to initiate a communication session with each other. The user application 512 can be a stand-alone app, a JavaScript program running in a web browser installed on the user device, or a web page implemented in a markup language and displayed on the web browser, and so on. Example user applications include video conferencing applications, VoIP applications, multi-player online video games that allow video communication between players, and other applications that facilitate video communication betw een user devices on a netw ork.

[0081] Each user device has access to a video synthesis system 514. The video synthesis system 514 can correspond to the video synthesis system 100 of FIG. 1. In some implementations, each user device locally implements an instance of the video synthesis system 514. That is, components of the video synthesis system 514 are stored within the user device. In other implementations, the video synthesis system 514 is accessible, e.g., through an application programming interface (API) or another data interface, by each user device over the network 530, where at least some components of the video synthesis system 514 are hosted remotely, e.g., in a data center, from the user device.

[0082] In some implementations, the user application 512 causes the video synthesis system 514 to perform the operations that will be described below; For example, in response to a selection in a user interface of the user application, the user application can cause the user device to switch from a conventional communication session to a communication session that uses the video synthesis system 514. In some other implementations, an operating system of the user device directly modifies a conventional communication session to use the video synthesis system 514, e.g., upon detecting a low data bandwidth condition (e.g., when the amount of data that can be transferred between the network 530 with a fixed time period is limited), to improve user experience with the communication session. [0083] The user device using the video synthesis system 514 receives live data 535 extracted from a corresponding driving video frame, and plays a predicted output video frame that is generated based on the live data 535, in place of an output video frame (the corresponding driving video frame) that is being transmitted from another user device, i.e., in place of the actual communications made by the user of the other user device. The live data 535 can include the driving expression parameters, the driving pose parameters, and the plurality of 3D driving frame keypoints that can be generated from the corresponding driving video frame by the video synthesis system 514.

[0084] Because the live data 535 has a relatively small data size than the output video frame and is thus more suitable for transmission over the network 530, e.g., even with network bandwidth limitations, the user device can play that predicted output video frame while the corresponding driving video frame is still in transit from its origin. In other words, the user device can play a predicted output video frame for a particular time step during the communication session before the user device even receives the output video frame for that time step.

[0085] FIG. 6 is a flowchart of an example process 600 for generating an output video frame in a communication environment. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system that implements the video synthesis system 100 of FIG. 1. appropriately programmed in accordance with this specification, can perform the process 600.

[0086] The system initiates a communication session between a first user device and a second user device (step 602). Each user device is configured to transmit data to and receive from one or more other user devices. The communication session can be initiated through a particular application running on both user devices. Initiating the communication session can include performing the following steps 604-608.

[0087] The system obtains, at the first user device, a source video frame which depicts the human face of a user of the first user device. For example, the source video frame can be captured by a camera of the first user device.

[0088] The system processes, at the first user device and by using the encoder subsystem and the 3D face modeling subsystem included in the video synthesis system, the source video frame to generate source appearance parameters, source expression parameters, source pose parameters, and a plurality of 3D source frame keypoints. [0089] The system transmits initiation data that includes the source appearance parameters, the source expression parameters, the source pose parameters, and the plurality of 3D source frame keypoints from the first user device to the second user device over a network.

[0090] Next, the system repeatedly performs the following steps 610-614 at each of multiple time steps during the communication session.

[0091] The system obtains, at the first user device, a driving video frame which depicts the human face of the user of the first user device at the time step (step 610). Like the source video frame, the driving video frame can be captured by a camera of the first user device. In process 600, the driving video frame and the source video frame depict the same human face. [0092] The system processes, at the first user device and by using the encoder subsystem and the 3D face modeling subsystem included in the video synthesis system, the driving video frame to generate driving expression parameters, driving pose parameters, and a plurality of 3D driving frame keypoints (step 612).

[0093] The system transmits live data that includes the driving expression parameters, the driving pose parameters, and the plurality of 3D driving frame keypoints from the first user device to the second user device over the network (step 614). Upon receiving the live data, the second user device is operable to generate, by using the feature warping subsystem and the image generation subsystem included in the video synthesis system, a predicted output video frame from the initiation data and the live data. In implementations the second user device can display the predicted output video frame as an output video frame at the time step that would be received from the first user device in a temporal sequence of output video frames to a user of the second user device.

[0094] FIG. 7 is a flowchart of an example process 700 for generating an output video frame which depicts a human face having a target expression. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video synthesis system, e.g., the video synthesis system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 700.

[0095] The system obtains target expression data which specifies a target expression (step 702). In some cases, the system can receive the target expression data from a user of the system, e.g., in the form of a user selection or input. For example, the system can present a list of facial expressions to the user and prompts for the user to make a selection of one of the facial expressions and. optionally, additional aspects relevant to the facial expression, e.g.. a degree or time duration of the facial expression. In other cases, the target expression data is prestored at a storage device accessible by the system, and the system can obtain the target expression data from the storage device.

[0096] The system receives a driving video frame and processes, using an encoder subsystem included in the video synthesis system, the driving video frame to generate driving appearance parameters and driving expression parameters (step 704). The driving video frame depicts a human face having an initial expression. The initial expression is generally different from the target expression. For example, the initial expression is sad, and the target expression is happy, and so on.

[0097] The system determines modified driving expression parameters by using an expression/emotion translation model included in the video synthesis system (step 706). The emotion translation model can generate the modified driving expression parameters based on modifying the driving expression parameters based on the target expression specified by the target expression data.

[0098] The system processes, using image generation subsystem included in the video synthesis system, the driving appearance parameters and the modified driving expression parameters to generate the output video frame (step 708). The output video frame depicts the same human face as the driving video frame but having the target expression.

[0099] FIG. 8 is a flowchart of an example process 800 for training a video synthesis system. For convenience, the process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system, e.g.. the video synthesis system 100 of FIG. 1 or another training system, appropriately programmed in accordance with this specification, can perform the process 800. The process 800 may be used to train any of the above described video synthesis systems such as the system 100 of FIG. 1.

[00100] The system can repeatedly perform iterations of the process 800 on each of one or more temporal sequences of training video frames to update the trainable parameters of the video synthesis system. For example, the parameters of an encoder subsystem which may include an encoder neural network and/or an appearance feature extractor network(s), the parameters of a feature warping subsystem which may include a motion field estimator, the parameters of an image generation subsystem which may include a generator neural network, and/or other trainable components of the video synthesis system where appropriate. Each temporal sequence of training video frames includes multiple video frames associated with different time points that each depict the same human face. [00101] The system can continue performing iterations of the process 800 until termination criteria for the training of the video synthesis system have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 800 have been performed.

[00102] The system samples, as a training source video frame, a first video frame from a temporal sequence of training video frames, and samples, as a training driving video frame, a second video frame from the temporal sequence of training video frames (step 802). For example, the first and second video frames can be video frames associated with two different time points within the temporal sequence of training video frames.

[00103] The system processes the training source video frame and the training driving video frame using the video synthesis system in accordance with current values of the parameters of the video synthesis system to generate a training output video frame (step 804). This may be carried out as described above with respect to generating an output video frame from source and driving video frames. In some implementations, to improve the robustness of the video synthesis system against the expression parameters the exact numbers of which may vary during inference, the system randomly drops out source expression parameters or driving expression parameters during training. For example, assuming the source expression parameters can in the form of a 50 dimensional vector, the system can do so by randomly sampling k E [0, 50] and dropping the last k dimensions in the 50 dimensional vector before providing it as input to the generator neural network.

[00104] The system evaluates a loss function (step 806). The loss function can include a perceptual loss term evaluated between the training output video frame and the training driving video frame. The perceptual loss term trains the video synthesis system to generate training output video frames that are more similar to, e.g.. closer in pixel distance to. the training driving video frame. As a particular example, the perceptual loss term can be determined using one of the loss functions mentioned in Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.

[00105] The loss function can also include an adversarial loss term evaluated with respect to the training output video frame. For example, a multi-resolution patch discriminator neural network can be used to determine the adversarial loss term. The adversarial loss term trains the video synthesis system to generate training output video frames that are more similar to, e.g., closer in pixel distance to, the training driving video frame, and in particular, to generate training output video frames that are indistinguishable by the multi-resolution patch discriminator neural network (or another discriminator neural network) from the training driving video frames when processing pairs of training output video frames and training driving video frames.

[00106] The system determines, from the loss function, one or more updates to the current values of the parameters of the encoder neural network, the appearance feature extractor network(s), the motion field estimator, and the generator neural network of the video synthesis system (step 808). The system can do this by computing respective gradients of the loss function with respect to the parameters of the video synthesis system by backpropagation through the appropriate parameters. The system can then determine the updates by applying an update rule, e.g., an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule, to the respective gradients.

[00107] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[00108] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. [00109] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[00110] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[00111] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[00112] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[00113] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [00114] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [00115] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory', media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[00116] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by⁷ sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[00117] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[00118] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow or J AX framework.

[00119] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[00120] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g.. for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[00121] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[00122] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[00123] This specification also provides the subject-matter of the following clauses:

Clause 1. A video synthesis system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an encoder subsystem configured to: process a source video frame to generate source appearance parameters, source expression parameters, and source pose parameters; and process a driving video frame to generate driving appearance parameters, driving expression parameters, and driving pose parameters, a three-dimensional (3D) face modeling subsystem configured to: process the source appearance parameters, the source expression parameters, and the source pose parameters to generate a plurality of 3D source frame keypoints; and process the driving appearance parameters, the driving expression parameters, and the driving pose parameters to generate a plurality of 3D driving frame key points, a feature warping subsystem configured to: process the source appearance parameters, the plurality of 3D source frame keypoints, and the plurality of 3D driving frame keypoints to generate a scene flow estimation between the source video frame and the driving video frame; and determine warped source appearance parameters by warping the source appearance parameters in accordance with the scene flow estimation, and an image generation subsystem configured to: process the warped source appearance parameters and the driving expression parameters to generate an output video frame.

Clause 2. The video synthesis system of clause 1, wherein the encoder subsystem comprises an encoder neural network that comprises a plurality of convolutional layers followed by a fully connected layer which outputs the source expression parameters and the source pose parameters.

Clause 3. The video synthesis system of any one of clauses 1-2, wherein the encoder subsystem comprises an appearance feature extractor network comprising one or more downsampling blocks followed by one or more convolutional layers followed by one or more residual blocks which output the source appearance parameters.

Clause 4. The video synthesis system of any one of clauses 1-3, wherein the 3D face modeling subsystem comprises a statistical face model configured to compute, in accordance with the source appearance parameters, the source expression parameters, and the source pose parameters, a 3D mesh with a predetermined number of vertices that define the plurality of 3D source frame keypoints.

Clause 5. The video synthesis system of any one of clauses 1-4, wherein the feature warping subsystem comprises a motion field estimator configured to: compute a respective estimated flow from each 3D source frame keypoint to a corresponding 3D driving frame keypoint; generate a 3D flow composition mask based on the respective estimated flows and the source appearance parameters; and generate the scene flow estimation based on the 3D flow composition mask and the respective estimated flows.

Clause 6. The video synthesis system of clause 5, wherein computing the respective estimated flow comprises using zeroth order approximation. Clause 7. The video synthesis system of any one of clauses 1-6, wherein the image generation subsystem comprises a generator neural network that comprises a plurality of spati ally-adaptive normalization blocks that are each configured to: process a combination of the warped source appearance parameters and the driving expression parameters using one or more convolutional layers to generate scaling modulation parameters and bias modulation parameters; process the warped source appearance parameters using a normalization layer to generate normalized warped source appearance parameters; and generate a block output based on (i) determining an elementwise product between the scaling modulation parameters and the normalized warped source appearance parameters and (ii) determining an elementwise summation between the bias modulation parameters and the elementwise product.

Clause 8. The video synthesis system of any one of clauses 1-7. wherein the driving appearance parameters are the same as the source appearance parameters.

Clause 9. One or more computer storage media storing instructions that when executed by one or more computer cause the one or more computer to implement the video synthesis system of any one of clauses 1-8.

Clause 10. A method comprising the operations that the video synthesis system of any one of clauses 1-8 is configured to perform.

Clause 11. A method of generating a temporal sequence of output video frames that each depict a first human face having a particular motion relative to a prior output video frame using the video synthesis system of any one of the preceding clauses, wherein the method comprises: processing, using the video synthesis system, (i) a source video frame which depicts the first human face and (ii) a driving video frame in a temporal sequence of driving video frames which depict a second human face having the particular motion to generate an output video frame in the temporal sequence of output video frames. Clause 12. A method of generating a temporal sequence of output video frames that depict a human face using the video synthesis system of any one of the preceding clauses, wherein the method comprises: initiating a communication session between a first user device and a second user device, wherein initiating the communication session comprises: obtaining, at the first user device, a source video frame which depicts the human face of a user of the first user device; processing, at the first user device, the source video frame to generate source appearance parameters, source expression parameters, source pose parameters, and a plurality of 3D source frame keypoints; and transmitting initiation data comprising the source appearance parameters, the source expression parameters, the source pose parameters, and the plurality of 3D source frame keypoints from the first user device to the second user device; and during the communication session, repeatedly: obtaining, at the first user device, a driving video frame which depicts the human face of the user of the first user device; processing, at the first user device, the driving video frame to generate driving expression parameters, driving pose parameters, and a plurality of 3D driving frame key points; transmitting live data comprising the driving expression parameters, the driving pose parameters, and the plurality of 3D driving frame keypoints from the first user device to the second user device so that the second user device is operable to generate an output video frame in the temporal sequence of output video frames from the initiation data and the live data.

Clause 13. A method generating an output video frame which depicts a human face having a target expression using the video synthesis system of any one of the preceding clauses, wherein the method comprises: processing, using the encoder subsystem, a driving video frame which depicts the human face having an initial expression to generate driving appearance parameters and driving expression parameters; determining modified driving expression parameters by using an emotion translation model in accordance with the target expression; and processing, using image generation subsystem, the driving appearance parameters and the modified driving expression parameters to generate the output video frame.

Clause 14. The method of clause 13, further comprising receiving target expression data which specifies the target expression.

Clause 15. A method of training a video synthesis system, wherein the training comprises: sampling a training source video frame and a training driving video frame from a temporal sequence of training video frames; processing the training source video frame and the training driving video frame using the video synthesis system to generate a training output video frame; evaluating a loss function that includes (i) a perceptual loss term evaluated between the training output video frame and the training driving video frame and (ii) an adversarial loss term evaluated with respect to the training output video frame; and determining, from the loss function, one or more updates to respective parameters of a encoder neural network, an appearance feature extractor network, a motion field estimator, and a generator neural network of the video synthesis system.

Clause 16. The method of clause 15, wherein processing the training source video frame and the training driving video frame using the video synthesis system to generate the training output video frame comprises: randomly dropping out source expression parameters or driving expression parameters.

Clause 17. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations of the respective method of any one of clauses 1 1-16.

Clause 18. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform operations of the respective method of any one of clauses 11-16.

[00124] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[00125] What is claimed is:

Claims

1. A video synthesis system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an encoder subsystem configured to: process a source video frame to generate source appearance parameters, source expression parameters, and source pose parameters; and process a driving video frame to generate driving appearance parameters, driving expression parameters, and driving pose parameters, a three-dimensional (3D) face modeling subsystem configured to: process the source appearance parameters, the source expression parameters, and the source pose parameters to generate a plurality of 3D source frame keypoints; and process the driving appearance parameters, the driving expression parameters, and the driving pose parameters to generate a plurality of 3D driving frame keypoints, a feature warping subsystem configured to: process the source appearance parameters, the plurality of 3D source frame keypoints, and the plurality of 3D driving frame keypoints to generate a scene flow estimation between the source video frame and the driving video frame; and determine warped source appearance parameters by warping the source appearance parameters in accordance with the scene flow estimation, and an image generation subsystem configured to: process the warped source appearance parameters and the driving expression parameters to generate an output video frame.

2. The video synthesis system of claim 1, wherein the encoder subsystem comprises an encoder neural network that comprises a plurality of convolutional layers followed by a fully connected layer which outputs the source expression parameters and the source pose parameters.

3. The video synthesis system of any one of claims 1-2, wherein the encoder subsystem comprises an appearance feature extractor neural network comprising one or more downsampling blocks followed by one or more convolutional layers followed by one or more residual blocks which output the source appearance parameters.

4. The video synthesis system of any one of claims 1-3, wherein the 3D face modeling subsystem comprises a statistical face model configured to compute, in accordance with the source appearance parameters, the source expression parameters, and the source pose parameters, a 3D mesh with a predetermined number of vertices that define the plurality of 3D source frame keypoints.

5. The video synthesis system of any one of claims 1-4, wherein the feature warping subsystem comprises a motion field estimator configured to: compute a respective estimated flow from each 3D source frame keypoint to a corresponding 3D driving frame keypoint; generate a 3D flow composition mask based on the respective estimated flows and the source appearance parameters; and generate the scene flow estimation based on the 3D flow composition mask and the respective estimated flows.

6. The video synthesis system of claim 5, wherein computing the respective estimated flow comprises using zeroth order approximation.

7. The video synthesis system of any one of claims 1-6, wherein the image generation subsystem comprises a generator neural network that comprises a plurality of spatially- adaptive normalization blocks that are each configured to: process a combination of the warped source appearance parameters and the driving expression parameters using one or more convolutional layers to generate scaling modulation parameters and bias modulation parameters; process the warped source appearance parameters using a normalization layer to generate normalized warped source appearance parameters; and generate a block output based on (i) determining an elementwise product between the scaling modulation parameters and the normalized warped source appearance parameters and (ii) determining an elementwise summation between the bias modulation parameters and the elementwise product.

8. The video synthesis system of any one of claims 1-7. wherein the driving appearance parameters are the same as the source appearance parameters.

9. One or more computer storage media storing instructions that when executed by one or more computer cause the one or more computer to implement the video synthesis system of any one of claims 1-8.

10. A method comprising the operations that the video synthesis system of any one of claims 1-8 is configured to perform.

11. A method of generating a temporal sequence of output video frames that each depict a first human face having a particular motion relative to a prior output video frame using the video synthesis system of any one of the preceding claims, wherein the method comprises: processing, using the video synthesis system, (i) a source video frame which depicts the first human face and (ii) a driving video frame in a temporal sequence of driving video frames which depict a second human face having the particular motion to generate an output video frame in the temporal sequence of output video frames.

12. A method of generating a temporal sequence of output video frames that depict a human face using the video synthesis system of any one of the preceding claims, wherein the method comprises: initiating a communication session between a first user device and a second user device, wherein initiating the communication session comprises: obtaining, at the first user device, a source video frame which depicts the human face of a user of the first user device; processing, at the first user device, the source video frame to generate source appearance parameters, source expression parameters, source pose parameters, and a plurality of 3D source frame keypoints; and transmitting initiation data comprising the source appearance parameters, the source expression parameters, the source pose parameters, and the plurality of 3D source frame keypoints from the first user device to the second user device; and during the communication session, repeatedly: obtaining, at the first user device, a driving video frame which depicts the human face of the user of the first user device; processing, at the first user device, the driving video frame to generate driving expression parameters, driving pose parameters, and a plurality of 3D driving frame keypoints; transmitting live data comprising the driving expression parameters, the driving pose parameters, and the plurality of 3D driving frame keypoints from the first user device to the second user device so that the second user device is operable to generate an output video frame in the temporal sequence of output video frames from the initiation data and the live data.

13. A method generating an output video frame which depicts a human face having a target expression using the video synthesis system of any one of the preceding claims, wherein the method comprises: processing, using the encoder subsystem, a driving video frame which depicts the human face having an initial expression to generate driving appearance parameters and driving expression parameters; determining modified driving expression parameters by using an expression translation model in accordance with the target expression; and processing, using image generation subsystem, the driving appearance parameters and the modified driving expression parameters to generate the output video frame.

14. The method of claim 13. further comprising obtaining target expression data which specifies the target expression.

15. A method of training a video synthesis system, wherein the training comprises: sampling a training source video frame and a training driving video frame from a temporal sequence of training video frames; processing the training source video frame and the training driving video frame using the video synthesis system to generate a training output video frame; evaluating a loss function that includes (i) a perceptual loss term evaluated between the training output video frame and the training driving video frame and (ii) an adversarial loss term evaluated with respect to the training output video frame; and determining, from the loss function, one or more updates to respective parameters of an encoder neural network, an appearance feature extractor neural network, a motion field estimator, and a generator neural network of the video synthesis system.

16. The method of claim 15, wherein processing the training source video frame and the training driving video frame using the video synthesis system to generate the training output video frame comprises: randomly dropping out source expression parameters or driving expression parameters.

17. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations of the respective method of any one of claims 11- 16.

18. A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform operations of the respective method of any one of claims 11-16.