WO2025200078A1

WO2025200078A1 - Face tracking based on spatial-temporal aggregation and rigid prior

Info

Publication number: WO2025200078A1
Application number: PCT/CN2024/090761
Authority: WO
Inventors: Ming Lu; Anbang YAO; Shandong WANG; Yangyuxuan KANG; Yurong Chen
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 2024-03-26
Filing date: 2024-04-30
Publication date: 2025-10-02
Anticipated expiration: 2026-09-26

Abstract

A computer vision system may extract feature maps from various frames in a video, e.g., a monocular video. The frames may include a present frame and one or more historical frames. The computer vision system may aggregate a feature map extracted from the present frame ( "present feature map" ) with a feature map extracted from each historical frame ( "historical feature map" ). The computer vision system may use a decoder to determine blendshape coefficients from the aggregated feature map. The blendshape coefficients may represent the predicted facial expression. Additionally, the computer vision system may use another decoder to determine pose coefficients from the aggregated feature map, e.g., after imposing a semantic-based rigid prior module to the aggregated feature map. The pose coefficients may represent the predicted head pose. The computer vision system may generate a face animation from the blendshape coefficients and pose coefficients.

Description

FACE TRACKING BASED ON SPATIAL-TEMPORAL AGGREGATION AND RIGID PRIOR

Cross-Reference to Related Application

This application claims the benefit of and hereby incorporates by reference, for all purposes, the entirety of the contents of International Application No. PCT/CN2024/083753, filed March 26, 2024, and entitled “FACE TRACKING BASED ON SPATIAL-TEMPORAL AGGREGATION AND SEMANTIC-AWARE RIGID PRIOR. ”

Technical Field

This disclosure relates generally to computer vision, and more specifically, face tracking based on spatial-temporal aggregation and rigid prior.

Background

The last decade has witnessed a rapid rise in AI (artificial intelligence) based data processing, particularly based on deep neural networks (DNNs) , such as convolutional neural networks (CNNs) , and so on. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. Computer vision tasks include methods for acquiring, processing, analyzing, or understanding visual images or videos to produce information, such as face tracking (tracking face expression, tracking face motion, etc. ) , motion detection, and so on.

Brief Description of the Drawings

Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

FIG. 1 illustrates an example computer vision system, in accordance with various embodiments.

FIG. 2 illustrates memory-based aggregation of spatial-temporal features, in accordance with various embodiments.

FIG. 3 illustrates an example read controller, in accordance with various embodiments.

FIG. 4 illustrates face tracking using a blendshape decoder and a pose decoder, in accordance with various embodiments.

FIG. 5 illustrates an example decoder module, in accordance with various embodiments.

FIG. 6 illustrates an example rigid prior module, in accordance with various embodiments.

FIG. 7 illustrates an example convolution, in accordance with various embodiments.

FIG. 8 illustrates an example DNN, in accordance with various embodiments.

FIG. 9 illustrates an AI-based visual recognition environment, in accordance with various embodiments.

FIG. 10 is a flowchart showing a method of face tracking, in accordance with various embodiments.

FIG. 11 is a block diagram of an example computing device, in accordance with various embodiments.

Detailed Description

Many currently available methods of three-dimensional (3D) face tracking from monocular videos lack sufficient accuracy and robustness due to depth ambiguity and self-occlusion of the monocular input. Some face tracking methods use DNNs that directly predict 3D face animation parameters from monocular videos. Such methods typically use a two-dimensional (2D) face detector to crop the face region in each video frame, then train a 3D face tracking network to regress the parameters in the frame-by-frame manner, and finally use Kalman filter to smooth the parameters over all the video frames. These DNN-based method suffer from limitations. For example, they require usage of Kalman filter as the post processing step to achieve temporally-consistent 3D face tracking results, which results in additional computational cost during inference. As another example, they usually lack rigid stabilization, which is important in monocular 3D face tracking due to the depth ambiguity of the monocular input.

Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing a method of face tracking (e.g., facial expression tracking, head pose tracking, etc. ) based on spatial-temporal aggregation and semantic-aware rigid prior. For instance, spatial-temporal features extracted from multiple frames may be aggregated for predicting a facial expression (e.g., smiling, frowning, eye rolling, appearing bored, appearing interested, etc. ) . By fusing the multi-frame features, the tracking accuracy can be improved. Also, as Kalman filter is not needed, the tracking efficiency can be improved by reducing the computational cost during inference. The aggregated spatial-temporal features can also be used in combination with a semantic-aware rigid prior to predict a head pose. The semantic-aware rigid prior can enable rigid stabilization and reduce or even eliminate depth shifting during face tracking and therefore, improve the accuracy of head pose prediction.

In various embodiments of the present disclosure, a computer vision system may extract feature maps from various frames in a video, e.g., a monocular video. The frames may include a present frame and one or more historical frames. The present frame may be the frame that has the most recent time stamp, while each historical frame has an earlier time stamp and is temporally precedent to the present frame in the video. The computer vision system may aggregate a feature map extracted from the present frame ( “present feature map” ) with a feature map extracted from each historical frame ( “historical feature map” ) . In an example, the computer vision system may use one or more transformer blocks to generate an intermediate feature map from the present feature map and the historical feature map (s) . The output of the transformer block (s) may be further concatenated with the feature map to generate an aggregated feature map. A transformer block may include a transformer encoder or transformer decoder.

The computer vision system may use a decoder to determine blendshape coefficients from the aggregated feature map. The blendshape coefficients may represent the predicted facial expression. Blendshapes may be models (e.g., 3D models) of facial expressions. For instance, a blendshape may be approximate semantic parameterization of a facial expression. Additionally, the computer vision system may generate a rigid transformation from the aggregated feature map by imposing a semantic-based rigid prior module to the aggregated feature map. The computer vision system may use the rigid transformation to predict a head pose, e.g., by using another decoder. The other decoder may output pose coefficients, e.g., rotation parameters and translation parameters. The face in the video may be reconstructed using the blendshape coefficients and pose coefficients. For instance, a 3D face animation may be generated.

For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or/and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

For the purposes of the present disclosure, the phrase “A or B” or the phrase "A and/or B" means (A) , (B) , or (A and B) . For the purposes of the present disclosure, the phrase “A, B, or C” or the phrase "A, B, and/or C" means (A) , (B) , (C) , (A and B) , (A and C) , (B and C) , or (A, B, and C) . The term "between, " when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

The description uses the phrases "in an embodiment" or "in embodiments, " which may each refer to one or more of the same or different embodiments. The terms "comprising, " "including, " "having, " and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above, " "below, " "top, " "bottom, " and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives “first, ” “second, ” and “third, ” etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

In the following detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

The terms “substantially, ” “close, ” “approximately, ” “near, ” and “about, ” generally refer to being within +/-20%of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., “coplanar, ” “perpendicular, ” “orthogonal, ” “parallel, ” or any other angle between the elements, generally refer to being within +/-5-20%of a target value as described herein or as known in the art.

In addition, the terms “comprise, ” “comprising, ” “include, ” “including, ” “have, ” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term “or” refers to an inclusive “or” and not to an exclusive “or. ”

The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

FIG. 1 illustrates an example computer vision system 100, in accordance with various embodiments. The computer vision system 100 includes a face tracking module 110, a training module 130, a validating module 140, and a datastore 150. In other embodiments, alternative configurations, different or additional components may be included in the computer vision system 100. Further, functionality attributed to a component of the computer vision system 100 may be accomplished by a different component included in the computer vision system 100 or by a different system.

The face tracking module 110 tracks faces in videos. The face tracking module 110 includes a DNN 160, a feature aggregation module 170, a blendshape module 180, and a pose module 190. The DNN 160 may extract features from video frames. In some embodiments, the DNN 160 may extract a feature map from a frame. The DNN 160 may operate in an online mode. For instance, a sequence of frames in a video may be input into the DNN 160 sequentially and the DNN 160 may generate a feature map from every frame as it receives the frame. The order in which the DNN 160 processes the frames may correspond to the order in which the frames are arranged in the video. The DNN 160 may include a plurality of layers, such as an input layer, one or more hidden layers, and an output layer. One or more layers in the DNN 160 may be convolutional layers.

A feature map generated by the DNN 160 may be a 2D or 3D tensor. A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (1D) tensor) , matrix (which is 2D tensor) , 3D tensors, four-dimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. Taking a convolutional layer for example, the input tensors include an activation tensor (also referred to as “input feature map (IFM) ” or “input activation tensor” ) including one or more activations (also referred to as “input elements” ) and a weight tensor. The weight tensor may be a kernel (a2D weight tensor) , a filter (a3D weight tensor) , or a group of filters (a4D weight tensor) . A convolution may be performed on the input activation tensor and weight tensor to compute an output activation tensor in the convolutional layer.

The feature aggregation module 170 receives features generated by the DNN 160 and aggregates features. In some embodiments, the feature aggregation module 170 may generate an aggregated feature map for each of a plurality of frames. To generate the aggregate feature map of a frame, the feature aggregation module 170 aggregates the feature map extracted from the frame by the DNN 160 with one or more other feature maps extracted from one or more other frames by the DNN 160. The frame may be referred to as the present frame or current frame. The feature map extracted from the frame may be referred to as the present feature map or current feature map. The one or more other frames may be temporally precedent to the frame and may be referred to as historical frame (s) or previous frame (s) . The one or more other feature maps extracted from one or more other frames may be referred to as historical feature map (s) or previous feature map (s) . The feature aggregation module 170 may aggregate the current feature map with the previous feature map (s) by using one or more transformer decoders. The output of the one or more transformer decoders may be further accumulated with the current feature map to generate the aggregated feature map. More details regarding the feature aggregation module 170 are provided below in conjunction with FIG. 2.

The blendshape module 180 predicts face expressions from aggregated feature maps generated by the feature aggregation module 170. In some embodiments, face expressions are represented using blendshapes. A blendshape may represent a face expression, such as smiling, frowning, and so on. In some embodiments, a blendshape may be a 3D representation (e.g., a 3D model) showing a face expression. In some embodiments, the blendshape module 180 may include one or more transformer decoders that can generate one or more blendshape parameters from an aggregated feature map. The one or more blendshape parameters may encode or may be used to generate the predicted blendshape, which represents the prediction of the facial expression shown in the current frame. The one or more blendshape parameters may be morphable models.

The pose module 190 predicts poses from aggregated feature maps generated by the feature aggregation module 170. In some embodiments, the pose module 190 may include one or more transformer decoders that can generate one or more pose parameters from an aggregated feature map. The one or more pose parameters may encode the prediction of the head pose shown in the current frame. The one or more pose parameters may include one or more translation parameters (e.g., global translation parameters) or one or more rotation parameters (e.g., global rotation parameters) . Translation parameter may encode predicted translation of the head. Rotation parameter may encode predicted rotation of the head.

In some embodiments, the pose module 190 includes a semantic-aware rigid prior that can enhance the accuracy of the pose prediction. The semantic-aware rigid prior may use a DNN to compute a reweighting mask from a segmentation probability map of the face. The aggregated feature map may be multiplied with the reweighting mask before being processed by the transformer decoder (s) in the pose module 190. By using the semantic-aware rigid prior, rigid regions of the face (e.g., forehead, nose, etc. ) may be enhanced, which can reduce or even eliminate depth drifting of the rigid regions and facilitate a more accurate prediction of rigid motion.

The face tracking module 110 can aggregate spatial-temporal features for accurate and robust 3D face tracking, instead of performing frame-by-frame prediction. Since rigid stabilization can be important in the task of real-time monocular face tracking given the presence of head pose ambiguity due to the depth ambiguity, the semantic-aware rigid prior introduced before the transformer decoders can enable rigid stabilization during 3D face tracking and improve the tracking accuracy of head pose. Compared with currently available face tracking methods, the face tracking module 110 can provide more accurate and efficient 3D face tracking.

The animation generator 120 generates face animations based on face tracking parameters (e.g., blendshape parameters and pose parameters) generated by the face tracking module 110. In some embodiments, the animation generator 120 may modify one or more blendshape templates of a face with the face tracking parameters determined by the face tracking module 110. A blendshape template may represent a specific face expression. The specific face expression may be a reference face expression. In an example, the reference face expression may be a neutral face expression, like a face expression that indicate no strong emotion. The animation generator 120 may generate a new blendshape representing the predicted face expression based on the blendshape parameters and one or more the blendshape templates. The animation generator 120 may also modify the pose of the blendshape based on the pose parameters. In some embodiments, the animation generator 120 generates a sequence of face animations for a sequence of frames processed by the face tracking module 110. Face animations can be used in various applications, such as digital human, video conference, movie, gaming, and so on.

The training module 130 trains the face tracking module 110, such as the DNNs in the face tracking module 110. The DNN in the face tracking module 110 may include the DNN 160, transformer decoders in the feature aggregation module 170, transformer decoders in the blendshape module 180, transformer decoders in the pose module 190, CNN in the pose module 190, and so on. The training module 130 may modify internal parameters of the face tracking module 110 based on the ground-truth labels of the training samples and the outputs of the face tracking module 110 that are generated by processing the training samples. Training samples may include video frames. The ground-truth labels of training sample may include verified or known face tracking parameters that encode face expressions and poses shown in the video frames. A training sample may include a sequence of frames. For each training sample, the training module 130 may sample a batch of consecutive N_F+1 frames from the training dataset. The training module 130 may also extract and store the feature of the first N_F frames and compute the loss based on the prediction of the last frame.

In some embodiments, the training module 130 modifies the internal parameters of the face tracking module 110 to minimize the error between labels of the training samples that are generated by the face tracking module 110 and the ground-truth labels. In some embodiments, the training module 130 uses a cost function or loss function to minimize the error. For instance, the training module 130 may use the Mean Squared Error (MSE) loss across blendshape parameters, rotation parameters, and translation parameters. The MSE loss may be denoted as:

wheredenotes the predicted blendshape parameters, β denotes the ground-truth blendshape parameters, denotes the predicted rotation parameters, denotes the ground-truth rotation parameters, denotes the predicted translation parameters, and t denotes the ground-truth translation parameters, andstands for loss.

In some embodiments, the training module 130 may also form validation datasets for validating performance of the face tracking module 110 after training by the validating module 140. A validation dataset may include validation samples and ground-truth labels of the validation samples. The validation dataset may include different samples from the training dataset used for training the face tracking module 110. In an embodiment, a part of a training dataset may be used to initially train the face tracking module 110, and the rest of the training dataset may be held back as a validation subset used by the validating module 140 to validate performance of the face tracking module 110. The portion of the training dataset not including the validation subset may be used to train the face tracking module 110.

The training module 130 also determines hyperparameters for training the face tracking module 110. Hyperparameters are variables specifying the training process. Hyperparameters are different from parameters inside the face tracking module 110 (e.g., weights of convolutional layers, weights in transformer decoders, etc. ) . In some embodiments, hyperparameters include variables determining the architecture of at least part of the face tracking module 110, such as number of hidden layers in the DNN 160, and so on. Hyperparameters also include variables which determine how the face tracking module 110 is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the face tracking module 110. The batch size is the same as or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the internal parameters of the face tracking module 110. An epoch may include one or more batches. The number of epochs may be 15, 150, 500, 1500, or even larger.

The training module 130 may define the architecture of the face tracking module 110 (or part of the face tracking module 110, e.g., a DNN in the face tracking module 110) , e.g., based on some of the hyperparameters. The architecture of the DNN may include an input layer, an output layer, and a plurality of hidden layers. The input layer may include tensors (e.g., a multi-dimensional array) specifying attributes of the IFM, such as the height of the IFM, the width of the IFM, and the depth of the IFM (e.g., the number of channels in the IFM) . The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, layer normalization layers, self attention layers, cross attention layers, Softmax or logistic layers, and so on.

The training module 130 may train the face tracking module 110 for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm will work through the entire training dataset. One epoch means that each sample in the training dataset has had an opportunity to update the internal parameters of the face tracking module 110. After the training module 130 finishes the predetermined number of epochs, the training module 130 may stop updating the internal parameters of the face tracking module 110, and the face tracking module 110 is considered trained.

The validating module 140 verifies accuracy of the face tracking module 110 after the face tracking module 110 is trained. In some embodiments, the validating module 140 inputs samples in a validation dataset into the face tracking module 110 and uses the outputs of the face tracking module 110 to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the validating module 140 determines may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The validating module 140 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN) , where precision may be how many the face tracking module 110 correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives) , and recall may be how many the face tracking module 110 correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives) . The F-score (F-score = 2 *PR / (P + R) ) unifies precision and recall into a single measure.

The validating module 140 may compare the accuracy score with a threshold score. In an example where the validating module 140 determines that the accuracy score is lower than the threshold score, the validating module 140 instructs the training module 130 to re-train the face tracking module 110. In one embodiment, the training module 130 may iteratively re-train the face tracking module 110 until the occurrence of a stopping condition, such as the accuracy measurement indication that the face tracking module 110 may be sufficiently accurate, or a number of training rounds having taken place.

The datastore 150 stores data received, generated, used, or otherwise associated with the computer vision system 100. For example, the datastore 150 stores the datasets used by the training module 130 and validating module 140. The datastore 150 may also store data generated by the training module 130 and validating module 140, such as the hyperparameters for training the face tracking module 110, internal parameters of the face tracking module 110, and so on. As another example, the datastore 150 may store video frames to be processed by the face tracking module 110 for performing face tracking tasks. The datastore 150 may also store outputs of the face tracking module 110 or components of the face tracking module 110, such as feature maps, blendshapes parameters, pose parameters, segmentation probability maps, and so on. In the embodiment of FIG. 1, the datastore 150 is a component of the computer vision system 100. In other embodiments, the datastore 150 may be external to the computer vision system 100 and communicate with the computer vision system 100 through a network.

FIG. 2 illustrates memory-based aggregation of spatial-temporal features, in accordance with various embodiments. The memory-based aggregation of spatial-temporal features in FIG. 2 is performed by a feature aggregation module 210, which receives outputs of a backbone network 220. The backbone network 220 may be an example of the DNN 160 in FIG. 1. As shown in FIG. 2, the backbone network 220 receives a video 201 that includes a sequence of frames 202 (individually referred to as “frame 202” ) . The video 201 captures one or more faces, e.g., a face of a person or other types of objects. The video 201 may be a monocular video. The frames 202 may be arranged in a temporal sequence. A frame 202 that has an earlier time stamp may be arranged before a frame 202 that has a later time stamp. In some embodiments, the backbone network 220 and the feature aggregation module 210 may operate in an online mode. For instance, the backbone network 220 may receive each frame 202 as the frame 202 is being displayed. The latest frame 202 that is being received by the backbone network 220 may be the present frame, and the other frame (s) 202 arranged before the present frame are historical frame (s) .

Each frame 202 received by the backbone network 220 is an input to the backbone network 220. The backbone network 220 may be a DNN that includes a plurality of layers, which may include one or more convolutional layers. The backbone network 220 processes the frame 202 using the layers and generates a feature map. A feature map may be a 2D or 3D tensor that includes data points (e.g., activations) representing features extracted by the backbone network 220 from the corresponding frame 202. The feature map of each historical frame is referred to as a previous feature map 203H. The feature map of the present frame is referred to as a current feature map 203P. The historical feature map (s) 203H and the current feature map 203P are collectively referred to as “feature maps 203” or “feature map 203. ”

The feature aggregation module 210 receives feature maps 203 from the backbone network 220 and generate spatial-temporal aggregated features from the feature maps 203. The feature aggregation module 210 may be an example of the feature aggregation module 170 in FIG. 1. As shown in FIG. 2, the feature aggregation module 210 includes a write controller 230, a buffer 240, and a read controller 250. In other embodiments, alternative configurations, different or additional components may be included in the feature aggregation module 210. Further, functionality attributed to a component of the feature aggregation module 210 may be accomplished by a different component included in the feature aggregation module 210 or by a different module or system.

The write controller 230 may write feature maps 203 generated by the backbone network 220 into the buffer 240. In some embodiments, the write controller 230 may be implemented by a queue with a maximum length of N_F so that no more than N_F feature maps 203 would be preserved in the buffer 240 at a time. Storing a certain number of historical frames can facilitate effective and efficient face tracking because changes in expression and pose can exhibit smooth transitions in 3D face tracking. Frames 202 closer to the current time can contain more useful information, while frames 202 beyond a certain time window can be unlikely to contribute to the prediction of the current frame and may even introduce noise to the prediction. By using the queue that has a limited length, feature maps 203 beyond a specific time window can be removed from the buffer 240 while feature maps 203 within the time window are preserved and stored, which can ensure accuracy of the prediction and mitigate the accumulation of errors during the prediction process.

The read controller 250 reads feature maps 203 stored in the buffer 240. The read controller 250 may aggregate the current feature map 203C with one or more previous feature maps 203P. The aggregation may be at least part of a spatial-temporal aggregation. In some embodiments, the read controller 250 includes one or more transformer decoders that perform the aggregation. In an example, the transformer decoder (s) may take the current feature map 203P, which may be denoted asas queries and take the previous feature map (s) 203P, which may be denoted asas keys and values. The transformer decoder (s) may output an intermediate feature map.

The intermediate feature map is then added with the current feature map 203C by a concatenator 260 via a skip connection. The skip connection may be denoted as:
h_st=f_d (h_t, h_P) +h_t

where f_d denote the transformer decoder (s) , anddenotes a spatial-tempoeral aggregated feature map 204.

FIG. 3 illustrates an example read controller 300, in accordance with various embodiments. The read controller 300 aggregates features extracted from video frames having different time stamps. In an example, the read controller 300 may aggregate features extracted from a temporal sequence of frames from a monocular video for predicting face expression or head pose captured in one of the frames. The read controller 300 may be an example of the read controller 250 in FIG. 2.

As shown in FIG. 3, the read controller 300 includes transformer decoders 310, individually referred to as “transformer decoder 310. ” Each transformer decoder 310 includes a plurality of layers, such as a self attention layer 320, a layer norm layer 330, a cross attention layer 340, another layer norm layer 350, a feed forward layer 360, and another layer norm layer 370. In other embodiments, a transformer decoder 310 may include different, fewer, or more layers. The layer read controller 300 may include a different number of transformer decoders 310.

The self attention layer 320 or cross attention layer 340 may have an attention function. In some embodiments, an attention function may map one or more queries and one or more key-value pairs to an output, where the query, each key, each value, or the output may be a tensor, such as a vector. The output may be computed as a weighted sum of the values. The weight assigned to each value may be computed by a compatibility function of the query with the corresponding key. The layer norm layer 330 may connect all positions with a constant number of sequentially executed operations. Self attention, sometimes called intra-attention, may be an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. The self attention layer 320 may allow each position in the corresponding transformer decoder 310 to attend to all positions in the transformer decoder 310 up to and including that position.

As shown in FIG. 3, the self attention layer 320 receives a vector 302C. The vector 302C is generated by flattening a current feature map 301C, which may be generated from a current frame in the video. In some embodiments, the current feature map 301C may be a 2D or 3D tensor, while the vector 302C may be a 1D tensor. The data points in the current feature map 301C may be rearranged so that the data points are all in one dimension to generate the vector 302C. In an example, the current feature map 301C may have three dimensions: X, Y, and Z, in which the Z dimension may be the channel dimension. The data points of the current feature map 301C may be rearranged to be all in one of the X, Y, and Z dimensions to form the vector 302C. The vector 302C is input into the self attention layer 320 and may be used as the query, keys, and values of the self attention function. In some embodiments, the self attention function in the self attention layer 320 may be denoted as:

where Q, K, V denotes the query, keys, and values, respectively; f (Q, K, V) denotes the self attention function; and σ denotes a Softmax function.

The output of the self attention layer 320 is further accumulated with the vector 302C by a concatenator 325. The result of the concatenation is input into the layer norm layer 330. The layer norm layer 330 applies a layer normalization operation on the output of the concatenator 325. The layer norm layer 330 may normalize its input (i.e., the output of the concatenator 325) across the feature dimension (instead of the batch dimension) . The feature dimension may be the dimension of the vector 302C or the vector 302P.

In some embodiments, the layer normalization operation may include a sequence of computations. The layer norm layer 330 may include a mean computation , which may be denoted aswhere A_xyz denotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and μ_xy denotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer norm layer 330 may perform data to convert μ_xy to a 3D tensor μ_xyz, e.g., by replicating every data element over z output points.

The layer norm layer 330 may also include an elementwise subtraction, which may be denoted as D_xyz=A_xyz-μ_xyz. The layer norm layer 330 may also perform a variance computation, which may be denoted asThe layer norm layer 330 may further include a division computation denoted asM_xy may be a 2D tensor. The layer norm layer 330 may also convert M_xy to a 3D tensor M_xyz, e.g., by replicating every data element over z output points. Further, the layer norm layer 330 may have an element multiplication denoted as The layer norm layer 330 may further compute LN_xyz may be the output of the layer normalization operation.

The output of the layer norm layer 330, which encodes the flattened current feature, may be input into the cross attention layer 340. The cross attention layer 340 also receives a vector 302P as another input. The vector 302P is generated by flattening previous feature maps 301P, which may be generated from previous frames in the video. In some embodiments, a previous feature map 301P may be a 2D or 3D tensor, while the vector 302P may be a 1D tensor. The data points in the previous feature maps 301P may be rearranged so that the data points are all in one dimension to generate the vector 302P. In an example, each a previous feature map 301P may have three dimensions: X, Y, and Z, in which the Z dimension may be the channel dimension. The data points of the previous feature maps 301P may be rearranged to be all in one of the X, Y, and Z dimensions to form the vector 302P. The data points in the same previous feature map may be arranged together, e.g., in a manner that is not interrupted by any data points in a different previous feature map. In some embodiments, the vector 302P is in the same dimension as the vector 302C. The vector 302C and the vector 302P are each represented by a sequence of boxes in FIG. 3 for the purpose of illustration. The number of boxes in the vector 302C or the vector 302P does not represent the number of data points in the vector 302C or the vector 302P. In some embodiments, the number of data points in the vector 302C may equal the number of data points in the current feature map 301C, and the number of data points in the vector 302P may equal the total number of data points in all the previous feature maps 301P.

The cross attention layer 340 has a cross attention function that is applied on the vector 302P and the vector 302C. The vector 302P may be used as the query, and the vector 302C may be used as the keys and values. The cross attention function may be denoted as:

where σ is the Softmax function; denotes the vector 302C; denotes the vector 302P; and W^Q, W^K, are projection matrixes for query, keys, and values respectively.

The output of the cross attention layer 340 may be accumulated with the output of the layer norm layer 330 by another concatenator 345. The output of the concatenator 345 is input into the layer norm layer 350 for performing another layer normalization operation in the layer norm layer 350. The output of the layer norm layer 350 is input into the feed forward layer 360. In some embodiments, the feed forward layer 360 may include a linear transformation and a non-linear transformation. The linear transformation may be the same or similar to the operation in a fully-connected layer, e.g., a fully-connected layer 830 in FIG. 8. The non-linear transformation may be an activation function. In some embodiments, the feed forward layer 360 may multiple the input with keys to get a weight for each key. The feed forward layer 360 may compute a weight sum of the values using the weights to produce the output.

The output of the feed forward layer 360 may be accumulated with the output of the layer norm layer 350 by a concatenator 365. The output of the concatenator 365 may be input into the layer norm layer 370 in which another layer normalization operation is performed. The output of the layer norm layer 370 may be input into the next transformer decoder 310 for further processing. The output of the last transformer decoder 310 may constitute the output of the read controller 300, which may be an intermediate feature map that represents an aggregation of the current feature map 301C and the previous feature maps 301P.

FIG. 4 illustrates face tracking using a blendshape decoder 410 and a pose decoder 430 coupled with a rigid prior module 420, in accordance with various embodiments. The blendshape decoder 410 receives a feature map 401. The feature map 401 may be an aggregated feature map generated from a feature map extracted from a current frame and one or more feature maps extracted from one or more previous frames. In some embodiments, the feature map 401 is generated by the feature aggregation module 170 in FIG. 1.

The blendshape decoder 410 also receives a plurality of latent representations 402, individually referred to as latent representation 402. Each latent representation 402 may be an implicit blendshape latent representation. The latent representations 402 may represent various blendshapes corresponding to various face expressions. Each latent representation 402 may be a learnable latent tensor. A learnable latent tensor may be denoted as where N_E is the number of blendshapes and C_E is the channel number of the latent representations. Each latent tensor may represent or encode a blendshape after training. The blendshape decoder 410 uses the feature map 401 and the latent representations 402 to predict a face expression of the face shown in the current frame. In some embodiments, The blendshape decoder 410 may include one or more transformer decoders that processes the feature map 401 and the latent representations 402.

The blendshape decoder 410 generates blendshape parameters 403 (also referred to as “blendshape coefficients” ) , which may encode the predicted face expression. The blendshape parameters 403 may be denoted as:

where

The feature map 401 is also input into the rigid prior module 420. The rigid prior module 420 may reduce or even eliminate depth shifting in the feature map 401 for achieving better rigid stabilization. The rigid prior module 420 may use a segmentation map 404 of the face to generate a rigid transformation from the feature map 401. The rigid transformation is input into the pose decoder 430. The pose decoder 430 also receives a plurality of latent representations 405, individually referred to as latent representation 405. Each latent representation 405 may be an implicit pose latent representation. Each latent representation 405 may be a learnable latent tensor. The latent representation 405 may represent various poses. A latent representation 405 may be a latent tensor denoted as where N_P is the number of pose parameters and C_P is the channel number of the latent representations 405. In some embodiments, the pose decoder 430 may include one or more transformer decoders that processes the rigid transformation and the latent representations 405 to predict a head pose.

The pose decoder 430 may output pose parameters 406 (also referred to as “pose coefficients” ) that encode the predicted head pose. The pose parameters 406 may be denoted asIn some embodiments, the pose parameters may be decomposed into a six-degree-of-freedom vector, where rotation is parameterized by three Euler anglesand translation is represented by a vector

whereare predicted rotation parameters and predicted translation parameters, respectively.

The blendshape parameters 403 and the pose parameters 406 may be combined with one or more blendshape templates 407 to generate a 3D face animation 408. The 3D face animation 408 shows the predicted face expression and the predicted head pose. In some embodiments, a blendshape template may be denoted as where N_V is the number of vertices. Given a specific set of blendshape parameters the predicted blendshape may be obtained by:

Combining the predicted blendshape and the predicted pose, the 3D face animation 307 may be obtained by:
Ψ (β, T; B) =TF (β; B)

Where F is F in homogeneous coordinates.

FIG. 5 illustrates an example decoder module 500, in accordance with various embodiments. The decoder module 500 may be an example of the blendshape decoder 410 or the pose decoder 430 in FIG. 4. As shown in FIG. 5, the decoder module 500 includes transformer decoders 510, individually referred to as “transformer decoder 510. ” Each transformer decoder 510 includes a plurality of layers, such as a self attention layer 520, a layer norm layer 530, a cross attention layer 540, another layer norm layer 550, a feed forward layer 560, and another layer norm layer 570. In other embodiments, a transformer decoder 510 may include different, fewer, or more layers. The layer decoder module 500 may include a different number of transformer decoders 510. A transformer decoder 510 may be the same or similar as the transformer decoder 310 in FIG. 3.

As shown in FIG. 5, the self attention layer 520 receives a vector 502. The vector 502 is generated by flattening a feature map 501, which may be an aggregated feature map. An example of the feature map 501 may be the feature map 401 in FIG. 1. In some embodiments, the feature map 501 may be a 2D or 3D tensor, while the vector 502 may be a 1D tensor. The data points in the feature map 501 may be rearranged so that the data points are all in one dimension to generate the vector 502. In an example, the feature map 501 may have three dimensions: X, Y, and Z, in which the Z dimension may be the channel dimension. The data points of the feature map 501 may be rearranged to be all in one of the X, Y, and Z dimensions to form the vector 502. The vector 502 is input into the self attention layer 520 and may be used as the query, keys, and values of the self attention function. In some embodiments, the self attention function in the self attention layer 520 may be denoted as:

The output of the self attention layer 520 is further accumulated with the vector 502 by a concatenator 525. The result of the concatenation is input into the layer norm layer 530. The layer norm layer 530 applies a layer normalization operation on the output of the concatenator 525. The layer norm layer 530 may normalize its input (i.e., the output of the concatenator 525) across the feature dimension (instead of the batch dimension) . The feature dimension may be the dimension of the vector 502.

The output of the layer norm layer 530 may be input into the cross attention layer 540. The cross attention layer 540 also receives latent representations 504 (individually referred to as “latent representation 504” ) as another input. The latent representations 504 may be implicit blendshape latent representations, e.g., in embodiments where the decoder module 500 is used for predicting blendshapes. In other embodiments, (e.g., embodiments where the decoder module 500 is used for predicting poses) , the latent representations 504 may be implicit pose latent representations.

In some embodiments the cross attention function applied on the latent representations 503 and the output of the layer norm layer 530. In some embodiments, the cross attention layer 540 may use the latent representations 503 as queries and use the output of the layer norm layer 530, which may represent the aggregated spatial-temporal features, as keys and values. The cross attention function, which may be the same or similar as the cross attention function described above in conjunction with FIG. 3 but with different query, keys, and values.

The output of the cross attention layer 540 may be accumulated with the output of the layer norm layer 530 by another concatenator 545. The output of the concatenator 545 is input into the layer norm layer 550 for performing another layer normalization operation in the layer norm layer 550. The output of the layer norm layer 550 is input into the feed forward layer 560. The feed forward layer 560 may have the same or similar computations as the feed forward layer 360.

The output of the feed forward layer 560 may be accumulated with the output of the layer norm layer 550 by a concatenator 565. The output of the concatenator 565 may be input into the layer norm layer 570 in which another layer normalization operation is performed. The output of the layer norm layer 570 may be input into the next transformer decoder 510 for further processing. The output of the last transformer decoder 510 may constitute the output of the decoder module 500, which may be blendshape coefficients or pose coefficients.

FIG. 6 illustrates an example rigid prior module 600, in accordance with various embodiments. The rigid prior module 600 receives a segmentation map 602. The segmentation map 602 is generated from an image 601 that captures a head. The image 601 may be a frame in a video. The segmentation map 602 includes information that indicates segmentation of the head in the image 601. For instance, the segmentation map 602 may include a plurality of regions, each of which may correspond to a distinct portion of the head. In an example, the segmentation map 602 includes a region corresponding to the hair, a region corresponding to the face, a region corresponding to the nose, a region corresponding to the glasses, and so on. The segmentation map 602 may be a segmentation probability map of the current frame. In some embodiments, the segmentation map 602 may be generated by a face segmentation model, which segments a face into distinct regions. The segmentation map 602 may be an example of the segmentation map 404 in FIG. 4.

Different regions may have different attributes, such as different weights. In some embodiments, one or more regions in the segmentation map 602 may be identified or labeled as rigid region (s) . A rigid region may correspond to a portion of the head that has no or minimum distortion or change when the head pose changes. Examples of rigid regions include forehead, nose, and so on. The segmentation map 602 may be used by the rigid prior module 600 to improve rigid stabilization for pose prediction.

A DNN 610 in the rigid prior module 600 receives the segmentation map 602 as an input and outputs a reweighting mask 603. The DNN 610 may be a CNN, an example of which may be the DNN 800 in FIG. 8. The reweighting mask 603 may be denoted as whose spatial dimension is the same as a feature map 604. The feature map 604 may be an aggregation of a feature map extracted from the current frame and feature maps extracted from previous frames. In some embodiments, the feature map 604 is generated by the feature aggregation module 170 in FIG. 7.

A multiplier 620 in the rigid prior module 600 multiplies the reweighting mask 603 with the feature map 604 and computes a rigid transformation 605. In some embodiments, the rigid transformation 605 is a tensor that has the same spatial size or shape as the feature map 604. The values of the data points in the rigid transformation 605 may be different from those in the feature map 604. By multiplying the feature map 604 with the reweighting mask 603, the rigid regions can be enhanced. The rigid transformation 605 may be used to predict the pose of the head. In some embodiments, the rigid transformation 605 may be processed by a pose decoder, such as the pose decoder 430 to generate pose parameters.

FIG. 7 illustrates an example convolution, in accordance with various embodiments. The convolution may be a deep learning operation in a convolutional layer of a DNN, e.g., the DNN 160 in FIG. 1, the backbone network 220 in FIG. 2, the DNN 610 in FIG. 6, and so on. The convolution may extract features from an input. The convolution can be executed on an input tensor 710 and filters 720 (individually referred to as “filter 720” ) . The result of the convolution is an output tensor 730. In some embodiments, the convolution is performed by a DNN accelerator.

In the embodiments of FIG. 7, the input tensor 710 includes activations (also referred to as “input activations, ” “elements, ” or “input elements” ) arranged in a 3D matrix. An input element is a data point in the input tensor 710. The input tensor 710 has a spatial size H_in× W_in×C_in, where H_in is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel) , W_in is the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 3D matrix of each input channel) , and C_in is the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels) . For the purpose of simplicity and illustration, the input tensor 710 has a spatial size of 7×7×3, i.e., the input tensor 710 includes three input channels and each input channel has a 7×7 2D matrix. Each input element in the input tensor 710 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the input tensor 710 may be different.

Each filter 720 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A filter 720 has a spatial size H_f×W_f×C_f, where H_f is the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel) , W_f is the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel) , and C_f is the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels) . In some embodiments, C_f equals C_in. For purpose of simplicity and illustration, each filter 720 in FIG. 7 has a spatial size of 7×3×3, i.e., the filter 720 includes 7 convolutional kernels with a spatial size of 3×3. In other embodiments, the height, width, or depth of the filter 720 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 710.

An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

In the convolution, each filter 720 slides across the input tensor 710 and generates a 2D matrix for an output channel in the output tensor 730. In the embodiments of FIG. 7, the 2D matrix has a spatial size of 5×5. The output tensor 730 includes activations (also referred to as “output activations, ” “elements, ” or “output element” ) arranged in a 3D matrix. An output activation is a data point in the output tensor 730. The output tensor 730 has a spatial size H_out×W_out×C_out, where H_out is the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel) , W_out is the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each output channel) , and C_out is the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels) . C_out may equal the number of filters 720 in the convolution. H_out and W_out may depend on the heights and weights of the input tensor 710 and each filter 720.

As a part of the convolution, multiply-accumulate (MAC) operations can be performed on a 3×3×3 subtensor 715 (which is highlighted with a dotted pattern in FIG. 7) in the input tensor 710 and each filter 720. The result of the MAC operations on the subtensor 715 and one filter 720 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution) , an output activation may include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution) , an output activation may include more than one byte. For instance, an output element may include two bytes.

After the MAC operations on the subtensor 715 and all the filters 720 are finished, a vector 735 is produced. The vector 735 is highlighted with slashes in FIG. 7. The vector 735 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 735 have the same (X, Y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. The dimension of the vector 735 along the Z axis may equal the total number of output channels in the output tensor 730. After the vector 735 is produced, further MAC operations are performed to produce additional vectors till the output tensor 730 is produced.

In some embodiments, the output activations in the output tensor 730 may be further processed based on one or more activation functions before they are stored or inputted into the next layer of the CNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. In some embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, and so on. The results of the post processing may be stored in a local memory of the compute block and be used as input to the next layer. In some embodiments, the input activations in the input tensor 710 may be results of post processing of the previous layer. Even though the input tensor 710, filters 720, and output tensor 730 are 3D tensors in FIG. 7, the input tensor 710, a filter 720, or the output tensor 730 may be a 2D tensor in other embodiments.

FIG. 8 illustrates an example DNN 800, in accordance with various embodiments. The DNN 800 in FIG. 8 may be an example of the DNN 160 in FIG. 1, the backbone network 220 in FIG. 2, the DNN 610 in FIG. 6, and so on. The DNN 800 is trained to receive grid-structured data and output information indicating conditions of objects. In the embodiments of FIG. 8, the DNN 800 includes a sequence of layers comprising a plurality of convolutional layers 810 (individually referred to as “convolutional layer 810” ) , a plurality of pooling layers 820 (individually referred to as “pooling layer 820” ) , and a plurality of fully-connected layers 830 (individually referred to as “fully-connected layer 830” ) . In other embodiments, the DNN 800 may include fewer, more, or different layers. In an inference of the DNN 800, the layers of the DNN 800 execute tensor computation that includes many tensor operations, such as convolution (e.g., MAC operations, etc. ) , pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc. ) , other types of tensor operations, or some combination thereof.

The convolutional layers 810 summarize the presence of features in the input to the DNN 800. The convolutional layers 810 function as feature extractors. The first layer of the DNN 800 is a convolutional layer 810. In an example, a convolutional layer 810 performs a convolution on an input tensor 840 (also referred to as IFM 840) and a filter 850. As shown in FIG. 8, the IFM 840 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 840 includes 3 input channels, each of which is represented by a 7×7 two-dimensional (2D) matrix. The 7×7 2D matrix includes 7 input elements (also referred to as input points) in each row and seven input elements in each column. The filter 850 is represented by a 3×3×3 3D matrix. The filter 850 includes 3 kernels, each of which may correspond to a different input channel of the IFM 840. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 8, each kernel is represented by a 3×3 2D matrix. The 3×3 kernel includes 3 weights in each row and three weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 850 in extracting features from the IFM 840.

The convolution includes MAC operations with the input elements in the IFM 840 and the weights in the filter 850. The convolution may be a standard convolution 863 or a depthwise convolution 883. In the standard convolution 863, the whole filter 850 slides across the IFM 840. All the input channels are combined to produce an output tensor 860 (also referred to as OFM 860) . The OFM 860 is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements (also referred to as output points) in each row and five output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 8. In embodiments where there are multiple filters, the standard convolution may produce multiple output channels in the OFM 860.

The multiplication applied between a kernel-sized patch of the IFM 840 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernel-sized patch of the IFM 840 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the “scalar product. ” Using a kernel smaller than the IFM 840 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 840 multiple times at different points on the IFM 840. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 840, left to right, top to bottom. The result from multiplying the kernel with the IFM 840 one time is a single value. As the kernel is applied multiple times to the IFM 840, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 860) from the standard convolution 863 is referred to as an OFM.

In the depthwise convolution 883, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an output channel. As shown in FIG. 8, the depthwise convolution 883 produces a depthwise output tensor 880. The depthwise output tensor 880 is represented by a 5×5×3 3D matrix. The depthwise output tensor 880 includes 3 output channels, each of which is represented by a 5×5 2D matrix. The 5×5 2D matrix includes 5 output elements in each row and five output elements in each column. Each output channel is a result of MAC operations of an input channel of the IFM 840 and a kernel of the filter 850. For instance, the first output channel (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots) , the second output channel (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips) , and the third output channel (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes) . In such a depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 893 is then performed on the depthwise output tensor 880 and a 1×1×3 tensor 890 to produce the OFM 860.

The OFM 860 is then passed to the next layer in the sequence. In some embodiments, the OFM 860 is passed through an activation function. An example activation function is rectified linear unit (ReLU) . ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 810 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 860 is passed to the subsequent convolutional layer 810 (i.e., the convolutional layer 810 following the convolutional layer 810 generating the OFM 860 in the sequence) . The subsequent convolutional layers 810 perform a convolution on the OFM 860 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 810, and so on.

In some embodiments, a convolutional layer 810 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions F×F×D pixels) , the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time) , and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 810) . The convolutional layers 810 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 800 includes 86 convolutional layers 810. In other embodiments, the DNN 800 may include a different number of convolutional layers.

The pooling layers 820 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 820 is placed between two convolution layers 810: a preceding convolutional layer 810 (the convolution layer 810 preceding the pooling layer 820 in the sequence of layers) and a subsequent convolutional layer 810 (the convolution layer 810 subsequent to the pooling layer 820 in the sequence of layers) . In some embodiments, a pooling layer 820 is added after a convolutional layer 810, e.g., after an activation function (e.g., ReLU, etc. ) has been applied to the OFM 860.

A pooling layer 820 receives feature maps generated by the preceding convolution layer 810 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics. Accordingly, the pooling operation improves the efficiency of the CNN and avoids over-learning. The pooling layers 820 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map) , max pooling (calculating the maximum value for each patch of the feature map) , or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2×2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 820 applied to a feature map of 8×6 results in an output pooled feature map of 3×3. The output of the pooling layer 820 is inputted into the subsequent convolution layer 810 for further feature extraction. In some embodiments, the pooling layer 820 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

The fully-connected layers 830 are the last layers of the CNN. The fully-connected layers 830 may be convolutional or not. The fully-connected layers 830 may also be referred to as linear layers. In some embodiments, a fully-connected layer 830 (e.g., the first fully-connected layer in the DNN 800) may receive an input operand. The input operand may define the output of the convolutional layers 810 and pooling layers 820 and includes the values of the last feature map generated by the last pooling layer 820 in the sequence. The fully-connected layer 830 may apply a linear transformation to the input operand through a weight matrix. The weight matrix may be a kernel of the fully-connected layer 830. The linear transformation may include a tensor multiplication between the input operand and the weight matrix. The result of the linear transformation may be an output operand. In some embodiments, the fully-connected layer may further apply a non-linear transformation (e.g., by using a non-linear activation function) on the result of the linear transformation to generate an output operand. The output operand may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 8, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 830 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function.

FIG. 9 illustrates an AI-based face tracking environment 900, in accordance with various embodiments. The AI-based face tracking environment 900 includes a computer vision system 910, client devices 920 (individually referred to as client device 920) , and a third-party system 930. In other embodiments, the AI-based face tracking environment 900 may include fewer, more, or different components. For instance, the AI-based face tracking environment 900 may include a different number of client devices 920 or more than one third-party system 930.

The computer vision system 910 performs face tracking tasks, e.g., tasks of generating face animations based on videos, including monocular videos. For instance, the computer vision system 910 may uses DNNs to predict face expressions and poses from video frames. In some embodiments, the computer vision system 910 may receive a video that captures one or more faces and perform face tracking using one or more frames in the video. The computer vision system 910 may receive the video from one or more client devices 920 or the third-party system 930. Also, the computer vision system 910 may transmit information indicating the results of face tracking (e.g., face tracking animations, face tracking parameters, etc. ) to one or more client devices 920 or the third-party system 930. An example of the computer vision system 910 is the computer vision system 100 in FIG. 1.

The client devices 920 are in communication with the computer vision system 910. For example, the client device 920 may receive 3D face animations from the computer vision system 910 and display the 3D face animations to one or more users associated with the client device 920. In some embodiments, a client device 920 may execute one or more applications allowing one or more users of the client device 920 to interact with the computer vision system 910. For example, a client device 920 executes a browser application to enable interaction between the client device 920 and the computer vision system 910. In another embodiment, a client device 920 interacts with the computer vision system 910 through an application programming interface (API) running on a native operating system of the client device 920, such asor ANDROID^TM.

A client device 920 may be one or more computing devices capable of receiving user input as well as transmitting and/or receiving data via the network 940. In one embodiment, a client device 920 is a conventional computer system, such as a desktop or a laptop computer. Alternatively, a client device 920 may be a device having computer functionality, such as a personal digital assistant (PDA) , a mobile telephone, a smartphone, an autonomous vehicle, or another suitable device. A client device 920 is configured to communicate via the network 940. In an embodiment, a client device 920 is an integrated computing device that operates as a standalone network-enabled device. For example, the client device 920 includes display, speakers, microphone, camera, and input device. In another embodiment, a client device 920 is a computing device for coupling to an external media device such as a television or other external display and/or audio output system. In this embodiment, the client device 920 may couple to the external media device via a wireless interface or wired interface and may utilize various functions of the external media device such as its display, speakers, microphone, camera, and input devices. Here, the client device 920 may be configured to be compatible with a generic external media device that does not have specialized software, firmware, or hardware specifically for interacting with the client device 920.

The third-party system 930 is an online system that may communicate with the computer vision system 910 or at least one of the client devices 920. In some embodiments, the third-party system 930 may provide data to the computer vision system 910 for 3D face tracking. The data may include videos, data for training DNNs, data for validating DNNs, and so on. The third-party system 930 may be a social media system, an online image gallery, an online searching system, and so on. Additionally or alternatively, the third-party system 930 may use results of 3D face tracking in various applications. For instance, the third-party system 930 may use face tracking results from the computer vision system 910 for virtual reality, augmented reality, film and game production, telepresence, and so on.

The computer vision system 910, client devices 920, and third-party system 930 are connected through a network 940. The network 940 may comprise any combination of local area and/or wide area networks, using both wired and/or wireless communication systems. In one embodiment, the network 940 may use standard communications technologies and/or protocols. For example, the network 940 may include communication links using technologies such as Ethernet, 9010.11, worldwide interoperability for microwave access (WiMAX) , 3G, 4G, code division multiple access (CDMA) , digital subscriber line (DSL) , etc. Examples of networking protocols used for communicating via the network 940 may include multiprotocol label switching (MPLS) , transmission control protocol/Internet protocol (TCP/IP) , hypertext transport protocol (HTTP) , simple mail transfer protocol (SMTP) , and file transfer protocol (FTP) . Data exchanged over the network 940 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML) . In some embodiments, all or some of the communication links of the network 940 may be encrypted using any suitable technique or techniques.

FIG. 10 is a flowchart showing a method 1000 of face tracking, in accordance with various embodiments. The method 1000 may be a method of 3D visual recognition. The method 1000 may be performed by the face tracking module 110 in FIG. 1. Although the method 1000 is described with reference to the flowchart illustrated in FIG. 10, many other methods for visual recognition may alternatively be used. For example, the order of execution of the steps in FIG. 10 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

The face tracking module 110 generates 1010 by a first neural network, a first feature map from a first frame in a video that captures a face. In some embodiments, the first neural network is the DNN 160 in FIG. 1. In some embodiments, the first frame is the current frame. The first feature map is the current feature map.

The face tracking module 110 generates 1020, by the first neural network, a second feature map from a second frame in a video. The second frame is temporally subsequent to the first frame in the video. In some embodiments, the first frame is a previous frame. The first feature map is a previous feature map.

The face tracking module 110 generates 1030 an aggregated feature map by using the first feature map with the second feature map. in some embodiments, the face tracking module 110 writes the first feature map to a buffer after generating the first feature map and reads the first feature map from the buffer after generating the second feature map.

The face tracking module 110 determines 1040 by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation. The one or more blendshape coefficients represent a predicted facial expression. In some embodiments, the second neural network may include a transformer decoder. The transformer decoder includes a cross attention module (e.g., a cross attention layer) that processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query. In some embodiments, the trained blendshape representation comprises a plurality of vectors, and each vector corresponds to a trained blendshape.

In some embodiments, the face tracking module 110 generates, by a third neural network, an intermediate feature map from the first feature map and the second feature map. In some embodiments, the third neural network may include one or more transformer decoders.

In some embodiments, the face tracking module 110 also generates a face segmentation probability map of the second frame. The segmentation probability map indicates segmentation of different portions of the face. The face tracking module 110 generates a reweighting mask by applying a third neural network on the face segmentation probability map. The face tracking module 110 generates a reweighted feature map by aggregating the second feature map with the reweighting mask. The face tracking module 110 also determines, by a fourth neural network, one or more pose coefficients by using the reweighted feature map. In some embodiments, the third neural network is a convolutional neural network, and the fourth neural network comprises a transformer decoder.

The face tracking module 110 generates 1150 an animation of the face based on the blendshape coefficients. In some embodiments, the face tracking module 110 generates the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients. In some embodiments, the face tracking module 110 generates the animation of the face further based on one or more blendshape templates.

FIG. 11 is a block diagram of an example computing device 1100, in accordance with various embodiments. In some embodiments, the computing device 1100 can be used as at least part of the computer vision system 100. A number of components are illustrated in FIG. 11 as included in the computing device 1100, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1100 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1100 may not include one or more of the components illustrated in FIG. 11, but the computing device 1100 may include interface circuitry for coupling to the one or more components. For example, the computing device 1100 may not include a display device 1106, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1106 may be coupled. In another set of examples, the computing device 1100 may not include an audio input device 1118 or an audio output device 1108, but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 1118 or audio output device 1108 may be coupled.

The computing device 1100 may include a processing device 1102 (e.g., one or more processing devices) . The processing device 1102 processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory. The computing device 1100 may include a memory 1104, which may itself include one or more memory devices such as volatile memory (e.g., DRAM) , nonvolatile memory (e.g., read-only memory (ROM) ) , high bandwidth memory (HBM) , flash memory, solid state memory, and/or a hard drive. In some embodiments, the memory 1104 may include memory that shares a die with the processing device 1102. In some embodiments, the memory 1104 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for performing 3D face tracking, e.g., the method 1000 described above in conjunction with FIG. 10 or some operations performed by the computer vision system 100 or a component of the computer vision system 100 (e.g., the face tracking module 110) described above in conjunction with FIG. 1. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1102.

In some embodiments, the computing device 1100 may include a communication chip 1112 (e.g., one or more communication chips) . For example, the communication chip 1112 may be configured for managing wireless communications for the transfer of data to and from the computing device 1100. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

The communication chip 1112 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family) , IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment) , Long-Term Evolution (LTE) project along with any amendments, updates, and/or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2" ) , etc. ) . IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1112 may operate in accordance with a Global System for Mobile Communication (GSM) , General Packet Radio Service (GPRS) , Universal Mobile Telecommunications System (UMTS) , High Speed Packet Access (HSPA) , Evolved HSPA (E-HSPA) , or LTE network. The communication chip 1112 may operate in accordance with Enhanced Data for GSM Evolution (EDGE) , GSM EDGE Radio Access Network (GERAN) , Universal Terrestrial Radio Access Network (UTRAN) , or Evolved UTRAN (E-UTRAN) . The communication chip 1112 may operate in accordance with CDMA, Time Division Multiple Access (TDMA) , Digital Enhanced Cordless Telecommunications (DECT) , Evolution-Data Optimized (EV-DO) , and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1112 may operate in accordance with other wireless protocols in other embodiments. The computing device 1100 may include an antenna 1122 to facilitate wireless communications and/or to receive other wireless communications (such as AM or FM radio transmissions) .

In some embodiments, the communication chip 1112 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet) . As noted above, the communication chip 1112 may include multiple communication chips. For instance, a first communication chip 1112 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1112 may be dedicated to longer-range wireless communications such as global positioning system (GPS) , EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1112 may be dedicated to wireless communications, and a second communication chip 1112 may be dedicated to wired communications.

The computing device 1100 may include battery/power circuitry 1114. The battery/power circuitry 1114 may include one or more energy storage devices (e.g., batteries or capacitors) and/or circuitry for coupling components of the computing device 1100 to an energy source separate from the computing device 1100 (e.g., AC line power) .

The computing device 1100 may include a display device 1106 (or corresponding interface circuitry, as discussed above) . The display device 1106 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD) , a light-emitting diode display, or a flat panel display, for example.

The computing device 1100 may include an audio output device 1108 (or corresponding interface circuitry, as discussed above) . The audio output device 1108 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

The computing device 1100 may include an audio input device 1118 (or corresponding interface circuitry, as discussed above) . The audio input device 1118 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output) .

The computing device 1100 may include a GPS device 1116 (or corresponding interface circuitry, as discussed above) . The GPS device 1116 may be in communication with a satellite-based system and may receive a location of the computing device 1100, as known in the art.

The computing device 1100 may include another output device 1110 (or corresponding interface circuitry, as discussed above) . Examples of the other output device 1110 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

The computing device 1100 may include another input device 1120 (or corresponding interface circuitry, as discussed above) . Examples of the other input device 1120 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

The computing device 1100 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a PDA, an ultramobile personal computer, etc. ) , a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1100 may be any other electronic device that processes data.

The following paragraphs provide various examples of the embodiments disclosed herein.

Example 1 provides a method, including generating, by a first neural network, a first feature map from a first frame in a video that captures a face; generating, by the first neural network, a second feature map from a second frame in a video, in which the second frame is temporally subsequent to the first frame in the video; generating an aggregated feature map by using the first feature map with the second feature map; determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression; and generating an animation of the face based on the one or more blendshape coefficients.

Example 2 provides the method of example 1, in which generating the aggregated feature map includes generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.

Example 3 provides the method of example 2, in which the third neural network includes a transformer decoder.

Example 4 provides the method of any one of examples 1-3, in which generating the aggregated feature map includes after generating the first feature map, writing the first feature map to a buffer; and after generating the second feature map, reading the first feature map from the buffer.

Example 5 provides the method of example any one of claims 1-4, in which the second neural network includes a transformer decoder that includes a cross attention module, and the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.

Example 6 provides the method of any one of examples 1-5, in which the trained blendshape representation includes a plurality of vectors, and each vector corresponds to a trained blendshape.

Example 7 provides the method of any one of examples 1-6, further including generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face; generating a reweighting mask by applying a third neural network on the face segmentation probability map; and generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.

Example 8 provides the method of example 7, in which the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

Example 9 provides the method of example 7 or 8, in which generating the animation of the face includes generating the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients.

Example 10 provides the method of any one of examples 1-9, in which generating the animation of the face includes generating the animation of the face further based on one or more blendshape templates, a blendshape template corresponding to a reference facial expression.

Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including generating, by a first neural network, a first feature map from a first frame in a video that captures a face; generating, by the first neural network, a second feature map from a second frame in a video, in which the second frame is temporally subsequent to the first frame in the video; generating an aggregated feature map by using the first feature map with the second feature map; determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression; and generating an animation of the face based on the one or more blendshape coefficients.

Example 12 provides the one or more non-transitory computer-readable media of example 11, in which generating the aggregated feature map includes generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.

Example 13 provides the one or more non-transitory computer-readable media of example12, in which the third neural network includes a transformer decoder.

Example 14 provides the one or more non-transitory computer-readable media of any one of examples 11-13, in which generating the aggregated feature map includes after generating the first feature map, writing the first feature map to a buffer; and after generating the second feature map, reading the first feature map from the buffer.

Example 15 provides the one or more non-transitory computer-readable media of any one of examples 11-14, in which the second neural network includes a transformer decoder that includes a cross attention module, and the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.

Example 16 provides the one or more non-transitory computer-readable media of any one of examples 11-15, in which the trained blendshape representation includes a plurality of vectors, and each vector corresponds to a trained blendshape.

Example 17 provides the one or more non-transitory computer-readable media of any one of examples 11-16, in which the operations further include generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face; generating a reweighting mask by applying a third neural network on the face segmentation probability map; and generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.

Example 18 provides the one or more non-transitory computer-readable media of example 17, in which the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

Example 19 provides the one or more non-transitory computer-readable media of example 17 or 18, in which generating the animation of the face includes generating the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients.

Example 20 provides the one or more non-transitory computer-readable media of any one of examples 11-19, in which generating the animation of the face includes generating the animation of the face further based on one or more blendshape templates, a blendshape template corresponding to a reference facial expression.

Example 21 provides an apparatus, including a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations including generating, by a first neural network, a first feature map from a first frame in a video that captures a face, generating, by the first neural network, a second feature map from a second frame in a video, in which the second frame is temporally subsequent to the first frame in the video, generating an aggregated feature map by using the first feature map with the second feature map, determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression, and generating an animation of the face based on the one or more blendshape coefficients.

Example 22 provides the apparatus of example 21, in which generating the aggregated feature map includes generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.

Example 23 provides the apparatus of example 21 or 22, in which generating the aggregated feature map includes after generating the first feature map, writing the first feature map to a buffer; and after generating the second feature map, reading the first feature map from the buffer.

Example 24 provides the apparatus of any one of examples 21-23, in which the operations further include generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face; generating a reweighting mask by applying a third neural network on the face segmentation probability map; and generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.

Example 25 provides the apparatus of example 24, in which the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

A method, comprising:

generating, by a first neural network, a first feature map from a first frame in a video that captures a face;

generating, by the first neural network, a second feature map from a second frame in a video, wherein the second frame is temporally subsequent to the first frame in the video;

generating an aggregated feature map by using the first feature map with the second feature map;

determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression; and

generating an animation of the face based on the one or more blendshape coefficients.
The method of claim 1, wherein generating the aggregated feature map comprises:

generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and

aggregating the intermediate feature map with the second feature map.
The method of claim 2, wherein the third neural network comprises a transformer decoder.
The method of any one of claims 1-3, wherein generating the aggregated feature map comprises:

after generating the first feature map, writing the first feature map to a buffer; and

after generating the second feature map, reading the first feature map from the buffer.
The method of claim any one of claims 1-4, wherein the second neural network comprises a transformer decoder that includes a cross attention module, and the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.
The method of any one of claims 1-5, wherein the trained blendshape representation comprises a plurality of vectors, and each vector corresponds to a trained blendshape.
The method of any one of claims 1-6, further comprising:

generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face;

generating a reweighting mask by applying a third neural network on the face segmentation probability map; and

generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and

determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.
The method of claim 7, wherein the third neural network is a convolutional neural network, and the fourth neural network comprises a transformer decoder.
The method of claim 7 or 8, wherein generating the animation of the face comprises:

generating the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients.
The method of any one of claims 1-9, wherein generating the animation of the face comprises:

generating the animation of the face further based on one or more blendshape templates, a blendshape template corresponding to a reference facial expression.
One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:

generating, by a first neural network, a first feature map from a first frame in a video that captures a face;

generating, by the first neural network, a second feature map from a second frame in a video, wherein the second frame is temporally subsequent to the first frame in the video;

generating an aggregated feature map by using the first feature map with the second feature map;

determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression; and

generating an animation of the face based on the one or more blendshape coefficients.
The one or more non-transitory computer-readable media of claim 11, wherein generating the aggregated feature map comprises:

generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and

aggregating the intermediate feature map with the second feature map.
The one or more non-transitory computer-readable media of claim12, wherein the third neural network comprises a transformer decoder.
The one or more non-transitory computer-readable media of any one of claims 11-13, wherein generating the aggregated feature map comprises:

after generating the first feature map, writing the first feature map to a buffer; and

after generating the second feature map, reading the first feature map from the buffer.
The one or more non-transitory computer-readable media of any one of claims 11-14, wherein the second neural network comprises a transformer decoder that includes a cross attention module, and the cross attention module processes the aggregated feature map as a key or value and processes the trained blendshape representation as a query.
The one or more non-transitory computer-readable media of any one of claims 11-15, wherein the trained blendshape representation comprises a plurality of vectors, and each vector corresponds to a trained blendshape.
The one or more non-transitory computer-readable media of any one of claims 11-16, wherein the operations further comprise:

generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face;

generating a reweighting mask by applying a third neural network on the face segmentation probability map; and

generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and

determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.
The one or more non-transitory computer-readable media of claim 17, wherein the third neural network is a convolutional neural network, and the fourth neural network comprises a transformer decoder.
The one or more non-transitory computer-readable media of claim 17 or 18, wherein generating the animation of the face comprises:

generating the animation of the face based on the one or more blendshape coefficients and the one or more pose coefficients.
The one or more non-transitory computer-readable media of any one of claims 11-19, wherein generating the animation of the face comprises:

generating the animation of the face further based on one or more blendshape templates, a blendshape template corresponding to a reference facial expression.
An apparatus, comprising:

a computer processor for executing computer program instructions; and

a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising:

generating, by a first neural network, a first feature map from a first frame in a video that captures a face,

generating, by the first neural network, a second feature map from a second frame in a video, wherein the second frame is temporally subsequent to the first frame in the video,

generating an aggregated feature map by using the first feature map with the second feature map,

determining, by a second neural network, one or more blendshape coefficients by using the aggregated feature map and a trained blendshape representation, the one or more blendshape coefficients representing a predicted facial expression, and

generating an animation of the face based on the one or more blendshape coefficients.
The apparatus of claim 21, wherein generating the aggregated feature map comprises:

generating, by a third neural network, an intermediate feature map from the first feature map and the second feature map; and

aggregating the intermediate feature map with the second feature map.
The apparatus of claim 21 or 22, wherein generating the aggregated feature map comprises:

after generating the first feature map, writing the first feature map to a buffer; and

after generating the second feature map, reading the first feature map from the buffer.
The apparatus of any one of claims 21-23, wherein the operations further comprise:

generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different portions of the face;

generating a reweighting mask by applying a third neural network on the face segmentation probability map; and

generating a reweighted feature map by aggregating the second feature map with the reweighting mask; and

determining, by a fourth neural network, one or more pose coefficients by using the reweighted feature map.
The apparatus of claim 24, wherein the third neural network is a convolutional neural network, and the fourth neural network comprises a transformer decoder.