RU2755396C1 - Neural network transfer of the facial expression and position of the head using hidden position descriptors - Google Patents

Abstract

FIELD: image data processing.SUBSTANCE: invention relates to computer technology, namely to computer graphics. A hardware apparatus for transferring a random facial expression and position of the head to an avatar of the person while maintaining the identity of the avatar of the person, containing: a detector for capturing images of the facial expression and position of the head of a person B, a tool for selecting images of a person A, an identity encoder block configured to retrieve an identity descriptor from the images of the person A, wherein the output of the position encoder does not contain information about the identity of the person A, a position encoder block configured to receive images of the person B at the input, the generator block is configured to output the received image of the avatar of the person A with the position of the head and the facial expression taken from the person B.EFFECT: increasing the quality of image synthesising.6 cl, 10 dwg, 2 tbl

Description

The technical field to which the invention relates

The invention relates to computer graphics, telepresence (for example, video communication), determining the posture of a person's head in an image, video production, tracking a person's face in video, augmented / virtual reality.

State of the art

In recent years, the quality and stability of the transfer of video images of the head has improved significantly. Today's most advanced systems [33, 22, 37, 30, 39, 42, 10, 36, 35] demonstrate convincing, almost photorealistic transfers of the "talking head". The most recent of them are able to provide such a result using deep neural generative networks, even if there is only one image of the target person [30, 42, 10, 36, 35].

Face / head transfer is an active area of research. In this area, it is possible to distinguish works in which changes and additions are localized within faces (transfer of a face image), for example [33, 30], and larger-scale approaches in which wide areas are modeled, including a significant part of clothing, neck, upper clothes (transfer of the image of the head), for example, [22, 37, 42].

An important aspect of transfer systems is posture presentation. As noted above, in most works, the transfer is carried out using landmarks [33, 22, 37, 42, 10, 36]. Another approach is to use elements of facial dynamics (AU) [9], as in face transfer [30] and head transfer [35]. Detecting dynamic elements still requires manual annotation and supervised learning. The X2Face system [39] uses hidden vectors that are trained to predict deformation fields.

A more classical approach is to simulate a face / head pose on a 3D morphing model (3DMM) platform [1] or apply a similar approach in a two-dimensional environment (eg active appearance model) [6]. However, training a 3DMM and fitting a trained 3DMM almost always involves finding landmarks and thus inherits many of the disadvantages of landmarks. In other cases, a 3D scan dataset is required to build a model to separate pose from identity on the 3DMM platform.

A number of recent works have explored how landmarks can be taught without involving a teacher [44, 19]. While uncontrolled cue points are generally very promising, they still contain human-specific information, as well as controlled cue points, and are therefore generally not suitable for transference from one person to another. The same applies to dense, multidimensional descriptors such as the DensePose body descriptor [14], and dense face-only descriptors [15, 34]. Finally, codec avatars [26] are trained on human-specific hidden pose descriptors and extractors based on reconstruction loss. However, the transfer of such descriptors from one person to another has not been considered.

Recent parallel work [32] has demonstrated that it is possible to use the relative movement of uncontrolled keypoints to transfer animation, at least without significant head rotation. A comprehensive comparison of the proposed approach with [32] is left for further work.

In addition to head / face transfer, there is a very large body of work on teaching split representations. Representative works in this area include works [8, 40], in which hidden pose or shape descriptors for arbitrary classes of objects are used for training using video datasets. Some approaches (eg [24]) aim at teaching the separation of style and content (which may roughly correspond to the separation of form and texture) using adversarial loss [12] and loss of cycle coherence [45, 17]. Alternatively, splitting can be done by directly fitting factorized distributions to the data (eg, [23]).

More importantly, a new posture representation is proposed (hereinafter “head posture” means a set of head orientation, position, and facial expression) for neural network transfer of facial expression and head posture. This representation of the pose plays a key role in the quality of the transfer. Most systems, including [33, 22, 37, 42, 10, 36], are based on the representation of a key point (landmark). The main advantage of this representation is that reliable and efficient “ready-made” landmark detectors are now available [21, 2].

The advent of large, untagged human video datasets, such as [29, 4, 5], makes it possible to teach hidden posture and expression descriptors without a teacher. This approach was first studied in [39, 38], where the hidden pose descriptors were trained in such a way that a dense flow between different frames could be deduced from the trained descriptors.

However, facial landmarks have several disadvantages. First, training a landmark detector requires an overwhelming annotation effort, and some important aspects of the pose are often missing from the annotated landmarks sets. For example, many landmark annotations lack pupils, and as a result, hyphenation will not completely control the direction of gaze. Second, many of the landmarks are not anatomically based and their annotations are ambiguous and error-prone, especially when obstructed by other objects. In practice, this ambiguity in annotation often leads to temporary instability in key point detection, which, in turn, affects the results of the transfer. In conclusion, it should be noted that landmarks as representations are specific to humans, since they contain a significant amount of posture-independent information about the geometry of the head.

This can be highly undesirable for head transfer, for example if one wishes to take a portrait photograph or painting with a target person with a different head geometry.

The essence of the invention

The present invention proposes to improve existing systems of neural network head transfer according to one example (one-shot), using two important approaches. First, the previous modern transfer system [42] is simply supplemented with the ability to predict foreground segmentation. Such prediction is required for various scenarios, for example, for telepresence, where transferring the original background to a new environment may not be desirable.

An alternative to the deformation-based approach has been proposed [39, 38]. In the proposed approach, training is conducted using small-sized, human-independent pose descriptors along with medium-sized human-independent and pose-independent descriptors, by superimposing a set of reconstruction losses on video frames in a large collection of video images.

It is important to note that when estimating reconstruction losses, it is proposed to remove the background so that its noise and frame-by-frame changes do not affect the trained descriptors.

The proposed system modifies and expands the transport model of Zakharov et al. [42]. First, the ability to predict segmentation has been added. Second, the system is trained to perform translation based on the latent vectors of the pose, rather than key points.

A simple training structure based on sampling many random frames from the same video material, combined with a large video dataset size, allows extractors to be trained for both descriptors, which works very well for transfer tasks, including transfer from one person to another.

In particular, the proposed transfer based on a new latent pose representation preserves the identity of the target person much better than using the FAb-Net [38] and X2Face [39] pose descriptors. In addition, the quality of the trained hidden pose descriptors for tasks such as predicting landmarks and searching based on posture is analyzed.

A critical component of a large number of human-centered tasks involving computer vision and graphics is a broad representation of a person's head posture and facial expressions. An effective approach is to train such views on the data without relying on human annotation, as it can use vast untagged datasets of video images of people. In this paper, we propose a new simple method for implementing such training. In contrast to previous works, the proposed training allows you to obtain human-independent descriptors that capture a person's posture and at the same time can be transferred from one person to another, which is especially useful for applications such as face transfer. It is also shown that, in addition to transferring a face, these descriptors are useful for other subsequent tasks, for example, to determine the orientation of a face. Meanwhile, the above-described method performed by an electronic device can be performed using an artificial intelligence model.

It is important to note that the proposed system can be successfully applied to video creation. Even if each video frame is created independently and without any temporal processing, the proposed system shows time-smoothed facial expressions thanks to pose additions and hidden pose representation. Previous methods, which used facial cue points to perform the transfer, would inherit the shakiness from cue point detectors.

A hardware device is proposed that contains a software product that performs a method for neural network transfer of facial expressions and head postures, comprising: an identity encoder unit configured to obtain an identity descriptor from an image of a person A, and the output of the pose encoder unit does not contain information about the identity of a person A; a posture encoder unit configured to obtain a head posture descriptor and a facial expression from an image of a person B, wherein the output of the pose encoder unit does not contain identity information of person B; a generator block that receives the outputs of the identity encoder and posture encoder blocks, and the generator block is configured to synthesize an avatar of a person A having a head pose and facial expression taken from a person B. In this case, the posture encoder block is a convolutional neural network that takes an image as input a person and outputs a vector describing the posture of his head and facial expression and does not describe his identity. In this case, the identity of person B means the skin color, face shape, eye color, clothing and jewelry of person B.

A method for synthesizing a photorealistic avatar of a person is proposed, which consists in the fact that: using an identity encoder unit, an identity descriptor is obtained from an image of a person A; obtaining, using a pose encoder unit, a descriptor for a head pose and a facial expression from an image of person B; are synthesized using a generator block that accepts the outputs of the identity encoder and pose encoder blocks, an avatar of a person A, having a head pose and a facial expression taken from a person B. In this case, the pose encoder block is a convolutional neural network that accepts an image of a person as input and an outstanding vector describing his head posture and facial expression and not describing his identity. In this case, identity means skin color, face shape, eye color, clothing, jewelry.

Description of drawings

The above and / or other aspects of the invention will become more apparent from the description of exemplary embodiments with reference to the accompanying drawings.

FIG. 1 shows the use of randomly selected people as samples of facial expressions and head posture (top row) to obtain realistic transfers of arbitrary talking heads (such as Mona Lisa, bottom row).

FIG. 2a shows the proposed training pipeline (for simplicity, the discriminator is not shown).

FIG. 2b shows the application of the proposed method.

FIG. 3 shows the assessment of the transfer systems in terms of their ability to represent the pose of the specimen and maintain the original identity (arrows indicate improvement).

FIG. 4 shows a comparison of transfer from one person to another for several systems on the VoxCeleb2 test set.

FIG. 5 shows translation by interpolation between two pose vectors along a spherical path in pose descriptor space.

FIG. 6 shows an additional comparison of human-to-human transfer for multiple systems on the VoxCeleb2 test suite.

FIG. 7 presents a quantitative assessment of how the ablation of several important features of the training sample affects the proposed system.

FIG. 8 shows a comparison of the transfer from one person to another for the proposed best model and its ablative versions.

FIG. 9 shows the effect of posture additions on the X2Face + and FAb-Net + models. Without additions, the identity divergence becomes noticeable.

Description of embodiments

A neural network transfer system for facial expression and head posture is proposed, which is based on the latent representation of the posture and is capable of predicting foreground segmentation along with an RGB image. The latent representation of the pose is trained as part of the entire transfer system, and the learning process is based solely on the loss of image reconstruction. Despite its simplicity, in the presence of a large and sufficiently diverse training data set, such training successfully separates a pose from an identity. After that, the resulting system can reproduce the facial expressions of a person-sample and, moreover, carry out a transfer from one person to another. In addition, trained descriptors are useful for other pose-related tasks such as predicting cue points and searching based on posture.

FIG. Figure 1 shows the use of randomly selected people as samples of facial expressions and head posture (top row) to create realistic hyphenation of free talking heads (such as Mona Lisa, bottom row). The proposed system can create realistic transfers of arbitrary talking heads (such as the Mona Lisa) using arbitrary people as models for facial expressions and head postures (top row). Despite the fact that teaching is performed without a teacher, this method allows you to successfully decompose the posture and identity, so that the identity of the person to whom the transfer is made is preserved.

The invention makes it possible to quickly obtain an accurate and flexible representation (descriptor) of the head posture and facial expression of a person from a single image of his head.

The invention makes it possible to determine the posture of the head (direction / rotation / angles of inclination) and facial expression (including the direction of gaze) from the image of a person. The invention can find application in the following areas:

telepresence systems (video communication);

remote control systems using faces (in TVs, smartphones, robots, etc.);

entertainment systems, games (managing or creating a virtual avatar);

messengers (for example, creating animated stickers / emoticons);

systems of augmented (AR) or virtual (VR) reality (determining the position of the head or gaze when drawing virtual objects from the correct angle).

The proposed system modifies and expands the transport model of Zakharov et al. [42]. First, the ability to predict segmentation has been added. Second, the system is trained to perform translation based on the latent vectors of the pose, rather than key points. As in [42], training is carried out on the VoxCeleb2 video sequence dataset [4]. Each sequence contains a speaking person and is obtained from the raw sequence by running a face detector, cropping the resulting face and resizing it to a fixed size (in the proposed case, 256x256). In addition, as in the case of [42], a stage of "meta-training" is provided, at which a large model responsible for reproducing all people in the dataset is trained through a sequence of training episodes, and a stage of additional training, at which the "meta-model" is subjected to additional training for a tuple of images (or one image) of a specific person.

As shown in FIG. 2a, at each stage of meta-learning, the proposed system selects a set of frames from a person's video. These frames are processed by two encoders. The larger encoder is applied to multiple video frames, and the smaller encoder is applied to the held frame. Withheld data usually refers to a piece of data that is not specifically shown to a particular model (i.e. held outside of it). In this case, the held frame is just another frame of the same person. The term "held" emphasizes that a given frame is guaranteed not to go to the identity encoder and only goes to the pose encoder. The received attachments are sent to the generator network, the purpose of which is to reconstruct the last (withheld) frame. Having an identity encoder that is more capacious than a pose encoder is essential for separating pose from identity in the hidden pose nesting space, which is a key component of the present invention. It is equally important to have a very bottleneck in the pose encoder, which in the proposed case is implemented through a neural network of lower performance in the pose encoder, and a smaller dimension of pose nesting than identity nesting. This forces the pose encoder to encode only the pose information into the pose attachments and ignore the identity information. Since the performance of the pose encoder is limited and its input does not quite match other frames with respect to identity (due to data padding), the system learns to extract all pose independent information through the identity encoder and uses a smaller encoder to capture only the pose related information, thereby providing separation poses and identities.

A method for neural network transfer of facial expression and head posture is proposed, i.e. an algorithm for synthesizing a photorealistic avatar of person A, where the facial expression and head posture are taken from the image of person B (as shown in Fig. 2a).

FIG. 2b illustrates the inference pipeline of the proposed method, i. E. the forecasting / rendering algorithm by the proposed system after its training.

It should be noted that the generator block is trained simultaneously with all other blocks. During training, an identity encoder block predicts identity nesting from multiple images of a person, and a pose encoder block predicts a pose nesting from that person's additional image. The generator block uses the averaged identity nesting and pose nesting and is trained to output the image that entered the pose encoder block, as well as the foreground mask for that image. New to this pipeline are: (1) a pose encoder block that does not directly involve the teacher during training, (2) use a foreground mask as a target, (3) pose augments (preserving the identity of the input distortion in a pose encoder block) as a method separation of posture and identity.

First of all, the identity embedding is calculated as the average of the outputs of the identity encoder over all available images of person A. there is only one person in the training set, (2) the pose encoder weights are kept frozen, (3) the identity embedding is converted to a trainable parameter. Finally, the image of any person B is passed through the pose encoder, and the predicted rendering and segmentation is generated as usual (i.e., as during training) from the resulting pose nesting and the retrained identity nesting.

1. To obtain an identity descriptor (skin color, face shape, eye color, clothing, jewelry, etc.) from the image of person A, the "identity encoder" software block is used (for example, using the system described in the article "Few-Shot Adversarial Learning of Realistic Neural Talking Head Models ".

2. To obtain a descriptor of a head pose and a facial expression from the image of a person B (dashed line in Fig. 2a), a "pose encoder" program block is used.

3. The outputs of these two blocks are fed to the input of the "generator" program block for synthesizing the desired avatar.

The pose encoder block is a trainable machine learning model (eg, a convolutional neural network). It takes an image of a person as input and outputs a vector that describes the head posture and facial expression and does not describe their identity.

The learning algorithm for this model: In the training pipeline of the "Few-Shot Adversarial Learning of Realistic Neural Talking Head Models" system, the input block "RGB & landmarks" is replaced by the "pose encoder" block according to the invention. The entire system is then trained as described in "Few-Shot Adversarial Learning of Realistic Neural Talking Head Models" with the additions described below. In particular: there is no downsampling in the generator (it does not belong to the image-to-image architecture (from one image to another)), since it starts from a constant trainable tensor, and not from a full-size image; the nesting of poses is additionally appended to an MLP input that predicts the AdaIN parameters; the identity encoder has a different architecture (ResNeXt-50 32x4d).

Arbitrary posture additions to the facial expression pattern and head posture are suggested; the generator predicts a foreground segmentation mask in the auxiliary output channel, and Dice loss is applied to match this prediction with true.

Compared to other existing solutions to this problem, the novelty lies in the following:

the output of the "pose encoder" block does not contain information about the identity of person B, therefore, this output is depersonalized, which has an advantage from the point of view of information security;

for the same reason as above, the proposed approach eliminates the need for a block for adapting the head posture descriptor and facial expression to the identity of person A (inside or outside the generator block), which means it improves computational efficiency.

Next, the meta-learning step will be described.

, а также SK+1 - карта сегментации переднего плана для IK+1, которая предварительно вычисляется с помощью готовой сети семантической сегментации.Each episode of the meta-teaching examines a separate video sequence. Then K + 1 random frames are selected from this sequence

as well as SK + 1 - a foreground segmentation map for IK + 1, which is pre-computed using a ready-made semantic segmentation network.

подаются в относительно высокопроизводительную сверточную сеть F, которая вызывает кодировщик идентичности. Она аналогична сети внедрения в [42], за исключением того, что не принимает ключевые точки на входе.Then the first K images

are fed into a relatively high performance convolutional network F, which calls the identity encoder. It is similar to the embedding network in [42], except that it does not accept cue points as input.

, который вызывает вложение идентичности Ii. Ожидается, что вложения идентичности содержат независимую от позы информацию о человеке (включая освещение, одежду и т.п.). При наличии K кадров получается один вектор идентичности

путем приведения к среднему

.For each image Ii, the identity encoder produces a di-dimensional vector

which invokes identity nesting Ii. Identity attachments are expected to contain posture-independent information about a person (including lighting, clothing, etc.). In the presence of K frames, one identity vector is obtained

by converting to the mean

...

, которое стремится стать не зависящим от человека дескриптором позы. The remaining image IK + 1 (pose source) is first transformed A for arbitrary pose addition, which will be described below. A (IK + 1) is then sent over a much lower performance network called the pose encoder, denoted as G. The pose encoder outputs a dp-dimensional embedding of the pose

, which seeks to become a person-independent posture descriptor.

The A transformation mentioned above is important for separating pose and identity. It keeps the person's posture intact, but it can change their identity. In particular, it arbitrarily scales the image independently along the horizontal and vertical axes and arbitrarily applies content saving operations such as blur, sharpening, contrast change, or JPEG compression. The invention draws on pose augmentation because it is applied at the pose source and can be considered a form of data augmentation.

размера 3×256x256 и

размера 1×256x256, которые он пытается согласовать с частью переднего плана изображения IK+1 и его маской сегментации SK+1, соответственно. Это достигается простым прогнозированием тензора 3×256x256 в конечном слое. После каждой свертки вставляются блоки AdaIN. Для получения коэффициентов AdaIN берутся соединенные вложения позы и идентичности, и этот (di+dp)-мерный вектор передается через MLP с обучаемыми параметрами как в StyleGAN [20].The pose and identity attachments are passed to the generator network, which attempts to reconstruct the IK + 1 image as accurately as possible. While [42] used rasterized cue points (stickman images) to pass the pose to their generator networks, the present inventors rely entirely on the AdaIN mechanism [16] to pass both pose and identity attachments to the generator. More specifically, the proposed upsampling generator starts with a 512 × 4x4 constant trainable tensor and outputs two tensors:

size 3 × 256x256 and

size 1 × 256x256, which it tries to match with the foreground portion of the image IK + 1 and its segmentation mask SK + 1, respectively. This is achieved by simply predicting a 3x256x256 tensor in the final layer. AdaIN blocks are inserted after each convolution. To obtain the AdaIN coefficients, the connected embeddings of the pose and identity are taken, and this (di + dp) -dimensional vector is transmitted through MLP with trained parameters as in StyleGAN [20].

и

будут максимально близки к

и SK+1, соответственно. Это достигается с помощью нескольких функций потерь. Карты сегментации сопоставляются с помощью потери коэффицтента Дайса [27]. С другой стороны, изображения головы с забитым задним планом сопоставляются с использованием той же комбинации потерь, что и в [42]. В частности, существуют потери контента на основании сопоставления активаций ConvNet для модели VGG-19, обученной классификации ImageNet, и модели VGGFace, обученной распознавать лица. Кроме того,

и

передаются через дискриминатор проекции (в данном случае отличие от [42] состоит в том, что в изобретении снова не предоставляются ему растеризованные ключевые точки) для вычисления состязательной потери, которая подталкивает изображения к реалистичности, потерь согласования признаков дискриминатора и параметра согласования вложения.Generator-generated

and

will be as close to

and SK + 1, respectively. This is achieved using several loss functions. Segmentation maps are compared using the loss of Dyes coefficient [27]. On the other hand, images of a head with a clogged background are compared using the same loss combination as in [42]. In particular, there is content loss based on the mapping of ConvNet activations for the VGG-19 model trained for ImageNet classification and the VGGFace model trained for face recognition. Besides,

and

are passed through the projection discriminator (in this case, the difference from [42] is that the invention again does not provide it with rasterized key points) to calculate the adversarial loss, which pushes the images towards realism, the matching losses of the discriminator's features and the nesting matching parameter.

идентичности, пропустив эти изображения через кодировщик идентичности и усреднив эти результаты поэлементно. Затем, подставив вектор позы y, извлеченный из изображения того же самого или другого человека, можно осуществить перенос этого человека, вычислив изображение

и его маску

переднего плана.Transfer and additional training. After meta-learning a model, you can use it to fit new identities that are not visible during meta-learning. Thus, in the presence of one or several images of a new person, it is possible to extract from them the vector

identity by passing these images through an identity encoder and averaging these results element by element. Then, substituting the pose vector y, extracted from the image of the same or another person, we can carry out the transfer of this person by calculating the image

and his mask

foreground.

сохраняется фиксированным во время дообучения (включение его в оптимизацию не привело к каким-либо различиям в предлагаемых экспериментах, так как число параметров во вложении

намного меньше, чем в MLP и сети-генераторе). Сеть G вложения позы также сохраняется фиксированной во время дообучения.To further reduce the identity discrepancy, it is proposed to follow [42] and carry out additional training of the model (namely, the MLP weights, generator and discriminator) with the same set of losses as in [42], plus with the loss of the Dice coefficient, considering the provided set of images of a new human beings and their segmentation as truth. Evaluated Identity Nesting

remains fixed during additional training (its inclusion in optimization did not lead to any differences in the proposed experiments, since the number of parameters in the nesting

much less than MLP and grid-generator). The net G nesting posture is also kept fixed during retraining.

извлекается из изображения того же человека X. Более удивительно то, что эта модель также может воспроизводить мимику, когда вектор позы извлекается из изображения другого человека Y. В этом случае просачивание идентичности от этого другого человека сводится к минимуму, т.е. полученное изображение все еще выглядит как изображение человека X.The main conclusion is that in relation to person X, trained as described above, the transfer model can successfully reproduce the facial expressions of a person in image I, when the pose vector

extracted from the image of the same person X. More surprisingly, this model can also reproduce the facial expressions when the pose vector is extracted from the image of another person Y. In this case, the leakage of identity from this other person is minimized, i.e. the resulting image still looks like an image of person X.

Initially, this separation of posture and identity should not have occurred and would require some form of adversarial learning [12] or loop compatibility [45, 17]. It turned out that in the case of (i) a sufficiently low performance of the network G of the pose extractor, (ii) applying the posture additions and (iii) removing the background, the separation occurs automatically, and the proposed experiments with additional loss conditions, such as, for example, in [8] do not give any further improvements. Obviously, with the three methods described above, the model prefers to extract all of the person-specific details from the identity source frame using the more powerful identity extractor network.

This separation effect will then be assessed, which came as a "pleasant surprise" and shown to be indeed stronger than other similar methods (ie, it better supports transference from one person to another with less identity leakage).

In addition, ablation studies are additionally performed to determine how the performance of the posture encoder, posture augmentation, segmentation, and dp dimension of the latent posture vector affect the ability of the proposed transfer system to maintain posture and identity.

The proposed training dataset is a collection of YouTube videos from VoxCeleb2 [4]. There are about 100,000 video images of about 6,000 people. For every 25 frames in each video, 1 frame was selected, so there are only about seven million total training images left. In each image, the annotated face was re-cropped, for which first the S3FD detector [43] captured its bounding rectangle, and then the rectangle was transformed into a square by enlarging the smaller side, the sides of the rectangle were enlarged by 80% while maintaining the center, and finally, the size of the cropped image changed to 256X256.

Human segmentation was performed using the Graphonomy model [11]. As in [42], the authors set K = 8, thereby using the identity vector extracted from eight random video frames to reconstruct the ninth frame from its pose descriptor.

In the proposed best model, the pose encoder has the MobileNetV2 architecture [31], and the identity encoder has the ResNeXt-50 (32 X 4d) architecture [41]. They were both not debugged and therefore contain packet normalization [18]. The pose and identity nest sizes dp and di are 256 and 512, respectively. No normalization or regularization is applied to these attachments. The module that converts them into AdaIN parameters is a ReLU perceptron with spectral normalization and one hidden layer of 768 neurons.

The proposed generator is based on the generator [42], but without downsampling blocks, since all inputs are delegated to the AdaIN located after each convolution. More precisely, the trainable constant tensor 512 × 4 × 4 is transformed by two residual blocks of constant resolution, followed by 6 residual blocks of upsampling. Starting with the fourth upsampling block, the number of channels begins to halve, so that the final resolution tensor (256 X 256) has 64 channels. This tensor is passed through the AdaIN layer, ReLU, 1X1 convolution and tanh, becoming a 4-channel image. Unlike [42], the authors do not use the self-attention mechanism. Spectral normalization [28] is applied everywhere in generator, discriminator and MLP.

Instead of alternating generator and discriminator updates, a single weight update is performed for all networks after accumulating gradients from all loss parameters.

The model was trained for 1,200,000 iterations with a mini-batch of 8 samples distributed across two NVIDIA P40 GPUs, which took about two weeks in total.

The proposed quantitative assessment assesses both the relative effectiveness of posture descriptors using auxiliary tasks and the quality of transfer from one person to another. Examples of transfer in scenarios with the same person and from one person to another are shown in terms of quality, as well as the results of interpolation in the trained pose space. Ablation studies in the supplementary material show the effect of various components of the proposed method.

The following is a comparison of the results of the present invention with the results of known methods and systems. The following posture descriptors are considered based on different degrees of teacher involvement.

The proposed invention. 256-dimensional hidden pose descriptors trained in the proposed system.

X2Face. 128-dimensional sample vectors trained in the X2Face transfer system [39].

FAB-Net. The 256-dimensional FAb-Net descriptors [38] are evaluated as representations of the pose. They are related to the proposed invention in that, although they are not human independent, they are also taught without teacher participation on the VoxCeleb2 video collection.

3DMM. The newest 3DMM system is considered [3]. This system extracts decomposed rigid posture, facial expression, and shape descriptor using a deep web. The pose descriptor is obtained by combining the rotation of the rigid pose (represented as a quaternion) and the facial expression parameters (29 coefficients).

The proposed descriptor is trained on the VoxCeleb2 dataset. The X2Face descriptor is trained on the smaller VoxCeleb1 dataset [29], and the FAb-Net on both. 3DMM descriptors are most closely monitored as 3DMM learns from 3D scans and requires a landmark detector (which in turn is trained with a teacher).

In addition, the following head transfer systems are considered based on these pose descriptors:

X2Face. The X2Face system [39] is based on native descriptors and deformation-based transfer.

X2Face +. In this variant, instead of the proposed pose encoder, the authors use the frozen pre-trained X2Face control network (up to the control vector) and leave the rest of the architecture unchanged relative to the proposed one. The identity encoder, the generator that depends on the latent vector of the X2Face pose and the assumed identity nesting, and the projection discriminator are trained.

FAB-Net +. Same as X2Face +, but with frozen FAb-Net instead of the suggested pose encoder.

3DMM +. Same as X2Face +, but with frozen Exp-Net [3] instead of the proposed pose encoder and with pose add-ons disabled. The pose descriptor is built from the ExpNet outputs as described above.

In the proposed invention, these 35-dimensional descriptors are additionally normalized to the element-wise mean and standard deviation calculated on the VoxCeleb2 training set.

FSTH. The original few-shot talking head system [42] driven by rasterized cue points.

FSTH +. The authors retrain the system [42] with several changes that make it more comparable to the proposed system and other basic aspects. The raw coordinates of key points are entered into the generator using the AdaIN mechanism (as in the proposed system). The generator predicts segmentation along with the image. The authors also use the same cuts, which are different from [42].

To understand how good the trained pose descriptors are when juxtaposing different people in the same pose, the Multi-PIE dataset [13] is used, which is not used to train any of the descriptors, but contains six annotations of emotion classes for people in different poses. This dataset is limited to near-frontal and semi-profile camera orientations (namely: 08_0, 13_0, 14_0, 05_1, 05_0, 04_1, 19_0), so there are 177,280 images left. In each camera orientation group, the authors randomly select a request image from it and select the nearest N images from the same group using the cosine similarity of descriptors. If a person with the same emotion tag is returned in response, the match will be correct.

This procedure is repeated 100 times for each group. Table 1 shows the overall ratio of correct matches in the top 10, top 20, top 50, and top 100 lists. For the 3DMM descriptor, the authors consider only 29 facial expression coefficients and ignore information about rigid posture as irrelevant to emotions.

Table 1. Accuracy of search results based on pose (expression) using various pose descriptors on the Multi-Pie dataset. See text for details.

Accuracy for top N queries (%) Descriptor N = 10 N = 20 N = 50 N = 100 Fab-Net 45.7 40.8 36.6 35.7 3DMM 47.3 45.6 41.9 41.1 X2Face 61.0 55.7 51.8 49.4 Ours 75.7 63.8 57.8 54.1

In this comparison, you can see that the hidden space of the proposed pose attachments is better grouped by emotion class than other facial expression descriptors, since the proposed result is much better for the top 10 and top 20 metrics, although it is similar to X2Face and better than the rest. for the top 50 and top 100. The vectors FAb-Net and X2Face contain identity information, so they are more likely to be close to vectors representing the same or similar person. As far as 3DMM is concerned, different hidden expression vectors are required to transform different shapes (people) into the same facial expression by design; therefore, the expression coefficients can easily match for different people showing different facial expressions.

Keypoint regression is not included in the proposed target applications, as keypoints contain human-specific information. However, this is a popular problem with which uncontrolled pose descriptors have been compared in the past, so the proposed method is performed on a standard benchmark in the MAFL test set [25]. To predict key points, a ReLU MLP with one hidden layer of size 768 is used, and in the proposed case, the authors use both pose and identity nesting as input. Using a standard normalized ocular basis, the authors obtain a distance error of 2.63. This is less than the 3.44 error obtained by FAb-Net, although it exceeds the level (2.54) achieved in [19] for this problem.

. Все модели, кроме X2Face, дообучаются по этим 32 кадрам за 600 шагов оптимизации. Используя эти модели, авторы вычисляют две метрики для каждой системы: ошибку идентичности

и ошибку реконструкции позы

.Quantitative assessment. The performance of the seven transfer systems listed above in the transfer scheme from one person to another is compared. To do this, 30 people are randomly selected from the VoxCeleb2 test group and the T1 talking head models are trained on them; ...; T30. Each Tk model is created from 32 random video frames

... All models, except for X2Face, are retrained with these 32 frames in 600 optimization steps. Using these models, the authors compute two metrics for each system: identity error

and posture reconstruction error

...

оценивает, насколько близко полученные говорящие головы похожи на подлинного человека k, для которого обучалась модель. Для этого используется сеть R распознавания лица ArcFace [7], которая выдает дескрипторы идентичности (векторы). Авторы вычисляют усредненный эталонный дескриптор

из датасета дообучения

и используют косинусное сходство (csim), чтобы сравнить его с дескрипторами, полученными из результатов переноса с одного человека на другого. Перенос с одного человека на другого выполняется путем прогона Tk со всеми другими 29 людьми. Чтобы получить окончательную ошибку, усредняются сходства (минус одно) для всех 30 человек в тестовом наборе.Identity error

evaluates how closely the received talking heads resemble a real person k for whom the model was trained. For this, the face recognition network R ArcFace [7] is used, which produces identity descriptors (vectors). Authors calculate the averaged reference descriptor

from the retraining dataset

and use cosine similarity (csim) to compare it to descriptors derived from transfer from one person to another. Transferring from one person to another is done by running Tk with all the other 29 people. To get the final error, the similarities (minus one) are averaged for all 30 people in the test set.

Formally,

предназначена для количественной оценки того, насколько хорошо система воспроизводит позу и выражение лица образца, и определяется в виде лицевых ориентиров. Поскольку наборы ориентиров можно сравнивать напрямую только для одного и того же человека, тестовый датасет ограничен парами самопереноса, т.е. прогон осуществляется только для Tk с Ik. Однако, поскольку Tk обучен на

, то во избежание переобучения авторы используют другие 32 удержанных кадра из того же видео

. Используется готовый алгоритм прогнозирования двухмерных лицевых ориентиров L [2] для получения ориентиров как в образце

, так и в результате переноса

.On the other hand, posture reconstruction error

is designed to quantify how well the system reproduces the pose and facial expression of the sample, and is defined in terms of facial landmarks. Since landmark sets can only be compared directly for the same person, the test dataset is limited to self-transfer pairs, i.e. the run is carried out only for Tk with Ik. However, since Tk is trained on

, then in order to avoid retraining, the authors use the other 32 retained frames from the same video

... A ready-made algorithm for predicting two-dimensional facial landmarks L [2] is used to obtain landmarks as in the sample

, and as a result of the transfer

...

того, насколько близко ориентиры

приближаются к эталонным ориентирам

, является среднее расстояние между соответствующими ориентирами, нормированное на глазной базис. Как и ранее, вычисляется d для всех образцов и среднее для всех 30 человек:In the proposed case, the measure

how close the landmarks are

approaching benchmarks

, is the average distance between the corresponding landmarks, normalized to the ocular basis. As before, d is calculated for all samples and the average for all 30 people:

и ошибке реконструкции позы

, соответственно, которые обе вычисляются из подмножества тестовой части датасета VoxCeleb2. Эти метрики подробно описаны ниже в разделе "Подробное описание" в подразделе "Количественная оценка". Каждая точка на графике соответствует известной системе переноса головы, которая сравнивается с данной предложенной системой. График на фиг. 3 содержит эти две метрики, оцененные для сравниваемых моделей. Идеальная система Т имела бы

, т.е. чем ближе к нижнему левому углу, тем лучше. В этих условиях предлагаемая полная модель явно лучше всех систем кроме FSTH+, которая немного лучше по одной из метрик, но намного хуже по другой, и выигрыш которой обусловлен использованием внешнего детектора ключевых точек.FIG. 3 shows the assessment of the transfer systems in terms of their ability to represent the pose of the specimen and maintain the initial identity (arrows indicate improvement). Horizontal and vertical axes correspond to identification error

and posture reconstruction error

, respectively, which are both calculated from a subset of the test part of the VoxCeleb2 dataset. These metrics are detailed below in the "Detailed Description" section in the "Quantification" subsection. Each point on the graph corresponds to a known head transfer system, which is compared to this proposed system. The graph in FIG. 3 contains these two metrics assessed for the compared models. An ideal system T would have

, i.e. the closer to the bottom left corner, the better. Under these conditions, the proposed complete model is clearly better than all systems except FSTH +, which is slightly better in one of the metrics, but much worse in the other, and the gain of which is due to the use of an external key point detector.

Qualitative comparison. FIG. 4 shows a comparison of transfer from one person to another for several systems on the VoxCeleb2 test set. The top left image is one of 32 frames of the identity source, the top left image shows one of the images defining the target identity from the VoxCeleb2 test portion, i.e. a person whom it would be desirable to visualize with a different facial expression and head posture. The rest of the images in this row are also taken from the test section of VoxCeleb2, and they define the pose that should be transmitted to the target person. The other images in the top row are examples of facial expressions and head posture. The proposed method better preserves the identity of the target person and successfully transmits facial expressions from the person-sample. Each of the remaining rows corresponds to some method of neural network translation of facial expression and head posture, which is compared with the "Proposed invention", which is the invention, and other methods corresponding to those shown in FIG. 3. The transfer results obtained by the corresponding method are shown.

FIG. 4 shows a qualitative comparison of the transfer systems described above. It can be seen that the FSTH system, guided by rasterized landmarks, relies heavily on the proportions of the specimen's face and therefore is not independent of humans. Its modified version of FSTH + gives a better result, as it has more representative power around the vectorized keypoints; however, there is a noticeable "identity leak" (eg, compare head widths in columns 1 and 2) and errors in meaningful facial expressions, eg closed eyes. X2Face's deformation-based method has already failed in small turns.

Two similar methods, X2Face + and FAb-Ne +, provide a convincing baseline despite some signs of identity discrepancy, such as eyeglass marks in column 7 and long hair emerging from the facial expression pattern and head posture in column 5. It is important to note that, although the pose descriptors from these methods are not human-independent, during training, pose additions are still applied. The ablative study described below demonstrates that the effectiveness of transference from one person to another is drastically reduced when posture additions are removed in the two methods.

The 3DMM + method has a very bottleneck in interpreted parameters and therefore its identity discrepancy is very small. However, apparently for the same reason, this method is not so good at rendering correct subtle facial expressions. The proposed complete system is capable of accurately representing the facial expression and head posture of the specimen while maintaining the identity of the target person.

In addition, the authors also show translation by interpolation in pose space for the proposed system in FIG. 5, which demonstrates smooth changes in posture. FIG. 5 shows translation by interpolation between two pose vectors along a spherical path in pose descriptor space. Each row shows the results of the transfer according to the proposed invention for a certain person from the test portion of VoxCeleb2. The first and last images in each row are computed using some of the pose source frames from the VoxCeleb2 test portion (not shown in the figure) displayed with the pose encoder in attachments A and B, respectively. The images in between are obtained by the normal implementation of the present invention, except that the pose nesting is obtained by spherical interpolation between A and B, and is not calculated by the pose encoder from some image. Specifically, the pose nesting in columns 2-5 is slerp (A, B; 0.2), slerp (A, B; 0.4), slerp (A, B; 0.6), slerp (A, B; 0.8).

FIG. 6 shows an additional comparison of the transfer from one person to another for several systems on the VoxCeleb2 test set. The top left image is one of the images defining the target identity from the VoxCeleb2 test portion, i.e. the person to be visualized with a different facial expression and head posture. The rest of the images in this row are also taken from the test part of VoxCeleb2 and define the pose that should be transmitted to the target person.

Each of the remaining rows corresponds to some method of neural network translation of facial expression and head posture, which are compared to the "Proposed invention", which is the invention, and other methods corresponding to those shown in FIG. 3. The transfer results obtained by the corresponding method are shown.

Temporary smoothness. An additional video demonstrates the ability of the proposed descriptor to create time-smoothed hyphenation without any temporal smoothing of the extracted pose (assuming the results of the bounding box detection are time-smoothed). At the same time, achieving time-smoothed transfer using key point driven systems (FSTH, FSTH +) requires a lot of key point smoothing.

Reducing the dimension of the pose vector, increasing the performance of the pose encoder, preserving the background in images, and eliminating pose augmentation are discussed.

The proposed best retraining model is implemented with different subsets of these changes, and an attempt is made to remove the pose add-ons from X2Face + and FAb-Net +. X2Face + and FAb-Net + have been modified by removing the posture add-ons, and the observed effects are discussed below.

All the considered models obtained are shown in Table 2. They are compared both in quantitative and qualitative aspects, as in Section 4.3.

The proposed study examines four ablation sizes that correspond to the columns in Table 2.

Table 2. Summary table of the systems compared in the ablation study

Model name Pose vector
dim, dp Pose encoder Erased background Pose addition Ours 256 MobileNetV2 + + -PoseDim 64 + + -Augm 256 + -Segm 256 + + PoseEnc 256 ResNeXt-50
(32 × 4d) + + + PoseEnc-Augm 256 + + PoseEnc-Segm 256 + X2Face + 128 X2Face (pretrained) + + X2Face + (- Augm) 128 + FAb-Net + 256 FAb-Net (pretrained) + + FAb-Net + (- Augm) 256 +

The dimension of the pose vector is dp. First, dp is reduced from 256 to 64 simply by changing the number of channels in the last trainable layer of the pose encoder. The proposed basic model (invention), but with pose vectors limited to size 64, is labeled in FIG. 7 as -PoseDim. FIG. 7 presents a quantitative assessment of how the ablation of several important features of the training scheme affects the proposed system. In addition, the effect of posture supplementation during training for X2Face + and FAb-Net + is illustrated.

Intuitively, a bottleneck like this should both limit the ability to represent different poses and induce the generator to extract human-specific information from a more meaningful identity nesting. According to the presented graph, the posture reconstruction error does increase slightly, while the system remains equally independent of the person. However, this difference in posture is qualitatively negligible.

Pose encoder performance. Second, the pose encoder is replaced by a stronger network, namely ResNeXt-50 (32 X 4), which makes it equal to the identity encoder (i.e., with the same architecture). The suggested best model with this modification is designated + PoseEnc. As noted above, the pose encoder and identity encoder are intentionally unbalanced, so the pose encoder is weaker and thus forces the optimization process to prefer extracting human-specific information from the identity source frames rather than from the sample frame. Both the metrics and hyphenation patterns for + PoseEnc suggest that this idea was not meaningless: a more powerful pose encoder begins to convey human-specific cues from the facial expression pattern and head posture. FIG. 8, it can be seen that the + PoseEnc result is influenced by the clothes of sample # 1, hair of samples # 6- # 8, and the face shape of sample # 9. This is also confirmed by the very large increase in the identity error in FIG. 7. On the other hand, such a system reconstructs the pose with higher accuracy, which may indicate that it may be a better choice for self-transference if the "identity leak" is not so relevant.

Erasing the background. Third, the proposed system is modified so that it does not predict foreground segmentation, does not use segmentation to calculate loss functions, and thus becomes a teacher-less system. "Our plus" in this change is Segm. The pose encoder in such a system wastes its productivity on encoding the background of the sample image rather than evaluating the subtle details of the facial expression. This is because the perceptual loss functions are too sensitive to discrepancies between the generated and targeted backgrounds compared to differences in facial expressions. More importantly, since the background often changes within the video, it is too difficult to reconstruct the background of the target image simply by looking at the images of the identity source. Therefore, in the optimization algorithm, it is tempting to transfer the work of the identity encoder to the pose encoder. This is evident from the graph and samples, in which the introduction of the background makes a large contribution to the identity discrepancy, and is even more noticeable when combined with a stronger pose encoder (+ PoseEnc-Segm model).

Supplementing the pose. The model has been retrained without arbitrary additions to the pose, i.e. A is set to transform identity. In this scheme, the system is trained to accurately reconstruct the image of the facial expression pattern and head posture, and therefore is more likely to degrade to an autoencoder (assuming the posture encoder is trained along with the entire system).

As best seen in FIG. 7 (Proposed invention! -Augm, + PoseEnc! + PoseEnc-Augm), despite the additional improvement in the ability to represent poses, it also significantly harms the preservation of identity. In fact, the system with the powerful ResNeXt-50 pose encoder trained without posture additions (+ PoseEnc-Augm) turned out to be the worst of the proposed models in terms of PT, but at the same time the best model in terms of posture transfer quality. Again, such a model can be very useful for self-transfer, but not at all suitable for creating a "puppet" (transfer from one person to another). However, even with self-transfer, care must be taken as this model can produce undesirable effects such as image quality rendering (eg, from sample # 8 in FIG. 8). FIG. 8 shows a comparison of the transfer from one person to another for the proposed best model and its ablated versions. The top left image in FIG. 8 is one of the images defining the target identity from the VoxCeleb2 test part, i.e. the person the authors would like to visualize with a different facial expression and head posture. The rest of the images in this row are also taken from the test part of VoxCeleb2 and determine the pose to be transmitted to the target person.

This effect is further confirmed by removing the pose add-ons from X2Face + and FAb-Net + (suffix (-Augm) added to each model). With voluntary additions enabled, despite the human-specific nature of the X2Face and FAb-Net pose descriptors, the generator still develops resistance to human-specific features of facial expression patterns and head posture. However, without the additions, the degree of "identity leakage" is fully explained by the specific identity of these ready-made descriptors. In addition, the pose reconstruction error should be reduced if the generator is no longer supposed to be so pattern-resistant, and so some of its performance is freed up and can be used to render more accurate poses. As expected, in FIG. 7 shows a significant increase in identity error and a sharp decrease in posture error for the two models. This proves once again that the X2Face and FAb-Net descriptors are not human independent. In addition, in FIG. 9, you can clearly see the discrepancy of identity. FIG. 9 illustrates the effect of posture additions on the X2Face + and FAb-Net + models. Without additions, the identity divergence becomes noticeable. The top left image in FIG. 9 is one of the images defining the target identity from the VoxCeleb2 test part, i.e. the person the authors would like to visualize with a different facial expression and head posture. The rest of the images in this row are also taken from the test part of VoxCeleb2 and determine the pose to be transmitted to the target person.

Each of the other rows corresponds to some comparable method of neural network transfer of facial expression and head posture. These compared methods are listed in the bottom four rows of Table 2, namely X2Face + and Fab-Net +, each scored with and without pose additions. The transfer results from the corresponding method are shown, for example glasses from sample # 7 or the face shape from sample # 1 are transferred to the result.

и ошибкой реконструкции позы

. Этот компромисс регулируется путем применения вышеуказанных изменений в зависимости от того, который из сценария самопереноса или сценария переноса с образца одного человека на другого более важен. Во втором случае лучше использовать предложенную лучшую модель "PROPOSED", тогда как для первого сценария хорошим кандидатом может быть +PoseEnc или +PoseEnc-Segm.In conclusion, it should be noted that there is a trade-off between the identity preservation error

and posture reconstruction error

... This trade-off is adjusted by applying the above changes depending on which of the self-transfer scenario or the transfer scenario from one person's sample to another is more important. In the second case, it is better to use the suggested best model "PROPOSED", while for the first scenario, + PoseEnc or + PoseEnc-Segm might be a good candidate.

Above, a neural network transfer of facial expressions and head postures was described and analyzed, which uses hidden posture descriptors and which allows you to achieve a realistic transfer. In contrast to the previous system [42], in which key points were used as the pose descriptor, the proposed system uses pose descriptors without the explicit participation of the teacher, solely on the basis of reconstruction losses. The only weak form of teacher involvement comes from segmentation masks. The proposed trained head pose posture descriptors are superior to previous descriptors without teacher involvement in posture-based search tasks as well as transference from one person to another.

The main and perhaps unexpected conclusion is that the limited performance of the pose extraction network in the proposed scheme is sufficient to separate pose and identity. At the same time, it is possible that the appropriate use of cyclic and / or adversarial losses can further improve the separation.

Possibly due to limited network performance, the proposed pose descriptors and the transfer system have problems capturing some subtle facial expressions, especially gaze direction (although this still works better than cue point descriptors, which have no gaze representation at all). Another obvious area of research is the training of the pose descriptor and the whole system with partial involvement of the teacher.

At least one of the plurality of blocks may be implemented through the AI model. AI related function can be performed via nonvolatile memory, volatile memory and processor.

A processor can include one or more processors. However, one or more processors may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processor such as a graphics processing unit (GPU), a machine vision processor (VPU), and / or a specialized AI processor such as a neural processor (NPU).

One or more processors control the processing of input data in accordance with a predefined work rule or artificial intelligence (AI) model stored in nonvolatile memory and volatile memory. A predefined model of a work rule or artificial intelligence is provided through teaching or learning.

In this context, provision by training means that by applying a training algorithm to a set of training data, a predefined working rule or AI model of the desired characteristic is created. This training can be performed on the device itself, in which the AI is performed in accordance with the embodiment, and / or can be implemented through a separate server / system.

An AI model can be composed of multiple layers of a neural network. Each layer has multiple weights and performs a layer operation by calculating the previous layer and a multi-weight operation. Examples of neural networks include, but are not limited to, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Trust Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN) ), generative adversarial networks (GANs), and deep Q-networks.

A learning algorithm is a method of training a given target device (eg, a robot) using a plurality of training data to induce, permit, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, part-time learning, or reinforcement learning.

According to this disclosure, in an electronic device, using the facial expression and head posture recognition method, it is possible to obtain output that recognizes an image or facial expression and head posture in an image using the image data as input to an artificial intelligence model. An artificial intelligence model can be obtained through training. In this context, "obtain by training" means that a predefined work rule or artificial intelligence model, configured to perform a desired function (or goal), is obtained by training a basic artificial intelligence model on multiple pieces of training data using a training algorithm. This artificial intelligence model can include many layers of a neural network. Each of the plurality of neural network layers has a plurality of weights and performs a neural network computation by computation between the result of the computation of the previous layer and the plurality of weights.

Visual awareness is a method of recognizing and processing objects similar to human vision, and it includes, for example, object recognition, object tracking, image search, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

In this case, the described method performed by an electronic device can be performed using an artificial intelligence model.

The above described exemplary embodiments of the invention are only examples and should not be construed as limiting. The description of the exemplary embodiments is also intended to be illustrative, and not to limit the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

List of sources

[1] V. Blanz, T. Vetter, et al. A morphable model for the synthesis of 3d faces. In Proc. SIGGRAPH, volume 99, pages 187-194, 1999,

[2] A, Bulat and G. Tzimlropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In Proc. ICCV, pages 1021-1030,2017.

[3] F.-J. Chang, A. T. Tran, T. Hassner, I. Masi, R. Nevatia, and G. Medioni, Expnet: Landmark-free, deep, 3d facial expressions. In In proc. FG, pages 122-129. IEEE, 2018.

[4] J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. In INTERSPEECH, 2018,

[5] J. S. Chung, A. Senior, 0. Vinyals, and A. Zisserman. Lip reading sentences in the 'wild. In Proc. CVPR, pages 3444-3453. IEEE, 2017.

[6] T. F. Cootes, G. J. Edwards, and C. J. Taylor. Active appearance models, T-PAMI, (6): 681-685, 2001.

[7] J. Deng, J. Guo, X. Niannan, and S, Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR,

[8] E. L. Denton and V, Birodkar. Unsupervised learning of

disentangled representations from, video. In Proc. NeurlPS, pages 4414-4423, 2017.

[9] P. Ekman. Facial action coding system. 1977.

[10] C. Fu, Y. Ni, X. Wu, G, Wang, Q. Zhang, and R. He. High fidelity face manipulation with extreme pose and expression. arXiv preprint arXiv: 1903.12 00 3, 2019,

[11] K. Gong, Y. Gao, X. Liang, X. Shen, M. Wang, and L. Lin. Graphonomy: Universal human parsing via graph transfer learning. In CVPR, 2019.

[12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Proc. NIPS, 2014.

[13] R. Gross, I. Matthews, J. Cohn, T, Kanade, and S. Baker. Multi-pie. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, IEEE Computer Society, September 2008.

[14] R. A. G "uler, N. Neverova, and I. Kokkinos. DensePose: Dense human pose estimation in the wild. In Proc. CVPR, June 2018.

[15] R, A. G "uler, G. Trigeorgis, E, Antonakos, P. Snape, S. Zafeiriou, and I. Kokkinos. DenseReg: Fully convolutional dense shape regression in-the-wild. In CVPR, volume 2 , page 5, 2017,

[16] X. Huang and S. Belongie. Arbitrary style transfer in realtime with adaptive instance normalization. In Proc. ICCV, 2017.

[17] X. Huang, M, -Y. Liu, S. Belongie, and J. Kautz, Multimodal unsupervised image-to-image translation. In Proc. ECCV, 2018.

[18] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML, ICML'15, pages 448-456, 2015.

Unsupervised learning of object landmarks through conditional image generation. In Proc. NeurlPS, pages 4016-4027, 2018.

[1] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proc. CVPR, June 2019.

[2] V. Kazemi and J, Sullivan. One millisecond face alignment with an ensemble of regression trees. Proc. CVPR, pages 1867-1874, 2014.

[3] H. Kim, P. Garrido, A. Tewari, W, Xu, J. Thies, M. Niefiner, P. P'erez, C. Richardt, M. Zollh "ofer, and C. Theobalt. Deep video portraits. In Proc. SIGGRAPH, 2018,

[4] H, Kim and A. Mnih. Disentangling by f act or i sing. In Proc. ICML, pages 2654-2663, 2018.

[5] M. Liu, X. Huang, A. Mallya, T. Karras, T. Aila, J. Lentinen, and J. Kautz, Few-shot unsupervised image-to-image translation. In Proc. ICCV, 2019.

[6] Z. Liu, P. Luo, X. Wang, and X. Tang, Deep learning face attributes in the wild. In Proc. ICCV, December 2015.

[26] S. Lombardi, J. Saragih, T. Simon, and Y. Sheikh. Deep appearance models for face rendering. ACM Transactions on Graphics (TOG), 37 (4): 68, 2018.

[1] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentationоn. pages 565-571, 10 2016.

[2] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.

[3] A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: a large-scale speaker identification dataset. In INTERSPEECH, 2017.

[4] A. Pumarola, A. Agudo, A. M. Martinez, A. Sanfeliu, and F. Moreno-Noguer. Ganimation: Anatomically-aware facial animation from a single image. In Proc. ECCV, pages 818 --- 833, 2018.

[5] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. CVPR, June 2,018.

[6] A. Siaronin, S. Lathuili'ere, S. Tulyakov, E, Ricci, and N. Sebe. First order motion model for image animation. In Proc. NeurlPS, pages 7135-7145, 2019.

[7] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman. Synthesizing Obama: learning lip sync from, audio. ACM Transactions on Graphics (TOG), 36 (4): 95, 2017.

[34] J. Thewlis, S. Albanie, H. Bilen, and A. Vedaldi. Unsupervised learning of landmarks by descriptor vector exchange. In Proc. ICCV, 2019.

[1] S. Tripathy, J. Kannala, and E. Rahtu. Icface: Interpretable and controllable face reenactment using gans. CoRR, abs / 1904.01909, 2019.

[2] T. Wang, M. Liu, A. Tao, G. Liu, J. Kautz, and B. Catanzazo, Few-shót video-to-video synthesis. CoRR, abs / 1910.12713, 2019.

[3] T. -C. Wang, M. - Y. Liu, J. - Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro, Video-to-video synthesis. Proc. NeurlPS, 2018.

[4] O. Wiles, A. Koepke, and A. Zisserman. Self-supervised learning of a facial attribute embedding from video. In Proc. BMVC, 2018.

[5] O. Wiles, A. Sophia Koepke, and A. Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proc. ECCV, September 2018.

[6] F. Xiao, P. Liu, and Y. J. Lee. Identity from here, pose from there: Self-supervised disentanglement and generation of objects using unlabeled videos. In Proc. ICCV, October 2019.

[7] S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proc.CVPR, July 2017.

[8] E. Zakharov, A. Shysheya, E. Burkov, and V. Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In Proc. ICCV, October 2019.

[9] S, Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, S3fd: Single shot scale-invariant face detector. In Proc. ICCV, Oct 2017.

[10] Y. Zhang, Y. Guo, Y. Jin, Y. Luo, Z. He, and H. Lee. IJnsupervised discovery of object landmarks as structural representations. In Proc. CVPR, pages 2694-2703, 2018.

[11] J.-Y. Zhu, T. Park, P. Isola, and A. A, Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks.In Proc.ICCV, 2017.

Claims (18)

1. A hardware device for transferring an arbitrary facial expression and head posture to a person's avatar while maintaining the identity of a person's avatar, containing: a detector for capturing images of facial expressions and head posture of person B; a means for selecting images of a person A; an identity encoder unit configured to obtain an identity descriptor from images of a person A, wherein the posture encoder output does not contain information about the identity of person A; a posture encoder unit configured to input images of a person B and configured to obtain a head posture descriptor and a facial expression from an image of person B, wherein the output of the pose encoder unit does not contain information about the identity of person B; a generator block receiving outputs of the identity encoder and posture encoder blocks, the generator block being configured to synthesize an avatar image of a person A having a head pose and facial expression taken from a person B; moreover, the generator block is configured to output the resulting image of an avatar of a person A having a head pose and facial expression taken from a person B. 2. The hardware device of claim 1, wherein the pose encoder block is a convolutional neural network that takes an image of a person as input and outputs a vector describing his head pose and facial expression and not describing their identity, and which is applied to the images arbitrary people, including people whose images were not included in the training set. 3. The hardware device of claim 1, wherein the identity of person B means the skin color, face shape, eye color, clothing and jewelry of person B. 4. The method of operation of the hardware device according to claim 1 for transferring an arbitrary facial expression and head posture to a person's avatar while maintaining the identity of the person's avatar, which consists in the following: capture, by means of the detector, images of facial expressions and postures of the head of person B; select images of person A; obtaining, using the identity encoder unit, an identity descriptor from images of person A; obtain, using the pose encoder unit, a descriptor of a head pose and a facial expression from images of person B; synthesized by means of a generator block that accepts the outputs of the identity encoder and posture encoder blocks, an image of an avatar of a person A with a head pose and a facial expression taken from a person B, outputting the resulting image of the avatar of person A, having a head pose and facial expression taken from person B. 5. The method according to claim 4, in which the pose encoder block is a convolutional neural network that accepts an image of a person as input and outputs a vector describing the head pose and facial expression and not describing their identity, and is applicable to images of arbitrary people, in including people whose images were not included in the training sample. 6. The method of claim 4, wherein identity means skin color, face shape, eye color, clothing, jewelry.

Images