RU2720361C1 - Multi-frame training of realistic neural models of speakers heads - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: present invention relates to computer engineering. Method comprises steps of: receiving at least one video frame from a single sequence of frames of said person; evaluating head orientation and facial expression descriptors for at least one video frame; computing an N-dimensional embedding vector which characterizes a person-specific information, based on at least one video frame using a meta-trained embedded network; implementing a generator network based on a set of parameters of a meta-trained generator network and an N-dimensional embedding vector; and fine tuning network parameters of generator to match at least one video frame, when the generator network is provided with head orientation and facial expression descriptors, synthesising a sequence of images of the talking persona head for said person using a finely tuned generator network and a previously unobserved sequence of head orientation and facial expression descriptors.

EFFECT: technical result consists in improvement of quality of synthesized images.

9 cl, 5 dwg

Description

Technical field

[0001] This application relates to a method and apparatus for training in several frames (few – shot learning) (photographs) of realistic neural models of the heads of talking persons, as well as to a method and apparatus for creating personalized photorealistic sequences of images of the head of a speaking person, i.e. . software and / or hardware modules that can synthesize sequences of plausible videos of the head of a particular individual during an emotional conversation and facial expressions. Computer-readable media that store computer-executable instructions are also disclosed that, when executed by a processor of a device, cause the processor to perform any one of the disclosed methods.

State of the art

[0002] This application considers the task of creating personalized photorealistic models of the head of a talking person, i.e. systems that can synthesize sequences of plausible videos of emotions and facial expressions when talking to a specific individual. More specifically, the problem of synthesizing photorealistic personalized images of the head is considered, with a given set of recognizable elements (landmarks) of the face that stimulate the animation of the model. This ability has practical applications for telepresence (telepresence), including video conferencing and multiplayer games, as well as the special effects industry. It is known that synthesizing sequences of realistic images of the head of a talking person is difficult for two reasons. Firstly, human heads are characterized by high photometric, geometric and kinematic complexity. This complexity is caused not only by face modeling (for which there are a large number of modeling methods), but also by modeling the oral cavity, hair and clothing. The second factor of complexity is the sharpness of the human visual system, even to the smallest errors in modeling the appearance of people's heads (the so-called “sinister valley effect” [24]). Such a small tolerance for modeling errors explains the fact that currently non-photorealistic caricature-like avatars prevail in many practically deployed teleconferencing systems.

Related work

[0003] To overcome the aforementioned problems, a number of works were presented on synthesizing related sequences of images of the head by deforming a single or several static frames. Both classical deformation algorithms [5, 28] and deformation fields synthesized using machine learning (including deep learning) [11, 29, 40] can be used for this purpose. Although deformation-based systems can create sequences of images of the head of a talking person even from a single image, the amount of movement, head rotation, and elimination of overlap (de-occlusion) that they can process without noticeable artifacts is limited.

[0004] Direct (no deformation) synthesis of video frames using competitively-trained deep convolutional networks (ConvNets) presents a new solution that has hopes for creating photorealistic images of the head of a talking person. Recently, some very realistic results have been demonstrated by such systems [16, 20, 37]. However, in order to succeed, such methods must train large networks where both the generator and the discriminator have tens of millions of parameters for each head of the talking person. Therefore, such systems require a few minutes of video [20, 37] or large datasets of photographs [16], as well as GPU training hours to create a new personalized model of the head of a talking person. Although such efforts are less than those required by systems that construct photorealistic models of the head of a talking person using complex physical and optical modeling [1], they are still exorbitant for most practical telepresence scenarios, where it is desirable to enable users to create their personalized models the heads of the talking person with the least effort possible.

[0005] A huge number of works are devoted to statistical modeling of the appearance of human faces [6], and very good results were obtained both by classical methods [35] and, recently, with the help of deep learning [22, 25] (only as a few examples). Although facial modeling is a task quite closely related to modeling the head of a speaking person, both of these tasks are not identical, since the latter also involves modeling non-facial parts such as hair, neck, oral cavity and often shoulders / outerwear. These non-face parts cannot be handled by some trivial extension of face modeling methods, since they are much less amenable to precise alignment and often have higher variability and higher complexity than face parts. In principle, the results of face modeling [35] or lip modeling [31] can be “sewn” into an existing head video. However, such a design does not allow full control over the rotation of the head in the resulting video and therefore does not result in a fully developed system for modeling the head of a talking person.

[0006] The model-insensitive learning module (MAML) [10] uses meta-training to obtain the initial state of the image classifier, from which it can quickly convert to class image classifier without preparation if several training samples are specified. This high-level idea is used in the manner proposed in this document, although its implementation is rather difficult. Several papers were presented that combine competitive learning with meta-learning. So, GAN with increasing data [3], meta-GAN [43], adversarial meta-learning [41] use ad-hoc trained networks to generate additional examples for classes not observed at the meta-learning stage. While these methods are focused on improving the classification efficiency for several frames, the method disclosed herein is aimed at training image generation models using competitive goals. As a result, in the present invention, adversarial fine-tuning is introduced into the meta-learning infrastructure. The first is applied after the initial state of the generator and discriminator networks is obtained through the meta-learning stage.

[0007] Finally, the last two works relate to generation from text to speech [4, 18]. Their purpose (training on several frames of generative models) and some of the components (autonomous integration network, fine tuning of the generator) are also used in the present invention. Nevertheless, the differences of the present invention are at least in the field of application, the use of competitive training, its specific adaptation to the meta-learning process and numerous implementation details. These differences are described in detail below in the section “Detailed Description of the Invention”.

SUMMARY OF THE INVENTION

[0008] The present application discloses a method, device and system for creating models of the head of a speaking person from a small number (“handful") of photographs (the so-called multi-frame training ). In fact, the proposed system can generate an acceptable result based on a single photograph ( training on a single frame ), although adding a few more photographs increases the accuracy of personalization. Like [16, 20, 37], “talking heads” created by the model disclosed here correspond to deep ConvNets, which synthesize video frames in a direct way through a sequence of convolution operations, rather than through deformation. The “talking heads” created by the system disclosed herein can therefore handle a wide variety of postures that go beyond the capabilities of deformation based systems.

[0009] The ability to learn in several frames was obtained through extensive preliminary training ( meta-learning, meta-learning ) on a large body (set) of videos of the heads of speaking people corresponding to different speakers with different appearance. During meta-training, the proposed system simulates learning tasks in several frames and learns to transform the positions of landmarks into realistic looking personalized photographs, given a small training set of images of a given person. After that, a small set of photographs of a new person poses a new problem of competitive training with a high-performance generator and discriminator, previously trained through meta-training. A new adversarial problem converts into a state that generates realistic and personalized images after several stages of training.

[0010] The architecture of the proposed system borrows much of the latest advances in generative image modeling. Thus, the architecture proposed in this document uses adversarial learning and, more specifically, the ideas underlying conditional discriminators, including projection discriminators. The meta-learning stage uses an adaptive mechanism to normalize an instance (sample), which has proved to be useful in large-scale conditional generation tasks. Thus, the present invention provides improved quality of synthesized images and eliminates the sinister valley effect from such images.

Brief Description of the Drawings

[FIG. 1] FIG. 1 illustrates a proposed meta-learning architecture in accordance with an embodiment of the present invention.

[FIG. 2] FIG. 2 illustrates an embodiment of a method for synthesizing a sequence of images of the head of a talking person for an arbitrary person in accordance with the present invention.

[FIG. 3] FIG. 3 illustrates an embodiment of training on multiple frames of a generator network in accordance with the present invention.

[FIG. 4] FIG. 4 illustrates an embodiment of meta-training of a generator network in accordance with the present invention.

[FIG. 5] FIG. 5 illustrates a simplified block diagram of a computing device that is capable of performing a method for synthesizing a head model of a talking person for an arbitrary person in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

обозначает

–ую последовательность видео и

обозначает

–ый кадр видео упомянутой последовательности видео. Во время мета–обучения, а также во время тестирования, местоположения ориентиров лица предполагаются доступными для всех кадров (стандартный код совмещения лиц [7] может быть использован для получения местоположений ориентиров лица). Ориентиры растризованы (преобразованы в растровый формат) в трехканальные изображения (т.е. изображения ориентиров лица) с использованием предопределенного набора цветов, чтобы связывать определенные ориентиры с линейными сегментами.

обозначает полученное в результате изображение ориентиров лица, вычисленное для

.[0011]Architecture and designation.Stage meta-training of the proposed method involves accessibilityMvideo sequences containing images of the heads of various talking persons.

denotes

Sequence of video and

denotes

–– video frame of the mentioned video sequence. During meta-training, as well as during testing, the locations of landmarks are assumed to be accessible to all frames (the standard face registration code [7] can be used to obtain locations of landmarks). Landmarks are rasterized (converted to raster format) into three-channel images (i.e. face landmark images) using a predefined set of colors to associate certain landmarks with linear segments.

denotes the resulting face landmark image calculated for

.

[0012] As shown in FIG. 1, the proposed meta-learning architecture uses an embedder network (which embeds the head images (with estimated face references) onto embedding vectors) that contain posture-dependent information, a generator network that maps face input landmarks to output frames through a set of convolutional levels that are modulated by embedding vectors through adaptive sample normalization (AdaIN). Usually, during the meta-learning stage, frames from the same video are passed through the embedder’s network, the resulting embeddings are averaged and used to predict the adaptive parameters of the generator network. Then, the landmarks of another frame are passed through the generator network, and the generated image is compared with control (true) data. The objective function includes losses due to perception (perceptual) and competition, the latter being realized through a discriminator projection network. The meta-learning architecture and related operations are described in more detail below.

[0013] Thus, at the meta-learning stage of the proposed method, the following three networks are trained (collectively referred to as adversarial network or generative adversarial network (GAN)) (see Fig. 1 ):

). Встраиватель сконфигурирован, чтобы брать кадр видео

, ассоциированное изображение ориентиров лица

и отображать эти входы на N–мерный вектор встраивания

. Кадр видео

берется из обучающей последовательности, т.е. из множества M последовательностей видео, содержащих изображения голов говорящих персон, отличающихся от произвольной персоны, для которой позже должна быть синтезирована модель головы говорящей персоны. Здесь,

обозначает параметры встраивателя, которые обучаются во время стадии мета–обучения. В принципе, целью стадии мета–обучения для встраивателя E является обучение

таким образом, что N–мерный вектор встраивания

содержит специфическую для видео информацию (такую как идентичность человека), которая инвариантна к позе и мимике в конкретном кадре s. N–мерные вектора встраивания s, вычисленные встраивателем, обозначаются как

.1. Embedded E (

) Embedded configured to take video frame

, associated face landmark image

and map these inputs to the N-dimensional embed vector

. Video frame

taken from the training sequence, i.e. from the set of M video sequences containing images of the heads of speaking persons different from an arbitrary person, for which a model of the head of the speaking person should later be synthesized. Here,

denotes the parameters of the embedder that are trained during the meta-learning stage. In principle, the goal of the meta-learning stage for Embedded E is to teach

so that the N-dimensional embedding vector

contains video-specific information (such as a person’s identity) that is invariant to posture and facial expression in a particular frame s . N – dimensional embedment vectors s calculated by the embedder are denoted as

.

). Генератор сконфигурирован, чтобы брать изображение ориентиров лица

для кадра видео

, не наблюдавшегося встраивателем E, и соответственно вычисленный N–мерный вектор встраивания

и генерировать синтезированный кадр видео

. Генератор G обучается максимизировать подобие между его выходами (т.е. синтезированными кадрами видео

) и соответствующими истинно верными кадрами. Все параметры генератора G разбиты на два набора: общие для человека параметры

и специфические для человека параметры

. Во время стадии мета–обучения, только общие для человека параметры

обучаются напрямую, в то время как специфические для человека параметры

предсказываются из N–мерного вектора встраивания

во время стадии тонкой настройки (которая подробно описана ниже) мета–обучения с использованием обучаемой матрицы проекции

:

.2. Generator G (

) The generator is configured to take facial landmarks

for frame video

, not observed by the embedder E, and, accordingly, the calculated N – dimensional embedding vector

and generate a synthesized video frame

. Generator G learns to maximize the similarity between its outputs (i.e. synthesized video frames

) and the corresponding truly faithful frames. All parameters of the generator G are divided into two sets: parameters common to humans

and human-specific parameters

. During the meta-training stage, only parameters common to humans

learn directly while human-specific parameters

predicted from the N-dimensional embedding vector

during the fine-tuning stage (which is described in detail below) of meta-training using a trained projection matrix

:

.

). Дискриминатор сконфигурирован, чтобы брать входной кадр видео

, ассоциированное изображение ориентиров лица

и индекс обучающей последовательности

и вычислять оценку реалистичности r (одиночный скаляр). Здесь,

обозначают параметры дискриминатора, которые обучаются во время стадии мета–обучения. Дискриминатор содержит часть V(

) сверточной сети (ConvNet), которая сконфигурирована, чтобы отображать входной кадр видео

и ассоциированное изображение ориентиров лица

на N–мерный вектор. Оценка реалистичности r затем вычисляется дискриминатором на основе упомянутого N–мерного вектора и параметров дискриминатора

. Оценка реалистичности r указывает, является ли входной кадр видео

реальным (т.е. не синтезированным) кадром видео

–ой последовательности видео, и соответствует ли входной кадр видео

ассоциированному изображению ориентиров лица

. Кадр видео

, который вводится в дискриминатор, может быть синтезированным кадром видео

; однако, тот факт, что входной кадр видео

является синтезированным, не известен дискриминатору. 3.Discriminator D(

) The discriminator is configured to take the input frame of the video

, associated face landmark image

and training sequence index

and calculate a realistic estimater(single scalar). Here,

denote the discriminator parameters that are trained during the meta-learning stage. The discriminator contains partV(

) a convolutional network (ConvNet) that is configured to display the input video frame

and associated face landmarks

on theN–measured vector. Realistic Ratingr then calculated by the discriminator based on the aboveN–dimensional vector and discriminator parameters

. Realistic Ratingrindicates whether the input frame of the video

real (i.e. not synthesized) video frame

–Th video sequence, and whether the input frame of the video matches

associated face landmarks

. Video frame

that is introduced into the discriminator may be a synthesized video frame

; however, the fact that the input frame of the video

is synthesized, not known to the discriminator.

и одиночный истинно верный кадр видео

из этой последовательности выбираются случайным образом. В дополнение к

, дополнительные K кадров видео

выбираются из той же самой обучающей последовательности видео

. Затем, во встраивателе E, N–мерный вектор встраивания

вычисляется для обучающей последовательности видео

путем усреднения N–мерных векторов встраивания

, вычисленных для дополнительных K кадров видео следующим образом:[0014]Stage of meta-training. During the meta-learning stage of the proposed method, the parameters of all three networks are trained in an adversarial way. This can be done by modeling episodes.K–Framework training.K was equal to 8 in experiments performed by the authors of the present invention. However, the present invention should not be limited to an embodiment in whichK= 8 sinceKmay be selected by a person skilled in the art to be more or less than 8, depending on the performance of the hardware used for the meta-learning stage, or depending on the accuracy of the images generated by the meta-trained GAN, and the purposes for which meta-training of such a GAN. In each episode, a training video sequence

and single truly true frame video

from this sequence are randomly selected. In addition to

additionalKvideo frames

are selected from the same video training sequence

. Then, in the embedE, N – dimensional embedding vector

calculated for training video sequence

by averaging N – dimensional embedding vectors

calculated for additionalK video frames as follows:

(1)

(1)

(т.е. реконструкция

–го кадра) вычисляется на основе вычисленного N–мерного вектора встраивания

, т.е.:[0015] Then, in the generator G , the synthesized video frame

(i.e. reconstruction

–Th frame) is calculated based on the calculated N – dimensional embedding vector

i.e.:

(2)

(2)

, состязательный член

и член соответствия встраивания

:The parameters of the embedder E and the generator G are then optimized to minimize the next objective function that contains the content loss term

adversarial member

and embed compliance member

:

(3)

(3)

измеряет различие между истинно верным кадром видео

и синтезированным кадром видео

с использованием меры перцепционного подобия. В качестве примера, может использоваться мера перцепционного сходства, которая соответствует сети VGG19, обучаемой для классификации ILSVRC, и сети VGGFace, обучаемой для верификации лица. Однако настоящее изобретение не должно быть ограничено таким примером меры перцепционного сходства, поскольку в нем могут быть использованы любые меры перцепционного сходства, известные из предшествующего уровня техники. Если сети VGG19 и VGGFace используются для получения меры перцепционного сходства, член потерь содержимого

может быть вычислен как взвешенная сумма потерь L ₁ между признаками упомянутых сетей.In the objective function (3), the term content loss

measures the difference between a true true frame of video

and synthesized video frame

using measures of perceptual similarity. As an example, a measure of perceptual similarity can be used, which corresponds to the VGG19 network trained for ILSVRC classification and the VGGFace network trained for face verification. However, the present invention should not be limited to such an example of a measure of perceptual similarity, since any measures of perceptual similarity known from the prior art can be used in it. If the VGG19 and VGGFace networks are used to obtain a measure of perceptual similarity, a member of the content loss

can be calculated as the weighted sum of losses L ₁ between the features of the mentioned networks.

в целевой функции (3) соответствует оценке реалистичности r, вычисленной дискриминатором D, которая должна быть максимизирована, и члену соответствия признаков

, который по существу является мерой перцепционного подобия, вычисленному с использованием дискриминатора (это улучшает стабильность мета–обучения):[0016] Adversarial Member

in the objective function (3) corresponds to the estimate of the realism r calculated by the discriminator D , which should be maximized, and to the term of correspondence of attributes

, which is essentially a measure of perceptual similarity calculated using a discriminator (this improves meta-learning stability):

(4)

(4)

содержат N–мерные векторы встраивания, которые соответствуют индивидуальным видео. Дискриминатор D сначала отображает свои входы (т.е. входной кадр видео

, ассоциированное изображение ориентиров лица

и индекс обучающей последовательности

) на N–мерный вектор V(

) и затем вычисляет оценку реалистичности r следующим образом:Following the projection discriminator approach, the columns of the matrix

contain N-dimensional embedment vectors that correspond to individual videos. The discriminator D first displays its inputs (i.e., the input frame of the video

, associated face landmark image

and training sequence index

) on N is the dimensional vector V (

) and then computes a realistic estimate of r as follows:

(5)

(5)

обозначает

–ый столбец матрицы

. В то же время,

и

не зависят от индекса видео, так что эти члены соответствуют обобщенной реалистичности

и ее совместимости с изображением ориентиров лица

.Where

denotes

–Th column of the matrix

. In the same time,

and

independent of the video index, so these terms correspond to generalized realism

and its compatibility with face landmarks

.

в дискриминаторе D. Член соответствия

в вышеуказанной целевой функции (3) стимулирует подобие двух типов N–мерных векторов встраивания путем штрафования L ₁–разности между

и W _i .[0017] Thus, there are two types of N-dimensional embedment vectors in the proposed system: one computed by the embedderE, and others that match the columns of the matrix

in discriminatorD. Compliance Member

in the above objective function (3) stimulates the similarity of two types of N-dimensional embedding vectors by fineL ₁–Differences between

and W _i .

встраивателя E и параметры

генератора G обновляются, параметры

дискриминатора D также обновляются. Обновление приводится в действие минимизацией следующей целевой функции “потери шарнира” (hinge–loss) (6), что стимулирует увеличение оценки реалистичности r по реальным (т.е. нефальсифицированным) кадрам видео

и ее снижение по синтезированным (т.е. фальсифицированным) кадрам видео

:[0018] When the parameters

Embedded E and parameters

generator G are updated, parameters

discriminator D is also updated. The update is driven by minimizing the next target function “hinge-loss” (6), which stimulates an increase in the estimate of the realism r from real (i.e., falsified) video frames

and its decrease in synthesized (i.e., falsified) video frames

:

(6)

(6)

и реального образца

и затем обновляет параметры дискриминатора, чтобы сдвигать эти оценки ниже –1 и выше +1 соответственно. Мета–обучение продолжается попеременными обновлениями параметров встраивателя E и генератора G, которые минимизируют потери

,

и

, с обновлениями дискриминатора D, которые минимизируют потери

.The objective function (6) thus compares the realism of a falsified sample

and real sample

and then updates the discriminator parameters to shift these ratings below –1 and above +1, respectively. Meta-learning continues with alternate updates of the parameters of the embedded E and generator G , which minimize losses

,

and

, with discriminator D updates that minimize loss

.

[0019]Multi-frame training through fine-tuning. After the convergence of meta-learning is achieved, the proposed system can be further trained to synthesize models of the head of a speaking person for a new person who was not observed during the meta-learning stage. As previously, synthesis is conditioned on landmark images. The system is trained in the way of training in several frames, assuming thatT training imagesx(1),x(2), ...,x(T) (i.e., T frames of the same video), and thaty(1),y(2), ...,y(T) are corresponding face landmark images. Note that the numberT frames do not have to be equalKused at the meta-learning stage. The proposed system can generate an acceptable result based on one photograph (single shot training, T= 1), while adding a few more photos (multi-training, T>1) increases the accuracy of personalization. The experiments performed by the authors of the present invention cover the range [1, 33] forT. However, the present invention should not be limited to an embodiment in whichT lies within the range [1, 33], sinceTcan be selected by a person skilled in the art to exceed this range, depending on the performance of the hardware used for training in multiple frames, or depending on the accuracy of the images generated by the student across multiple GAN frames (which was previously meta-trained as described above), and the goals for which the meta-trained GAN is trained in several frames (i.e., is subject to fine-tuning).

для нового человека, для которого должна быть синтезирована модель головы говорящей персоны. Иными словами, вычисление

может выполняться в соответствии со следующим:[0020] Preferably, the meta-trained embedder E is used in the current learning stage in multiple frames to calculate the N-dimensional embedment vectors

for a new person for whom the model of the head of the talking person should be synthesized. In other words, the calculation

may be performed in accordance with the following:

(7)

(7)

встраивателя E, ранее полученные на стадии мета–обучения. Простым способом генерировать новые синтезированные кадры, соответствующие новым изображениям ориентиров, является тогда применение генератора G, использующего вычисленный N–мерный вектор встраивания

и полученные мета–обучением параметры

генератора G, а также матрицы проекции P. Авторами настоящего изобретения установлено, что при таком выполнении синтезированные изображения головы говорящей персоны являются верными и реалистичными, однако часто имеется значительный пробел в идентичности, который является неприемлемым для большинства приложений, нацеленных на высокую степень персонализации.reusing parameters

Embedded E , previously obtained at the meta-learning stage. An easy way to generate new synthesized frames corresponding to new landmark images is then to use the generator G using the calculated N-dimensional embedding vector

and meta-learning parameters

generator G , as well as the projection matrix P. The authors of the present invention found that with this embodiment, the synthesized images of the head of a talking person are true and realistic, however, there is often a significant gap in identity, which is unacceptable for most applications aimed at a high degree of personalization.

[0021] This gap in identity can often be closed by the fine-tuning step proposed herein. The fine-tuning process can be considered as a simplified version of meta-training with a single video sequence and a small number of frames. The fine-tuning process includes the following components:

) теперь заменяется генератором G’( y

). Как ранее, генератор G’ сконфигурирован, чтобы получать изображение ориентиров лица y

и генерировать синтезированный кадр видео

. Важно, что специфические для человека параметры генератора, которые теперь обозначены как

, напрямую оптимизируются на стадии обучения по нескольким кадрам, вместе с общими для человека параметрами

. Вычисленные N–мерные векторы встраивания

и матрица проекции P, полученная на стадии мета–обучения, могут по–прежнему использоваться, чтобы инициализировать специфические для человека параметры генератора

, т.е.

. 1. Generator G (

) is now replaced by the generator G '( y

) As previously, the generator G ′ is configured to receive landmark images of the face y

and generate a synthesized video frame

. It is important that the human-specific parameters of the generator, which are now designated as

are directly optimized at the training stage for several frames, together with parameters common to humans

. Computed N-dimensional embedding vectors

and the projection matrix P obtained at the meta-learning stage can still be used to initialize human-specific generator parameters

, i.e.

.

), как ранее на стадии мета–обучения, сконфигурирован, чтобы вычислять оценку реалистичности r. Параметры

части V(

) ConvNet дискриминатора D’ и смещение b инициализируются в те же самые параметры

, b, полученные на стадии мета–обучения. Инициализация

описана ниже.2. DiscriminatorD '(

), as previously in the meta-learning stage, is configured to calculate a realistic scorer. Parameters

partsV(

) ConvNet discriminatorD 'and offsetbinitialized to the same parameters

,bobtained at the stage of meta-learning. Initialization

described below.

[0022] During the fine-tuning stage, an estimate of the realism r of the discriminator D 'is obtained in the same way as in the meta-learning stage:

(8)

(8)

на стадии тонкой настройки является той же самой, что и роль вектора

на стадии мета–обучения. Для инициализации

на стадии обучения по нескольким кадрам, аналог

не доступен для нового человека (поскольку кадры видео этого человека не использовались в обучающем наборе данных мета–обучения). Однако член соответствия

в процессе мета–обучения гарантирует подобие между N–мерными векторами встраивания дискриминатора и N–мерными векторами встраивания, вычисленными встраивателем. Поэтому,

инициализируется на стадии обучения по нескольким кадрам в сумму w ₀ и

.As can be seen from a comparison of expressions (5) and (8), the role of the vector

at the stage of fine tuning is the same as the role of the vector

at the meta-training stage. To initialize

at the training stage in several frames, analog

not available for a new person (since this person’s video frames were not used in the training meta-training data set). However member of compliance

in the process of meta-learning, it guarantees the similarity between the N-dimensional vectors of embedding the discriminator and the N-dimensional vectors of embedding calculated by the embedder. Therefore,

initialized at the training stage for several frames in the amount of w ₀ and

.

и общие для человека параметры

генератора G’ оптимизируются, чтобы минимизировать упрощенную целевую функцию:[0023] Once a new learning problem has been identified, the loss functions of the fine-tuning stage directly follow from the meta-learning options. Thus, human-specific parameters

and parameters common to humans

generators G 'are optimized to minimize the simplified objective function:

(9)

(9)

where t ∈ {1 ... T } is the number of the training example.

, w _new, b оптимизируются путем минимизации той же самой функции hinge loss, что и в (6):[0024] Discriminator Parameters

, w _new , b are optimized by minimizing the same hinge loss function as in (6):

(10)

(10)

[0025] In most situations, a fine tuning generator provides much better learning sequence compliance. The initialization of all parameters through the meta-learning stage is thus critical. As revealed in experiments, such initialization first introduces a highly realistic image of the head of a talking person, which allows the disclosed model to extrapolate and predict realistic images for poses (orientations) with changing head orientations and facial expressions.

) может быть основана на архитектуре переноса изображения к изображению, предложенной Johnson et. al. [19], но слои понижающей дискретизации и повышающей дискретизации должны быть заменены остаточными блоками с нормализацией образца. Специфические для человека параметры

служат в качестве аффинных коэффициентов слоев нормализации образца, следуя методу адаптивной нормализации образца, известному в технике, хотя слои регулярной (неадаптивной) нормализации образца в блоках понижающей дискретизации, которые кодируют изображения ориентиров лица

, все еще используются.[0026] Implementation Details and specific embodiments. Generator Network G (

) may be based on the image-to-image transfer architecture proposed by Johnson et. al. [19], but the layers of downsampling and upsampling should be replaced by residual blocks with normalization of the sample. Human Specific Parameters

serve as the affine coefficients of the normalization layers of the sample, following the method of adaptive normalization of the sample known in the art, although layers of regular (non-adaptive) normalization of the sample in downsampling units that encode face landmark images

are still in use.

) и части V(

) ConvNet дискриминатора, могут использоваться подобные сети, которые состоят из остаточных блоков понижающей дискретизации (тех же, что и те, которые используются в генераторе, но без слоев нормализации). Сеть дискриминатора, по сравнению со встраивателем, имеет дополнительный остаточный блок на конце, который работает при пространственном разрешении 4×4. Чтобы получить векторизованные выходы в обеих сетях, может выполняться глобально суммарный опрос по пространственным размерностям с последующим выпрямленным линейным блоком (ReLU).[0027] For the integration module E (

) and part V (

) ConvNet discriminator, similar networks can be used, which consist of residual downsampling blocks (the same as those used in the generator, but without normalization layers). The discriminator network, in comparison with the embedder, has an additional residual block at the end, which operates at a spatial resolution of 4 × 4. To obtain vectorized outputs in both networks, a global survey of spatial dimensions can be performed globally followed by a rectified linear block (ReLU).

[0028] Spectral normalization can be used for all convolutional and fully related layers in all networks. Self-service units are also used. They are inserted with a spatial resolution of 32 × 32 in all parts of the downsampling networks and with a resolution of 64 × 64 in terms of upsampling the generator.

, потеря L ₁ может оцениваться между активациями слоев Conv1,6,11,20,29 VGG19 и слоев Conv1,6,11,18,25 VGGFace для реалистичных и фальсифицированных изображений. Эти потери с весами равными 1⋅10^–2 для VGG19 и 2⋅10^–3 для членов VGGFace могут суммироваться. Могут использоваться Caffe–обученные версии для обеих из этих сетей. Для

, могут использоваться активации после каждого остаточного блока сети дискриминатора и веса равные 1⋅10¹. Наконец, для

вес устанавливается на 8⋅10¹.[0029] To calculate

, the loss of L ₁ can be estimated between activations of the Conv1,6,11,20,29 VGG19 layers and the Conv1,6,11,18,25 VGGFace layers for realistic and falsified images. These losses with weights of 1–10 ^–2 for VGG19 and 2–10 ^–3 for VGGFace members can be added up. Caffe-trained versions for both of these networks can be used. For

, activations after each residual block of the discriminator network and weights of ¹ × 10 ¹ can be used. Finally for

weight is set to 8⋅10 ¹ .

[0030] The minimum number of channels in the convolutional layers can be set to 64, and the maximum number of channels, as well as the size N of the embedding vectors can be set to 512. As a result, the embedder has 15 million parameters, the generator has 38 million parameters. The ConvNet discriminator part has 20 million parameters. Networks can be optimized using the Adam method. The learning speed of the embedded and generator networks can be set to 5 × 10 ^–5 and 2 × 10 ^–4 for the discriminator, thus performing two update steps for the latter to one for the first. The present invention should not be limited to the above specific approaches, meanings and details, since some changes and modifications to the above approaches, meanings and details will be apparent to a person skilled in the art without any creative effort. Thus, such changes and modifications are considered to be included in the scope of the claims.

The best option for implementing the above method

. См. Фиг. 2.[0031] A method for synthesizing a sequence of images of the head of a talking person for an arbitrary person using a generator network configured to map head orientation descriptors and facial expressions to one or more images of a sequence of images of the head of a talking person on a computing device (20) is provided in accordance with the first aspect of the present disclosure. The method comprises the steps of executing (S105) training on several frames of a generator network meta-trained (S100) on a plurality of M video sequences containing images of the heads of talking persons other than an arbitrary person, and synthesizing (S110) a sequence of images of the heads of the talking person for said person using a finely tuned generator network and a previously unseen sequence of head orientation and facial expressions descriptors,

. See FIG. 2 .

из единственной последовательности кадров упомянутой персоны, для которой должна быть синтезирована последовательность изображений головы говорящей персоны; оценивания (S105.2) дескрипторов ориентации головы и мимики

для по меньшей мере одного кадра видео

; вычисления (S105.3) N–мерного вектора встраивания

, характеризующего специфическую для персоны информацию, на основе по меньшей мере одного кадра видео

с использованием мета–обученной сети встраивателя; реализации (создания экземпляра) (S105.4) сети генератора на основе набора параметров мета–обученной сети генератора и N–мерного вектора встраивания

’; и тонкой настройки (S105.5) параметров сети генератора, чтобы соответствовать по меньшей мере одному кадру видео

, когда сеть генератора обеспечивается дескрипторами ориентации головы и мимики

. Набор параметров мета–обученной сети генератора является входом на этап (S105), а набор параметров тонко настроенной сети генератора является выходом этапа (S105). См. Фиг. 3.[0032] The step of performing (S105) training on several frames of a generator network meta-trained (S100) on the setMvideo sequences containing images of the heads of talking people different from an arbitrary person, contains sub-steps: receiving (S105.1) at least one frame of the video

from a single sequence of frames of the said person for which a sequence of images of the head of the talking person should be synthesized; evaluating (S105.2) descriptors of head orientation and facial expressions

for at least one frame of video

; computing (S105.3) the N-dimensional embedding vector

characterizing information specific to a person, based on at least one frame of video

using the meta-trained network of the embedder; implementation (creating an instance) (S105.4) of the generator network based on a set of parameters of the meta-trained generator network and the N-dimensional embedding vector

’; and fine-tuning (S105.5) parameters of the generator network to correspond to at least one video frame

when the generator network is provided with heading descriptors and facial expressions

. The parameter set of the meta-trained generator network is the input to step (S105), and the parameter set of the finely tuned generator network is the output of the step (S105). Cm.FIG. 3.

и

могут содержать, но без ограничения, ориентиры лица. Дескрипторы ориентации головы и мимики

используются вместе с по меньшей мере одним кадром видео

, чтобы вычислять N–мерный вектор встраивания

.[0033] Descriptors of orientation of the head and facial expressions

and

may include, but are not limited to, landmarks of the face. Descriptors of orientation of the head and facial expressions

used together with at least one frame of video

to calculate the N-dimensional embedment vector

.

из одной последовательности видео из множества M последовательностей видео, содержащих изображения головы говорящих персон, отличающихся от произвольной персоны; оценивания (S100.2) дескрипторов ориентации головы и мимики

для по меньшей мере одного кадра видео

; вычисления (S100.3) N–мерного вектора встраивания

, характеризующего специфическую для персоны информацию, на основе по меньшей мере одного кадра видео

; реализации (S100.4) сети генератора на основе текущего набора параметров сети генератора и N–мерного вектора встраивания

; и обновления (S100.5) набора параметров сети генератора и набора параметров сети встраивателя на основе соответствия между выходом сети генератора для оцененных дескрипторов ориентации головы и мимики

и последовательности из по меньшей мере одного кадра видео

. См. Фиг. 4.[0034] Meta-training of the generator network and the embedded network is performed in episodesK–Framework whereK is a predefined integer, and each of the episodes comprises the steps of: receiving (S100.1) at least one frame of a video

from one sequence of video from manyMvideo sequences containing images of the heads of speaking persons different from an arbitrary person; evaluating (S100.2) heading descriptors and facial expressions

for at least one frame of video

; computing (S100.3) the N-dimensional embedding vector

characterizing information specific to a person, based on at least one frame of video

; implementations (S100.4) of the generator network based on the current set of generator network parameters and the N-dimensional embedment vector

; and updates (S100.5) of the generator network parameter set and the embedder network parameter set based on the correspondence between the generator network output for the estimated descriptors of head orientation and facial expressions

and sequences of at least one frame of video

. See FIG. 4.

[0035] The generator network and the embedder network may be conventional convolutional networks. During the implementation steps (S105.4, S100.4), the normalization coefficients in the implemented generator network are calculated based on the N-dimensional embedding vectors calculated by the embedder network. The meta-discriminator network is trained together with the generator network and the embedder’s network, the method further comprising the step of: calculating, using the discriminator’s network, estimates of the realism r of the generator network output, and updating the generator network and the embedder’s parameters based on the realisticness r ; updating the discriminator network parameters in order to increase the realistic estimate of r by video frames from the set of M video sequences and reduce the realistic estimate of r by the output (i.e., the synthesized image) of the generator network.

, который обучается для каждой из последовательностей видео во множестве M последовательностей видео. Различия между N–мерным вектором встраивания

и N–мерным вектором встраивания w штрафуются во время мета–обучения (S100), дискриминатор проекции используется во время стадии тонкой настройки (S105), и N–мерный вектор встраивания w дискриминатора проекции инициализируется в N–мерный вектор встраивания

’ в начале тонкой настройки.[0036] The discriminator network is a projection discriminator network that is configured to calculate an estimate of the realisticness r of the output of the generator network using an N-dimensional embedding vector w different from the N-dimensional embedding vector

that is trained for each of the video sequences in a plurality of M video sequences. Differences between the N-dimensional embedding vector

and the N-dimensional embedment vector w are fined during meta-training (S100), the projection discriminator is used during the fine-tuning stage (S105), and the N-dimensional embedment vector w of the projection discriminator is initialized to the N-dimensional embedment vector

'at the beginning of fine tuning.

[0037] Computing device (20) is provided in accordance with a second aspect of the present disclosure. All steps of the above method can be performed by the computing device (20) illustrated in FIG. 5 . The computing device (20) comprises a processor (20.1) and a memory (20.3). The memory (20.3) is configured to store computer-executable instructions that, when executed by the processor (20.1) of the computing device, cause the processor to perform the disclosed method of synthesizing a model of the head of a talking person for an arbitrary person using a generator network. Optionally, the computing device (20) may comprise a camera (20.2) configured to capture a video or a single frame of a person for whom a model of the head of the talking person should be synthesized. Computing device (20) can be a smartphone, tablet, PC, laptop, or any other user computing device such as, for example, AR glasses, VR glasses, smart watches, etc. The processor (20.1) of the computing device (20) can be any computing tool, including, but not limited to, a specialized integrated circuit (ASIC), a user-programmable gate array (FPGA), or a system on a chip (SoC). Any data types can be processed by the proposed methods, devices and systems.

[0038] Various methods, devices, and systems providing photorealistic avatars can be practiced based on the foregoing disclosure. Additionally, various methods, devices and systems for training in several frames of realistic neural models of the heads of a talking person and / or using the models / networks thus trained to provide photorealistic avatars can be created based on the foregoing disclosure. Embodiments of the disclosure may also be implemented as a non-transitional (non-temporary) computer-readable medium transferring computer-executable instructions that, when executed by a processing unit of a device, cause the processing unit to perform the disclosed method of synthesizing a model of the head of a speaking person for an arbitrary person using an adversarial network .

[0039] The proposed method can also be implemented as a system for synthesizing a model of the head of a talking person for an arbitrary person using an adversarial network. In such a system, the specific steps of the method can be implemented as various functional blocks, circuits, and / or processors. However, it should be obvious that any suitable distribution of functionality between different functional blocks, circuits, and / or processors can be used without deviating from the described embodiments.

[0040] Embodiments may be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. Embodiments may optionally be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. Elements and components of any embodiment may be physically, functionally, or logically implemented in any suitable manner. In fact, functionality can be implemented in one unit, in multiple units, or as part of other general purpose units.

[0041] The preceding descriptions of embodiments of the invention are illustrative, and modifications to the configuration and implementation are intended to be included within the scope of the claims. For example, while embodiments of the invention are mainly described with respect to process steps, these descriptions are illustrative. Although the claimed subject matter is described in terms specific to structural features and methodological actions, it should be understood that the claimed subject matter as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and acts described above are disclosed as exemplary forms of implementing the claims. Moreover, the invention is not limited to the illustrated order of the steps of the method, the order can be modified by a person skilled in the art without the application of creative efforts. Some or all of the steps of the method may be performed sequentially or simultaneously. Accordingly, it is understood that the scope of an embodiment of the invention should be limited only by the following claims.

Claims (24)

1. A method of synthesizing a sequence of images of the head of a talking person for an arbitrary person using a generator network configured to map head orientation descriptors and facial expressions to one or more images of a sequence of images of a head of a talking person on a computing device (20), the method comprising the steps of: performing training on several frames (S105) of the generator network, meta-trained (S100) on the set M of video sequences containing images of the heads of talking persons different from the said arbitrary person, by: receiving (S105.1) at least one frame

video from a single sequence of frames of the said person for which a sequence of images of the head of the talking person should be synthesized; evaluating (S105.2) descriptors

head orientation and facial expressions for at least one frame

video; calculation (S105.3)N–Dimensional vector

embedding characterizing information specific to a person, based on at least one frame

video using the meta-trained network of the embedder; and implementations (S105.4) of a generator network based on a set of parameters of a meta-trained generator network and an N- dimensional vector

embedding; and fine-tuning (S105.5) parameters of the generator network to correspond to at least one frame

video when the generator network is provided by descriptors

head orientation and facial expressions; synthesizing (S110) a sequence of images of the head of a talking person for said person using a finely tuned generator network and a previously not observed sequence of descriptors

orientation of the head and facial expressions. 2. The method according to claim 1, wherein the descriptors

and

head orientations and facial expressions contain landmark images of the face. 3. The method of claim 1, wherein the descriptors

 head orientations and facial expressions are used together with at least one frame

 video to calculateN–Dimensional vector

 embedding. 4. The method according to claim 1, wherein the meta-training of the generator network and the embedded network is performed in episodes of K- frame training, K being a predetermined integer, and each of the episodes contains the steps: receiving (S100.1) at least one frame

video from a single video sequence from a plurality of M video sequences containing images of the heads of speaking persons different from said arbitrary person; evaluating (S100.2) descriptors

head orientation and facial expressions for at least one frame

video; calculation (S100.3)N–Dimensional vector

 embedding characterizing information specific to a person, based on at least one frame

 video; implementations (S100.4) of the generator network based on the current set of generator network parameters and the N- dimensional vector

embedding; and updating (S100.5) the generator network parameter set and the embedder network parameter set based on the correspondence between the generator network output for the estimated descriptors

head orientation and facial expressions and sequences of at least one frame

video. 5. The method according to any one of paragraphs. 1–4, wherein the generator network and the embedder’s network are convolutional networks, and during the implementation stages (S105.4, S100.4), the normalization coefficients within the implemented generator network are calculated based on N- dimensional embedding vectors calculated by the embedding network. 6. The method according to claim 4, wherein the meta-discriminator network is trained together with the generator network and the embed network, the method further comprising the steps of: computing, using a discriminator network, estimates of the r realistic output of the generator network; and updating the parameters of the generator network and the embedded network based on an assessment of r realism; updating the discriminator network parameters in order to increase the estimate of r realism by video frames from the set of M video sequences and reduce the estimate of r realism by the output of the generator network. 7. The method of claim. 6, the discriminator network is a network discriminator projection configured to estimate r realistic estimate generator output network using the N -dimensional vector w embedding different from N -dimensional vectors

embeddings that have been trained for each of the video sequences in a plurality of M video sequences. 8. The method according to p. 7, the differences between the N- dimensional vector

embedding and an N- dimensional embedding vector w are fined during meta-training (S100), the projection discriminator is used during the fine-tuning step (S105), and the N- dimensional embedding vector of the projection discriminator w is initialized to the N- dimensional vector

embedding at the beginning of fine tuning. 9. A computing device (20) containing memory (20.3) storing computer-executable instructions that, when executed by the processor (20.1) of the computing device, cause the processor to perform a method of synthesizing a sequence of images of the head of a talking person for an arbitrary person using a generator network in accordance with any one of paragraphs. 1-8.

Images