RU2775825C1 - Neural-network rendering of three-dimensional human avatars - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: invention relates to means of creating and rendering a three-dimensional model of an avatar of the user and can be applied in augmented and virtual reality systems, video games, and teleconferences. Method for rendering a avatar includes the stages of: obtaining the body shape parameters s characterising the body shape of the user, the pose parameters p characterising the desired pose of the avatar of the user, and the camera parameters C characterising the desired angle; creating a deformable polygonal structure as a three-dimensional model of the body of the user; overlaying an L-channel neural network texture characterising the physical features of the user; and rendering an avatar of the user by means of a pre-trained rendering neural network, wherein the neural network texture and the rendering neural network are trained by readjusting the parameters of the neural network texture and the rendering neural network until the resulting images of the avatar of the user correspond to one or more frames of the image of the user.

EFFECT: possibility of creating or rendering a photorealistic three-dimensional avatar of the user in the desired pose and/or from the desired angle based on only one or more photo images of the user.

16 cl, 3 dwg

Description

The field of technology to which the invention belongs

The present invention relates generally to the processing of three-dimensional images, and more specifically to the rendering of a three-dimensional (3D) model of a user's avatar, and can be applied to augmented and virtual reality (AR/VR) systems, video games, teleconferencing, and the like.

State of the art

Avatar - a graphic representation of a user in a virtual environment, designed to individualize the user and reflect certain specific character traits, appearance, status, etc. this particular user. Until recently, in a virtual network environment such as, for example, websites, Internet forums, instant messaging services, and the like, a fixed or moving two-dimensional (2D) image selected or created by a specific user was used as an avatar. As the possibilities of virtual and augmented reality, multiplayer online video games, teleconferencing services, etc. expand. as an avatar, a three-dimensional (3D) image has also begun to be used, in particular a 3D model, which can be individualized to one degree or another in accordance with the preferences of a particular user, depending on the capabilities of the corresponding environment in which the 3D model is used.

However, in some applications, primarily in teleconferencing, multiplayer online video games, and the like. there is a need for a three-dimensional user avatar, which could, on the one hand, reflect the appearance, clothing (real or modified in accordance with the user's preferences), etc. in as much detail as possible. a specific user, and on the other hand, to repeat movements, facial expressions, etc. as accurately and in detail as possible. the user that the avatar represents.

In recent years, complex and powerful 3D models of the human body without textures have been developed that mimic the shape of the body, including deformities of the facial muscles and hands (see references [30, 44]). These models are based on polygonal geometry and are formed by learning from several available data sets obtained by scanning the human body. However, clothing and hair modeling literally adds an extra layer of complexity to the development of such 3D models, which can be even more problematic when modeling the user's appearance, for which only a small amount of 3D data is available. At the same time, the creation of realistic 3D avatars is impossible without clothing and hair modeling.

The well-known solution disclosed in [23] uses an early generative model to create a three-dimensional model of a person in clothes, which converts the renders of a parametric model of the human body into analysis (semantic segmentation) maps of a person, which are then transformed into images of a person.

Another solution, disclosed in reference [27], uses pose-based human imaging techniques that synthesize images of a person in new poses or new clothing using a single image input by reshaping elements when converting from pose to image. (see sources [36, 10, 9, 13]), predicting coordinates on the surface in the source and target frames (see source [4]), or samples from texture maps in the RGB color space (see sources [28, 12] ).

In the situation of video image processing, the source [42] discloses a solution in which a system is trained that converts a sequence of body poses into a corresponding time-continuous monocular video image, and the source [34] discloses a solution in which renders of body joints are converted into body surface coordinates in clothes , which are used to form samples from the trained texture stack in the RGB color space, which allows for improved image quality.

Geometric texture decomposition helps in detecting blind spots and generalizing to render images for poses not shown in the input images, however, errors in surface coordinate regression lead to numerous visible image artifacts, especially in the face and hands of the simulated person.

Sources [3, 2, 6, 24, 15] describe solutions in which, based on one image or multiple images of a person, a texture and displacement map is formed for the body model, which can then be used to form renders from any viewpoint and depicting a person in an arbitrary position. However, the quality of generating renders for a user avatar in known solutions is limited by the characteristics of classical graphic processing processes.

The sources [41, 46, 25, 35] describe the transformation of a pose into an image when processing a video image in a low frame rate mode, in which the target appearance of the user model is determined by several images. However, this provides a low quality of the generated image.

Disclosure of invention

This section, which discloses various aspects and embodiments of the claimed invention, is intended to provide a brief description of the claimed objects of the invention and embodiments. A detailed description of technical means and methods that implement combinations of features of the claimed inventions is given below. Neither this disclosure nor the following detailed description and accompanying drawings should be construed as defining the scope of the claimed invention. The scope of legal protection of the claimed invention is determined solely by the attached claims.

Taking into account the above disadvantages of the known solutions from the prior art in the field of imaging a three-dimensional user model for various spatial positions, angles, poses, etc., the technical problem solved by the present invention is to improve the quality and reliability of the generated user avatar images with a larger number of different poses and angles for which an avatar image can be generated.

The objective of the invention is to provide a method and system for imaging a three-dimensional model of a user's avatar with improved rendering of a three-dimensional model using one or more neural networks.

The technical result achieved by using the invention is to create a photorealistic three-dimensional user avatar with the possibility of its animation based on body posture parameters, including facial expressions and hand positions.

To solve the above problem, in the first aspect of the invention, a method for creating a user avatar is provided, comprising the steps of: receiving input data in the form of one or more user image frames; extracting from the received input data body shape parameters s characterizing the user's body shape, pose parameters p characterizing the user's pose on one or more user image frames, and camera parameters C characterizing the angle from which said one or more user image frames were taken; forming a deformable polygonal structure as a three-dimensional model of the user's body; forming an L-channel neural network texture characterizing the features of the user's appearance, based on the processing of said one or more frames of the user's image; impose L-channel neural network texture on the deformable polygonal structure; rendering the user's avatar in a new pose and/or from a new angle by means of a rendering neural network; and training the rendering neural network parameters and the neural network texture by adjusting them to match the resulting user avatar images and one or more user image frames from the received input data.

In the second aspect, this problem is solved by a method for rendering a user avatar, comprising the steps of: receiving input data in the form of body shape parameters s characterizing the shape of the user's body, pose parameters p characterizing the avatar pose, camera parameters C characterizing the angle from which the user should be depicted as an avatar; forming a deformable polygonal structure as a three-dimensional model of the user's body; imposing the trained L-channel neural network texture on the deformable polygonal structure; and rendering the user avatar in a new position and/or from a new angle by means of the trained rendering neural network, wherein the training of the neural network texture and the rendering neural network is carried out by adjusting the parameters of the neural network texture and the rendering neural network to match the resulting images of the user avatar and one or more user image frames.

, где

- параметры нейросети, R - функция нейротекстурирования, M - деформируемая полигональная структура, определяемая параметрами p позы и параметрами s формы тела, C - параметры камеры, T - L-канальная нейротекстура.In one or more embodiments of the methods of the first and second aspects of the present invention above, the rendering neural network may have a convolutional architecture. The number L channels of the L-channel neural network texture may be 16. The rendering neural network parameters may be trained together with the L-channel neural network texture generator model, and the L-channel avatar neural network texture may be obtained by inference in the trained generator model and further adjustment. The training of the rendering neural network and the generating model of L-channel neural network textures can be carried out according to the principle of adversarial networks. The rendering neural network in accordance with the methods according to the invention is defined by the expression

, where

are the neural network parameters, R is the neurotexturing function, M is the deformable polygonal structure determined by the parameters p of the pose and the parameters s of the body shape, C are the camera parameters, T is the L-channel neurotexture.

In a third aspect, the above problem is solved by a system for creating a user avatar, comprising: a memory in which computer program instructions are stored; and one or more processors which, under the control of said computer program instructions, is(are) configured to: receive input data in the form of one or more user image frames; extracting from the received input data body shape parameters s characterizing the user's body shape, pose parameters p characterizing the user's pose on one or more user image frames, and camera parameters C characterizing the angle from which said one or more user image frames were taken; forming a deformable polygonal structure as a three-dimensional model of the user's body; forming an L-channel neural network texture that characterizes the features of the user's appearance, based on the processing of said one or more frames of the user's image; overlaying an L-channel neural network texture on a deformable polygonal structure; rendering the user's avatar in a new pose and/or from a new angle by means of a rendering neural network; and training the rendering neural network parameters and the neural network texture by adjusting them to match the received user avatar images and one or more user image frames from the received input data.

In a fourth aspect, the above problem is solved by a user avatar rendering system, comprising: a memory in which computer program instructions are stored; and one or more processors which, under the control of said computer program instructions, are configured to: receive input data in the form of body shape parameters s characterizing the user's body shape, pose parameters p characterizing the avatar's pose, camera parameters C characterizing the angle from which the avatar should be shown; forming a deformable polygonal structure as a three-dimensional model of the user's body; overlaying a trained L-channel neural network texture on a deformable polygonal structure; and rendering the user avatar in a new pose and/or from a new angle by means of the trained rendering neural network, wherein the training of the neural network texture and the rendering neural network is carried out by adjusting the parameters of the neural network texture and the rendering neural network to match the resulting images of the user avatar and one or more user image frames.

In the fifth and sixth aspects of the present invention, the above object is achieved by computer-readable media on which computer program instructions are stored causing the system for creating a user avatar or the system for rendering a user avatar to execute the methods for creating a user avatar or rendering a user avatar according to the above aspects of the present invention when executing the computer program instructions. one or more processors.

Those skilled in the art will appreciate that, in addition to the foregoing objects of the invention, the inventive concept underlying the present invention may be embodied in the form of other objects of the invention, such as one or more devices, a computer program product, a computer program, a system, a method, etc. .P.

Brief description of the drawings

The drawings are given in this document to facilitate understanding of the essence of the present invention. The drawings are schematic and not to scale. The drawings are for illustrative purposes only and are not intended to define the scope of the present invention.

On FIG. 1 illustrates a neural network model for rendering the surface of a 3D user avatar according to the present invention;

On FIG. 2 shows a schematic diagram of a user avatar imaging method according to the invention;

On FIG. 3 illustrates an example of input images of two people and avatars obtained by the method according to the invention, in various poses and differently oriented with respect to the camera.

Implementation of the invention

The present invention is based on the use of an artificial intelligence model that includes one or more neural network layers. The proposed approach, also referred to in the context of the present invention as neural network rendering of the user avatar, allows the creation of three-dimensional realistic full-length avatars or their rendering from one or more images of the user. More specifically, the methods of the invention may use a video image of the user, but in principle one or more photographic images may be sufficient to render the user's avatar in accordance with the invention.

The proposed approach is based on the use of a deformable three-dimensional polygonal structure (also referred to in the context of the present invention as the word "mesh" (from the English mesh - network) or "three-dimensional (3D) model of a" naked "person"). As a non-limiting example, the SMPL-X model can be used as a polygonal structure (see, for example, the source [30]). The polygonal structure is used in the invention, in particular, to model and animate the general geometry of the avatar's body in 3D.

On top of the mentioned three-dimensional body model, the proposed invention uses a multi-channel neural network texture (to determine the neural network texture, see, for example, the source [38]) and a rendering neural network, which ultimately allow you to generate images of the user's avatar with clothes and hair.

Creating a user avatar can be done by adjusting the neural network texture and rendering neural network parameters to a set of user images, performed in order to train the neural network texture and/or rendering neural network. The parameters are adjusted using gradient optimization, and the gradient of the divergence between the predicted and actual images is calculated using the backpropagation method. When creating an avatar in this way, the number of user images must be large enough (in particular, the user must be photographed from all angles).

Thus, the first contribution of the invention in relation to the prior art lies in the fact that the combination of a deformable three-dimensional polygonal structure and neural network textures ("neural textures") allows you to simulate the appearance of the user's avatar in full growth in the presence of free clothes and hair in the avatar and allows you to take into account geometric features of the user's three-dimensional avatar, which are absent in parametric models of the user's body.

A second contribution of the invention to the prior art is to enable a 3D user avatar to be created or rendered in a desired pose and/or from a desired angle based on just one or more photographic images of the user through the use of a generative model.

In accordance with one aspect of the present invention, a method for creating a user avatar is provided, comprising the steps of: receiving input data in the form of one or more frames of a user's image; extracting from the received input data body shape parameters s characterizing the user's body shape, pose parameters p characterizing the user's pose on one or more user image frames, and camera parameters C characterizing the angle from which said one or more user image frames were taken; forming a deformable polygonal structure as a three-dimensional model of the user's body; forming an L-channel neural network texture characterizing the features of the user's appearance, based on the processing of said one or more frames of the user's image; impose L-channel neural network texture on the deformable polygonal structure; rendering the user's avatar in a new pose and/or from a new angle by means of a rendering neural network; and adjusting (training) the parameters of the rendering neural network and the neural network texture, by adjusting them to achieve matching of the received images of the user's avatar and one or more frames of the user's image from the received input data.

In accordance with another aspect of the present invention, a rendering of a user avatar is proposed, comprising the steps of: receiving input data in the form of body shape parameters s characterizing the shape of the user's body, pose parameters p characterizing the avatar pose, camera parameters C characterizing the angle from which an avatar must be displayed; forming a deformable polygonal structure as a three-dimensional model of the user's body; imposing the trained L-channel neural network texture on the deformable polygonal structure; and rendering the user avatar in a new position and/or from a new angle by means of the trained rendering neural network, wherein the training of the neural network texture and the rendering neural network is carried out by adjusting the parameters of the neural network texture and the rendering neural network to match the resulting images of the user avatar and one or more user image frames.

The rendering neural network used in the methods of the invention preferably has a convolutional architecture. It should be understood that other neural network architectures suitable for use in the practice of the present invention may be apparent to those skilled in the art.

As a non-limiting example, the number L of channels of the L-channel neural network texture is 16. However, this number of channels of the neural network texture corresponds to one preferred embodiment of the invention, and the number of channels of the neural network texture in the context of the present invention is not limited to this value, but may be more or less channels. The rendering neural network parameters are trained together with the L-channel neural network texture generator model, and the avatar L-channel neural network texture is obtained by inferring in the trained generator model and further tuning, as will be described in detail below. The training of the rendering neural network and the generative model of L-channel neural network textures is carried out according to the principle of adversarial networks.

The key component of the proposed solution is a generating (generating) neural network for creating the neurotexture of the user's avatar. The generating neural network is based, as a non-limiting example, on the StyleGANv2 generator (see, for example, the source [20]). Thus, to build a three-dimensional (3D) model of a user avatar in accordance with the present invention, the following three components are required: the formation (eg, rendering) of a polygonal structure, the formation of neurotextures, and rendering through a rendering neural network. These three components of the invention are essentially combined into one overall process that is adversarially trained on a large data set of full body images of people.

In addition, in accordance with the invention, the problem of ensuring that the appearance of the formed avatar remains unchanged in various poses and angles is solved, which is provided by an additional discriminator neural network, as well as by modifying the learning process of the aforementioned process, consisting of these three components.

Next, the first of the above components of the process according to the invention will be considered, namely the formation of a polygonal structure corresponding in general to a three-dimensional model of the human body, which is used as the basis for rendering the user's avatar. As stated above, this polygonal structure can also be referred to as essentially a "naked" person model in the sense that it corresponds to a generalized three-dimensional model of the human body without regard to hair, clothing, etc.

The polygon structure in question can also be referred to as a fixed topology polygon structure that is controlled by pose parameters and one or more body shape parameter vectors obtained as input from the results of processing one or more input images. To form a polygonal structure, the parameters of the pose, body shape from the input data received as a sequence of images, such as a video image of a user or at least one photo image of a user, are used.

The process according to the invention in one or more embodiments will be described below, in which the formation of a deformable polygonal structure is combined with the formation of a neurotexture and the use of a rendering neural network to obtain a user avatar based on an input video image.

, управляемую вектором параметров p позы и вектором параметров s формы тела. К полигональной структуре с фиксированной топологией может быть применена заданная функция нейротекстурирования, определяемая как R(M;T;C), в которой задействована полигональная структура с фиксированной топологией M, L-канальная нейротекстура T и камера C, при этом результатом работы этой функции нейротекстурирования является L-канальное растрирование полигональной структуры с нейротекстурой с использованием z-буферного алгоритма.The deformable polygon structure model (also referred to in the context of this invention as the naked person model), which can be based on, as a non-limiting example, the well-known SMPL-X model, forms a fixed topology polygon structure

, controlled by the parameter vector p of the pose and the vector of parameters s of the body shape. A given neurotexturing function defined as R(M;T;C) can be applied to a polygonal structure with a fixed topology, in which a polygonal structure with a fixed topology M, an L-channel neurotexture T and a camera C are used, while the result of this neurotexturing function is an L-channel rasterization of a polygonal structure with a neurotexture using a z-buffer algorithm.

As a neurotexture in the present invention, an L-channel texture T is used, which, as a non-limiting example, has the number of channels L=16. It should be noted, however, that the number of channels in the neurotexture is not limited to 16, and other smaller or larger positive integer channels may be used. The L-channel neurotexture is used to encode local photometric and geometric information (including the features of the geometry of the user's avatar model, which are absent in the parametric deformable polygonal structure.

с обучаемыми параметрами

для преобразования результата функции нейротекстурирования (в виде растрированного изображения) R в четырехканальное изображение I того же размера, в котором первые три канала соответствуют красному, зеленому и синему (RGB) цветовым каналам, а четвертый канал соответствует маске переднего плана.Next, to the neurotexture applied over the deformable polygonal structure, the rendering neural network f

with learnable parameters

to convert the result of the neurotexturing function (in the form of a rasterized image) R into a four-channel image I of the same size, in which the first three channels correspond to the red, green and blue (RGB) color channels, and the fourth channel corresponds to the foreground mask.

In the proposed invention, the avatar A is characterized by body shape parameters s _A and neurotexture T _A . By using a pre-trained rendering neural network f _Θ , a user avatar can be rendered for an arbitrary pose p and arbitrary camera parameters (in other words, the angle or viewpoint from which the viewer "looks" at the avatar) C. As a non-limiting example, according to With the present invention, the rendering process of a 3D user avatar is performed at a speed of about 25 frames per second at an image resolution of one megapixel.

, где

. Выполняется сегментирование человека и фона, например, с применением способа, описанного в источнике [11]. Следует отметить, что данный способ сегментирования указан исключительно в качестве неограничивающего примера, и могут также применяться и другие способы сегментирования, которые будут очевидны специалистам в данной области техники.The rendering neural network is pre-trained using the following process. If there is a collection of video images of several people, their avatars can be rendered by adapting the neural network model according to the invention. So, suppose that for person i, there is a set of video frames

, where

. Segmentation of the person and the background is performed, for example, using the method described in the source [11]. It should be noted that this segmentation method is given solely as a non-limiting example, and other segmentation methods may also be used, as will be apparent to those skilled in the art.

, а также параметры камеры

, которые соответствуют отдельным кадрам. Применение этих параметров с адаптацией к конкретной модели основано, в качестве неограничивающего примера, на модифицированном алгоритме SMPLify-X (см., например, источник [30]), который накладывает ограничения на параметры формы тела, которые должны быть общими для множества кадров.Next apply body shape parameters S _i and pose parameters

, as well as camera settings

, which correspond to individual frames. The application of these parameters with adaptation to a specific model is based, as a non-limiting example, on a modified SMPLify-X algorithm (see, for example, the source [30]), which imposes restrictions on body shape parameters that must be common to multiple frames.

нейросети рендеринга, а также нейротекстур T_i для всех людей из коллекции видеоизображений с использованием обратного распространения ошибки путем минимизации функции перцептуальных потерь (см., например, источник [17]), функции состязательных потерь (см., например, источник [16]) и функции потерь на сопоставление признаков (см., например, источник [40]) между контрольными изображениями

и изображениями, полученными путем рендеринга

.Then the parameters are optimized

rendering neural networks, as well as neural textures T _i for all people from a collection of video images using error backpropagation by minimizing the perceptual loss function (see, for example, the source [17]), the adversarial loss function (see, for example, the source [16]) and loss functions for feature matching (see, for example, the source [40]) between control images

and images obtained by rendering

.

The mentioned loss functions are used in this case to fill the color channels, while the cross entropy loss function is used for reference images and predictive masks. It is important to note that when predicting the mask, all pixels related to the parametric body model are assigned to the foreground.

Pre-training a rendering neural network based on a collection of video images of several people allows rendering of a user avatar based on at least one input video image. Thus, as a non-limiting example, in one practical implementation of the method according to the invention, a rendering neural network was pre-trained on a dataset of 56 people for which many thousands of image frames were collected. As a result, by using a pretrained rendering neural network, estimating the pose, body shape, and camera parameters for a limited number of frames depicting a user, generating a neurotexture for a given user, and then optimizing the neurotexture by backpropagating the corresponding loss functions, an avatar was created. After creating the avatar, the avatar was rendered for this user in various poses and from various angles.

The invention has been described above mainly in relation to rendering a user avatar based on a video image of a given user that contains a large number of user image frames. However, the invention is not limited to using a user's video as input, and in one or more implementations, using a small number (one or more) of a given user's photo images is sufficient as input for rendering a user's avatar. In this case, in addition to the means described above, the invention uses a generative (generating) model, which is essentially an addition to the process of neural network rendering of the user avatar described above.

Avatar backpropagation training makes it possible to obtain avatars from relatively short video images or even from several suitably spaced frames of a photo image (so that each part of the user's surface is shown in at least one view). In some practical embodiments, a full-length avatar rendering from a single image of the user in the most informative pose can be implemented. However, to implement this, the system implementing the process according to the invention needs to be able to evaluate that part of the avatar that is hidden in this image. To solve this problem, in one or more embodiments of the invention, a generative model for neurotextures is used, which can be used when rendering a user's avatar based on one or more frames of the user's photo image.

The generative (generating) model used, called the StylePeople model by the authors of the present invention, is based on developments in the field of application of neural networks in modeling two-dimensional images with high resolution (see, for example, sources [18, 7, 19]), and in particular it is a further development of the well-known neural network model Style-GANv2 (see, for example, the source [20]). It should be noted that when training the generative model, a pairwise discriminator is used, which guarantees a match between poses for the same avatar.

и сверточной нейросети

, которая принимает в качестве входных данных набор из 512-мерных «стилевых векторов», управляющих генерацией с различными величинами разрешающей способности (от 4×4 до 512×512) посредством механизма модуляции-демодуляции. Кроме того, используется также набор из N «шумовых карт» с упомянутыми величинами разрешающей способности. В результате получается порождающая модель

, в которой случайный вектор z обрабатывается посредством многослойного перцептрона (MLP) и затем используется в качестве входных данных в нейросеть рендеринга.The Style-GANv2 model (see, for example, reference [20]) is generally used to model high-resolution images. Multilayer Perceptron (MLP) is being trained

and convolutional neural network

, which takes as input a set of 512-dimensional "style vectors" that control generation at various resolutions (from 4x4 to 512x512) through a modulation-demodulation mechanism. In addition, a set of N "noise maps" with the mentioned resolution values is also used. The result is a generating model

, in which a random vector z is processed by a multilayer perceptron (MLP) and then used as input to the rendering neural network.

The prediction (inference) algorithm in the generative model used to obtain a user avatar from one or more photographic images is based on the so-called extended generation space (see, for example, sources [19, 1]), training of auxiliary coding neurons (see, for example, sources [20, 26]), the use of feature matching losses in the discriminator (see, for example, source [29]), as well as fine tuning of the generative model to achieve the best fit (see, for example, sources [47, 5, 29 ]).

On FIG. 2 illustrates an exemplary embodiment of the invention using a generative model based on the StyleGANv2 model mentioned above, as well as the rendering neural network described above. In this case, the generative model is used to generate neurotextures that are superimposed on SMPL-X polygonal structures, after which the combination of neurotextures and polygonal structure is processed through the mentioned rendering neural network. Also in FIG. Figure 2 shows a discriminator that is used in the adversarial training phase of the rendering neural network and that evaluates a pair of images of the same person.

The generation model according to the considered embodiment of the invention has two significant differences from the well-known StyleGANv2 model from the source [20]. First, as noted above, in this case it outputs an L-channel neurotexture, not an RGB image. Secondly, in the proposed approach, the input data of several subsequent layers of the rendering neural network is concatenated with a 16-channel map of spectral coordinates for the vertices of the polygonal structure, transformed into the space of neurotextures, to provide the generator with certain information about the topology of parts of the neurotexture. During training, a selection of poses and body shapes from the training data set is also performed, as well as the imposition of the formed neurotextures on the deformable polygonal structures of the “naked” person model. Then the resulting polygonal structures with superimposed neurotextures are processed by the rendering neural network, which is trained as part of the generative model.

In general, the images generated by the generative model are calculated using the following expression:

, (1)

, (one)

where z and N are variables that are selected from unit normal distributions, while variables characterizing posture, camera, and body shapes are selected from an empirical distribution in the training dataset.

Adversarial training of the neural network model of artificial intelligence is controlled by a discriminator similar to that used in the StyleGANv2 architecture, however, in the general case, several types of discriminators are used in the process according to the invention:

1) The unary discriminator generally corresponds to that used in the well-known Style-GANv2 model, and it is based on the use of individual images (both generated by the generative model and real images) and is mainly used to evaluate image quality.

и

, где z, N (и, соответственно, нейротекстура), а также параметры s формы тела являются общими для обоих экземпляров. Параметры p ₁ , p ₂ позы и положения C₁, C₂ камеры являются различными в каждой паре изображений и соответствуют двум кадрам одного и того же видеоизображения. Таким образом, назначение дискриминатора состоит как в оценке визуальной «реалистичности» примеров, так и в обеспечении сохранения идентичности при смене положения камеры и смене позы.2) The binary discriminator is basically based on the same known model, but it uses pairs of images rather than single individual images. Each "real" example is taken from two segmented frames of the same segmented video image. To get a "fake" example, two instances are formed -

and

, wherez, N (and, accordingly, neurotexture), as well as parameterss body shapes are common to both instances. Optionsp _one , p ₂ postures and positions C_one, C₂ the cameras are different in each pair of images and correspond to two frames of the same video image. Thus, the purpose of the discriminator is both to evaluate the visual "realism" of the examples, and to ensure that the identity is preserved when changing the camera position and changing the pose.

3) The face discriminator uses individual images obtained by cropping around the face area (for both real images and synthesized ones) and is used to improve the quality of face rendering in the generative model.

, который использует сформированные изображения

и пытается восстановить вектор

. Потери при обучении

компенсируются путем обратного распространения ошибки по всей порождающей модели, что гарантирует целостность изображений, формируемых с использованием одной и той же нейротекстуры.When training an artificial intelligence model in the method according to the invention, an additional regularization step can be performed to ensure identity is preserved, since the use of a pairwise discriminator alone is not sufficient. Regularization uses three additional techniques. The first trick is to train the predictor

, which uses generated images

and trying to restore the vector

. Loss in training

are compensated by backpropagating the error throughout the generative model, which guarantees the integrity of images generated using the same neurotexture.

применяют случайное жесткое преобразование Tr в пределах некоторой плоскости, обрабатывают исходные и преобразованные входные данные посредством сети

рендеринга, после чего применяют штрафную функцию к разности между

и

.The second trick in regularization is to ensure that the rendering neural network is covariant with respect to the geometric transformation of its input data within a certain plane. This is ensured by the fact that to rasterized images

apply a random hard transformation Tr within a certain plane, process the original and transformed input data through the network

rendering, after which a penalty function is applied to the difference between

and

.

Finally, the third technique is performed to reduce the impact of poorly segmented images from the training dataset, as well as those images where at least part of the user's body is not visible, the foreground masks in the generated samples are forced to cover the entire polygonal structure received as input data. To do this, instead of the discretized segmentation mask, its combination with the binary mask of the polygonal structure is used as the final segmentation of the foreground.

изображением путем регулирования получаемой текстуры при помощи обратного распространения ошибки. При наличии изображения человека I (с оцененной маской сегментирования) оцениваются параметры s формы тела, параметры p позы для полигональной структуры SMPL-X, и параметры камеры C. Целью дальнейшей обработки является формирование такой нейротекстуры T, при которой нейросеть рендеринга

выдает изображение, которое соответствует наблюдаемому изображению. Параметризация нейротекстуры осуществляется посредством сверточного генератора

, и таким образом оптимизация выполняется по отношению к 512-мерным «стилевым векторам» w ⁴ ,... w ⁵¹² и тензорам шума N. Стилевые векторы инициализируются выходными данными предиктора

, и в процессе оптимизации применяется регуляризация тензоров шума N. При необходимости, в качестве последнего этапа оптимизации выполняется тонкая регулировка параметров

порождающей модели

. Параметры

нейросети рендеринга не изменяются процессом адаптации.As stated above, the proposed generative model can be used to create and/or render avatars of real users based on one or more (multiple) photo images. So, if there is one full-length image of a person, the segmentation mask, the parameters p of the pose and the parameters s of the body shape, as well as the parameters C of the camera are evaluated. Then, the adaptation loss between the segmented input image and the rendered one is minimized.

image by adjusting the resulting texture using backpropagation. Given an image of a person I (with an estimated segmentation mask), body shape parameters s, pose parameters p for the SMPL-X polygonal structure, and camera parameters C are estimated. The goal of further processing is to form a neurotexture T such that the rendering neural network

produces an image that matches the observed image. Parameterization of the neurotexture is carried out by means of a convolutional generator

, and thus the optimization is performed with respect to the 512-dimensional "style vectors" w ⁴ , ... w ⁵¹² and the noise tensors N. The style vectors are initialized with the output of the predictor

, and the optimization process applies the regularization of the noise tensors N . If necessary, the parameters are fine-tuned as the last stage of optimization

generative model

. Options

Rendering neural networks are not changed by the adaptation process.

. Инициализируются латентные переменные w путем пропускания изображений I через одну или более предварительно обученных кодирующих нейросетей. Если доступно более одного изображения на входе (т.е. N > 1), прогнозируемые латентные переменные усредняются для всей совокупности изображений. Для простоты рассмотрим пример, в котором на входе имеется только одно изображение I. Оптимизация выполняется по (i) латентным переменным w, (ii) параметрам генератора h, и (iii) тензорам шума N для дополнительной минимизации различия между полученным изображением

и I. В качестве последнего этапа оптимизации выполняется непосредственная оптимизация значений нейротекстуры только за 100 итераций.In one or more specific embodiments of the present invention, the optimization process receives as input a set of images of the same person I, and more specifically

. Latent variables w are initialized by passing images I through one or more pre-trained encoding neural networks. If more than one input image is available (i.e., N > 1), the predicted latent variables are averaged over the entire population of images. For simplicity, consider an example where there is only one image I at the input. Optimization is performed on (i) latent variables w, (ii) generator parameters h, and (iii) noise tensors N to further minimize the difference between the resulting image

and I. As the last optimization step, the neurotexture values are directly optimized for only 100 iterations.

The optimization uses many loss functions, for example, the LPIPS loss function (see source [48]) between the received image and the input image, the standard deviation (Mean Squared Error (MSE)) between the received image and the input image, the mean absolute error (Mean Absolute Error (MAE)) in the deviation of the latent variables w from the initialization values predicted by the encoding neural network, MAE in the deviation of the generator parameters h from the initial values, MAE in the deviation of the neurotexture from the neurotexture values optimized at the beginning of the mentioned last stage, the LPIPS loss function ([48] ) for face areas in acquired and input images, the feature matching loss function based on the trained face discriminator (see reference [29]). If several images are available at the input, the loss functions are averaged over the entire set of images.

If the texture parameterization is redundant (i.e., the number of neurotexture elements typically exceeds the number of observations in I), very different sets of latent variables are available to adapt to the observed image, which lead to different levels of generalization with respect to new poses and camera parameters.

посредством выборки z значений и их обработки посредством перцептрона. Тензоры шума получают из нормального распределения, и сверточный генератор формирует случайную нейротекстуру. Затем нейротекстура накладывается на случайное изображение тела из обучающей выборки, находящегося в позе, приближенной к «А»-образной позе и снятое по существу фронтально расположенной камерой.Thus, in one or more specific embodiments of the present invention, the training of so-called encoding neural networks is used, which initialize the latent variables w to obtain a good generalization with respect to new poses and camera parameters. To do this, based on the generative model for generating neurotextures, a data set of synthetic samples is formed. In particular, to obtain the kth sample, randomly and independently, the values

by fetching z values and processing them by the perceptron. The noise tensors are obtained from the normal distribution and the convolutional generator generates a random neurotexture. The neurotexture is then superimposed on a random image of the body from the training sample, in a pose close to the "A"-shaped pose and filmed with an essentially frontal camera.

для восстановления векторов

из изображения I_k, который обучается посредством функции потерь L1 на синтетических данных, полученных только от порождающей модели, описанной выше.The rendering neural network then generates an image I _k for a random avatar in an "A"-shaped pose. After that, the so-called A-coding neural network is trained.

to restore vectors

from the image I _k , which is trained by the loss function L1 on synthetic data obtained only from the generative model described above.

получает пары реальных изображений

, извлеченных из одного и того же видеоизображения одного и того же человека, прогнозирует латентные переменные

на основании

, дополняет их случайными тензорами шума N, в результате чего получается нейротекстура

, которая далее накладывается на полигональную структуру, которая деформируется для создания определенной позы, после чего выполняется рендеринг на основании параметров SMPL-X

и параметров камеры

, полученных из изображения

. Затем применяется упомянутая выше функция потерь LPIPS между

и

, которая используется в качестве функции потерь по отношению к реальным данным. К синтетическим данным, аналогично описанной выше кодирующей нейросети

, используется функция потерь L1 между синтетическими и прогнозируемыми латентными векторами. Эти две функции потерь равным образом суммируются для обучения параметров кодирующей нейросети

.In addition to the aforementioned A-coding neural network trained only on synthetic data and suitable for images of avatars in an "A"-shaped pose, in one or more embodiments of the invention, a "general" encoding neural network (so-called G-coding neural network) can be additionally trained , whose training is based both on synthetic data and on a real subset of pairs of frames from the video image. For learning from real data G-coding neural network

gets pairs of real images

extracted from the same video image of the same person predicts latent variables

based

, complements them with random noise tensors N, resulting in a neurotexture

, which is then overlaid on a polygon structure, which is deformed to create a specific pose, after which it is rendered based on the SMPL-X parameters

and camera settings

obtained from the image

. The LPIPS loss function mentioned above is then applied between

and

, which is used as a loss function with respect to real data. To synthetic data, similar to the coding neural network described above

, the L1 loss function between synthetic and predicted latent vectors is used. These two loss functions are summed equally to train the parameters of the encoding neural network

.

The main point of using coding neural networks is that each level of the coding neural network predicts a latent vector corresponding to the resolution of the generator at this level. The authors of the invention adopted the EfficientNet-B7 network (see, for example, the source [37]) pre-trained on the basis of ImageNet (see, for example, the source [32]) as the basis for training coding neural networks.

Due to the use in the method according to the invention of a polygonal structure with a neurotexture superimposed on it and its processing by means of a rendering neural network trained on the basis of a generative model, a technical effect is achieved, which consists in rendering the user's avatar with the possibility of its animation based on body posture parameters, including facial expression parameters and hand positions. The result is a realistic video image of a three-dimensional model of the user's avatar.

The proposed solution according to the invention can be implemented using any computing device with sufficient computing power (eg GPU) and a display screen. As a non-limiting example, the present invention also provides a system for creating a user avatar and a system for rendering a user avatar, comprising a memory in which computer program instructions are stored and one or more processors which, under the control of said computer program instructions, execute(s) with the ability to implement functions corresponding to the steps of the methods for creating a user avatar and rendering a user avatar based on the above-described means and methods.

Those skilled in the art will appreciate the various combinations of hardware and software that can implement the system according to the second aspect of the present invention. As stated above, the system comprises one or more processors, as a non-limiting example, one or more graphics processing units (GPUs), as well as a memory (Read Only Memory (ROM), Random Access Memory (RAM), etc.) storing computer program instructions for implementing the respective processing algorithms and other operations described above for the method of the first aspect of the present invention. It will be apparent to those skilled in the art that said program instructions may be implemented in any suitable programming language and/or using any suitable programming environment, in computer executable code, and the like. In addition, computer program instructions controlling one or more processors for implementing the method of the invention may be stored on any suitable form of computer readable medium, such as a permanent computer readable medium, volatile and/or nonvolatile computer readable medium, and may also be transmitted over any suitable wired and/or wireless data network. The processing for executing the algorithms underlying the method according to the invention may be performed on the basis of one or more public computers located in one place or distributed and connected by one or more data networks, and the like. In addition to one or more processors, one or more programmable logic integrated circuits (FPGAs), one or more microprocessors, and the like may also be used. The scope of the invention is not limited to any particular combination of software and/or hardware for implementing the algorithms described above.

Creating and/or rendering avatars requires photo and video input. Appropriate software or firmware may be used to analyze images, as well as to implement one or more of the neural networks underlying the present invention, which may be implemented in the form of one or more computer programs, computer program elements, program modules, and the like. . Said software may be stored in one or more memory elements of the system according to the second aspect of the invention.

The invention can be applied in various scenarios for implementing a virtual "remote presence" of a user, in particular, in augmented and virtual reality (AR / VR) systems, video games, teleconferencing, display on three-dimensional (3D) displays, etc. In addition, the invention can also be applied to any scenarios for displaying user images on conventional two-dimensional (2D) displays. Unlike the known solutions, the present invention implements image rendering based on an artificial intelligence neural network model to generate at least one user avatar image in a different position and/or from a different angle compared to the input images, displaying such user avatar image elements as clothing and/or hair, over a body model based on a parametric polygonal structure, as well as a combination of deep generative models for two-dimensional (2D) images used for neural network texture generation and neural network rendering.

The methods of the invention may be implemented by an electronic device and/or system capable of implementing the neural networks described above.

In contrast to the prior art solutions, which require at least several dozen images for each scene, in accordance with the present invention, it is proposed to use a deformable polygonal structure (the so-called “naked” person model) with an L-channel superimposed on it. a neurotexture simulating the structural features of the body, as well as clothes, hair, etc., with subsequent processing through a rendering neural network. The use of a generative model as described above makes it possible to create a photorealistic user avatar based on several frames of a photo or video image.

Those skilled in the art will appreciate that what has been described and shown in the drawings above are only some of the possible examples of techniques and facilities by which embodiments of the present invention may be implemented. The above detailed description of embodiments of the invention is not intended to limit or determine the scope of legal protection of the present invention.

Other embodiments that may be within the scope of the present invention may be contemplated by those skilled in the art upon careful reading of the foregoing description with reference to the accompanying drawings, and all such obvious modifications, alterations and/or equivalent substitutions are deemed to be within the scope of the present invention. All prior art references cited and discussed in this document are hereby incorporated into this specification by reference as far as applicable.

While the present invention has been described and illustrated with reference to various embodiments, it will be understood by those skilled in the art that various changes may be made to its form and specific details without departing from the scope of the present invention, which is defined only the following claims and their equivalents.

Bibliography

[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? Published in Proc. ICCV, pp. 4432-4441, 2019.

[2] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, and Gerard Pons-Moll. Learning to reconstruct people in clothing from a single rgb camera. Published in Proc. CVPR, pp. 1175-1186, 2019.

[3] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll. Video based reconstruction of 3d people models. Published in Proc. CVPR, pp. 8387-8397, 2018.

[4] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. Published in Proc. CVPR, pp. 7297-7306, 2018.

[5] David Bau, Hendrik Strobelt, William S. Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Trans. Graph., 38(4):59:1-59:11, 2019.

[6] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll. Multi-garment net: Learning to dress 3d people from images. Published in Proc. ICCV, pp. 5420-5430, 2019.

[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. Published in International Conference on Learning Representations, 2018.

[8] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body-driven attention. Published in Proc. ECCV, 2020.

[9] Haoye Dong, Xiaodan Liang, Ke Gong, Hanjiang Lai, Jia Zhu, and Jian Yin. Soft-gated warping-gan for pose-guided person image synthesis. Published in Proc. NeurIPS, pp. 474-484, 2018.

[10] Patrick Esser, Ekaterina Sutter, and Björn Ommer. A variational u-net for conditional appearance and shape generation. Published in Proc. CVPR, pp. 8857-8866, 2018.

[11] Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, and Liang Lin. Graphonomy: Universal human parsing via graph transfer learning. Published in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7450-7459, 2019.

[12] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate-based texture inpainting for pose-guided image generation. Published in Proc. CPR, 2019.

[13] Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott. Clothflow: A flow-based model for clothed person generation. Published in Proc. ICCV, pp. 10471-10480, 2019.

[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Published in I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pp. 6626-6637. 2017.

[15] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. Arch: Animatable reconstruction of clothed humans. arXiv preprint arXiv:2004.04572 , 2020.

[16] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. Published in Proc. CPR, 2017.

[17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. Published in Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Proc. ECCV, 2016.

[18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. Published in International Conference on Learning Representations, 2018.

[19] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Published in Proc. CVPR, pp. 4401-4410, 2019.

[20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen and Timo Aila. Analyzing and improving the image quality of StyleGAN. CoRR, abs/1912.04958, 2019.

[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

[22] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. JCV, 2020.

[23] Christoph Lassner, Gerard Pons-Moll, and Peter V Gehler. A generative model of people in clothing. Published in Proc. ICCV, pp. 853-862, 2017.

[24] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll. 360-degree textures of people in clothing from a single image. Published in Proc. 3DV, pp. 643-653. IEEE, 2019.

[25] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. Published in Proc. ICCV, pp. 5904-5913, 2019.

[26] Elizaveta Logacheva, Roman Suvorov, Oleg Khomenko, Anton Mashikhin, and Victor Lempitsky. Deeplandscape: Adversarial modeling of landscape videos. Published in Proc. ECCV, pp. 256-272. Springer, 2020.

[27] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. Published in Proc. NeurIPS, pp. 406-416, 2017.

[28] Natalia Neverova, Riza Alp Guler, and Iasonas Kokkinos. Dense pose transfer. Published in Proc. ECCV, pp. 123-138, 2018.

[29] Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. Proc. ECCV, 2020.

[30] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. Published in Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2019.

[31] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. arXiv preprint arXiv:2008.00951 , 2020.

[32] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.

[33] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. Published in D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pp. 2234-2242. 2016.

[34] Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, et al. Textured neural avatars. Published in Proc. CVPR, pp. 2387-2397, 2019.

[35] Aliaksandr Siarohin, Stéphane Lathuiliére, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Published in Proc. NeurIPS, pp. 7135-7145, 2019.

[36] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliére, and Nicu Sebe. Deformable gans for pose-based human image generation. Published in Proc. CVPR, pp. 3408-3416, 2018.

[37] Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 , 2019.

[38] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.

[39] Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. Published in Proc. CVPR, pp. 1526-1535, 2018.

[40] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. Published in Proc. CVPR, pp. 8798-8807. IEEE Computer Society, 2018.

[41] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Bryan Catanzaro, and Jan Kautz. Few-shot video-to-video synthesis. Published in Proc. NeurIPS, pp. 5014-5025, 2019.

[42] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. Published in Proc. NeurIPS, pp. 1144-1156, 2018.

[43] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.

[44] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. Monocular total capture: Posing face, body, and hands in the wild. In Proc. CPR, 2019.

[45] Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu and Hyun Soo Park. Humbi: A large multiview dataset of human body expressions, 2020.

[46] Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for poseguided human video generation. Published in Proc. BMVC, 2019.

[47] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. Published in Proc. ICCV, pp. 9459-9468, 2019.

[48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Published in Proc. CVPR, pp. 586-595, 2018.

[49] Kaiyang Zhou, Xiatian Zhu, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. Learning generalisable omni-scale representations for person re-identification. arXiv preprint arXiv:1910.06827 , 2019.

Claims (44)

1. A method for creating a user avatar, comprising the steps of: - receive input data in the form of one or more frames of the user's image; - extracting from the received input data body shape parameters s characterizing the user's body shape, pose parameters p characterizing the user's pose on one or more user image frames, and camera parameters C characterizing the angle from which said one or more user image frames were taken; - forming a deformable polygonal structure as a three-dimensional model of the user's body using body shape parameters s and pose parameters p; - form an L-channel neural network texture that characterizes the features of the user's appearance, based on the processing of the mentioned one or more frames of the user's image; - impose an L-channel neural network texture on a deformable polygonal structure; - rendering the user's avatar in a new position and/or from a new angle by means of a rendering neural network, and - training the parameters of the rendering neural network and the neural network texture by adjusting them to match the received images of the user's avatar and one or more frames of the user's image from the received input data. 2. The method according to claim 1, wherein the rendering neural network has a convolutional architecture. 3. The method according to claim 1, in which the number L of channels of the L-channel neural network texture is 16. 4. The method of claim 1, wherein the rendering neural network parameters are trained together with the L-channel neural network texture generation model, and the L-channel neural network texture of the avatar is obtained by inference in the trained generator model and further adjustment. 5. The method according to claim 4, in which the training of the rendering neural network and the generative model of L-channel neural network textures is carried out according to the principle of adversarial networks. 6. The method according to claim 1, wherein the rendering neural network is defined by the expression

, where

- neural network parameters, R - neurotexturing function, M - deformable polygonal structure determined by the parameters p of the pose and parameters s of the body shape, C - camera parameters, T - L-channel neurotexture, i - the user for whom the avatar is created, j - image number in the image set

. 7. A method for rendering a user avatar, comprising the steps of: receiving input data in the form of body shape parameters s characterizing the user's body shape, pose parameters p characterizing the avatar's pose, camera parameters C characterizing the angle from which the avatar should be shown; - forming a deformable polygonal structure as a three-dimensional model of the user's body using body shape parameters s and pose parameters p; - impose a trained L-channel neural network texture on a deformable polygonal structure and - rendering the user's avatar in a new position and/or from a new angle using a trained rendering neural network, wherein the training of the neural network texture and the neural network of rendering is carried out by adjusting the parameters of the neural network texture and the neural network of rendering to match the received images of the user's avatar and one or more frames of the user's image. 8. The method of claim 7, wherein the rendering neural network has a convolutional architecture. 9. The method according to claim 7, wherein the number L of channels of the L-channel neural network texture is 16. 10. The method of claim 7, wherein the rendering neural network parameters are trained together with the L-channel neural network texture generation model, and the L-channel neural network texture of the avatar is obtained by inferring in the trained generator model and further tuning. 11. The method according to claim 10, in which the training of the rendering neural network and the generative model of L-channel neural network textures is carried out according to the principle of adversarial networks. 12. The method according to claim 1, wherein the rendering neural network is defined by the expression

, where

- neural network parameters, R - neurotexturing function, M - deformable polygonal structure determined by the parameters p of the pose and parameters s of the body shape, C - camera parameters, T - L-channel neurotexture, i - the user for whom the avatar is created, j - image number in the image set

. 13. A system for creating a user avatar, comprising: - a memory in which computer program instructions are stored; and - one or more processors which, under the control of said computer program instructions, is(are) configured to: - receiving input data in the form of one or more frames of the user's image; - extracting from the received input data body shape parameters s characterizing the user's body shape, pose parameters p characterizing the user's pose on one or more user image frames, and camera parameters C characterizing the angle from which said one or more user image frames were taken; - forming a deformable polygonal structure as a three-dimensional model of the user's body using body shape parameters s and pose parameters p; - formation of an L-channel neural network texture that characterizes the features of the user's appearance, based on the processing of the mentioned one or more frames of the user's image; - overlaying an L-channel neural network texture on a deformable polygonal structure; - rendering the user's avatar in a new pose and/or from a new angle through the rendering neural network, and - training the parameters of the rendering neural network and the neural network texture by adjusting them to match the received images of the user's avatar and one or more frames of the user's image from the received input data. 14. User avatar rendering system, comprising: - a memory in which computer program instructions are stored; and - one or more processors which, under the control of said computer program instructions, is(are) configured to: - obtaining input data in the form of body shape parameters s characterizing the user's body shape, pose parameters p characterizing the avatar's pose, camera parameters C characterizing the angle from which the avatar should be displayed; - forming a deformable polygonal structure as a three-dimensional model of the user's body using body shape parameters s and pose parameters p; - overlaying a trained L-channel neural network texture on a deformable polygonal structure and - rendering the user's avatar in a new position and/or from a new angle using a trained rendering neural network, wherein the training of the neural network texture and the neural network of rendering is carried out by adjusting the parameters of the neural network texture and the neural network of rendering to match the received images of the user's avatar and one or more frames of the user's image. 15. A computer-readable medium that stores computer program instructions that cause the system to create a user avatar to perform the method of creating a user avatar according to any one of paragraphs. 1-6 as computer program instructions are executed by one or more processors. 16. A computer-readable medium that stores computer program instructions for causing the user avatar rendering system to execute the user avatar rendering method according to any one of claims. 7-12 as the computer program instructions are executed by one or more processors.

Images