RU2813485C1 - Method of creating full-length animated avatar of person from one image of person, computing device and machine-readable medium for implementation thereof - Google Patents

Abstract

FIELD: image processing means.

SUBSTANCE: invention relates to synthesis and processing of images based on artificial intelligence. An image of a person and a calculated parametric model of the body are obtained; determining a texturing function; human body RGB texture sampling is performed and a map of selected pixels is obtained; neural texture by concatenating RGB texture, map of selected pixels and created texture; non-observed texture regions of the neural texture are drawn using the trained diffusion painting model; rasterized image of an avatar of a person is transformed into a full-length image in a new position using a trained neural renderer into a rendered image of a full-height person in a new position.

EFFECT: simplification of the process of obtaining an avatar of a person and expansion of poses possible for processing.

20 cl, 9 dwg, 1 tbl

Description

Field of technology to which the invention relates

[0001] The present invention relates generally to the field of artificial intelligence (AI)-based image synthesis and processing, and in particular to a computer-implemented method for creating a life-size animated avatar of a person from a single image of that person, and to a computing device and a machine-readable medium for implementing this method.

State of the art

[0002] One of the drivers of the current increase in interest in photorealistic full-body avatars is the use of full-body avatars in virtual and augmented reality applications. In addition to the realism and accuracy of avatars, the ease of obtaining new personalized avatars is of great importance. To do this, a number of works propose methods for restoring a textured 3D human model from a single image, however, such models require additional effort to create a skeleton (rigging) for animation. Using additional rigging techniques greatly complicates the process of obtaining an avatar and often limits the poses that can be processed. At the same time, some of the latest methods use textured parametric models of the human body using inpainting in texture space. However, existing texture methods lack photorealism and rendering quality.

[0003] An alternative to using classic RGB textures directly is to use deferred neural rendering. Such approaches allow the creation of human avatars defined by a parametric model. The resulting avatars are more photorealistic and easier to animate. However, existing methods require a video sequence to create an avatar. The StylePeople system, also based on deferred neural rendering and a parametric model, allows you to create avatars from individual images, but suffers from poor rendering quality of body parts that are not visible in the image.

[0004] Many avatar creation systems are based on parametric models of the human body, the most popular of which are the SMPL body model, as well as the SMPL-X model, which augments the SMPL with facial expressions and hand articulation. Such models represent the human body without clothes and hair. Methods based on deferred neural rendering (DNR) or neural emission fields (NeRF) can be used to add clothing and perform photorealistic rendering. DNR uses a multi-channel trained neural texture and a convolutional network renderer to realistically render the resulting avatars. This makes it easier to animate the resulting avatar. NeRF renders by sampling along a ray in implicit space, allowing it to extract accurate and consistent geometry.

[0005] In single-shot avatar generation methods, human body avatars are reconstructed from a single image. In early works this was achieved by adding partial RGB textures. Such methods did not allow realistic modeling of avatars with clothing. More recent work on single-shot body modeling uses implicit geometry and luminance models to predict opacity and color using a multilayer perceptron driven by feature vectors extracted from the input image. Although this line of work can often recover complex geometric details, the reconstructed texture of parts not visible in the image usually has limited realism.

[0006] The ARCH system uses canonical space with a constructed skeleton to create avatars suitable for animation. ARCH++ improves the quality of resulting avatars by revising the ARCH milestones. They also solve the problem of unobservable surfaces in the image by synthesizing the rear view of a person from the front view. PHORHUM improves geometry modeling by adding judgment about scene illumination and surface albedo. ICON uses local features to decouple the reconstructed geometry from the global pose. In this method, individual models from each species are first estimated, and then these models are combined using SCANimate. This method uses RGB textures applied to the reconstructed geometry, which limits the photorealism of the rendering.

[0007] An alternative approach to deriving avatars from a single snapshot is to use avatar generation models as proposed in StylePeople. The authors eliminate the need to reconstruct unobservable parts of the image by exploiting the connectivity in the GAN's latent space. Unfortunately, imperfections in the underlying generative model often lead to implausible appearances of unobservable parts.

[0008] A well-known alternative to generative models are diffusion models. In the field of human modeling, diffusion models have shown high efficiency for solving the problem of generating human movements. In RODIN, a diffusion model is used to create head avatars without a constructed skeleton in the form of neural radiation fields represented by 2D feature maps. TEXTure uses text guidance and a pre-trained diffusion model to generate an RGB view-consistent texture for a given geometry. However, to the best of the inventors' knowledge, diffusion models have not been used to generate neural textures for 3D objects to date.

The essence of the invention

[0009] The present invention overcomes or at least partially solves one or more of the prior art problems discussed above. Thus, a new method has been proposed for creating photorealistic animated human avatars from a single photograph. To make these avatars suitable for animation, a neural texture-based approach is used along with the SMPL-X parametric body model. A new architecture is proposed for generating neural textures, in which the texture contains both an RGB part, explicitly extracted by sampling pixels from the input image, and additional neural channels obtained by mapping the image into a latent vector space and decoding the result into texture space. Training texture generation using a rendering network is carried out using an end-to-end method. A diffusion model was developed to recover neural texture for unobservable parts of the human body in the image. This approach makes it possible to obtain photorealistic human avatars from single images. Given multiple images, neural textures corresponding to different images can be combined, and still missing parts can be restored through diffusion inpainting. The difference between the present invention and the prior art, including the StylePeople model, based on a generative adversarial structure, for finishing textures of the human body, is, at least, in the use of diffusion for finishing. The use of diffusion has been found to reduce problems with learning collapse and to produce plausible samples from complex multimodal distributions.

[0010] Thus, the contributions of the present invention to the prior art include:

• a new approach to modeling human avatars based on neural textures combining RGB components and hidden components; And

• a diffusion model adapted to neural textures, which, as has been demonstrated and confirmed experimentally, is capable of completing such textures.

[0011] The ability of the proposed method to create realistic animated avatars from a single image or photograph has been demonstrated and experimentally verified. Experimental data are given at the end of this description. The disclosed method makes it possible to obtain a photorealistic animated avatar of a person from a single image. For example, for illustration, having received one image or photograph of a person from the front, the proposed method will allow photorealistic and reliable reconstruction of the view of a person from behind. The effectiveness and accuracy of the proposed method was demonstrated and experimentally validated on real images taken from the publicly available Snapshot People dataset and on images of people in natural poses.

[0012] Based on the foregoing, a first aspect of the present invention provides a computer-implemented method for creating a full-length animated human avatar from a single image of a person. The method includes the steps of: obtaining an image I _rgb of a person and a calculated parametric model of the body, determined by the values of the pose parameters p and the shape parameters s of the human body in the image, and the values of the camera parameters C used in capturing the image; based on the parametric model of the body, a texturing function F _UV is determined, which specifies the mapping between each pixel of the human body observed in the image and the corresponding texture coordinates in texture space, and for all pixels of the human body unobserved in the image, the texturing function additionally sets the corresponding texture coordinates in texture space; sample the RGB texture T _rgb of the human body based on the display and obtain a map B _smp of the selected pixels, wherein the RGB texture T _rgb contains for each pixel of the human body observed in the image the corresponding pixel value, and also contains one or more unobserved texture areas corresponding to the texture coordinates unobservable pixels of the human body in the image; create a texture T _gen of the human body observed in the image by passing the image through a trained network of encoder E - generator G; obtain a neural texture T by concatenating the RGB texture T _rgb , the map B _smp of the selected pixels and the generated texture T _gen ; carry out additional drawing of unobserved texture areas of the neural texture T using a trained diffusion model of additional drawing; convert the rasterized image R of a full-length human avatar in a new pose using a trained neural renderer θ into a rendered image I _rend of a full-length human avatar in a new pose, wherein the rasterized image R is obtained from the completed neural texture T _inpainted and the display specified by the texturing function F _UV , in which the parametric body model is modified based on target pose parameters p _target and/or camera parameters are modified into target camera parameters C _target , wherein target pose parameters p _target and/or target camera parameters C _target correspond to said new human avatar pose in full height.

[0013] As a development of the first aspect of the invention, the method additionally performs additional drawing in the selected RGB texture of gaps not exceeding a given threshold size by averaging neighboring pixels and obtaining a B _fill map of additional drawn pixels, and the B _fill map of additional drawn pixels is additionally used during concatenation at the stage obtaining the neural texture of T.

[0014] In a further development of the first aspect of the invention, the method step of drawing S125 of unobserved texture regions of the neural texture T is performed by adding Gaussian noise to the neural texture T and then iteratively performing a noise removal procedure on the neural texture T with the Gaussian noise T ^noise times using a trained diffusion models until the completed neural texture T _inpainted appears instead of Gaussian noise.

[0015] In a further development of the first aspect of the invention, the parametric body model is based on a mesh M with a fixed topology, controlled by the values of the pose parameters p and the shape parameters s.

[0016] In a further development of the first aspect of the invention, the encoder E of the trained encoder network E - generator G is based on the StyleGAN2 discriminator and is trained to compress the I _rgb image into a feature vector , and the generator G of the trained encoder network E - generator G is based on the StyleGAN2 generator and is trained to create texture T _gen from a feature vector .

[0017] In a further development of the first aspect of the invention, the display is created by UV unwrapping performed on a front view of the human body in the image.

[0018] In a further development of the first aspect of the invention, the method further finds an occluded region of the human body in the image I _rgb and excludes this occluded region when sampling the RGB texture T _rgb .

[0019] In a further development of the first aspect of the invention, the trained diffusion inpainting model is based on a noise dispersion probabilistic model (DDPM) having a noise-removing U-Net architecture trained to inpaint unobserved texture regions of a neural texture T to produce _{an inpainted} neural texture T in which the denoising The noise-removing U-Net architecture has BigGAN residual blocks for upsampling and downsampling and attention layers at a number of levels in the feature hierarchy of the denoising U-Net architecture.

[0020] In a further development of the first aspect of the invention, the trained diffusion inpainting model further comprises a vector quantized generative adversarial network (VQGAN)-based autoencoder including an E _VQ encoder trained to encode the neural texture T into a lower dimensional latent representation of the neural texture T input into a U-Net denoising architecture, and a D _VQ decoder trained to decode the output of the U-Net denoising architecture into a T _inpainted neural texture.

[0021] In a further development of the first aspect of the invention, the training of the encoder network E - generator G, the diffusion model of completion and the neural renderer θ is carried out in two stages of training:

at the first stage of training:

said encoder network E - generator G and said neural renderer θ, based on a U-Net rendering architecture having ResNet blocks that form a pipeline, are trained end-to-end based on sets of multi-view images used as training data,

at each stage of learning from the many stages of learning during learning at the first stage of learning:

select two different images from the same set of multi-view image sets: a first image showing a person in a pose having input pose parameters p _input used as the input image, and a second image showing a person in a pose having different target pose parameters p _target , used as the reference image (I _G ),

passing the first image and target pose parameters p _target through a pipeline trained to render a render image I _rend showing a person's avatar in a pose having target pose parameters p _target .

calculating, based on at least the render image I _rend and the reference image I _GT , a loss value in accordance with one or more of the following loss functions: loss L ₂ differences between the render image I _rend and the corresponding reference image I _GT , loss of learned perceptual similarity between image patches, LPIPS, between the render image I _rend and the corresponding reference image I _GT , a non-saturating adversarial loss based on the StyleGAN2 discriminator with R ₁ regularization, and a Dice loss between the predicted segmentation mask s _pred and the reference segmentation mask s _GT,

calculate gradients based on the loss value, and

updating the parameters of the encoder network E - generator G and neural renderer θ based on these gradients; And

after the value of losses according to one or more loss functions is minimized, the learned parameters of the network of encoder E - generator G and neural renderer B are fixed and proceed to the second stage of training;

at the second stage of training:

a diffusion inpainting model comprising at least a denoising U-Net architecture is added to the pipeline and trained using conditional diffusion learning to inpaint unobserved texture regions of the neural texture T, wherein

at each stage of training from the many stages of training during training in the second stage of training:

obtain a pooled neural texture used as a reference neural texture, wherein the pooled neural texture is obtained by combining partial neural textures obtained from two or more images covering different viewing angles and selected from the same set of multi-view images from the sets of multi-view images used as training data,

add Gaussian noise to the combined neural texture according to a random step t, where t=[0, T ^noise ],

concatenate to the combined neural texture with added Gaussian noise the incomplete neural texture from the incomplete neural textures used to obtain the combined neural texture,

pass the combined neural texture with added Gaussian noise and the concatenated incomplete neural texture through an inpainting diffusion model, which predicts at step t the noise that should be removed to obtain the inpainted combined neural texture,

calculating, at least based on the predicted noise and the added Gaussian noise, a loss value in accordance with the loss function L ₁ ,

calculate gradients based on the loss value, and

update the parameters of the U-Net denoising architecture used for gradient-based diffusion inpainting; And

Once the loss value is minimized, the pretrained pipeline is fine-tuned by performing many additional training steps.

[0022] In a further development of the first aspect of the invention, fine tuning includes fixing the weights and biases of the neural renderer θ and propagating gradients from the differentiable neural renderer θ to the RGB channels of the RGB T _rgb texture, said gradients being calculated based on the loss value obtained in the first stage of training . For example, a neural texture may contain, as will be detailed below, 20 or 21 channels. Three of the specified channels are RGB channels, and the remaining channels are neural. In the described propagation process, RGB channels are represented as one or more modifiable value tensors, RGB colors are adjusted by changing the values in said tensors using gradients based on a loss function.

[0023] In a further development of the first aspect of the invention, if an (optional) VQGAN-based autoencoder is further included in the trained inpainting diffusion model, then the VQGAN-based autoencoder is further trained in the first training stage to encode the neural textures T into lower-dimensional latent representations of the neural textures T and decode lower-dimensional latent representations of neural textures T by minimizing a loss value calculated from one or more loss functions used in the first stage of training, and then training in the second stage of training is performed in the latent space of the pretrained VQGAN-based autoencoder.

[0024] In a further development of the first aspect of the invention, in the first training stage, the L ₂ difference loss and the LPIPS loss are calculated for the entire image and are further calculated with a given weight for the region of the image containing the face.

[0025] In a further development of the first aspect of the invention, the fusion is based on weighted averaging, pyramidal mixing, or Poisson mixing of neural textures.

[0026] In a further development of the first aspect of the invention, the second stage of training uses probability decimation in the residual blocks of the U-Net denoising architecture.

[0027] In a further development of the first aspect of the invention, sets of multi-angle images contain images of people wearing different clothes, different body types, different skin tones, different genders and in a variety of different poses, taken from different viewing angles.

[0028] In a further development of the first aspect of the invention, the viewing angle is defined as the angle between the normal vectors of the corresponding grid point M with a fixed topology and the camera direction vector.

[0029] In a second aspect of the invention, there is provided a computing device comprising a processor and a memory storing processor-executable instructions and parameters containing at least weights and biases of a trained encoder network pipeline E - generator G, a diffusion inflation model, and a neural renderer θ, wherein, when the processor executes processor-executable instructions, the processor causes the computing device to execute a method of creating a life-size animated avatar of a person from a single image of a person in accordance with the first aspect of the invention or any development thereof.

[0030] In a further development of the second aspect of the invention, the computing device further comprises a camera configured to capture an I _rgb image of a person.

[0031] In the third aspect of the invention, a non-volatile computer-readable medium is proposed that stores computer-executable instructions and parameters containing at least the weights and biases of a trained encoder network pipeline - generator G, a diffusion model of completion and a neural renderer θ, characterized in that the execution by the computing device of computer-executable The instructions cause the computing device to execute a method of creating a life-size animated human avatar from a single image of a person in accordance with the first aspect of the invention or any development thereof.

Brief description of drawings

[0032] The above and other features of the present invention will be described in detail in the following section "Detailed Description of the Invention" with reference to the accompanying drawings, in which:

Fig. 1 is a flow diagram of a computer-implemented method for creating a full-length animated human avatar from a single image of a person in accordance with the first aspect of the present invention.

Fig. 2 is a non-limiting schematic diagram of an entire pipeline proposed for a computer-implemented method of creating a full-body animated human avatar from a single image of a person in accordance with the first aspect of the present invention.

Fig. 3 is a non-limiting schematic diagram illustrating the operation and other details of a proposed diffusion inpainting model comprising an (optional) VQGAN-based autoencoder and U-Net denoising architecture used in a method in accordance with the first aspect of the present invention; The figure shows details of the "VQGAN" block shown in FIG. 1, and also shows in general the stages of training and use of the U-Net noise-removing architecture.

Fig. 4 is a schematic illustration of a non-limiting implementation of an E encoder architecture from an E encoder-G generator network used in a method according to a first aspect of the present invention for generating a neural texture core portion T _gen .

Fig. 5 is a schematic illustration of a non-limiting implementation of a neural renderer θ used in the method according to the first aspect of the present invention for converting a rasterized full-length human avatar image R in a new pose into a render image I _rend .

Fig. 6 is a simplified representation of the neural texture fusion step to obtain a complete neural texture from different images of a multi-view image set covering different viewing angles.

Fig. 7 is a schematic diagram of an optional step of detecting an occluded region of a person's body in an I _rgb image.

Fig. 8 is an illustration of animated full-body avatars of people in various poses created by the method according to the first aspect of the present invention based on the corresponding single images (shown on the left side of the figure).

Fig. 9 is a non-limiting schematic diagram of a computing device in accordance with a second aspect of the present invention.

Detailed Description of the Invention

[0033] Diffusion models are probabilistic models for learning the distribution p(x) by gradually removing noise from a normally distributed variable. This noise removal corresponds to learning the inverse process for a fixed Markov chain of length T ^noise . The most successful image generation models use a reweighted version of the variational lower bound p(x). These models can also be interpreted as noise-removing autoencoders ∈ _ω (x _t ,t); t=1,…, T ^noise with common weights. Such autoencoders can be trained to predict x _t-1 with reduced noise compared to x _t . It is now known that these denoising models can be trained using a simplified loss function:

where E (mathematical expectation) represents the averaging operation, t is selected uniformly from {1,…, T ^noise }, ∈ is noise from the normal Gaussian distribution N(0, 1) with mean 0 and variance 1, ∈ _W is noise predicted denoising U-Net with trainable parameters w. Typically, at each training step, the denoising U-Net takes as input a noisy image x _t and a number of steps t corresponding to the amount of added Gaussian noise, and predicts the noise that needs to be removed to obtain a denoised image. The loss L ₁ between the added Gaussian noise and the predicted noise is then calculated. To complete neural texture inpainting, the proposed method uses latent diffusion, which has been shown to be effective in inpainting RGB images. As used herein, the term "neural texture" generally means a texture having an arbitrary number of channels, the values of which are selected by gradient methods based on the calculation of a loss function. Rendering such a texture to RGB is done by a neural renderer, which converts the rasterized 3D model (i.e. mesh) with such a texture into an RGB image.

[0034] A computer-implemented method for creating a life-size animated avatar of a person from a single image of that person consists of two main parts: avatar generation components and a diffusion augmentation model. This division is optional, but is used in this description for ease of explanation. In fig. 1 shows a block diagram of the proposed method. In fig. 2 shows a non-limiting schematic representation of the entire conveyor used in this method; in this representation, the diffusion model of the inpainting is schematically represented by the "VQGAN" block, and everything else shown in Fig. 2, corresponds to the avatar creation components. In fig. Figure 3 shows a non-limiting schematic diagram illustrating the operation and other details of the proposed diffusion completion model.

[0035] In general, the proposed method reconstructs a neural texture from an input image of a person using two paths, and then uses a texturing operation and neural rendering to synthesize plausible avatar images corresponding to a person in different poses, and, optionally, from different viewing angles. A sequence of such plausible images can be used to animate an avatar. In this description, the rendering model is referred to as a "diffusion rendering model" because it is trained using a noise dispersion probabilistic model (DDPM) on top of pretrained avatar generation components.

[0036] The method in accordance with the first aspect of the present invention generates 3D avatars with a built skeleton of clothed people using a pipeline including at least an E-generator encoder network G, a diffusion inflation model and a neural renderer trained together in an end-to-end manner . In step S100, the method receives an RGB image I _rgb and a parametric body model as input. In a preferred embodiment, the parametric body model is an SMPL-X parametric body model. However, other known parametric body models can be used. During training, SMPL-X models are fit to sequential images using the SMPLifyX principle with an additional segmentation Dice loss between the predicted segmentation mask and the reference segmentation mask, allowing for improved matching of human silhouettes.

[0037] More specifically, one embodiment of the present invention uses a fixed topology SMPL-X M(p; s) mesh controlled by sets of pose parameters p and shape parameters s. In method step S105, a UV map function F _UV (M(p _target , s), C _target ) for texture mapping is determined. For SMPL-X, you can optionally use a specially created UV-unfolder with a front cut to avoid difficult to finish seams in the rear view. However, other types of incisions may be used. The rendering process takes as input the mesh M and the parameters of the desired camera C _target . The pose parameters p _target can be used to rig the mesh. The texturing function F _UV generates a UV map of size H × W × 2, where H and W determine the size of the output image, and for each (observed or unobserved) pixel the texture coordinates [i, j] are specified on the L-channel texture T. In other words A UV map can be thought of as a two-channel "texture" that indicates which texel (texture element) corresponds to which vertex of the body model. Therefore, the texturing function F _UV can be used in the rasterizer R (F _UV , T) to map the pixels of the output image into the texel features of the neural texture T. Thus, the rasterizer R produces an image of size H × W × L.

[0038] The parameters of the rasterizer R can be set such that H and W match the height and width of the input RGB image I _rgb . In this case, the UV map function F _UV can be used not only to display feature vectors from the neural texture T, but also to sample color values from the input image I _rgb into texture space: T _rgb =ξ (F _UV (M (p _input , s), C _input ), I _rgb ). Here the number of channels is L=3, p _input corresponds to the human pose in I _rgb , and C _input ^is the camera parameters recovered from I _rgb . The mapping ξ transfers the color value from I _rgb to the texture point T _rgb determined by F _UV in method step S110. This RGB texture allows you to explicitly preserve information about high-frequency details and original colors (as discussed below), which are difficult to preserve when mapping the entire image to a limited-dimensional vector. Additionally, a simple average padding may be required to remove gaps that appear in the texture due to the discrete sampling grid. Therefore, it is possible to apply padding to small gaps (for example, gaps not exceeding a specified threshold size of n pixels, where n is, for example, 20 pixels or less, 15 pixels or less, 10 pixels or less, 5 pixels or less, 2 pixels, or even one pixel), for example, by averaging neighboring pixels to fill gaps in T _rgb . The binary map B _smp of the selected pixels and the binary map B _fill of the selected and completed pixels can be saved for later use when generating a neural texture.

[0039] To perform RGB texture sampling in method step S110, the following non-limiting RGB texture sampling algorithm may be used, averaging pixels adjacent to the gap(s):

Algorithm 1. RGB texture sampling algorithm

Requires: RGB (size × size × 3)

Requires: UV (size × size × 2)

# Initialize texture to zeros

T ← zeros (texture size × texture size × 3)

C ← zeros (texture size × texture size)

# Filling texels with the average of neighbors

for ∀x, y ∈ [0. size] execute

(i, j) ← UV [x; y]

for ∀k, m ∈ [-1, 0, 1] execute

T [i+k, j+m]+=RGB[x; y]

With [i+k, j+m]+=1

finish for

finish for

T=T/S

# Filling exact values in texels that do not require additional painting

for ∀x, y ∈ [0. size] execute

(i, j) ← UV [x; y]

T [i, j] ← RGB[x; y]

finish for

[0040] Algorithm 1 presented above should not be interpreted as the only possible algorithm for RGB texture sampling, since one skilled in the art will be able to suggest another algorithm to implement the same or similar functions. Therefore, Algorithm 1 above should be interpreted as a non-limiting example.

[0041] The main part of the neural texture is T _gen . It has a number of channels L=16 and is created in method step S115 using the trained encoder network E - generator G. Therefore, T _gen =G (E (I _rgb )). Encoder E compresses the input image I _rgb into a feature vector . In fig. Figure 4 schematically shows a non-limiting implementation of the E encoder architecture. As the E encoder architecture, the inventors adapted the StyleGAN2 discriminator architecture with some modifications. In particular, three inputs are supplied to the network input: an RGB image I _rgb , a segmentation mask S and an additional single-channel noise. Additional single-channel noise is introduced to provide additional freedom to the generative model when training the GAN. The effectiveness of using noise in generative neural networks has been demonstrated by the authors of StyleGAN. The images received as input are concatenated by channels and passed through a feature extractor with an architecture equivalent to the StyleGAN discriminator consisting of ResNet blocks. The head block of the model was modified to output a feature vector of dimension 512. This vector is then used as the generator input , and the proposed encoder is trained end-to-end using this generator and renderer. Generator has a StyleGAN2 generator architecture and transforms a feature vector into the main part of the T _gen neural texture. T _gen has the number of channels L=16, as in StylePeople.

[0042] The final (but without additionally drawn unobservable areas in the image) neural texture T used in this method has a size of 256 × 256 × 20 or 256 × 256 × 21. The final neural texture T is obtained at step S120 of the method by concatenating the RGB texture T _rgb (256 × 256 × 3), map B _smp of selected pixels, generated texture T _gen (256 × 256 × 16) and, if necessary, map B _{fill of} completed pixels:

T=T _gen ⊕ T _rgb ⊕ B _smp (2a), or

T=T _gen ⊕ T _rgb ⊕ B _smp ⊕ B _fill (2b)

[0043] After obtaining the final neural texture, the method proceeds to step S125 of inpainting the unobserved texture regions of the neural texture T using the trained diffusion inpainting model to obtain T _inpainted . The neural renderer θ (R(F _UV , T _inpainted )) then converts the rasterized image R (F _UV , T _inpainted ) with L channels into an output RGB image I _rend in method step S130. The neural renderer θ has a U-Net architecture with ResNet blocks. In fig. Figure 5 schematically shows a non-restrictive implementation of the neural renderer θ. The neural renderer θ takes three images as input: a rasterized image R of the SMPL-X body model with a completed neural texture T _inpaintedf a UV renderer and a UV mask. A UV render is a rasterized 3D model (i.e. a mesh) in the form of a two-channel image, where each pixel is a coordinate on the texture from which the corresponding color value is taken. The UV mask determines which UV render pixels have texture coordinates and which UV render pixels do not have texture coordinates. In the latter case, pixels that do not have specified texture coordinates can be specified in the UV mask, for example, with the value "0". Each input image is passed through a convolutional network consisting of two LeakyReLU-enabled convolutions and BatchNorm layers. The output features are concatenated and fed into a U-Net network containing ResNet blocks. U-Net has 3 layers connected by feature concatenation. The output of U-Net is passed through two additional convolutional networks to predict the rendered RGB image I _rend of the avatar and its mask.

[0044] Thus, it is possible to obtain a rasterized image R from the completed neural texture T _inpainted and the mapping specified by the texturing function F _UV , in which the parametric body model is modified based on the parameters of the target pose, and/or the camera parameters are modified into the parameters of the target camera C _target . The target pose parameters p _target and/or the target camera parameters C _target correspond to said new full body human avatar pose. It should be understood that throughout the present invention, the term "new full-length human avatar pose" includes any human avatar poses that can be obtained by changing at least one parameter of the pose parameters and/or at least one parameter of the parameters cameras. Therefore, a new full-body avatar pose of a person obtained by changing only the camera parameter(s) is also considered a new full-body avatar pose of a person, even if the pose of the person in the image remains unchanged.

[0045] The neural renderer is trained together with the E-generator encoder network G. Therefore, rendering a full-body avatar of a person in a new pose p _target based on a single input RGB image I _rgb depicting a person in pose p _input can take the following form:

T _inpainted =G (E (I _rgb )) ⊕ ξ (F _UV (M (p _input , s), C _input ), I _rgb ) ⊕ B _smp (3a), or

T _inpainted =G (E (I _rgb )) ⊕ ξ (F _UV (M (p _input , s), C _input ), I _rgb ) ⊕ B _smp ⊕ B _fill (3b)

I _rend =θ (R (F _UV (M (p _target , s), C _target ) T _inpainted )) (4)

[0046] During training, one or more of the following losses are minimized: the loss of L ₂ differences between the render image I _rend and the corresponding reference image I _GT , the loss of LPIPS (Learned Perceptual Image Patch Similarity) of the learned perceptual similarity between image patches between the render image I _rend and the corresponding reference image I _GT , a non-saturating adversarial loss based on the StyleGAN2 discriminator with R ₁ -regularization, and a Dice loss between the predicted segmentation mask S _pred and the reference segmentation mask S _GT . In the preferred embodiment, a combination of all the above loss functions is minimized (see mathematical expression (5) below). The _L2 difference loss and LPIPS loss are calculated for the entire image and can be calculated additionally with a weight for the face region, since the face is of great importance for human perception. This weight is preferably 0.1, but in other embodiments it may be greater (eg, 0.11 to 0.2) or smaller (eg, 0.09 to 0.01). The non-saturating adversarial loss can be used to make I _rend more believable and clear. This non-saturating Adv adversarial loss can be used with discriminator D from StyleGAN2 with R ₁ -regularization. Thus, the total losses used in the preferred embodiment may take the following form:

[0047] The selection of hyperparameters λ _1...5 will be described below.

[0048] To improve the quality of the avatar at the inference stage, several methods can be additionally applied. To improve the texture details in the observed part of the image, several (specifically 64) optimization steps of the RGB channels with gradients from the differentiable renderer can be performed on the input image. In particular, you can use gradients from the neural renderer to do this, obtained by comparing the render image I _rend with the input RGB image I _rgb . Such gradients can be applied to texels with weights corresponding to the angles between the normal vectors and the camera direction, as will be discussed in detail with reference to Fig. 5 below. This ensures that only those texels that are visible in the input image I _rgb will be optimized, with priority given to the most frontal ones. The L ₂ difference loss and LPIPS loss can be used to drive color matching, and the adversarial Adv loss with R ₁ regularization, similar to the loss used in the overall loss math expression (5), can be used to enhance granularity.

[0049] It is also optional to apply a linear adjustment to the RGB channels of the VQGAN decoding output (FIG. 3) to improve color matching between the front and back views after the finishing stage:

T _rgb =T _rgb α+β (8)

[0050] In this case, all texels have common alpha and beta trainable parameters, which can be optimized using neural renderer gradients obtained by pixels observed in the input image I _rgb . As a result, the RGB channels of the neural texture at the output of the VQGAN enhance the color matching to the selected RGB texture T _rgb . This minimizes the seam(s) after merging textures.

[0051] Additionally, due to the imperfections of SMPL-X meshes, there is a problem with SMPL-X fitting imperfections where pixels are erroneously selected from one part of the body to another in areas that occlude themselves (eg, arms located in front of the body). These SMPL-X fit imperfections can result in unrealistic renders. To solve this problem, occluded areas of the human body, including those areas that the person occludes on himself, are detected in the input image I _rgb , and the texture within the outline of this occlusion is not selected in method step S110. The unselected texture contained in the overlap outline, as well as all other unobservable texture areas, are then further painted in method step S125. As shown in FIG. 7, one implementation may use palette rasterization as a texture to find overlapping areas. In particular, each limb can be assigned a different color in the palette, and the transition between these limbs can be made smooth using a color gradient. This avoids seams when rasterizing. Edges in palette rasterization can be found using Canny's algorithm or any other edge detector algorithm known in the art. The person's contours can then be determined by SMPL-X binary rasterization. By deriving these contours from the boundaries, a map of the closed area(s) of the human body is obtained. The resulting map is then used to mask areas in the UV render. This allows you to rely on finishing later in the pipeline rather than sampling pixels in overlapping areas.

[0052] Training is performed on sets of multi-view images, such as sets of video frames (or sets of 3D model renders). In a non-restrictive implementation of learning in the first stage of training, at each stage of training, two different frames are taken from the same set, one serving as the input image I _rgb with the input pose parameters p _input and the other as the target image with the target pose parameters p _target and/or parameters target camera C _target . Thus, these two images have different camera parameters, as well as two different body pose parameters of the same person. In one training implementation, two frames may be selected from the same set of multi-view images such that the differences in pose parameters and/or camera parameters between the two images are greater than or equal to the corresponding predetermined minimum difference threshold(s) to eliminate image selection that have insufficient differences in pose parameters and/or camera parameters. This is important for the conveyor to generalize to new positions of camera C and pose p. To do this, the neural renderer θ and the encoder network E - generator G are trained to complete the drawing of texture areas that are not observed in I _rgb .

[0053] Although neural renderer B and encoder network E of generator G are trained to compensate for the small number of unobserved texture regions that may be present in the target view, this capability has been found to be largely limited to small changes in body pose and/or camera parameters. The easiest way to get an avatar that can be rendered at arbitrary angles is to create it from several images by combining the corresponding neural textures. For this purpose, a simple mixing scheme can be used, schematically shown in Fig. 6. In particular, suppose that given N input images man with different camera settings , which gives N neural textures T ⁱ . These textures naturally cover different regions of the human body observed in different input images. To combine these textures, you can use the function F(T ¹ ... T ^N , λ ¹ ... λ ^N ), schematically shown in Fig. 6. λ ¹ is auxiliary information about the texture region observed or visible in the corresponding input image .

[0054] The angles between the normal vectors of the corresponding grid point M ⁱ and the camera direction vector can be used as auxiliary information λ ¹ . Thus, λ ¹ determines how frontal each texture point is in relation to the camera. Using this information, texture merging is performed, highlighting the most frontal pixels for each T ⁱ . The textures T ⁱ are then aggregated using a weighted average with weights calculated as . The factor τ determines the sharpness of the boundaries at the junction of the combined textures. It should be noted that weights can be calculated in other ways. So the final merged texture can be calculated as follows:

[0055] This method allows obtaining avatars from multiple snapshots by combining avatars from single snapshots for different views. More complex mixing patterns such as pyramidal mixing or Poisson mixing can also be used.

[0056] The final part of the proposed method uses a diffusion inpainting model incorporating a denoising U-Net architecture and optionally a VQGAN-based autoencoder. A diffusion inpainting model can be trained supervised, using incomplete textures based on individual photographs as inputs, and pooled textures aggregated from multiple views as reference images.

[0057] It is important to note that since the distribution of plausible ("correct") full textures given an input partial texture is typically very complex and multimodal, the present inventors proposed to use a working approach of a noise dissipation probabilistic model, DDPM, and train a U-noise removal architecture. Net instead of directly converting input to output. As described above, in a non-limiting implementation, the neural texture T has a resolution of 256x256x20 (without the B _fill map of additional pixels) or 256x256x21 (with the B _fill map of additional pixels). This entails high memory requirements during training of the diffusion model. To reduce memory consumption and improve network convergence, firstly, the neural texture size can be reduced using a VQGAN-based autoencoder. As shown in FIG. 2, a VQGAN-based autoencoder is added as an alternative branch to the input of the neural renderer θ. After pre-training the VQGAN-based autoencoder, the pipeline can be fine-tuned from start to finish to adapt the neural renderer θ to the VQGAN-based autoencoder's decompression artifacts in the pre-painted neural texture T _inpainted . Several loss functions can be used to train a VQGAN-based autoencoder to more accurately restore neural textures. To improve the visual quality of the avatar after texture decompression, loss functions in RGB space can be used. During training, the avatar is rendered as described in expression (4) above, with the T _inpainted neural texture added, and then the loss function is optimized (5). To further regularize and preserve neural texture properties across all views, an additional _L2 loss in texture space can be used .

[0058] After adding the VQGAN-based autoencoder to the neural texture completion pipeline, as shown in Figure 3, the DDPM model-based denoising U-Net architecture is trained in the latent space of the pre-trained VQGAN-based autoencoder. Therefore, the noise-removing U-Net architecture based on the DDPM model is applied to T ⁰ _c =V (T ⁰ ) of size 64 × 64 × 3 obtained after single-view neural texture compression by the VQGAN E _VQ encoder. The denoising U-Net architecture based on the DDPM model can be trained with an attention mechanism, such as one or more attention layers. The noise-removing U-Net architecture is conditioned by E _VQ (T ⁰ ) ⊕ b(B ⁰ _fil l), where b is the bilinear resizing to the spatial dimension T ⁰ _c . Thus, the concatenation of the condition and texture T _c , distorted by normally distributed (Gaussian) noise corresponding to the diffusion step t, is fed as input to the noise-removing U-Net architecture. In this case, the U-Net denoising architecture is trained to remove noise from the input by minimizing the loss L _DM (1). As explained above, the L _DM loss can be based on the L ₁ loss function.

[0059] As noted above, the diffusion model of finishing illustrated in FIG. 3 is trained using the concatenated neural textures (Fig. 6) as reference neural textures. In this case, fused neural textures can be generated from a dataset of 3D human scans. You can also use a dataset of multi-view photographs of people with good angular coverage as training data. As noted above, the U-Net denoising architecture recovers the texture T _c (namely, its latent representation) from the noise based on the condition. Then the texture T c _is converted into a reconstructed full-size neural texture T using the VQGAN D _VQ decoder. To preserve all the details of the input image I _rgb , the reconstructed neural texture T is then combined with the input texture T ⁰ using a mask to get the completed neural texture T _inpainted :

[0060] The resulting texture has all the detail observed in the input image , and parts of the person not visible in the input image ,are reconstructed using the latent space diffusion,inpainting model of a VQGAN-based autoencoder. In fig. Figure 8 shows examples of animated full-length human avatars in various poses generated in the manner described above for the corresponding single input images I _rgb shown on the left side of the figure.

[0061] In FIG. 9 is a non-limiting schematic diagram of a computing device 50 in accordance with a second aspect of the present invention. It should be understood that computing device 50 may be configured to perform the above-described method in accordance with the first aspect or in accordance with any development of the first aspect. The computing device 50 includes a processor 50.1 and a memory 50.2, which stores instructions and parameters executable by the processor, including at least the weights and biases of the trained pipeline of the encoder network E - generator G, the diffusion model of completion and the neural renderer θ. Upon execution of processor executable instructions by processor 50.1, processor 50.1 causes computing device 50 to execute a method of creating a life-size animated avatar of a person from a single image of that person in accordance with the first aspect or any other development of the first aspect.

[0062] The computing device 50 may be of any type, for example, it may be, but is not limited to, a general purpose computer, a specialized computer, a computer network, a laptop, a smartphone, a tablet, a smart watch, smart glasses, an AR/VR headset, or another programmable device. The 50.1 processor can be any type of processor. The processor 50.1 may be, without limitation, one or more of the following processors: a general purpose processor (e.g., CPU), a digital signal processor (DSP), an application processor (AP), a graphics processing unit (GPU), a vision processor (VPU) , a processor dedicated to AI (for example, a neural processor, NPU). The processor may be implemented as a system-on-chip (SOC), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete logic gate, transistor logic, discrete hardware components, or any of these combinations. The processor may be divided into units, each of which performs one or more steps of the method described above.

[0063] Memory 50.2 may be, but is not limited to, read-only memory (ROM) and random access memory (RAM). Any type of RAM and ROM can be used. The computing device 5 0 may further include any type of camera 50.3 capable of capturing images. Computing device 50 may run on any operating system and may include any other necessary software, firmware, and hardware (eg, communications unit, input/output interface, camera, power supply, etc., not shown).

[0064] One skilled in the art will appreciate that for convenience and brevity the detailed description of the operating process of said computing device 50 should be referred to the corresponding process in the method embodiments described above. Their detailed description is not repeated.

[0065] Training data and network training data will be discussed next. The details discussed below should not be construed as limitations. Instead, the following description and non-limiting examples may be taken to demonstrate the sufficiency of the disclosure of the present invention as provided in this patent application. The E-G generator encoder network, the inpainting diffusion model, the θ neural renderer, or any aspect of their training can be implemented using open source machine learning libraries such as TensorFlow, Keras, PyTorch, and so on. Training can be done offline or online. Training of the encoder network pipeline E - network generator G, the diffusion model of repainting and the neural renderer θ was performed on RGB images with a resolution of 512 × 512. However, the resolution of the RGB training images may be greater or less than the aforementioned 512×512 resolution. For each input image of size 3 × 512 × 512, a neural texture of size 21 × 256 × 256 was generated. In this neural texture, the first 16 channels were generated by the G generator of the trained E encoder network - the G generator, the next three channels were RGB channels, and the remaining two channels were sampling and finishing masks. Generator G has a StyleGAN2 architecture and takes a feature vector as input , generated by encoder E of the trained network of encoder E -generator G. Encoder E has a StyleGAN2 discriminator architecture with a modified head to output a vector of length 512. The resulting texture was applied to the SMPL-X model and rasterized. The rasterized image had a size of 21 × 512 × 512 and was fed to the neural renderer, which has a U-Net architecture with ResNet blocks. The output of neural renderer B was a final image with a resolution of 3 × 512 × 512. Generator G and neural renderer θ were trained end-to-end with a batch size of four. However, the packet size may be greater or less than four.

[0066] A combination of one or more loss functions can be used for training (in the preferred embodiment, all of the following loss functions were used): loss L ₂ of the difference between the reconstructed image I _rend and the corresponding reference image I _GT , loss of learned perceptual similarity between image patches (LPIPS) between the render image I _rend and the corresponding reference image I _GT , a non-saturating adversarial loss based on the StyleGAN2 discriminator with R ₁ -regularization, and a Dice loss between the predicted segmentation mask S _pred and the reference segmentation mask S _GT . In the preferred embodiment, a weighted sum of the above losses was used with the following weights: difference loss L ₂ with weight λ1=2.2; loss of LPIPS with weight λ ₂ =1.0; Dice loss with weight λ ₃ =1.0; adversarial loss with weight λ ₄ =0.01. It should be noted that the weight values presented above should not be understood in a restrictive sense, since this method may well achieve the technical result(s) with modified scales. Such modification will be apparent to one skilled in the art upon reading this disclosure. For every 16 iterations, lazy regularization R ₁ was applied with a weight λ ₅ =0.1. To calculate the LPIPS loss, random frames of 256 × 256 were taken from images of size 512 × 512. The encoder network pipeline E - generator G, the diffusion inpainting model and the neural renderer θ were trained in 100,000 steps using the ADAM optimizer with a learning rate of 2e-3

[0067] A VQGAN-based autoencoder was trained to compress neural textures to 6x64x64 tensors consisting of vectors of length six from a trained 8192-element dictionary. First, the VQGAN-based autoencoder was trained in 300,000 steps. The pipeline (with the added pretrained VQGAN-based autoencoder) was then tuned end-to-end in an additional 20,000 steps to reduce neural renderer artifacts when processing neural textures processed by the VQGAN-based autoencoder. After this, the diffusion inpainting model was trained to restore the missing parts of the texture. The diffusion inpainting model used a denoising U-Net architecture with BigGAN residual blocks for upsampling and downsampling and with attention layers at three levels of the feature hierarchy. To further eliminate overfitting, decimation was used with probability 0.5 in the residual blocks of the denoising U-Net architecture. The inpainting diffusion model was trained for 50,000 iterations using the AdamW optimizer with a batch size of 128 and a learning rate of 1.0e-6.

[0068] Only 2D images obtained by rendering the Texel dataset were used to train the pipeline. Generator G network encoder E - generator G and neural renderer θ were pre-trained on 2D images of 13,000 people in various poses. It has been observed that different poses are critical for training realistically animated avatars. For each image, a segmentation mask was obtained using Graphonnomy segmentation, and the SMPL-X parametric body model was fitted using SMPLify-X. Dice segmentation loss was additionally used to improve the body shape of the fitted SMPL-X parametric body model.

[0069] Renders from the Texel dataset were used to train the VGQAN autoencoder and the diffusion inpainting model. From the Texel dataset, 3333 human scans were obtained. They depict people in different clothes, with different body types, skin tones and genders. Each scan was rendered from 8 different views (or camera angles) to create a multi-view dataset. The renders were also refined by changing camera angles and changing colors. Thus, 72 scans were obtained for each person in the data set. It should be noted that any images from different species are suitable for training this model, not necessarily obtained from 3D scans.

[0070] The generated avatars and their animations were qualitatively evaluated on the AzurePeople dataset. This dataset contains people wearing various clothes and standing in natural poses. Additionally, the trained pipeline was quantified on the publicly available SnapshotPeople benchmark. It contains 24 videos of people spinning in pose A. From each video, frames from the front and back views were selected to measure the accuracy of the reconstruction of the rear view. For each image, a segmentation mask and SMPL-X fit were obtained as described above.

[0071] Quantitative results. The following Table 1 provides a quantitative comparison:

Table 1: Comparison of metrics on the SnapshotPeople benchmark. The described method is compared not only with other approaches based on parametric models (StylePeople), but also with approaches that restore geometry and require the use of additional methods for rigging (PIFu, Phorhum) or restoration of geometry in a canonical pose (ARCH, ARCH++).

[0072] The present inventors compared the invention to various methods for creating an avatar from a single image, including those requiring additional rigging steps for animation. For clarity, Table 1 above is divided into three parts. PIFu and PHORHUM reconstruct a 3D mesh of a person in the pose shown in the input image. This places strict restrictions on the pose of the person in the input image if you want to animate it. ARCH and ARCH++ restore the 3D mesh to canonical space, which is easier to rig and animate on. StylePeople and the present invention are based on a parametric human model, so they are the easiest to animate and are not affected by rigging imperfections.

[0073] A number of metrics have been reported for the numerical evaluation of generated avatars, namely: multi-scale structural similarity (MS-SSIM↑), peak signal-to-noise ratio (PSNR↑), loss of learned perceptual similarity between image patches (LPIPS↓). These benchmark metrics are measured on front view avatars for the SnapshotPeople public benchmark. According to these metrics, for the front view, the proposed method performs on par with methods without rigging. To assess the quality of the rear view (and thus the ability to generalize to new views), kernel formation measurements (KID↓) are reported. This metric evaluates the quality of the generated images and is more suitable for small amounts of data than FID. The proposed method produced the highest KID value compared to other methods. A measure based on the FlipReID model of person reidentification was used to estimate the identity preservation reidentification score (ReID↓). The proposed method gave the best results in preserving the identity of a person between front and back views. To further check texture quality and measure structural similarity in cases with unaligned reference images, the Deep Image Structure and Texture Similarity (DISTS↓) measure was used. Table 1 shows measurements for front and rear views. The proposed method produced the most natural-looking avatars for both species.

[0074] In this case, the proposed method realistically reconstructs the texture of the fabric of clothing on the back (for example, folds on trousers), which increases the realism of renderings. Using all the information from a given image avoids copying unnecessary patterns from the front view to the rear view (as is typically done by pixel alignment methods when reconstructing texture for the rear view). Using a selected RGB texture as a complement to a neural texture allows for photorealistic facial detail while preserving high-frequency detail. PIFu was observed to accurately reproduce avatar color and perform good geometry reconstruction. However, this method does not preserve high-frequency details, causing avatars to lack photorealism. PHORHUM creates very photorealistic avatars, but often suffers from color shifts. Another methodological disadvantage of this approach is the lack of a priori information for the human body. Therefore, the model may be overtrained on human poses from the training dataset, which may lead to incorrect work with new poses. Avatars generated by ARCH contain strong color artifacts and are prone to geometry reconstruction errors. ARCH++ significantly improves the geometry and color quality of the front view, but the rear view still suffers from color shift and artifacts. An avatar using the StylePeople method is based on a parametric human model and can be easily animated without the use of third-party methods or additional rigging. However, the latent space coverage of their model is limited, leading to overfitting and poor generalization to new people when performing inference from a single view.

[0075] Disclosed herein is a new method for modeling human avatars based on neural textures that combine RGB components and hidden components. RGB components are used to preserve high-frequency details, and neural components add hair and clothing to the base SMPL-X mesh. Using the SMPL-X parametric model as a basis makes it easy to animate the resulting avatar. The proposed method restores missing parts of the texture using an adapted diffusion platform for completing such textures. Thus, this method allows you to create avatars with a constructed skeleton, also improving the quality of rendering of body parts that are not visible in the image compared to modern methods for reconstructing human models without constructing a skeleton.

[0076] One skilled in the art will also appreciate that the various illustrative logical blocks and steps presented in embodiments of the invention may be implemented by electronic hardware, computer software, or a combination thereof. The choice to implement these functions in hardware or software depends on the specific application and design requirements for the overall system. One skilled in the art may use various techniques to implement the described functionality for each particular application, but such implementation should not be considered beyond the scope of embodiments of a given application.

[0077] The present invention also provides a non-transitory computer-readable storage medium. The computer-readable medium stores computer-executable instructions and parameters including at least the weights and biases of a trained encoder network pipeline E - generator G, a diffusion infusion model, and a neural renderer θ, wherein the computer-executable instructions are executed by a computing device (e.g., the computing device 50 described above ) causes this computing device to perform a method of creating a life-size animated avatar of a person from a single image of that person in accordance with the first aspect of the present invention or any development thereof.

[0078] All or some of the described embodiments may be implemented using software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, the embodiments may be implemented in whole or in part as computer-executable instructions. When machine instructions are downloaded and executed on a computing device, a procedure or functions are generated, in whole or in part, in accordance with the embodiments. The machine instructions may be stored on a computer-readable storage medium or may be transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, machine instructions may be transmitted from a website, computing device, server, or data center to another website, computing device, server, or data center over a wired communications link (e.g., coaxial cable, fiber optic cable, or digital subscriber line (DSL). DSL)) or wireless communication line (for example, infrared, radio or microwave). A computer-readable storage medium can be any suitable medium that can be accessed by a computing device or data storage device, such as a server or data center integrated with one or more suitable media. Suitable media may be magnetic media (eg, floppy disk, hard disk, or magnetic tape), optical media (eg, digital video disc (DVD)), semiconductor media (eg, solid state drive (SSD)), or the like.

[0079] It should also be noted that the order of execution of the steps of the disclosed method is not strict, i.e. some (one or more) stages can be swapped and/or combined. Throughout this specification, reference to an element in the singular does not preclude the presence of several such elements in the actual practice of the invention, and, conversely, reference to an element in the plural does not exclude the presence of just one such element in the actual practice of the invention. Any value of the parameters mentioned above should not be interpreted in a restrictive sense, but rather should be considered as representing the midpoint of a certain range, defined as the midpoint ± up to approximately 20%.

[0080] The above descriptions are merely specific implementations of the present invention and are not intended to limit the scope of its protection. Any change or substitution that can be readily contemplated by one skilled in the art within the scope disclosed in this application shall fall within the scope of protection of this invention. Thus, the scope of protection of the present invention is established in accordance with the scope of protection of the claims.

Claims (50)

1. A computer-implemented method for creating a full-length animated human avatar from a single image of a person, comprising the steps of: obtain (S100) an image (I _rgb ) of a person and a calculated parametric model of the body, determined by the values of the pose parameters (p) and shape parameters (s) of the human body in the image, and the values of the camera parameters (C) used in capturing the image; based on the parametric model of the body, a texturing function (F _UV ) is determined (S105) specifying the mapping between each pixel of the human body observed in the image and the corresponding texture coordinates in texture space, and for all pixels of the human body unobserved in the image, the texturing function additionally sets the corresponding texture coordinates in texture space; sample (S110) the RGB texture (T _rgb ) of the human body based on the display and obtain a map (B _smp ) of the selected pixels, wherein the RGB texture (T _rgb ) contains for each pixel of the human body observed in the image a corresponding pixel value, and also contains one or several unobservable texture areas corresponding to the texture coordinates of human body pixels unobservable in the image; creating (S115) a texture (T _gen ) of the human body observed in the image by passing the image through a trained encoder (E)-generator (G) network; obtain (S120) a neural texture (T) by concatenating the RGB texture (T _rgb ), the map (B _smp ) of the selected pixels and the generated texture (T _gen ); carry out completion (S125) of unobserved texture regions of the neural texture (T) using the trained diffusion model of completion; transform (S130) a rasterized image (R) of a full-length human avatar in a new pose using a trained neural renderer (θ) into a rendered image (I _rend ) of a full-length human avatar in a new pose, wherein the rasterized image (R) is obtained from the completed neural texture (T _inpainted ) and the display specified by the texturing function (F _UV ), in which the parametric model of the body is modified based on the target pose parameters (p _target ) and/or the camera parameters are modified to the target parameters camera (C _target ), wherein the target pose parameters (p _target ) and/or target camera parameters (C _tarqet ) correspond to said new full-length human avatar pose. 2. The method according to claim 1, in which additionally the selected RGB texture is filled with gaps not exceeding a specified threshold size by averaging neighboring pixels and obtaining a map (B _fill ) of completed pixels, wherein the map (B _fill ) of completed pixels is additionally used during concatenation at the stage of obtaining the neural texture (T). 3. The method of claim 1, wherein the step of completing (S125) the unobserved texture regions of the neural texture (T) is performed by adding Gaussian noise to the neural texture (T) and then iteratively performing a denoising procedure on the neural texture (T) with the Gaussian noise T ^noise times using a trained diffusion inpainting model until the Gaussian noise is replaced by an inpainted neural texture (T _inpainted ). 4. The method according to claim 1, in which the parametric body model is based on a mesh M with a fixed topology controlled by the values of the pose parameters (p) and shape parameters (s). 5. The method according to claim 1, in which The encoder (E) of the trained encoder (E) - generator (G) network is based on the StyleGAN2 discriminator and is trained to compress an image (I _rgb ) into a feature vector , And generator (G) of the trained encoder network (E) - generator (G) is based on the StyleGAN2 generator and trained to create a texture (T _gen ) from a feature vector . 6. The method according to claim 1, wherein the display is created using UV unwrapping performed on a front view of the human body in the image. 7. The method according to claim 1, in which the closed region of the human body in the image is additionally found (I _rgb ) and this closed region is excluded when sampling the RGB texture (T _rgb ). 8. The method of claim 1, wherein the trained diffusion inpainting model is based on a noise dispersion probabilistic model (DDPM) having a noise-removing U-Net architecture trained to inpaint unobserved texture regions of the neural texture (T) to obtain an inpainted neural texture (T _inpainted) . ), in which the U-Net denoising architecture has residual BigGAN blocks for upsampling and downsampling and attention layers at a number of levels in the feature hierarchy of the U-Net denoising architecture. 9. The method of claim 8, wherein the trained diffusion inpainting model further comprises a vector quantized generative adversarial network (VQGAN)-based autoencoder, including an encoder (E _VQ ) trained to encode the neural texture (T) into a latent representation of the neural texture (T) of lower dimensionality input into the denoising U-Net architecture, and a decoder (D _VQ ) trained to decode the output of the denoising U-Net architecture into an inpainted neural texture (T _inpainted ). 10. The method according to claim 1, in which the training of the encoder (E) - generator (G) network, the diffusion model of completion and the neural renderer (θ) is carried out at two stages of training: at the first stage of training: said encoder (E)-generator (G) network and said neural renderer (θ), based on a U-Net rendering architecture having ResNet blocks that form a pipeline, are trained end-to-end based on sets of multi-view images used as training data, at each stage of learning from the many stages of learning during learning at the first stage of learning: select two different images from the same set of multi-view image sets: a first image showing a person in a pose having the input pose parameters (p _input ) used as the input image, and a second image showing a person in a pose having other parameters target pose (p _target ), used as reference image (I _GT ), pass the first image and target pose parameters (p _target ) through a pipeline trained to render a render image (I _rend ) showing a person's avatar in a pose having target pose parameters (p _target ), calculating, based on at least the render image (I _rend ) and the reference image (I _GT ), a loss value in accordance with one or more of the following loss functions: loss L ₂ differences between the render image (I _rend ) and the corresponding reference image ( I _GT ), the learned perceptual similarity loss between image patches (LPIPS) between the render image (I _rend ) and the corresponding reference image (I _GT ), the non-saturating adversarial loss based on the StyleGAN2 discriminator with R ₁ regularization, and the Dice loss between the predicted segmentation mask (S _pred ) and segmentation reference mask (S _GT ), calculate gradients based on the loss value and update parameters of the encoder (E) - generator (G) and neural renderer (θ) network based on these gradients; And after the loss value for one or more loss functions is minimized, the learned parameters of the encoder (E) - generator (G) and neural renderer (θ) network are recorded and proceed to the second stage of training; at the second stage of training: a diffusion inpainting model comprising at least a denoising U-Net architecture is added to the pipeline and trained using conditional diffusion learning to inpaint unobserved texture regions of the neural texture (T), wherein at each stage of training from the many stages of training during training in the second stage of training: obtain a pooled neural texture used as a reference neural texture, wherein the pooled neural texture is obtained by combining partial neural textures obtained from two or more images covering different viewing angles and selected from the same set of multi-view images from the sets of multi-view images used as training data, add Gaussian noise to the combined neural texture according to a random step t, where t=[0, T ^noise ], concatenate to the combined neural texture with added Gaussian noise the incomplete neural texture from the incomplete neural textures used to obtain the combined neural texture, pass the combined neural texture with added Gaussian noise and the concatenated incomplete neural texture through an inpainting diffusion model, which predicts at step t the noise that should be removed to obtain the inpainted combined neural texture, calculating, at least based on the predicted noise and the added Gaussian noise, a loss value in accordance with the loss function L ₁ , calculate gradients based on the loss value, and update the parameters of the U-Net denoising architecture used for gradient-based diffusion inpainting; And Once the loss value is minimized, the pretrained pipeline is fine-tuned by performing many additional training steps. 11. The method of claim 10, wherein the fine tuning includes fixing weights and biases of the neural renderer (θ) and propagating gradients from the differentiable neural renderer (θ) to RGB channels of the RGB texture (T _rgb ), said gradients being calculated based on loss values obtained at the first stage of training. 12. The method according to claim 10, in which, if a VQGAN-based autoencoder is additionally included in the trained diffusion model of completion, then the VQGAN-based autoencoder is additionally trained at the first stage of training to encode neural textures (T) into hidden representations of neural textures (T) of a smaller dimensions and decode lower-dimensional latent representations of neural textures (T) by minimizing the loss value calculated from one or more loss functions used in the first stage of training, and then training in the second stage of training is performed in the latent space of the pretrained VQGAN-based autoencoder. 13. The method of claim 10, wherein in the first training stage, the L ₂ difference loss and the LPIPS loss are calculated for the entire image and are further calculated with a given weight for the image region containing the face. 14. The method of claim 10, wherein the fusion is based on weighted averaging, pyramidal mixing, or Poisson mixing of the neural textures. 15. The method according to claim 10, in which at the second stage of training, decimation is used with a given probability in the residual blocks of the noise-removing U-Net architecture. 16. The method of claim 10, wherein the multi-angle image sets comprise images of people wearing different clothes, different body types, different skin tones, different genders, and in a variety of different poses taken from different viewing angles. 17. The method according to claim 10, in which the viewing angle is defined as the angle between the normal vectors of the corresponding grid point M with a fixed topology and the camera direction vector. 18. A computing device (50) containing a processor (50.1) and a memory (50.2) storing instructions and parameters executable by the processor, containing at least the weights and biases of the trained pipeline of the encoder (E) - generator (G) network, diffusion model completion and neural renderer (θ), characterized in that when the processor (50.1) executes processor-executable instructions, the processor (50.1) causes the computing device (50) to execute the method according to any one of claims. 1-17 creating an animated full-length human avatar from a single image of a person. 19. The computing device (50) of claim 18, further comprising a camera (50.3) configured to capture an image (I _rgb ) of a person. 20. A machine-readable medium storing machine-executable instructions and parameters containing at least the weights and biases of a trained encoder (E) - generator (G) network pipeline, a diffusion model of finishing and a neural renderer (θ), characterized in that the computing device executes machine-executable instructions causes the computing device to perform the method of any one of claims. 1-17 creating an animated full-length human avatar from a single image of a person.