RU2726160C1 - Repeated synthesis of image using direct deformation of image, pass discriminator and coordinate-based remodelling - Google Patents

Abstract

FIELD: data processing.SUBSTANCE: invention relates to image processing. System comprises an input image input module, a forward deformation module configured to predict for each pixel of the original image of the corresponding position in the target image, wherein forward deformation module is configured to predict forward deformation field, which is aligned with initial image, and a pass filling module configured to fill the blanks resulting from the use of the straightening module.EFFECT: technical result consists in improvement of accuracy of repeated synthesis of image based on input image.13 cl, 7 dwg

Description

FIELD OF THE INVENTION

The present invention relates generally to image processing and, more particularly, to the re-synthesis of images for synthesizing new views (projections) for a person or object based on an input image using machine learning methods.

state of the art

Recently, there has been a growing interest in learning-based re-synthesis of the image. In this context, the task of machine learning is to train in the synthesis of new projections, for example, for a person or object of a certain type based on one or more input images of a person or object. In extreme cases, only one input projection is available. In this sense, the new projection corresponds to the new position of the camera and / or the new pose of the human body. In repeated image synthesis, the quality of the target projection is measured and the quality of the intermediate representation, which often implicitly or explicitly matches the scene model (for example, 3D reconstruction), is not given importance. Optimizing the quality of the target projection directly often means that the quality of the target projection is better, especially when modeling the scene is difficult.

Several trends are different. First, the solution to the complex prediction problem that accompanies image re-synthesis necessitates deep convolution networks (ConvNets) (see [15]). Secondly, many prior art solutions avoid predicting pixel values directly from a high-dimensional non-convolutional representation. Instead, most architectures resort to some kind of deformation inside ConvNet (see, for example, [5, 30, 20, 3, 23]). It is well known that in many cases, the prior art uses reverse deformation [13], in which for each pixel in the target image the location in the original image is predicted from which the pixel will be copied. The deformation process is often followed by post-processing, such as brightness correction (see [5]), or post-processing networks.

Below will be described some approaches to the problems associated with the objective technical problem solved by the present invention.

Re-synthesis based on deformation. There is considerable interest in using deep convolutional networks to form realistic images (see, for example, [6]). When new images are formed by changing the geometry and representation of the input images, it was shown that the use of deformation modules significantly improves the quality of re-synthesized images (see, for example, [5, 30]). In this case, the deformation modules are based on a differentiable layer of the (inverse) trellis discretizer, which was first introduced as part of the spatial transducer (STN) networks (see, for example, [13]).

Competitive image reconstruction. There are also prior art solutions aimed at reconstructing images based on deep convolutional networks. Special versions of convolutional architectures adapted to the presence of gaps in the input data include Shepard convolutional neural networks (see, for example, [21]), sparse-invariant convolution networks (see, for example, [25]), networks with partial convolutions (see, for example, [17]), networks with gated convolutions (see, for example, [28]). The latter option is also used in the approach proposed in this disclosure.

Since reconstruction tasks make it necessary to conditionally synthesize image content, approaches from the prior art to reconstruction are based largely on options for generative competitive (competitive) training (see, for example, [6]). In particular, the prior art suggests the use of pairs of discriminators that focus on discrimination between real and fake examples on two different scales (see, for example, [16, 11, 28, 4]), where one of the scales can correspond to individual fragments (patches ) (reminiscent of the idea of a GAN fragment from [12]). Here, a new type of discriminator is introduced, which has a similar architecture with some of the local discriminators and the GAN fragment, however it distinguishes between two different classes of pixels (known pixels in fake images with respect to unknown pixels, also in fake images).

Frontalization of the face. Prior art solutions that focus on re-synthesizing an image (such as generating new projections and / or changing poses (body positions) for 3D objects based on one input photographic image or multiple input photographic images) use face images as the primary area. The frontalized face projection can be used as a normalized representation to simplify face recognition and improve its quality. Some prior art solutions use an inverse sampler for this task. For example, the HF-PIM system, which can be considered as the most typical example of such an approach, predicts a cylindrical texture map and a back deformation field required to convert a cylindrical texture map to a frontalized face projection. The result of the deformation is then refined by another network. Many other methods that are currently considered to be highly effective, such as CAPG-GAN (see, for example, [9]) LB-GAN (see, for example, [2]), CPF (see, for example, [26 ]), FF-GAN (see, for example, [27]), are based on encoder-decoder networks that directly perform the desired conversion by representing the image in a low-dimensional latent space. Additionally, the re-synthesis network is usually trained in a GAN configured to create an output representation of the face that is realistic and to prevent various kinds of artifacts. Many of these methods use additional information, such as landmarks (see, for example, [9, 29]), local fragments (see, for example, [10]), 3D transformable model (3DMM, see, for example, [1 ]), an estimate (see, for example, [27]). Such additional information can then be used, either to determine the re-synthesis process, or to formulate additional losses by measuring to what extent the synthesized image is consistent with the additional information available.

Warp-based re-synthesis of the whole body. In the prior art, deformation is used to synthesize new human projections under the condition of a single input projection (see, for example, [24, 23, 19]). This approach also uses dense body positioning (see, for example, [8]) in the network to represent the target pose of a person in a re-synthesized image.

It should be noted that all of the above types of prior art approaches to image re-synthesis have certain drawbacks, and the present invention is aimed at eliminating or at least reducing at least some of the drawbacks of the prior art. In particular, the disadvantages of the available solutions from the prior art relate to the use of back deformation in image re-synthesis, in which, for each pixel in the target image, the location in the original image from which the pixel will be copied is predicted.

DISCLOSURE OF INVENTION

An object of the present invention is to provide a new image re-synthesis method that eliminates or at least reduces all or at least some of the aforementioned drawbacks of existing prior art solutions.

The technical result achieved by the present invention consists in increased accuracy of image re-synthesis for synthesizing new projections of a person or object based on at least one input image.

In one aspect, this result is achieved by an image re-synthesis system comprising: an input image input module; a direct deformation module configured to predict, for each pixel of the source image, a corresponding position in the target image, the direct deformation module configured to predict a direct deformation field that is aligned with the original image; and a gap filling module configured to fill the gaps resulting from the use of the direct deformation module.

In an embodiment, the omission filling module may further comprise a deformation error correction module configured to correct direct deformation errors in the target image.

The image re-synthesis system may further comprise a texture transfer architecture configured to: predict strain fields for the source image and the target image; mapping the original image into the texture space through direct deformation, restoring the texture space to the full texture; and display the full texture back to a new body position (posture) using reverse deformation.

The image re-synthesis system may further comprise a texture extraction module adapted to extract the texture from the original image. At least the direct deformation module and the gap filling module can be implemented as deep convolutional neural networks.

In an embodiment, the gaps filling module may comprise a gaps reconstructor, the gaps reconstructor comprising: a coordinate assignment module, configured to assign to each pixel p = (x, y) an input image of a coordinate pair (u, v) of a texture in accordance with a fixed predetermined mapping textures to provide a two-channel map of x and y values in the texture coordinate system; a texture map completion module, configured to provide a complete texture map, where for each pixel (u, v) of the texture the corresponding pixel (x [u, v], y [u, v]) of the image is known; a final texture generating unit configured to produce a final texture by mapping image values from positions (x [u, v], y [u, v]) to a texture at positions (u, v) to provide a complete color final texture; a final texture re-mapping module configured to re-display the final texture in a new projection by providing another mapping from the coordinates of the image pixels to the coordinates of the texture.

At least one of the deep convolutional networks can be trained using a real / fake image discriminator configured to distinguish between true site images and reconstructed images. The image re-synthesis system may further comprise an image refinement module configured to correct defects in the output image.

In another aspect, there is provided a system for training a gap filling module configured to fill gaps as part of image re-synthesis, the system being configured to teach a gap filling module in parallel and together with a gap discriminator network, wherein the gap discriminator network is trained to predict the binary gap mask, and the gap filling module learns to minimize the accuracy of the gap discriminator network.

In another aspect, the invention relates to a method for re-synthesizing an image, comprising the steps of: inputting an original image; predicting, for each pixel of the original image, the corresponding position in the target image, and the field of direct deformation is predicted, which is aligned with the original image; predicting the binary mask of the gaps that result from direct deformation, filling in the gaps based on said binary mask of the gaps by generating a texture image by predicting a pair of coordinates in the original image for each pixel in the texture image; and mapping the entire texture back to a new body position using reverse deformation.

In an embodiment, the gap filling step may comprise the steps of: assigning to each pixel p = (x, y) an input image of a coordinate pair (u, v) of a texture in accordance with a fixed predetermined texture mapping to provide a two-channel map of x and y values in the coordinate system textures; providing a complete texture map, where for each pixel (u, v) of the texture the corresponding pixel (x [u, v], y [u, v]) of the image is known; forming the final texture by mapping the image values from the positions (x [u, v], y [u, v]) to the texture at the positions (u, v) to provide a complete color final texture; re-mapping the final texture to a new projection by providing another mapping from the coordinates of the image pixels to the coordinates of the texture.

In yet another aspect, the invention provides a method for training a gap filling module configured to fill gaps as part of image re-synthesis, the method comprising training the gap filling module in parallel with and together with a gap discriminator network, wherein the gap discriminator network is trained to predict a binary gap mask, and the gap filling module is trained to minimize the accuracy of the gap discriminator network.

In yet another aspect, a computer program product is provided comprising computer program code that, when executed by one or more processors, causes one or more processors to implement the method of the second aforementioned aspect.

In yet another aspect, a permanent computer-readable medium is provided having a computer program product stored thereon in accordance with the aforementioned aspect.

Based on the study and understanding of the description below, those skilled in the art should understand that the claimed invention may also take other forms. The various steps of the method and system components can be implemented in hardware, software and firmware, or any suitable combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

Following the brief description of the invention presented above, the following is a detailed description of the principles of the invention as an example and with reference to the accompanying drawings, which are given only as an illustration and are not intended to limit the scope of the claimed invention or to determine its essential features. In the drawings:

FIG. 1 shows the difference between forward and reverse deformation, explained from the point of view of the frontalization of the face;

FIG. 2 illustrates a machine learning process for reconstructing with skip discriminators in accordance with an embodiment of the present invention;

FIG. 3 illustrates a frontalization process of a face by direct deformation in accordance with an embodiment of the present invention;

FIG. 4 illustrates a texture transfer architecture in accordance with an embodiment of the present invention;

FIG. 5 illustrates a texture completion process for re-synthesizing a new body position (posture) in accordance with an embodiment of the present invention;

FIG. 6 illustrates a whole-body re-synthesis process using coordinate-based texture reconstruction in accordance with an embodiment of the present invention.

FIG. 7 shows a flowchart of an image re-synthesis method in accordance with an embodiment of the present invention.

The implementation of the invention

The present detailed description is provided to facilitate understanding of the invention. It should be noted that the description is directed to exemplary embodiments of the invention, and based on a careful study of the description with reference to the attached drawings, other modifications, variations and equivalent replacements in the described subject matter may be made by a person skilled in the art. All such obvious modifications, variations, and equivalents are intended to be included within the scope of the claimed invention. No reference numerals or conventions indicated in this detailed description, as well as in the attached claims, are intended to limit or determine the scope of the claimed invention in any way.

The present invention provides a new approach for re-synthesizing an image based on at least one input image. The methods and systems according to the invention are based on various neural networks with the possibility of training on various data sets, such as deep neural networks. One of ordinary skill in the art may understand that implementations of the present invention are not limited to the neural networks specifically described herein, but can be implemented using other types of networks that may be suitable for the task within the context of the present invention. Neural networks suitable for implementing the present invention can be implemented by material and technical means well known to specialists in this field of technology, for example, but not limited to, one or more processors, universal or specialized computers, graphic processors (GPUs), etc. D., managed by one or more computer programs, elements of computer programs, program code, etc., for implementing the methodologies according to the invention described below.

Firstly, the claimed invention will be described in terms of one or more machine learning models based on deep convolutional neural networks, previously trained or trained to perform a specific processing, the result of which is repeated image synthesis for synthesizing new projections of a person or object based on at least one input image.

As a result, the proposed approach is based on two contributions compared to the prior art. As a first contribution, a re-synthesis architecture based on direct deformation is proposed. The deformation process used inside the deformation stages according to the approaches of the prior art described above has been redesigned, in particular, by replacing the back deformation, widely used in the prior art, with the direct deformation provided by a module that predicts the corresponding position in the target image for each pixel of the original Images. The inventors have found that predicting the direct deformation field from the original image is a simpler task since the direct deformation field is aligned with the original image. This is fundamentally different from the back deformation field, which is spatially aligned with the target image and is not spatially aligned with the original image. The presence of spatial alignment between the original image and the direct deformation field makes the prediction display easier to learn for convolutional architecture.

However, the results of direct deformation contain gaps that must be filled. Most of the solutions from the prior art approach the problem of reconstructing passes using competitive (competitive) architectures. Thus, the second contribution of the proposed invention is a new type of skip discriminator, which is specific for reconstruction tasks. Skip discriminators are trained only on “fake” (ie, reconstructed) images and do not require “real” images. For each fake image, the gap discriminator is trained to predict the binary mask of the gaps that were introduced into the reconstruction network. As a result, training with respect to the discriminator of gaps encourages reconstruction networks to fill gaps in a way that makes gaps indistinguishable.

Both of the proposed contributions to the prior art are not independent, but complement each other, forming a new approach to re-synthesis, which was evaluated by the inventors for various tasks, such as frontalization of the face and re-synthesis of the whole body.

A new methodology for the re-synthesis of the whole body is also proposed. In this methodology, the coordinates of the body texture are estimated using the so-called DensePose method (tight (by all pixels) estimation of the posture (body position)). The texture is filled (completed) using a deep convolutional network. A deep convolution network can even be used to predict the color of unknown pixels. Alternatively, a deep network is used, which predicts for each pixel in the texture image a coordinate pair in the original image (coordinate-based reconstruction). The latter scheme (coordinate-based reconstruction) gives a much sharper texture. The completed texture is used to form a new projection of the full body (posture) specified by the coordinates of the body surface for each foreground pixel in the target image. Optionally, another deep network can be used to form the final target image, taking the generated overlay image and some other images as input.

In accordance with a first aspect, the present invention provides an image re-synthesis system 100 comprising:

a source image input unit 110;

direct deformation module 120;

module 130 filling the passes.

The gap filling module 130 further comprises a deformation error correction module 131 configured to correct forward deformation errors in the target image. Forward strain module 120 is configured to predict, for each pixel of the source image, a corresponding position in the target image, the forward strain module is capable of predicting a forward strain field that is aligned with the original image. The gap filling module 130 is configured to fill the gaps resulting from the use of the direct deformation module 120.

In an embodiment, the image re-synthesis system 100 further comprises a texture transfer architecture 150 configured to: predict warp fields for the source image and the target image; mapping the original image into the texture space through direct deformation, restoring the texture space to the full texture; and display the full texture back to a new pose using reverse deformation.

In an exemplary embodiment, the image re-synthesis system 100 further comprises a texture extraction module 160 configured to extract the texture from the original image. At least the direct deformation module 120 and the gap filling module 130 can be implemented as deep convolution networks. At least one of these deep convolutional networks is trained using a real / fake image discriminator, capable of distinguishing between images of true sections and reconstructed images.

In an embodiment, the skipping module 130 comprises a skip reconstructor 132, which, in turn, may comprise at least one of:

a coordinate assignment module 1321, configured to assign to each pixel p = (x, y) an input image of a coordinate pair (u, v) of a texture in accordance with a fixed predetermined texture mapping to provide a two-channel map of x and y values in the texture coordinate system;

module 1322 texture map completion, configured to provide a complete texture map, where for each pixel (u, v) texture is known the corresponding pixel (x [u, v], y [u, v]) of the image;

a final texture generating unit 1323 configured to form a final texture by mapping image values from positions (x [u, v], y [u, v]) to a texture at positions (u, v) to provide a full color final texture;

module 1342 re-display the final texture, configured to re-display the final texture on a new projection by providing another display from the coordinates of the image pixels in the coordinates of the texture.

In an embodiment, the image re-synthesis system 100 further comprises an image refinement module 170 configured to correct defects in the output image.

In another aspect of the present invention, a system 200 is provided for training a gap filling module 130. The system 200 is configured to train the gap filling module in parallel and in conjunction with the gap discriminator network 210, while the gap discriminator network 210 is trained to predict the binary gap mask, and the gap filling module 130 is trained to minimize the accuracy of the gap discriminator network 210.

With reference to FIG. 7, in yet another aspect, the present invention relates to a method 300 for re-synthesizing an image, comprising the steps of:

input source image (S310);

predictions for each pixel of the source image of the corresponding position in the target image (S320), and the direct deformation field that is aligned with the original image is predicted;

predictions of the binary mask of gaps resulting from direct deformation (S330),

filling gaps (S340) based on said binary ghost mask by forming a texture image by predicting a coordinate pair in the original image for each pixel in the texture image; and

displaying the full texture back to a new pose using reverse deformation (S350).

In an exemplary embodiment, the gap filling step (340) comprises the steps of:

assigning (S341), to each pixel p = (x, y) of the input image, a coordinate pair (u, v) of the texture in accordance with a fixed predetermined texture mapping, to provide a two-channel map of x and y values in the texture coordinate system;

providing (S342) a complete texture map, where for each pixel (u, v) of the texture the corresponding pixel (x [u, v], y [u, v]) of the image is known;

forming (S343) the final texture by mapping the image values from the positions (x [u, v], y [u, v]) to the texture at the positions (u, v) to provide a complete color final texture;

re-mapping (S344) the final texture to the new projection by providing another mapping from the coordinates of the image pixels to the coordinates of the texture.

Also provided is a computer program product 400 comprising computer program code 410, which, when executed by one or more processors, causes one or more processors to implement the method in accordance with the previous aspect. The computer program product 400 may be stored on a permanent computer-readable medium 500.

With reference further to FIG. 1, the difference between the forward and reverse deformation is illustrated, explained from the point of view of the frontalization problem of the face. In both schemes, a deformation field (bottom; hue = direction, saturation = magnitude) from the input image (top) is predicted, and deformation (right) is applied. In the case of direct deformation, the input image and the predicted field are aligned (for example, the movement of the tip of the nose is predicted at the position of the tip of the nose). In contrast, in the case of reverse deformation, the input image and the deformation field are not aligned. The direct deformation methodology in the context of the present invention will be described in more detail below.

One of skill in the art can readily understand that the methodology described below is adapted to be performed by deep convolutional neural networks that can implement elements of the image re-synthesis system 100 according to the invention and the steps of the image re-synthesis method 300, as mentioned above. A detailed description of the methodology given below with reference to mathematical operations and relations between different data elements can be based on the corresponding functions, and not on specific elements of the system 100 or method 300, as described above, and in such cases, a person skilled in the art can easily to obtain links between the mentioned system elements and / or the steps of the method, on the one hand, and the corresponding functions mentioned below, on the other hand, without the need to strictly limit the scope of various ways of implementing the mentioned functions to a specific relationship between each function and the corresponding system element and / or step way. System elements and / or method steps implemented by deep convolutional neural networks are meant to be illustrative and non-restrictive in the context of implementing the re-image synthesis approach of the invention, as described in detail below.

Re-synthesis by direct deformations

, где

является вектором некоторых обучаемых параметров, которые обучаются на определенном наборе данных.Assume that x is a source image and y is a target image, and suppose that x [p, q] denotes an image element (sample) at an integer location (p, q) (which may be, for example, an RGB value). Let w [p, q] = ( u [p, q], v [p, q]) be a deformation field. Often, this warp field will be predicted from x by means of a convolution network

where

is a vector of some learning parameters that are trained on a particular data set.

The standard approach to warp-based image re-synthesis uses warping to warp the original image x into the target image y :

where discretization in fractional positions is defined bilinearly. More formally, the result of reverse deformation is defined as:

where the bilinear core K is defined as follows:

so for each (p, q) the summation in (2) is performed for i =

The reverse deformation methodology was initially implemented for deep image recognition (see, for example, [13]) and was later widely used for deep re-synthesis of an image (see, for example, [5, 30, 20, 3, 23]), becoming standard layer in deep learning packages. It was found that, for re-synthesis problems with significant geometric transformations, the use of back-deformation layers provided a significant improvement in the quality and possibility of generalization compared to architectures with re-synthesis using convolutional layers alone (see, for example, [3]).

, предсказанный сетью

для пиксела (p, q), определяет движение для части объекта, которая первоначально проецировалась на пиксел

. Например, рассмотрим задачу фронтализации лица, где желательно, чтобы сеть предсказывала поле деформации фронтализации при заданном входном изображении, содержащем нефронтальное изображение лица. Предположим, что положение (p, q) в начальном изображении соответствует кончику носа, в то время как для фронтализованного лица то же самое положение соответствует центру правой щеки. Когда обратное деформирование используется для повторного синтеза, предсказание сети

для положения (p, q) должно содержать фронтализующее движение центра правой щеки. В то же время, рецептивное поле выходного блока сети при (p, q) во входном изображении соответствует кончику носа. Таким образом, сеть должна предсказывать движение щеки при наблюдении появления фрагмента, центрированного на носу (см., фиг. 1). Если фронтализующее движение мало, такое рассогласование может быть обработано достаточно глубокой сверточной архитектурой с достаточно большими рецептивными полями. Однако, когда движения становятся большими, такое отображение становится прогрессивно более трудным для обучения для сверточной архитектуры.However, back deformation is limited by the lack of alignment between the original image and the deformation field. In fact, as can be seen from the expression (1) presented above, the vector

predicted by the network

for a pixel (p, q), defines the motion for the part of the object that was originally projected onto the pixel

. For example, consider the face frontalization problem, where it is desirable for the network to predict the frontalization deformation field for a given input image containing a non-frontal face image. Suppose that the position (p, q) in the initial image corresponds to the tip of the nose, while for the frontalized face, the same position corresponds to the center of the right cheek. When back warping is used for re-synthesis, network prediction

for position (p, q) it should contain a frontalizing motion of the center of the right cheek. At the same time, the receptive field of the output network block at (p, q) in the input image corresponds to the tip of the nose. Thus, the network should predict the movement of the cheek when observing the appearance of a fragment centered on the nose (see, Fig. 1). If the frontalizing movement is small, such a mismatch can be processed by a sufficiently deep convolutional architecture with sufficiently large receptive fields. However, when movements become large, such a mapping becomes progressively more difficult to learn for convolutional architecture.

Therefore, direct deformation performed by the direct deformation module is used in re-synthesis architectures in accordance with the present invention instead of back deformation. The direct deformation operation is defined so that the following equality is approximately performed for the output image y _fw :

Thus, in the case of direct deformation, the deformation vector in the pixel [p, q] determines the motion of this pixel. To implement direct deformation, the bilinear core is used to rasterize the source pixels to the target image in the following way. Firstly, all contributions from all pixels are aggregated using a convolutional core into the aggregator map a :

At the same time, the total weight of all contributions to each pixel is accumulated in a separate aggregator w :

Finally, the value in the pixel is determined by normalizing:

предотвращает численные нестабильности. Формально, для каждого целевого положения (i, j) суммирование в (5) и (6) пробегает по всем исходным пикселам (p, q). Однако, поскольку для каждого исходного пиксела (p, q) билинейное ядро

принимает ненулевые значения только в цетырех положенниях в целевом изображении, вышеуказанное суммирование может быть вычислено эффективным образом с использованием одного прохода по пикселам исходного изображения. Отметим, что подобные методы используются для частичных сверток (см., например, [17]). Так как операции (5)-(7) являются кусочно-дифференцируемыми по отношению к входному изображению x и полю деформирования (u, v), можно обратно распространить градиенты посредством операции прямого деформирования при обучении сверточной сети.where is the small constant

prevents numerical instabilities. Formally, for each target position (i, j), the summation in (5) and (6) runs through all the original pixels (p, q). However, since for each source pixel (p, q) the bilinear core

takes nonzero values only in four positions in the target image, the above summation can be calculated efficiently using one pass through the pixels of the original image. Note that similar methods are used for partial convolutions (see, for example, [17]). Since operations (5) - (7) are piecewise differentiable with respect to the input image x and the deformation field ( u, v ), gradients can be redistributed through the direct deformation operation when training a convolutional network.

The main advantage of direct deformation over reverse deformation is that the input image and the predicted deformation field are aligned in the case of direct deformation, since the network prediction in the pixel (p, q) now corresponds to the 2D movement of the part of the object that is projected onto (p, q) in the input image. In the frontalization example shown above, the convolutional network should predict the frontalizing movement of the tip of the nose, based on the receptive field centered on the tip of the nose. This mapping is easier to train for a convolutional network than in the case of back deformation, and this effect has been experimentally demonstrated.

However, on the other hand, in most situations, the output y _{fw of the} direct deformation operation contains a series of empty pixels, which were not reflected in any of the original pixels. The binary mask of pixels that are non-empty is denoted by m, ie m [i, j] = [ w [i, j]> 0]. The next stage of reconstruction is then necessary to fill such gaps.

FIG. 2 illustrates the process of training a neural network to perform gap reconstruction using gap discriminators. The inventors of the present invention trained a reconstruction network to fill in the gaps in the input image (where the known pixels are identified by a mask) while minimizing the reconstruction loss with respect to the “truth section” (“truth truth” is information obtained by direct observation, as opposed to the information displayed). In parallel, the segmentation network (also referred to here as the discriminator of omissions) was trained to predict the mask from the result of the fill operation with minimization of the loss of mask prediction. The reconstruction network was trained on a competitive basis to the pass discriminator network by maximizing the loss of mask prediction, which caused the filled parts in the reconstructed image to be indistinguishable from the original parts.

The process of “reconstruction” of gaps resulting from the previous stage of direct deformation will be described in more detail below, to illustrate, but not limit.

Reconstruction with skip discriminators

завершения изображения с обучаемыми параметрами

отображает изображение y_fw и маску m на завершенное (реконструированное) изображение y_inp:Function

completion of the image with training parameters

maps the image y _fw and the mask m to the completed (reconstructed) image y _inp :

, выбор функции потерь для ее обучения играет решающую роль. Наиболее часто, обучение

выполняется в контролируемой настройке, которая предусматривает обеспечение набора данных завершенных изображений, проектирование случайного процесса, который закрывает части этих изображений, и обучение сети, чтобы реверсировать этот случайный процесс. Минимизация последующих потерь затем выполняется во время обучения:The use of deep networks with gated convolutions for processing reconstruction tasks has been experimentally confirmed as effective in providing good architectures for reconstructing gaps resulting from deformation during image re-synthesis. Regardless of architecture

, the choice of the loss function for its training plays a crucial role. Most often, training

performed in a controlled setting, which provides for providing a set of completed image data, designing a random process that covers parts of these images, and training the network to reverse this random process. Minimization of subsequent losses is then performed during training:

обозначает завершенные изображения. Норма в выражении (9) может быть выбрана как L1 норма (т.е. сумма абсолютных разностей по каждой координате) или как более сложные перцепционные потери, основанные не на разностях между пикселами, а на разностях между представлениями высокого уровня признаков изображения, извлеченными из предварительно обученной сверточной нейронной сети (см., например, [14]).where i iterates through the training examples, and

indicates completed images. The norm in expression (9) can be selected as the L1 norm (i.e., the sum of the absolute differences for each coordinate) or as more complex perceptual losses based not on differences between pixels, but on differences between representations of a high level of image features extracted from pre-trained convolutional neural network (see, for example, [14]).

классификации параллельно с сетью

. Цель обучения для

состоит в дискриминации между реконструированными и исходными (неискаженными) изображениями:When empty pixels form large continuous gaps, learning results with pixel or perceptual losses are usually suboptimal and lack a reliable large-scale structure due to the intrinsic multi-modality of the task. The use of competitive training (see, for example, [6]) provides significant support in this case. Competitive Learning Teaches a Separate Network

classification in parallel with the network

. Learning purpose for

consists of discrimination between reconstructed and original (undistorted) images:

затем расширяется отдельным условием, которое измеряет вероятность дискриминатора, чтобы классифицировать реконструированное изображение как реальное:Learning purpose for

then expanded by a separate condition that measures the probability of the discriminator to classify the reconstructed image as real:

Approaches from the prior art to competitive reconstruction offer the use of two discriminators based on the same principle, but focused on different parts of the images. One of them, referred to as the global discriminator, focuses on the full image, while the other, the local discriminator, focuses on the most important part, for example, in the immediate vicinity of the gap or the central part of the face (see, for example, [4]) .

пропуска обучается предсказывать маску m из реконструированного изображения путем минимизации взвешенных потерь кросс-энтропии для двоичной сегментации:The present invention proposes the use of a different kind of discriminators (referred to herein as omission discriminators) for reconstructing the omission. The inventors of the present invention have found that people tend to judge the success of a reconstruction operation by their (in) ability to identify gaps in a reconstructed image. Interestingly, people do not need to know some kind of “true plot” for such a judgment. To mimic this idea, a discriminator

the pass learns to predict the mask m from the reconstructed image by minimizing the weighted cross-entropy losses for binary segmentation:

Here, (⋅) denotes the elementwise product (summation over all pixels) and | m | denotes the number of nonzero pixels in the mask m . As the training of the skip discriminator develops, the reconstruction network learns to obfuscate the skip discriminator by maximizing the same cross-entropy losses (12) (thus realizing a zero-sum game). New losses can be used in parallel with дицион traditional ’competitive losses (11), as well as any other losses. The proposed new losses are applicable to any reconstruction / completion problem, not necessarily due to direct deformation.

истинного участка поступает с двоичной маской

известных пикселов. Эта маска должна отличаться от входной маски

(иначе, процесс обучения может сходиться к тривиальному решению идентичности для сети реконструирования). В таких ситуациях, потери, характеризуемые приведенными выше выражениями (9)-(11), адаптируются так, что

и

заменяются на

и

, соответственно. Интересно, что новые конкурентные потери не учитывают завершенные изображения истинных участков. Поэтому, даже когда завершенные изображения истинных участков недоступны, потери, характеризуемые в выражении (12) выше, могут все еще применяться без модификации (как для обучения дискриминатора пропусков, так и для обучения сети реконструирования).Learning with an unfinished true site. In some situations, such as texture reconstruction tasks, complete images of the true patch are not available. Instead, each image

true plot comes with a binary mask

known pixels. This mask must be different from the input mask.

(otherwise, the learning process may converge to a trivial identity solution for the reconstruction network). In such situations, the losses characterized by the above expressions (9) - (11) are adapted so that

and

replaced by

and

, respectively. Interestingly, the new competitive losses do not take into account the completed images of the true sites. Therefore, even when complete images of the true sections are not available, the losses described in expression (12) above can still be applied without modification (both for training the discriminator of omissions and for training the reconstruction network).

FIG. 3 shows an example of frontalization of a face by direct deformation in accordance with at least one embodiment of the present invention. In this example, an algorithm trained on 80% of random samples from the Multi-PIE dataset was visually evaluated based on two randomly selected objects from the control plot. Each input photograph (1st row in Fig. 3) independently passed through the deformation field regressor, resulting in a deformed image (2nd row in Fig. 3), and then through the reconstructor, resulting in a reconstructed image with filled gaps and corrected deformation errors (3rd row).

FIG. 4 illustrates an example texture transfer architecture in accordance with at least one embodiment of the present invention. The texture transfer architecture predicts warp fields for both the source and target images. Then, after the original image is displayed in the texture space through direct deformation, it is restored to the full texture and then displayed back to a new position using reverse deformation, the result of which is then refined. In FIG. 4, 0F and 0B are the forward and reverse deformations, respectively, while WF and WF are the predicted and true desepose deformation fields.

FIG. 5 illustrates an example of texture completion for the task of re-synthesizing a new pose in accordance with at least one embodiment of the present invention. The texture is extracted from the input image of the person (the first column in Fig. 5). Then it is reconstructed using a deep convolutional network. The third column in FIG. 5 shows the result of a reconstruction using a network trained without a discriminator of omissions. The addition of the skip discriminator (fourth column in Fig. 5) gives more distinct and more reliable results in the reconstructed areas. The resulting textures are then superimposed on the image of the person observed in a new pose (columns five and six in Fig. 5, respectively). The last column in FIG. 5 shows an image of a true site for a person in a new pose.

FIG. 6 illustrates an example of whole body re-synthesis using coordinate-based texture reconstruction. Input image A is used to form texture image B. A coordinate-based reconstruction is used that predicts the coordinates of the pixels in the original image for each texture image. The result is shown in image C, where texture color pixels are sampled from the original image in the prescribed coordinates. The image of a person in a new position (target image) is synthesized by taking the prescribed texture coordinates for the pixels of the target image and transferring colors from the texture (image D). Finally, a separate deep refinement network transforms the image into a new image (image E). Image F shows the true image of a person in a new pose.

End-to-end training

Since the networks of both direct deformation and reconstruction are differentiable in a cross-cutting manner (i.e., partial derivatives of any loss function with respect to the parameters of all layers, including the layers in front of the direct deformation module and the layers with the reconstruction network, can be calculated using the backpropagation process ), an integrated system (direct deformation and reconstruction) can be trained end-to-end by applying a gap discriminator to try to predict the locations of the gaps resulting from the direct deformation process for an integrated network.

Coordinate Based Reconstruction

The purpose of coordinate-based reconstruction is to complete the texture of, for example, the person represented in the image, based on portions of the texture recovered from the original image. More specifically, starting from the original image, the following steps are performed:

1. A pre-trained deep neural network is launched, which assigns, to each pixel p = (x, y) of the input image, a pair of coordinates (u, v) of the texture in accordance with a fixed predetermined texture mapping. As a result, the pixel coordinates are assigned to the subset of texture pixels, resulting in a two-channel map of x and y values in the texture coordinate system with a large number of texture pixels for which this mapping is unknown.

2. As the next step, launch the second deep convolutional neural network h with training parameters μ, so that the x- and y-cards are completed, resulting in a map of the completed texture, where for each pixel (u, v) of the texture the corresponding pixel is known ( x [u, v], y [u, v]) images.

3. The final texture is obtained by taking the image values (for example, in the red, green and blue channels) in position (x [u, v], y [u, v]) and placing them on the texture in position (u, v) , the result what is the finished color texture.

4. After the completed texture is obtained, it is used to texture the new projection of the person in a different position, which provides a different mapping from the coordinates of the image pixels to the coordinates of the texture.

The parameters μ can be further optimized, so that the sequence of steps 1-4 described above forms a close match with the second image in the pair when the first image of the pair was used to create the texture. Any standard loss (e.g., per-pixel, perceptual) can be used to measure the degree of proximity. The skip discriminator can be added to training the network to complete texture.

Finally, a separate refinement network can be used to transform the re-textured image obtained in the sequence of steps 1-4 in order to improve visual quality. This refinement network can be trained separately or in conjunction with the texture completion network.

The following are some examples of the practical implementation of the approaches according to the invention as an illustration, but not limitation. The approach according to the invention based on direct deformation followed by a reconstruction network trained with skipping discriminators is applied to a variety of problems with different levels of complexity.

Frontalization of the face

As a first task, the face frontalization approach is considered, which is intended to deform the image of a non-frontally oriented face into a frontalized one, while maintaining identity, facial expression and lighting. The method is trained and evaluated on the Multi-PIE dataset, which is described, for example, in [7], and the Multi-PIE dataset is a dataset of more than 750,000 upper body images of 337 people whose images were obtained for four sessions with varying (and famous) projections, lighting conditions and facial expressions. The architecture used was U-Net (see, for example, [22]) (N convolutional layers, N for direct deformation, and an hourglass-like architecture for the reconstruction network).

Face and upper body rotation

For face rotation, the methodology according to the invention was trained and evaluated on a Multi-PIE data set (see, for example, [7]). For each subject, 15 projections were simultaneously taken using several cameras, 13 of which were placed around the subject at regular intervals of 15 °, in the range from -90 ° to 90 °, in the same horizontal plane, and 2 at an elevated level. Each collection of multiple projections was shot under 19 different lighting conditions, up to 4 sessions and 4 facial expressions. In the experiments, only 13 cameras were used, placed around the subject in the same horizontal plane. In experiments for the upper torso, raw (original) images were used, while in experiments for faces, the MTCNN face detector was used to find the boundary frame of the face and crop it with a gap of 10 pixels. 128 × 128 was the standard resolution for the experiment, and all images were eventually resized to this resolution before entering the training algorithm. Frontalization is considered as the most important special case of the rotation problem in experiments.

с обучаемыми параметрами

, которая следует архитектуре U-Net (см., например, [22]). Обеспечено входное изображение (и два дополнительных массива meshgrid (ячеистой решетки), кодирующих строки и столбцы пикселов), и ConvNet формирует поле смещений, закодированное двумя 2D массивами

. Это поле затем трансформируется в поле прямой деформации путем простого суммирования

и подается на прямой решетчатый дискретизатор. В описываемом случае,

кодирует движение пиксела (p,q) на входном изображении. Отметим, однако, что так же самая конструкция могла бы потенциально использоваться для регрессии поля обратной деформации, если дополнительно применяется обратный дискретизатор.The proposed image re-synthesis conveyor consists of two large parts: a deformation field regressor implemented in the form of a direct deformation module, and a reconstructor implemented in the form of a gap filling module. The warp field regressor is a convolutional network (ConvNet)

with learning parameters

which follows the U-Net architecture (see, for example, [22]). An input image is provided (and two additional meshgrid (mesh) arrays encoding rows and columns of pixels), and ConvNet generates an offset field encoded by two 2D arrays

. This field is then transformed into a direct strain field by simple summation

and fed to a direct trellis sampler. In this case,

encodes the motion of a pixel (p, q) in the input image. Note, however, that the same design could potentially be used to regress the backward deformation field if an inverse sampler is additionally used.

с обучаемыми параметрами

также на основе архитектуры U-Net (хотя и без связей пропуска) со всеми свертками, замененными на стробированные свертки. Они являются аттентивными слоями, впервые предложенными в [28], чтобы эффективно обрабатывать сложные задачи реконструирования. Используется стробированная свертка, как определено в [28]:The second part, the reconstructor, is the network

with learning parameters

also based on the U-Net architecture (albeit without skip connections) with all convolutions replaced with gated convolutions . They are attentive layers, first proposed in [28], to efficiently handle complex reconstruction tasks. A gated convolution is used, as defined in [28]:

(13)

(13)

является входным изображением,

являются тензорами весов, и

и ELU являются сигмоидальной функцией и функцией экспоненциально-линейной единичной активации, соответственно. Реконструктор получает деформированное изображение с пропусками, маску пропусков и положения кодирования тензора ячеистой решетки пикселов и предсказывает реконструированное изображение.Where

is an input image

are tensors of weights, and

and ELU are a sigmoid function and an exponentially linear unit activation function, respectively. The reconstructor receives a deformed image with gaps, a mask of gaps and the encoding position of the pixel mesh lattice tensor and predicts the reconstructed image.

реального/поддельного изображения может быть организован как стек плоской и пошаговой сверток, в основном следуя архитектуре части экстрактора признаков VGG-16, за которым следует усредненный опрос и сигмоидальная функция. Результирующее число указывает предсказанную вероятность того, что изображение является «реальным». Второй дискриминатор является дискриминатором

пропусков, который нацелен на восстановление маски пропусков из реконструированного изображения путем решения проблемы сегментации. Генератор GAN, напротив, пытается «ввести в заблуждение» дискриминатор пропусков путем формирования изображений с реконструированными областями, неотличимыми от нереконструированных областей.The model is trained in the structure of a generative competitive network (GAN) and two networks of discriminators are added. The first discriminator, the discriminator of the real / fake image, is aimed at distinguishing the output images of the true portion from the reconstructed images generated by the generative reconstructing network. Discriminator

a real / fake image can be organized as a stack of flat and step-by-step convolutions, mainly following the architecture of the part of the feature extractor VGG-16, followed by an average survey and sigmoid function. The resulting number indicates the predicted probability that the image is "real." The second discriminator is the discriminator

skipping, which aims to restore the skipping mask from the reconstructed image by solving the segmentation problem. The GAN generator, by contrast, is trying to “mislead” the skip discriminator by forming images with reconstructed areas indistinguishable from unreconstructed areas.

As mentioned above, end-to-end conveyor training is a difficult task that requires a careful balance between the various loss components. The L _generator loss value for generative ConvNet, which contains the strain field regressor followed by the reconstructor, is optimized as follows:

(14)

(fourteen)

штрафует деформированное изображение и поле деформации, а

штрафует только реконструированное изображение,

и

являются штрафами генератора, соответствующими конкурентному обучению с первым дискриминатором, дискриминатором реального/поддельного изображения, и вторым дискриминатором, дискриминатором пропусков, соответственно. Следовательно, эти компоненты разлагаются на следующие базовые функции потерь:Where

fines the deformed image and the deformation field, and

fines only the reconstructed image,

and

are generator penalties corresponding to competitive training with the first discriminator, the real / fake image discriminator, and the second discriminator, the skip discriminator, respectively. Therefore, these components are decomposed into the following basic loss functions:

(15)

(15)

являются деформированным изображением и маской без пропусков, полученной прямым дискретизатором

является полем прямой деформации и x_i является i-ой входной выборкой. Здесь и далее, решетчатая сетка в качестве входа в регрессор поля деформации и реконструктор опущена для ясности.Where

are a deformed image and a mask without gaps obtained by the direct sampler

is a direct deformation field and x _i is the i-th input sample. Hereinafter, the lattice grid as an input to the strain field regressor and the reconstructor is omitted for clarity.

(16)

(sixteen)

where v is an extractor of an identity tag. Light-CNN-29, previously trained on the MS-Celeb-1M dataset, is used as a source of identity-invariant embedding. During training, v weights are fixed.

следует выражению (11), и

определяется аналогичным образом:

follows the expression (11), and

defined in the same way:

(17)

(17)

Together with the generator, both discriminators are updated with the aforementioned losses (10) and (12).

With reference to FIG. 3, illustrates the effectiveness of an algorithm trained on a Multi-PIE subset of 80% random samples and evaluated on the remaining 20% of the data. The results shown in FIG. 3 correspond to random samples from the control part.

Assess body texture and posture transfer

The methods of direct deformation and discriminator of gaps are used for the texture transfer problem. The DeepFashion dataset (see, for example, [18]) was used to demonstrate the effectiveness of the constructed model, which restores the full texture of the human body, which can be projected onto the body of any shape and posture to form the target image.

The model performs texture transfer in four steps:

1. Display the initial image in the texture space and detect its missing areas

2. Restore missing areas to restore full texture

3. Project the restored texture into a new body pose

4. Refine the resulting image to eliminate defects that appear after re-projecting the texture.

All modules that perform these steps can be trained simultaneously in an end-to-end manner, although separate loss functions are applied to the outputs of each module. The model diagram can be seen in FIG. five.

24 textures of various body parts placed on a single RGB image are used to predict the coordinates of the texture image for each pixel in the original image in order to map the input images to the texture space by forming their warp fields. The warp fields for both the source and target images are formed by the same network with a UNet-like architecture. Since the deformation fields establish a correspondence between the coordinates of the initial image and the texture space, it becomes possible to form texture images from a photograph using direct deformation and reconstruct a human image from a texture using reverse deformation with the same deformation field. To train the generator of the deformation field, we use the visualization of the true section uv generated by the densepose model (see, for example, [8]) and fine the resulting deformation fields with losses L1 (the sum of the absolute value of the differences):

(18)

(18)

является исходным изображением, WF является полем деформирования истинного участка, и

является генератором поля деформирования.Where

is the original image, WF is the deformation field of the true portion, and

is a deformation field generator.

After the original image is mapped to texture space, the resulting texture image has many missing parts (gaps) due to the mutual closure of the parts in the original image. These missing parts are then reconstructed by the gap filling module, which is the second part of the training model.

The trained model uses a gated hourglass architecture (see, for example, [28]) and a skip discriminator to form reliable textures along with l1 losses. The loss function for the reconstructor is as follows:

(19)

(nineteen)

является реконструктором, и L_gap вычисляется, как указано в выражении (12), в то время как веса дискриминатора пропусков обновляются по отношению к функции потерь 10.Here t _source and t _target are the source texture and target structure, respectively,

is the reconstructor, and the L _{gap is} calculated as indicated in expression (12), while the weights of the omission discriminator are updated with respect to the loss function 10.

After the reconstructed texture is formed, it can be re-projected back onto any body encoded with its deformation field. Having a deformation field for the target image formed by the predictor of the deformation field, the image of the initially selected person in the target position can be formed by deforming the texture back with the target deformation field.

Although a reliable reconstruction of the texture can be obtained in this way, the resulting deformed image can have many defects caused by discrepancies observed when different parts of the texture join together, as well as some small areas that are not in the texture space. These problems are difficult to solve in the texture space, but they can easily be solved in the space of the original image. For this purpose, an image refinement module embodied as a refinement network can be used to eliminate these defects. The refinement network output is the final result of the model. The VGG losses between it and the real target image are calculated together with the discriminator of the real / fake image, which tries to distinguish between the images from the data set and the generated images:

(20)

(20)

является обратным деформированием, VGG является VGG потерями, т.е. l2 расстоянием между признаками, извлеченными VGG-16 сетью, и

является дискриминатором реального/поддельного изображения, функции потерь которого выражены следующим образом:Where

is the reverse deformation, VGG is the VGG loss, i.e. l2 the distance between tags extracted by the VGG-16 network, and

is a discriminator of a real / fake image, the loss functions of which are expressed as follows:

(21).

(21).

After describing one or more machine learning models based on deep neural networks, pre-trained or trained to perform the task of image re-synthesis, the image re-synthesis system 100 according to the invention, which implements the methodology in accordance with the present invention, can be characterized as containing, in particular :

a source image input unit 110; a direct deformation module 120 configured to predict, for each pixel of the source image, a corresponding position in the target image, the direct deformation module 120 being configured to predict a direct deformation field that is aligned with the original image; a gap filling module 130 comprising a skipping discriminator 210 configured to predict a binary skipping mask that is a result of direct deformation and a skipping reconstructor 132 configured to fill the skips based on said binary skipping mask by generating a texture image by predicting a coordinate pair in the original image for each pixel in the texture image; and a target image output unit 180.

The direct deformation module 120 may further comprise a deformation field regressor 121, resulting in an image obtained by direct deformation. The gap filling module 130, as mentioned above, may further comprise a deformation error correction module 131 configured to correct for direct deformation errors in the target image.

In accordance with at least one embodiment, the system according to the invention may further comprise a texture transfer architecture 150 configured to: predict warp fields for the source image and target image; mapping the original image into the texture space through direct deformation, restoring the texture space to the full texture; and display the full texture back to a new position using reverse deformation.

The system may also further comprise a texture extraction module 160 configured to extract the texture (s) from the original image. According to the invention, at least the direct deformation module 120 and the gap filling module 130 can be implemented as deep convolutional neural networks. At least one of the deep convolutional networks can be trained using a real / fake image discriminator configured to distinguish between true plots and reconstructed images. The skip discriminator 210 of the system of the invention can be trained in the form of a separate classification network to distinguish between reconstructed and original images by predicting mask m from the reconstructed image by minimizing weighted cross-entropy losses for binary segmentation. The skip reconstructor 132 of the system according to the invention may also comprise: a coordinate assignment module 1321, configured to assign to each pixel p = (x, y) an input image of a coordinate pair (u, v) of a texture in accordance with a fixed predetermined texture mapping to provide a two-channel map x and y values in the texture coordinate system; a texture map completion module 1322, configured to provide a complete texture map, where for each pixel (u, v) of the texture the corresponding pixel (x [u, v], y [u, v]) of the image is known; a final texture generating unit 1323 configured to produce a final texture by mapping image values from positions (x [u, v], y [u, v]) to a texture at positions (u, v) to provide a complete color final texture; a final texture re-mapping unit 1324, configured to re-display the final texture on a new projection by providing another mapping from the coordinates of the image pixels to the coordinates of the texture.

In at least one embodiment, the image re-synthesis system may comprise an image refinement module 170 adapted to correct defects in the output image caused by discrepancies observed where different parts of the texture are connected.

For a person skilled in the art it should be clear that the above-mentioned modules of the system according to the invention can be implemented by various software tools, hardware, firmware, which are well known in the art. For example, to perform the functions and / or processing described above, various combinations of hardware and software may be provided, and these combinations will be obvious to a person skilled in the art based on a thorough study of the above description. The claimed invention is not limited to any specific implementation forms or combinations, as mentioned above, but can be implemented in various forms depending on the specific task (s) of the re-synthesis of the image, which must be solved.

The above is a detailed description of specific exemplary embodiments of the invention, which are intended to illustrate, but not limit, the material and technical means that implement the corresponding components of the system and the steps of the image processing method, their functional properties and the relationship between them, as well as the mode of operation of the system and image processing method according to the invention. Other embodiments that fall within the scope of the present invention may become apparent to a person skilled in the art based on a thorough study of the description above with reference to the attached drawings, and all such obvious modifications, variations and / or equivalent replacements are intended to be included in scope of the present invention. The order in which the steps of the method according to the invention are mentioned in the claims does not necessarily determine the actual sequence in which the steps of the method are implied, and some steps of the method can be performed substantially simultaneously, one after another or in any suitable sequence, unless otherwise specifically specified defined and / or determined by the context of this disclosure. Some steps of the method can be performed once or an appropriate number of times, although this is not indicated in the claims or elsewhere in the application materials.

It should also be noted that the invention may also take other forms than what is described above, and some components, modules, elements, functions can be implemented as software, hardware, firmware, integrated circuits, FPGAs, etc. . where applicable. The claimed invention or at least its specific parts, components, modules or steps may be implemented by a computer program stored on a computer-readable medium, the program being executed on a general-purpose computer, GPU, multifunction device, or any suitable image processing device, causes said device to execute some or all steps of the inventive method and / or control at least some of the components of the inventive system for re-synthesis of images in such a way that they work as described above. Examples of computer-readable media suitable for storing said computer program or code, instructions, or elements or modules of a computer program may include any type of permanent computer-readable medium that is known to one skilled in the art.

All non-patent documents [1] - [30] of the prior art mentioned and discussed herein and listed below are hereby incorporated into this disclosure by reference, where applicable.

Additional aspects of the invention may become apparent to a person skilled in the art based on a study of the above description with reference to the drawings. One skilled in the art will appreciate that other embodiments of the invention are possible and that the details of the invention can be modified in a number of aspects without departing from the concept of the invention. Therefore, the drawings and description should be considered as illustrative, but not restrictive. The scope of the claimed invention is determined only by the attached claims.

List of cited sources

[1] V. Blanz and T. Vetter. Face recognition based on fitting a 3d morphable model. T-PAMI, 25 (9): 1063-1074, 2003.2

[2] J. Cao, Y. Hu, B. Yu, R. He, and Z. Sun. Load balanced gans for multi-view face image synthesis. arXiv preprint arXiv: 1802.07447, 2018.2

[3] J. Cao, Y. Hu, H. Zhang, R. He, and Z. Sun. Learning a high fidelity pose invariant model for high-resolution face frontalization. arXiv preprint arXiv: 1806.08472, 2018.1,3

[4] J. Deng, S. Cheng, N. Xue, Y. Zhou, and S. Zafeiriou. UV-gan: adversarial facial uv map completion for pose-invariant face recognition. In Proc. CVPR, pages 7093-7102, 2018.1, 2, 4

[5] Y. Ganin, D. Kononenko, D. Sungatullina, and V. Lempitsky. Deepwarp: Photorealistic image resynthesis for gaze manipulation. In European Conference on Computer Vision, pages 311-326. Springer, 2016.1,2,3

[6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.2,4

[7] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker. Multi-pie. Image and Vision Computing, 28 (5): 807-813, 2010.5

[8] R. A. Guler, N. Neverova, and I. Kokkinos. DensePose: Dense human pose estimation in the wild. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.2, 8

[9] Y. Hu, X. Wu, B. Yu, R. He, and Z. Sun. Pose-guided photo-realistic face rotation. In Proc. CVPR, 2018.2

[10] R. Huang, S. Zhang, T. Li, R. He, et al. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. arXiv preprint arXiv: 1704.04086, 2017.2

[11] S. Iizuka, E. Simo-Serra, and H. Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36 (4): 107, 2017.1, 2

[12] P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proc. CVPR, pages 5967-5976, 2017.2

[13] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. In Proc. NIPS, pages 2017-2025, 2015.1,2,3

[14] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proc. ECCV, pages 694-711, 2016.4

[15] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1 (4): 541-551, 1989.1

[16] Y. Li, S. Liu, J. Yang, and M.-H. Yang Generative face completion. In Proc. CVPR, volume 1, page 3, 2017.1,2

[17] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro. Image inpainting irregular holes using partial convolutions. arXiv preprint arXiv: 1804.07723, 2018.1,2,3

[18] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proc. CVPR, pages 1096-1104, 2016.6

[19] N. Neverova, R. A. Guler, and I. Kokkinos. Dense pose transfer. In The European Conference on Computer Vision (ECCV), September 2018.2

[20] E. Park, J. Yang, E. Yumer, D. Ceylan, and A. C. Berg. Transformation-grounded image generation network for novel 3d view synthesis. In Proc. CVPR, pages 702-711. IEEE, 2017.1,3

[21] J. S. Ren, L. Xu, Q. Yan, and W. Sun. Shepard convolutional neural networks. In Proc. NIPS, pages 901-909, 2015.1, 2

[22] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, pages 234-241. Springer, 2015.5

[23] A. Siarohin, E. Sangineto, S. Lathuilire, and N. Sebe. De¬formable gans for pose-based human image generation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 1, 2, 3

[24] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.2

[25] J. Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger. Sparsity invariant cnns. In International Conference on 3D Vision (3DV), pages 11-20. IEEE, 2017.1,2

[26] J. Yim, H. Jung, B. Yoo, C. Choi, D. Park, and J. Kim. Rotating your face using multi-task deep neural network. In Proc. CVPR, pages 676-684, 2015.2

[27] X. Yin, X. Yu, K. Sohn, X. Liu, and M. Chandraker. Towards large-pose face frontalization in the wild. In Proc. ICCV, pages 1-10, 2017.2

[28] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv: 1806.03589, 2018.1, 2, 5, 6, 8

[29] J. Zhao, L. Xiong, P. K. Jayashree, J. Li, F. Zhao, Z. Wang, P. S. Pranata, P. S. Shen, S. Yan, and J. Feng. Dual-agent gans for photorealistic and identity preserving profile face synthesis. In Proc. NIPS, pages 66-76, 2017.2

[30] T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros. View synthesis by appearance flow. In Proc. ECCV, pages 286¬301, 2016.1,2,3.

Claims (34)

1. The system of re-synthesis of the image, containing: input image input module; a direct deformation module configured to predict, for each pixel of the source image, a corresponding position in the target image, the direct deformation module configured to predict a direct deformation field that is aligned with the original image; and a gap filling module configured to fill gaps resulting from the application of direct deformation by predicting a binary mask of gaps, which are the result of direct deformation, filling gaps based on said binary mask of gaps by generating a texture image by predicting a coordinate pair in the original image for each pixel in the image texture. 2. The image re-synthesis system according to claim 1, wherein the omission filling module further comprises a deformation error correction module configured to correct direct deformation errors in the target image. 3. The image re-synthesis system according to claim 1, further comprising a texture transfer architecture configured to: prediction of warp fields for the source image and the target image; mapping the original image into the texture space through direct deformation; restore texture space to full texture and display the full texture back into a new pose using reverse deformation. 4. The image re-synthesis system of claim 1, further comprising a texture extraction module configured to extract the texture from the original image. 5. The image re-synthesis system according to claim 1, wherein at least the direct deformation module and the gap filling module are implemented as deep convolutional neural networks. 6. The system for re-synthesis of the image according to claim 1, in which the module for filling in the blanks contains a gap designer, and the gap designer contains: a coordinate assignment module, configured to assign to each pixel p = (x, y) an input image of a pair of coordinates (u, v) of the texture in accordance with a fixed predetermined texture mapping to provide a two-channel map of x and y values in the texture coordinate system; a texture map completion module, configured to provide a complete texture map, where for each pixel (u, v) of the texture the corresponding pixel (x [u, v], y [u, v]) of the image is known; a final texture generating unit configured to produce a final texture by displaying image values from positions (x [u, v], y [u, v]) on a texture at positions (u, v) to provide a complete color final texture; a final texture re-mapping module configured to re-display the final texture on a new projection by providing another mapping from the coordinates of the image pixels to the coordinates of the texture. 7. The image re-synthesis system according to claim 5, wherein at least one of the deep convolutional networks is trained using a real / fake image discriminator configured to distinguish between images of the true region and reconstructed images. 8. The image re-synthesis system according to claim 4, further comprising an image refinement module configured to correct defects in the output image. 9. A system for training the gap filling module configured to fill the gaps as part of image re-synthesis, the system being configured to teach the gap filling module in parallel and together with the gap discriminator network, the gap discriminator network learning to predict the binary gap mask, and the fill module badges are trained to minimize the accuracy of the bad discriminator network. 10. A method for re-synthesizing an image comprising the steps of: enter the original image; for each pixel of the source image, a corresponding position in the target image is predicted, and a direct deformation field is predicted that is aligned with the original image; predict the binary mask of gaps, which are the result of direct deformation, filling gaps based on said binary ghost mask by forming a texture image by predicting a pair of coordinates in the original image for each pixel in the texture image; and display the full texture back to a new pose using reverse deformation. 11. The method of re-synthesis of the image according to claim 10, in which filling in the gaps contains stages in which: assigning to each pixel p = (x, y) of the input image a pair of coordinates (u, v) of the texture in accordance with a fixed predetermined texture mapping to provide a two-channel map of x and y values in the texture coordinate system; provide a complete texture map, where for each pixel (u, v) of the texture the corresponding pixel (x [u, v], y [u, v]) of the image is known; forming the final texture by mapping the image values from the positions (x [u, v], y [u, v]) to the texture at the positions (u, v) to provide a complete color final texture; re-display the final texture on a new projection by providing another mapping from the coordinates of the image pixels to the coordinates of the texture. 12. A training method for the gap filling module configured to fill the gaps as part of image re-synthesis, the method comprising the step of teaching the gap filling module in parallel with and together with the gap discriminator network, the gap discriminator network being trained to predict the binary gap mask, and the gap filling module is trained to minimize the accuracy of the gap discriminator network. 13. A permanent computer-readable medium on which a computer program product is stored containing computer program code, which, when executed by one or more processors, causes one or more processors to implement the method according to any one of claims. 10 or 11.

Images