CN113222144B

CN113222144B - Training method of image restoration model, image restoration method, device and equipment

Info

Publication number: CN113222144B
Application number: CN202110604235.1A
Authority: CN
Inventors: 王伟; 袁泽寰; 王长虎
Original assignee: Beijing Youzhuju Network Technology Co Ltd
Current assignee: Beijing Youzhuju Network Technology Co Ltd
Priority date: 2021-05-31
Filing date: 2021-05-31
Publication date: 2022-12-27
Anticipated expiration: 2041-05-31
Also published as: CN113222144A; WO2022252883A1

Abstract

The embodiment of the application discloses a training method of an image restoration model, an image restoration method, a device and equipment, wherein a real first pixel quality image, a false real first pixel quality image and a reconstructed second pixel quality image are obtained by performing image feature extraction on the real first pixel quality image and a first artificially synthesized pixel quality image and by a real first pixel quality image generator and a real second pixel quality image generator; and respectively calculating the domain alignment loss, the image generation loss and the image reconstruction loss, and training by using the obtained loss. The image restoration model obtained after training has better generalization capability, is more accurate in image restoration and has better model performance. And repairing the first pixel quality image to be repaired by utilizing the image characteristic encoder and the real second pixel quality image generator which are obtained by training, so that a second pixel quality image with a good repairing effect can be obtained, and the requirements of image use are met.

Description

Training method of image restoration model, image restoration method, device and equipment

Technical Field

The present application relates to the field of image processing, and in particular, to a method, an apparatus, and a device for training an image inpainting model.

Background

In the process of generating and processing the image, the image may be affected by the equipment, so that the pixel quality of the image is low, and the requirement of using the image cannot be met. In order to improve the pixel quality of the image with lower pixel quality, the image with lower pixel quality can be processed by adopting an image super-resolution reconstruction technology, so that the pixel quality of the image with lower pixel quality is improved, and the image with higher pixel quality is obtained.

At present, the image super-resolution reconstruction technology can be specifically realized by an image restoration model constructed by a convolutional neural network. The image inpainting model needs to be generated through training image training. The training images used to train the image inpainting model have a large impact on the performance of the generated image inpainting model. The training images typically use artificially constructed lower pixel quality images and higher pixel quality images. And the quality of the artificially constructed image with low pixel quality is low, so that the performance of the image restoration model obtained by training is poor, and the requirement of image pixel quality restoration cannot be met. Therefore, how to train and generate an image restoration model with better performance is an urgent problem to be solved.

Disclosure of Invention

In view of this, embodiments of the present application provide a training method, an apparatus, and a device for an image inpainting model, and an image inpainting method, an apparatus, and a device, which can train to obtain an image inpainting model with better performance, and implement inpainting on an image with lower pixel quality based on the image inpainting model to obtain an image with higher pixel quality more accurately.

In order to solve the above problem, the technical solution provided by the embodiment of the present application is as follows:

in a first aspect, an embodiment of the present application provides a method for training an image inpainting model, where the method includes:

inputting a real first pixel quality image into an image feature encoder to obtain a first image feature;

inputting the artificially synthesized first pixel quality image into the image characteristic encoder to obtain a second image characteristic; the artificially synthesized first pixel quality image is obtained by blurring a real second pixel quality image;

inputting the first image characteristic into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

inputting the second image characteristic into the real first pixel quality image generator to obtain a pseudo-real first pixel quality image;

inputting the second image characteristics into a real second pixel quality image generator to obtain a reconstructed second pixel quality image;

calculating a domain alignment loss from the first image feature and the second image feature;

calculating an image generation loss from the regenerated true first pixel quality image, the pseudo-true first pixel quality image and the true first pixel quality image;

calculating an image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image;

training the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator according to the domain alignment loss, the image generation loss and the image reconstruction loss, and repeatedly executing the steps of inputting the real first pixel quality image into the image feature encoder to obtain a first image feature and the subsequent steps until a preset condition is reached;

wherein the image sharpness of the first pixel quality is lower than the image sharpness of the second pixel quality.

In a second aspect, an embodiment of the present application provides an image inpainting method, where the method includes:

inputting a first pixel quality image to be restored into an image characteristic encoder to obtain target image characteristics;

inputting the target image characteristics into a real second pixel quality image generator to obtain a repaired second pixel quality image;

the image feature encoder and the real second pixel quality image generator are obtained by training according to the training method of the image inpainting model in any one of the embodiments.

In a third aspect, an embodiment of the present application provides an apparatus for training an image inpainting model, where the apparatus includes:

the first execution unit is used for inputting a real first pixel quality image into the image feature encoder to obtain a first image feature;

the second execution unit is used for inputting the artificially synthesized first pixel quality image into the image feature encoder to obtain a second image feature; the artificially synthesized first pixel quality image is obtained by blurring a real second pixel quality image;

a third execution unit, configured to input the first image feature into a true first pixel quality image generator, so as to obtain a regenerated true first pixel quality image;

a fourth execution unit, configured to input the second image feature into the true first pixel quality image generator, so as to obtain a pseudo-true first pixel quality image;

a fifth execution unit, configured to input the second image feature into a real second pixel quality image generator, so as to obtain a reconstructed second pixel quality image;

a first calculation unit configured to calculate a domain alignment loss from the first image feature and the second image feature;

a second calculation unit for calculating an image generation loss from the regenerated real first pixel quality image, the pseudo-real first pixel quality image and the real first pixel quality image;

a third calculation unit for calculating an image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image;

a training unit, configured to train the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator according to the domain alignment loss, the image generation loss, and the image reconstruction loss, and repeatedly execute the step of inputting the true first pixel quality image into the image feature encoder to obtain a first image feature and subsequent steps until a preset condition is reached;

In a fourth aspect, an embodiment of the present application provides an image restoration apparatus, including:

the eighth execution unit is used for inputting the first pixel quality image to be restored into the image feature encoder to obtain the target image feature;

a ninth execution unit, configured to input the target image feature into a real second pixel quality image generator, so as to obtain a repaired second pixel quality image;

In a fifth aspect, an embodiment of the present application provides an electronic device, including:

one or more processors;

a storage device having one or more programs stored thereon,

when the one or more programs are executed by the one or more processors, the one or more processors implement the method for training an image inpainting model according to any one of the above embodiments, or the method for inpainting an image according to any one of the above embodiments.

In a sixth aspect, the present application provides a computer-readable medium, on which a computer program is stored, where the program is executed by a processor to implement the training method of the image inpainting model according to any one of the above embodiments, or the image inpainting method according to any one of the above embodiments.

Therefore, the embodiment of the application has the following beneficial effects:

the embodiment of the application provides a training method, a device and equipment of an image restoration model, wherein a real first pixel quality image and a artificially synthesized first pixel quality image are respectively input into an image characteristic encoder to obtain a corresponding first image characteristic and a corresponding second image characteristic; inputting the first image characteristic and the second image characteristic into a real first pixel quality image generator respectively, and correspondingly obtaining a regenerated real first pixel quality image and a false real first pixel quality image; inputting the second image characteristics into a real second pixel quality image generator to obtain a reconstructed second pixel quality image; respectively calculating domain alignment loss, image generation loss and image reconstruction loss based on the obtained image features and pixel quality images, and training an image feature encoder, a real first pixel quality image generator and a real second pixel quality image generator based on the calculated loss; and repeatedly executing the training process until a preset condition is reached. Based on the domain alignment loss, the image generation loss and the image reconstruction loss, the difference between a real lower pixel quality image and a artificially synthesized lower pixel quality image can be better reduced, the difference between a generated lower pixel quality image under a simulated real scene and a generated lower pixel quality image under an actual real scene is reduced, and the difference between a generated higher pixel quality image under the simulated real scene and a generated higher pixel quality image under the actual real scene is reduced, so that the image restoration model obtained based on the calculated loss training has better generalization capability and restoration capability and better model performance. According to the image restoration method, the image restoration device and the image restoration equipment, the image characteristic encoder and the second pixel quality image generator which are generated by the image restoration model are used for restoring the first pixel quality image to be restored, so that the second pixel quality image with a good restoration effect can be obtained, and the use requirement of the image is met.

Drawings

Fig. 1 is a schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application;

fig. 2 is a flowchart of a training method of an image inpainting model according to an embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of an image restoration model according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of another image restoration model provided in an embodiment of the present application;

fig. 5 is a schematic structural diagram of another image restoration model provided in an embodiment of the present application;

fig. 6 is a schematic structural diagram of a feature discriminator according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an image feature encoder according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a real second pixel quality image generator according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of an RRDB unit according to an embodiment of the present disclosure;

fig. 10 is a flowchart of an image restoration method according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of a training apparatus for an image inpainting model according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of an image repairing apparatus according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present application more comprehensible, embodiments accompanying the drawings are described in detail below.

In order to facilitate understanding and explaining the technical solutions provided by the embodiments of the present application, the following description will first describe the background art of the present application.

After researching the traditional image restoration method, the inventor finds that the image with lower pixel quality in the training image used for training the image restoration model at present is obtained by performing image quality degradation on the image with higher pixel quality. The process of obtaining the image with lower pixel quality after processing the image with higher pixel quality through artificial simulation of blur and noise is difficult to simulate the process of generating the image with lower pixel quality. Resulting in a larger distribution difference between the artificially synthesized lower pixel quality image and the true lower pixel quality image. The image restoration model generated by training by using the artificially synthesized image with lower pixel quality as training data has poor generalization capability, and is difficult to reconstruct the image with lower pixel quality to be restored with higher quality.

Based on this, the embodiment of the application provides a training method, a device and equipment for an image restoration model, wherein a real first pixel quality image and a artificially synthesized first pixel quality image are respectively input into an image feature encoder to obtain a corresponding first image feature and a corresponding second image feature; inputting the first image characteristic and the second image characteristic into a real first pixel quality image generator respectively, and correspondingly obtaining a regenerated real first pixel quality image and a false real first pixel quality image; inputting the second image characteristics into a real second pixel quality image generator to obtain a reconstructed second pixel quality image; and respectively calculating domain alignment loss, image generation loss and image reconstruction loss based on the obtained image characteristics and the generated pixel quality image, and repeatedly executing the training process based on the calculated loss image characteristic encoder, the real first pixel quality image generator and the real second pixel quality image generator until a preset condition is reached. Based on the domain alignment loss, the image generation loss and the image reconstruction loss, the difference between a real lower pixel quality image and a artificially synthesized lower pixel quality image can be better reduced, the distribution difference between a generated lower pixel quality image under a simulated real scene and a generated lower pixel quality image under an actual real scene is reduced, and the difference between a generated higher pixel quality image under the simulated real scene and a generated higher pixel quality image under the actual real scene is reduced, so that the image restoration model obtained based on the calculated loss training has better generalization capability and restoration capability and better model performance. According to the image restoration method, the image restoration device and the image restoration equipment, the image characteristic encoder and the second pixel quality image generator which are generated by the image restoration model are used for restoring the first pixel quality image to be restored, so that the second pixel quality image with a good restoration effect can be obtained, and the image use requirements are met.

In order to facilitate understanding of the image restoration method provided in the embodiment of the present application, the following description is made with reference to a scene example shown in fig. 1. Referring to fig. 1, the drawing is a schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application.

In practical application, a first pixel quality image to be restored, which has lower image pixels and needs to be restored, exists, the first pixel quality image is input into an image feature encoder, and image features of the first pixel quality image can be extracted to obtain target image features. And inputting the obtained target image characteristics into a real second pixel quality image generator to obtain a repaired second pixel quality image. The second pixel quality image is an image with higher image pixels. The image feature encoder and the real second pixel quality image generator are both generated in advance through a training method of an image restoration model.

Those skilled in the art will appreciate that the block diagram shown in fig. 1 is only one example in which embodiments of the present application may be implemented. The scope of applicability of the embodiments of the present application is not limited in any way by this framework.

In order to facilitate understanding of the present application, a method for training an image inpainting model provided in an embodiment of the present application is described below with reference to the accompanying drawings.

Referring to fig. 2, which is a flowchart of a training method of an image inpainting model according to an embodiment of the present application, as shown in fig. 2, the method may include S201-S209:

s201: and inputting the real first pixel quality image into an image characteristic encoder to obtain a first image characteristic.

Referring to fig. 3, the figure is a schematic structural diagram of an image inpainting model provided in an embodiment of the present application. The real first pixel quality image is an image with a lower pixel quality generated in a real scene. The true first pixel quality image may be a lower pixel quality image with some blurring and/or noise, which may be affected by the device or data processing during image generation or processing of the image. True first pixel quality images may suffer from various blurring and noise degradation problems.

The image feature encoder is an encoder for extracting features of an image. The real first pixel quality image is input into the image feature encoder, and feature extraction can be performed on the real first pixel quality image to obtain a first image feature corresponding to the real first pixel quality image.

S202: inputting the artificially synthesized first pixel quality image into an image feature encoder to obtain a second image feature; the artificially synthesized first pixel-quality image is obtained by blurring the real second pixel-quality image.

Wherein the image sharpness of the first pixel quality is lower than the image sharpness of the second pixel quality. The artificially synthesized first pixel-quality image is an artificially constructed image of lower pixel quality. The artificially synthesized first pixel-quality image may be obtained by blurring the true second pixel-quality image. Wherein the real second pixel quality image is a higher pixel quality image generated in a real scene. By blurring the real second pixel quality image, the pixel quality of the real second pixel quality image can be degraded, and the artificially synthesized first pixel quality image with lower pixel quality can be obtained. The blurring process is an image processing mode for reducing pixel quality, and specifically, the blurring process may be adding gaussian noise and gaussian blurring to an image.

It should be noted that, in the embodiments of the present application, it is not limited whether the real first pixel quality image and the real second pixel quality image are images of the same content. The true first pixel-quality image and the true second pixel-quality image may be of the same content, only images of different pixel qualities, i.e. pairs of images of different pixel qualities. The actual first pixel quality image and the actual second pixel quality image may also be different content images with different pixel qualities, i.e. unpaired images with different pixel qualities.

And inputting the artificially synthesized first pixel quality image into an image feature encoder, and performing feature extraction on the artificially synthesized first pixel quality image to obtain a second image feature corresponding to the artificially synthesized first pixel quality image.

S203: inputting the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image.

The real first pixel-quality image generator is for generating a corresponding lower pixel-quality image in a real scene based on the input image features. Inputting the first image feature into the true first pixel quality image generator, a regenerated true first pixel quality image may be obtained.

The regenerated true first pixel quality image is also a lower pixel quality image. The regenerated true first pixel quality image is an image corresponding to the true first pixel quality image that simulates a lower pixel quality image in a true scene.

S204: and inputting the second image characteristics into a real first pixel quality image generator to obtain a pseudo-real first pixel quality image.

The second image feature is input into the true first pixel quality image generator, possibly to a false true first pixel quality image. The pseudo-true first pixel quality image is a lower pixel quality image. The pseudo-real first pixel quality image is an image of lower pixel quality in a simulated real scene corresponding to the artificially synthesized first pixel quality image.

S205: and inputting the second image characteristics into a real second pixel quality image generator to obtain a reconstructed second pixel quality image.

The real second pixel-quality image generator is for generating a corresponding higher pixel-quality image in the real scene based on the input image features. Inputting the second image feature into the true second pixel-quality image generator, a reconstructed second pixel-quality image may be obtained.

The second pixel quality image is reconstructed to a higher pixel quality image. The reconstructed second pixel quality image is an image corresponding to the artificially synthesized first pixel quality image, which simulates a higher pixel quality in a real scene.

S206: a domain alignment penalty is calculated based on the first image feature and the second image feature.

The first image feature is an image feature corresponding to a true first pixel quality image, and the second image feature is an image feature corresponding to a synthetic first pixel quality image. Based on the first image feature and the second image feature, a comparison may be made between the true first pixel quality image and the artificially synthesized first pixel quality image.

From the obtained first image feature and the second image feature, a domain alignment loss may be calculated. The domain alignment loss may reflect the difference between the true first pixel quality image and the artificially synthesized first pixel quality image from a feature perspective.

In a possible implementation manner, an embodiment of the present application provides a specific implementation manner for calculating a domain alignment loss according to a first image feature and a second image feature, which is described in detail below.

S207: and calculating image generation loss according to the regenerated real first pixel quality image, the false real first pixel quality image and the real first pixel quality image.

The regenerated true first pixel quality image is generated by a true first pixel quality image generator based on the first image feature. The pseudo-true first pixel quality image is generated by the true first pixel quality image generator based on the second image feature. Based on the regenerated true first pixel quality, the false true first pixel quality image and the true first pixel quality image, a loss of a lower pixel quality image of the true first pixel quality image generator in generating the simulated true scene, that is, an image generation loss, can be calculated. Image generation loss may be a measure of the difference between the generated lower pixel quality image and the true lower pixel quality image.

In a possible implementation manner, the present application provides a specific implementation manner for calculating an image generation loss according to the regenerated real first pixel quality image, the false real first pixel quality image, and the real first pixel quality image, which is described in detail below.

S208: an image reconstruction loss is calculated from the reconstructed second pixel quality image and the true second pixel quality image.

The reconstructed second pixel quality image is generated by a real second pixel quality image generator based on the second image feature. And reconstructing a higher pixel quality image under the real scene corresponding to the second pixel quality image into a real second pixel quality image. Based on the reconstructed second pixel quality image and the true second pixel quality image, a loss of a higher pixel quality image of the true second pixel quality image generator in generating the simulated true scene, i.e., an image reconstruction loss, may be calculated. Image reconstruction loss can be a measure of the difference between the resulting higher pixel quality image and the true higher pixel quality image.

In a possible implementation manner, an embodiment of the present application provides a specific implementation manner for calculating an image reconstruction loss according to a reconstructed second pixel quality image and a real second pixel quality image, which is described in detail below.

S209: training an image feature encoder, a real first pixel quality image generator and a real second pixel quality image generator according to the domain alignment loss, the image generation loss and the image reconstruction loss, repeatedly executing the step of inputting the real first pixel quality image into the image feature encoder to obtain the first image feature and the subsequent steps until the preset condition is reached.

Based on the calculated domain alignment loss, image generation loss, and image reconstruction loss, an image feature encoder, a true first pixel quality image generator, and a true second pixel quality image generator may be trained. Jointly adjusting the image feature encoder, the true first pixel quality image generator and the true second pixel quality image generator by using the calculated loss. And repeatedly performing the above-described S201-S209, training the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator. And stopping training the image characteristic encoder, the real first pixel quality image generator and the real second pixel quality image generator until the preset condition is met, and realizing the training of the image restoration model. Specifically, the preset condition may be the number of times of repeatedly performing the training, for example, the preset condition may be 500 times of repeatedly performing the training.

In a possible implementation manner, an embodiment of the present application further provides a specific implementation manner of training the image feature encoder, the real first pixel quality image generator, and the real second pixel quality image generator according to the domain alignment loss, the image generation loss, and the image reconstruction loss, which is described in detail below.

Based on the above-mentioned relevant contents of S201-S209, by extracting image features with an image feature encoder, a corresponding lower pixel quality image is generated by a true first pixel quality image generator, and a corresponding higher pixel quality image is generated by a true second pixel quality image generator. By using the obtained image features, the generated lower pixel quality image, the generated higher pixel quality image, the corresponding lower pixel quality image in the real scene and the corresponding higher pixel quality image in the real scene, the corresponding image feature loss, the image generation loss and the image reconstruction loss can be calculated. Based on the calculated loss, the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator are trained, so that the difference between a real lower pixel quality image and a artificially synthesized lower pixel quality image can be reduced, the difference between a generated lower pixel quality image under a simulated real scene and a generated lower pixel quality image under an actual real scene can be reduced, the difference between a generated higher pixel quality image under a simulated real scene and a generated higher pixel quality image under an actual real scene can be reduced, the image feature encoder and the real second pixel quality image generator which have better generalization capability and better repairing effect can be obtained, and the purpose of obtaining a higher pixel quality image with better repairing effect based on a trained image repairing model can be realized.

In one possible implementation, when the performance of the real first pixel-quality image generator is not good, the generated false real first pixel-quality image may also have a difference in image content from the artificially synthesized first pixel-quality image.

In view of the foregoing problems, an embodiment of the present application provides a method for training an image inpainting model, where in addition to the foregoing S201-S209, the method further includes the following steps:

a1: and inputting the pseudo-real first pixel quality image into an image characteristic encoder to obtain a third image characteristic.

Referring to fig. 4, the figure is a schematic structural diagram of another image inpainting model provided in the embodiment of the present application.

And extracting image characteristics of the false and real first pixel quality image, and inputting the false and real first pixel quality image into an image characteristic encoder to obtain a third image characteristic corresponding to the false and real first pixel quality image.

A2: and calculating content consistency loss according to the second image characteristics and the third image characteristics.

The second image feature is an image feature of the artificially synthesized first pixel-quality image. From the second image feature and the third image feature, a content consistency loss may be calculated. The content consistency loss is used to measure the image gap between the artificially synthesized first pixel quality image and the false-true first pixel quality image. The real first pixel quality image generator can be adjusted based on the content consistency loss, so that the image content of the generated simulated real scene lower pixel quality image and the image content of the corresponding artificially synthesized first pixel quality image are unchanged, and only the degradation mode of the image pixel is changed, thereby improving the image quality of the generated simulated real scene lower pixel quality image and ensuring that the image content is not changed.

Correspondingly, an embodiment of the present application provides a specific implementation manner for training an image feature encoder, a true first pixel quality image generator, and a true second pixel quality image generator according to a domain alignment loss, an image generation loss, and an image reconstruction loss, including:

an image feature encoder, a true first pixel quality image generator, and a true second pixel quality image generator are trained according to a domain alignment loss, an image generation loss, an image reconstruction loss, and a content consistency loss.

And training the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator according to the calculated domain alignment loss, the calculated image generation loss, the calculated image reconstruction loss and the calculated content consistency loss.

In a possible implementation manner, an embodiment of the present application provides a specific implementation manner of calculating a content consistency loss according to the second image feature and the third image feature, which is described in detail below.

In the embodiment of the application, by calculating the content consistency loss and training the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator by using the content consistency loss, the model performance of the generated image restoration model can be further improved, and the image content of the restored image is ensured to be consistent with that of the image to be restored.

Furthermore, a higher pixel quality image under a corresponding simulated real scene can be obtained by using the obtained third image characteristics through a real second pixel quality image generator, the corresponding image reconstruction loss is calculated, and a better-performance image restoration model is obtained by using image reconstruction loss training.

Correspondingly, the training method for the image inpainting model provided by the embodiment of the application can further comprise the following steps of:

and inputting the third image characteristics into a real second pixel quality image generator to obtain a pseudo-real reconstructed second pixel quality image.

Referring to fig. 5, the figure is a schematic structural diagram of another image restoration model provided in the embodiment of the present application.

The third image feature is an image feature of a false-true first pixel quality image. The third image feature is input into the real second pixel quality image generator, so that a higher pixel quality image corresponding to the false real first pixel quality image under the simulated real scene can be obtained, namely the false real reconstructed second pixel quality image.

Correspondingly, an embodiment of the present application provides a specific implementation manner for calculating an image reconstruction loss according to a reconstructed second pixel quality image and a real second pixel quality image, including the following two steps:

b1: a first image reconstruction loss is calculated from the reconstructed second pixel quality image and the true second pixel quality image.

The reconstructed second pixel quality image is a generated higher pixel quality image in a simulated real scene corresponding to the real second pixel quality image. The first image reconstruction loss may be calculated from the reconstructed second pixel quality image and the true second pixel quality image, and the first image reconstruction loss may be used to measure a difference between the reconstructed second pixel quality image and the true second pixel quality image.

In a possible implementation manner, the present application provides a specific implementation manner for calculating the reconstruction loss of the first image according to the reconstructed second pixel quality image and the real second pixel quality image, which is described in detail below.

B2: calculating a second image reconstruction loss according to the pseudo-true reconstructed second pixel quality image and the true second pixel quality image; the first image reconstruction loss and the second image reconstruction loss constitute an image reconstruction loss.

The pseudo-true reconstructed second pixel quality image is also a higher pixel quality image in a simulated true scene generated from the lower pixel quality image corresponding to the true second pixel quality image. The second image reconstruction loss can be calculated according to the pseudo-true reconstructed second pixel quality image, and the second image reconstruction loss can be used for measuring the difference between the pseudo-true reconstructed second pixel quality image and the true second pixel quality image.

And forming image reconstruction loss by using the obtained first image reconstruction loss and the second image reconstruction loss.

Based on the above, a pseudo-true reconstructed second pixel quality image is generated based on the third image feature by using the true second pixel quality image, and the image reconstruction loss is calculated based on the reconstructed second pixel quality image and the pseudo-true reconstructed second pixel quality image and the true second pixel quality image, respectively. The obtained image reconstruction loss can more comprehensively measure the difference between the generated simulated higher pixel quality image in the real scene and the generated higher pixel quality image in the actual real scene, and the generated result of the real second pixel quality image generator is restrained, so that an image restoration model with better model performance can be obtained based on image reconstruction loss training.

In a possible implementation manner, an embodiment of the present application provides a method for calculating an image generation loss according to a regenerated true first pixel quality image, a false true first pixel quality image, and a true first pixel quality image, which specifically includes the following two steps:

c1: calculating a first image generation loss from the regenerated true first pixel quality image and the true first pixel quality image.

The regenerated true first pixel quality image is a lower pixel quality image of the generated simulated true scene corresponding to the true first pixel quality image. From the regenerated true first pixel quality image and the true first pixel quality image, a first image generation loss may be calculated, which may be used to measure a difference between the regenerated true first pixel quality image and the true first pixel quality image.

In a possible implementation manner, the embodiment of the present application provides a specific implementation manner for calculating the first image generation loss according to the regenerated real first pixel quality image and the real first pixel quality image, which is described in detail below.

C2: calculating a second image generation loss from the pseudo-true first pixel quality image and the true first pixel quality image; the first image generation loss and the second image generation loss constitute an image generation loss.

The pseudo-real first pixel quality image is a lower pixel quality image generated based on artificially synthesizing the first pixel quality image to simulate a real scene. The second image generation loss may be calculated from the pseudo-true first pixel quality image and the true first pixel quality image. The second image generation penalty may be used to measure the difference between the pseudo-true first pixel quality image and the true first pixel quality image.

Based on the resulting first image generation loss and second image generation loss, an image generation loss may be composed.

In one possible implementation, the embodiment of the present application provides a method for calculating a second image generation loss according to a pseudo-true first pixel quality image and a true first pixel quality image; the first image generation loss and the second image generation loss constitute specific embodiments of the image generation loss, see in particular below.

In the embodiment of the present application, by calculating the first image generation loss and the second image generation loss using the regenerated real first pixel quality image and the false real first pixel quality image, and the real first pixel quality image, respectively, more accurate image generation loss can be obtained based on two aspects. A constraint on the generation result of the true first pixel-quality image generator is implemented. Based on the image generation loss, the difference between the generated image with lower pixel quality and the image with lower pixel quality in the real scene can be reduced, and the image restoration model with better model performance is obtained by training the obtained image generation loss.

In a possible implementation manner, an embodiment of the present application provides a specific implementation manner for calculating a domain alignment loss according to a first image feature and a second image feature, and includes the following three steps:

d1: and inputting the first image characteristic into a characteristic discriminator to obtain a first probability value.

First, it should be noted that the real first pixel quality image corresponding to the first image feature and the artificially synthesized first pixel quality image corresponding to the second image feature are obtained by processing different images. The image features can be input into a feature discriminator to obtain corresponding probability values, and domain alignment loss is calculated.

Referring to fig. 6, the figure is a schematic structural diagram of a feature discriminator according to an embodiment of the present application. The feature discriminator may consist of 6 modules, each module consisting of one 3*3 convolution layer and one activation function layer. The activation function layer may specifically adopt a LeakyRelu activation function.

The feature discriminator is used for outputting a probability value of the generated first image feature, corresponding to the first image feature, of the image feature corresponding to the image in the real scene. By inputting the first image feature into the feature discriminator, a first probability value corresponding to the first image feature output by the feature discriminator can be obtained.

In particular, the first image feature Fx may be denoted Fx = E _C (I _X ) Wherein, I _X Representing a true first pixel quality image, E _C (I _X ) A first image feature representing a true first pixel quality image extracted by an image feature encoder, wherein E _C An image feature encoder may be represented. The first probability value of the feature discriminator output may be expressed as logD _C (Fx). Wherein D is _C Representing a discriminator for a feature.

D2: and inputting the second image characteristics into the characteristic discriminator to obtain a second probability value.

By inputting the second image feature into the feature discriminator, a second probability value corresponding to the second image feature output by the feature discriminator may be obtained.

It should be noted that the second image feature is an image feature of the artificially synthesized first pixel quality image, and a probability value that the second image feature is an image feature simulating an image in a real scene needs to be obtained by the feature discriminator.

In particular, the second image feature F _Z Can be expressed as F _Z ＝E _C (I _Z ) Wherein, I _Z Representing artificially synthesized first pixel quality images，E _C (I _Z ) Representing second image features of the artificially synthesized first pixel-quality image extracted by the image feature encoder, wherein E _C An image feature encoder may be represented. The second probability value output by the feature discriminator may be expressed as log [1-D _C (F _Z )]。

D3: and calculating the domain alignment loss according to the first probability value and the second probability value.

Based on the first probability value and the second probability value output by the feature discriminator, a domain alignment loss may be calculated.

In one possible implementation, the domain alignment is lost

Can be calculated by the following formula:

wherein, E _X [logD _C (Fx)]Indicating the expected value, E, corresponding to the calculated first probability value _Z {log[1-D _C (F _Z )]Indicates the expected value for the calculated second probability value. The domain alignment loss is derived by calculating a sum of the expectation for the first probability value and the expectation for the second probability value.

In the embodiment of the application, the probability values corresponding to the first image characteristic and the second image characteristic are obtained through calculation of the characteristic discriminator, and then the domain alignment loss is calculated by using the probability values obtained through calculation, so that the obtained domain alignment loss is accurate. The corresponding loss value can be calculated by the characteristic discriminator based on the images obtained by different image processing, namely the unpaired images, so that the real second pixel quality image used when generating the artificially synthesized first pixel quality image can be an image with the same content as the real first pixel quality image, and the limitation of generating the training image is reduced.

Further, an embodiment of the present application provides a specific implementation manner of calculating a content consistency loss according to a second image feature and a third image feature, including:

calculating a 1-norm between the second image feature and the third image feature;

and calculating the content consistency loss according to the 1-norm between the second image characteristic and the third image characteristic.

Specifically, a 1-norm between the second image feature and the third image feature is calculated. Wherein the second image feature may be represented as F _Z The third image feature may be denoted as F _Z '. The corresponding second image features and 1-norm between the third image features may be expressed as F _Z' －F _Z || ₁ 。

And calculating the content consistency loss based on the calculated 1-norm between the second image characteristic and the third image characteristic.

In one possible implementation, the content consistency is lost L _pix Can be calculated by the following formula:

L _pix (F _Z ′，F _Z )＝E _Z [||F _Z ′－F _Z || ₁ ] (2)

wherein E is _Z [||F _Z' －F _Z || ₁ ]Represents calculated | | | F _Z ′－F _Z || ₁ The corresponding expected value may specifically be | | | F obtained _Z ′－F _Z || ₁ The corresponding average is calculated.

Based on the above contents, the constraint on a pixel-by-pixel basis is realized by calculating the 1-norm between the second image feature and the third image feature, the calculated content consistency loss can better represent the difference between the artificially synthesized first pixel quality image and the false-true first pixel quality image, and further, an image restoration model with better performance can be obtained through training.

Further, an embodiment of the present application provides a specific implementation manner for calculating a reconstruction loss of a first image according to a reconstructed second pixel quality image and a real second pixel quality image, including:

calculating a 1-norm between the reconstructed second pixel quality image and the true second pixel quality image;

the first image reconstruction loss is calculated from the 1-norm between the reconstructed second pixel quality image and the true second pixel quality image.

In particular, reconstructing the second pixel quality image may be by I _Z→Y Indicating that the true second pixel quality image can be passed through I _Y And (4) showing. The 1-norm between the reconstructed second pixel-quality image and the true second pixel-quality image may be expressed as | | I _Z→Y －I _Y || ₁ 。

A 1-norm between the reconstructed second pixel-quality image and the true second pixel-quality image is calculated.

The 1-norm between the reconstructed second pixel quality image and the true second pixel quality image can be represented by:

L _pix (I _Z→Y ，I _Y )＝E _Y [||I _Z→Y －I _Y || ₁ ] (3)

wherein, E _Y [||I _Z→Y －I _Y || ₁ ]Represents calculated | | | I _Z→Y －I _Y || ₁ The corresponding expected value.

In the embodiment of the application, the constraint can be performed pixel by calculating the 1-norm between the reconstructed second pixel quality image and the real second pixel quality image, the more accurate reconstruction loss of the first image can be calculated, and then the image restoration model with better performance can be obtained through training.

Further, in a possible implementation manner, an embodiment of the present application provides a specific implementation manner for calculating a reconstruction loss of a second image according to a pseudo-true reconstructed second pixel quality image and a true second pixel quality image, including:

calculating a 1-norm between the pseudo-true reconstructed second pixel quality image and the true second pixel quality image;

and calculating the reconstruction loss of the second image according to the pseudo-real reconstructed second pixel quality image and the 1-norm between the real second pixel quality images.

The false-true reconstructed second pixel quality image is obtained by processing a false-true first pixel quality image and processing a different image with the true second pixel quality image. And obtaining a corresponding probability value through the feature discriminator to calculate the domain alignment loss.

In particular, the pseudo-true reconstruction of the second pixel quality image may be by I _Z→X→Y Indicating that the true second pixel quality image can be passed through I _Y And (4) showing. The 1-norm between the reconstructed second pixel-quality image and the true second pixel-quality image may be expressed as | | I _Z→X→Y －I _Y || ₁ 。

A 1-norm between the pseudo-true reconstructed second pixel quality image and the true second pixel quality image is calculated.

The 1-norm between the pseudo-true second pixel-quality image and the true second pixel-quality image may be represented by:

L _pix (I _Z→X→Y ，I _Y )＝E _Y [||I _Z→X→Y －I _Y || ₁ ] (4)

wherein E is _Y [||I _Z→X→Y －I _Y || ₁ ]Represents calculated | | | I _Z→X→Y －I _Y || ₁ The corresponding expected value.

In the embodiment of the application, the 1-norm between the pseudo-real reconstructed second pixel quality image and the real second pixel quality image is calculated, so that the constraint can be performed pixel by pixel, the accuracy of the reconstruction loss of the second image is further improved, and an image restoration model with better performance is obtained through training.

In one possible implementation, an embodiment of the present application provides a method for calculating a first image generation loss according to a regenerated real first pixel quality image and a real first pixel quality image, including:

calculating a 1-norm between the regenerated true first pixel quality image and the true first pixel quality image;

calculating a first image generation loss based on the regenerated true first pixel quality image and a 1-norm between the true first pixel quality images.

In particular, the regenerated true first pixel quality image may be passed through I _X→X Indicating that the true first pixel quality image can be passed through I _X And (4) showing. The regenerated true first pixel quality image and the 1-norm between the true first pixel quality images may be expressed as | | I _X→X －I _X || ₁ 。

The 1-norm between the regenerated true first pixel quality image and the true first pixel quality image can be represented by:

L _pix (I _X→X ，I _X )＝E _X [||I _X→X －I _X || ₁ ] (5)

wherein E is _X [||I _X→X －I _X || ₁ ]Represents calculated | | | I _X→X －I _X || ₁ The corresponding expected value.

In the embodiment of the application, the 1-norm between the regenerated real first pixel quality image and the real first pixel quality image is calculated, so that the constraint can be performed on a pixel-by-pixel basis, and the accuracy of the first image generation loss is further improved, so that an image restoration model with better performance can be trained.

In a possible implementation manner, an embodiment of the present application provides a method for calculating a second image generation loss according to a pseudo-true first pixel quality image and a true first pixel quality image, including:

inputting the pseudo-real first pixel quality image into a feature discriminator to obtain a third probability value;

inputting the real first pixel quality image into a feature discriminator to obtain a fourth probability value;

and calculating the second image generation loss according to the third probability value and the fourth probability value.

The pseudo-true first pixel-quality image is generated by a process of artificially synthesizing the first pixel-quality image, and belongs to a different image from the true first pixel-quality image. The pseudo-real first pixel quality image and the real first pixel quality image are respectively input into the feature discriminator to obtain corresponding probability values, so that the calculation of the generation loss of the second image is realized.

And inputting the pseudo-real first pixel quality image into a feature discriminator to obtain a third probability value output by the feature discriminator. The third probability value is used to represent a probability value that the pseudo-real first pixel quality image is a lower pixel quality image generated under a simulated real scene.

In particular, the false real first pixel quality image may be represented as I _Z→X . Correspondingly, the third probability value may be expressed as log [1-D ] _X (I _Z→X )]. Wherein D is _X To represent a discriminator for an image.

And inputting the real first pixel quality image into the feature discriminator to obtain a fourth probability value output by the feature discriminator. The fourth probability value is used to represent a probability value that the real first pixel quality image is a lower pixel quality image generated under the real scene.

In particular, the true first pixel quality image may be denoted as I _X . Correspondingly, the fourth probability value can be expressed as logD _X (I _X )。

Based on the third probability value and the fourth probability value output by the feature discriminator, a second image generation loss can be calculated.

In one possible implementation, the second image is lost L _adv Can be calculated by the following formula:

L _adv (I _Z→X ，I _X )＝E _X [logD _X (I _X )]+E _Z {log[1-D _X (I _Z→X )]} (6)

wherein, E _X [logD _X (I _X )]Indicating the expected value, E, corresponding to the calculated fourth probability value _Z {log[1-D _X (I _Z→X )]Means obtained by calculationAnd (3) an expected value corresponding to the third probability value. The second image loss is obtained by calculating a sum of the expectation for the third probability value and the expectation for the fourth probability value.

In the embodiment of the application, the probability values corresponding to the pseudo-real first pixel quality image and the real first pixel quality image are obtained through calculation of the feature discriminator, the loss of the second image is calculated by using the probability values obtained through calculation, and the obtained loss of the second image is accurate. The images resulting from different image processing, i.e. unpaired images, can be constrained by a feature discriminator.

In a possible implementation manner, in the image restoration model provided in the embodiment of the present application, the image feature encoder includes at least one residual module;

the true first pixel quality image generator comprises at least one residual module;

the real second pixel quality image generator includes a first convolution layer, a base unit, a second convolution layer, an up-sampling layer, a third convolution layer, an activation function layer, and a fourth convolution layer, the base unit including at least one RRDB (Residual-in-Residual Dense Block) unit.

Fig. 7 is a schematic structural diagram of an image feature encoder according to an embodiment of the present disclosure. When training the image restoration model, the image feature encoder is used for extracting image features and may be composed of a plurality of residual modules. Wherein each residual module may be composed of a convolutional layer of 3*3, an activation function layer, and a convolutional layer of 3*3.

The true first pixel quality image generator is used for training assistance and may be composed of a plurality of residual modules. The structure of the real first pixel quality image generator may be identical to the structure of the image feature encoder, and will not be described herein.

Referring to fig. 8, the diagram is a schematic structural diagram of a real second pixel quality image generator according to an embodiment of the present application. The real second pixel quality image generator performs image restoration after training is completed, needs a complex structure, and realizes more accurate image restoration. The real second pixel quality image generator is composed of a first convolution layer, a basic unit, a second convolution layer, an up-sampling layer, a third convolution layer, an activation function layer and a fourth convolution layer. The first and fourth convolutional layers may be 1*1 convolutional layers, and the second and third convolutional layers may be 3*3 convolutional layers. The activation function used by the activation function layer may specifically be a LeakyRelu activation function.

Specifically, the basic unit may be a plurality of RRDB units. Referring to fig. 9, the diagram is a schematic structural diagram of an RRDB unit according to an embodiment of the present disclosure. The RRDB unit comprises five modules, wherein the first module consists of a convolution layer of 3*3 and an activation function layer, and the second module to the fifth module consist of a convolution layer of 1*1, a convolution layer of 3*3 and an activation function layer. The activation function used by the activation function layer in the RRDB unit may specifically be a LeakyRelu activation function.

In the embodiment of the application, by adopting a simpler model structure of the image feature encoder and the real first pixel quality image generator and adopting a more complicated model structure of the real second pixel quality image generator, the model performance of the generated image restoration model can be ensured on the premise of ensuring that the model structure is more simplified.

Based on the training method of the image inpainting model provided by the embodiment, the embodiment of the application further provides an image inpainting method.

Referring to fig. 10, which is a flowchart of an image repairing method provided in an embodiment of the present application, as shown in fig. 10, the method may include S1001-S1002:

s1001: and inputting the first pixel quality image to be restored into an image characteristic encoder to obtain the target image characteristic.

The first pixel quality image to be restored is an image with lower pixel quality, and image restoration is required to obtain a corresponding image with higher pixel quality.

Inputting the first pixel quality image to be restored into an image characteristic encoder, and extracting the image characteristics of the first pixel quality image to be restored to obtain the corresponding target image characteristics.

The image feature encoder is generated by training according to the image restoration model training method. The image feature encoder generated by training through the image restoration model training method can better extract features of the first pixel quality image to be restored, so that more accurate image restoration is realized.

S1002: and inputting the target image characteristics into a real second pixel quality image generator to obtain a repaired second pixel quality image.

The image feature encoder and the real second pixel quality image generator are obtained by training according to the training method of the image inpainting model in any of the embodiments.

And inputting the obtained target image characteristics into a real second pixel quality image generator. The true second pixel quality image generator is for generating a corresponding higher pixel quality image, i.e. a restored second pixel quality image, based on the input target image features.

The real second pixel quality image generator is generated by training through the image inpainting model training method. The second pixel quality image generator generated by training through the image restoration model training method can generate a restored second pixel quality image which corresponds to the first pixel quality image to be restored and has a better restoration effect based on the target image characteristics.

Based on the relevant contents of the above S1001-S1002, it can be known that the image restoration can be better performed on the first pixel quality image to be restored through the trained image feature encoder and the real second pixel quality image generator, so as to obtain the restored second pixel quality image with better restoration effect, and meet the needs of image use.

Based on the method for training the image inpainting model provided by the embodiment of the method, the embodiment of the application also provides a device for training the image inpainting model, and the device for training the image inpainting model is described below with reference to the accompanying drawings.

Fig. 11 is a schematic structural diagram of a training apparatus for an image inpainting model according to an embodiment of the present application. As shown in fig. 11, the training apparatus for image restoration models includes:

a first execution unit 1101, configured to input a true first pixel quality image into an image feature encoder, to obtain a first image feature;

a second execution unit 1102, configured to input the artificially synthesized first pixel quality image into the image feature encoder to obtain a second image feature; the artificially synthesized first pixel quality image is obtained by blurring a real second pixel quality image;

a third executing unit 1103, configured to input the first image feature into a real first pixel quality image generator, to obtain a regenerated real first pixel quality image;

a fourth execution unit 1104, configured to input the second image feature into the real first pixel quality image generator, so as to obtain a pseudo-real first pixel quality image;

a fifth executing unit 1105, configured to input the second image feature into a real second pixel quality image generator, so as to obtain a reconstructed second pixel quality image;

a first calculating unit 1106, configured to calculate a domain alignment loss according to the first image feature and the second image feature;

a second calculation unit 1107 for calculating an image generation loss from the regenerated real first pixel quality image, the pseudo-real first pixel quality image and the real first pixel quality image;

a third calculation unit 1108 for calculating an image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image;

a training unit 1109, configured to train the image feature encoder, the real first pixel quality image generator, and the real second pixel quality image generator according to the domain alignment loss, the image generation loss, and the image reconstruction loss, and repeatedly execute the step of inputting the real first pixel quality image into the image feature encoder to obtain a first image feature and subsequent steps until a preset condition is reached;

In one possible implementation, the apparatus further includes:

a sixth execution unit, configured to input the pseudo-true first pixel quality image into the image feature encoder, so as to obtain a third image feature;

a fourth calculation unit configured to calculate a content consistency loss according to the second image feature and the third image feature;

the training unit 1109 is specifically configured to train the image feature encoder, the real first pixel quality image generator, and the real second pixel quality image generator according to the domain alignment loss, the image generation loss, the image reconstruction loss, and the content consistency loss.

In one possible implementation, the apparatus further includes:

a seventh execution unit, configured to input the third image feature into the true second pixel quality image generator, so as to obtain a pseudo-true reconstructed second pixel quality image;

the third calculating unit 1108 includes:

a first calculation subunit, configured to calculate a first image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image;

a second calculation subunit, configured to calculate a second image reconstruction loss from the pseudo-true reconstructed second pixel quality image and the true second pixel quality image; the first image reconstruction loss and the second image reconstruction loss constitute an image reconstruction loss.

In a possible implementation manner, the second calculating unit 1107 includes:

a third computing subunit for computing a first image generation loss from the regenerated true first pixel quality image and the true first pixel quality image;

a fourth calculation subunit configured to calculate a second image generation loss from the pseudo-true first pixel quality image and the true first pixel quality image; the first image generation loss and the second image generation loss constitute an image generation loss.

In a possible implementation manner, the first calculating unit 1106 is specifically configured to input the first image feature into a feature discriminator to obtain a first probability value;

inputting the second image feature into the feature discriminator to obtain a second probability value;

and calculating the domain alignment loss according to the first probability value and the second probability value.

In a possible implementation manner, the fourth calculating unit is specifically configured to calculate a 1-norm between the second image feature and the third image feature;

and calculating content consistency loss according to the 1-norm between the second image characteristic and the third image characteristic.

In a possible implementation manner, the first calculating subunit is specifically configured to calculate a 1-norm between the reconstructed second pixel quality image and the true second pixel quality image;

and calculating the reconstruction loss of the first image according to the 1-norm between the reconstructed second pixel quality image and the real second pixel quality image.

In a possible implementation manner, the second calculating subunit is specifically configured to calculate a 1-norm between the pseudo-true reconstructed second pixel quality image and the true second pixel quality image;

In a possible implementation manner, the third computing subunit is specifically configured to compute a 1-norm between the regenerated real first pixel quality image and the real first pixel quality image;

calculating a first image generation loss based on said regenerated true first pixel quality image and a 1-norm between said true first pixel quality images.

In a possible implementation manner, the fourth calculating subunit is specifically configured to input the pseudo-true first pixel quality image into a feature discriminator to obtain a third probability value;

inputting the real first pixel quality image into the feature discriminator to obtain a fourth probability value;

In one possible implementation, the image feature encoder includes at least one residual module;

the real second pixel quality image generator includes a first convolution layer, a base unit, a second convolution layer, an up-sampling layer, a third convolution layer, an activation function layer, and a fourth convolution layer, the base unit including at least one RRDB unit.

Based on the image restoration method provided by the above method embodiment, an embodiment of the present application further provides an image restoration device, and the image restoration device will be described below with reference to the accompanying drawings.

Fig. 12 is a schematic structural diagram of an image repairing apparatus according to an embodiment of the present disclosure. As shown in fig. 12, the image restoration apparatus includes:

an eighth execution unit 1201, configured to input the first pixel quality image to be repaired into an image feature encoder, so as to obtain a target image feature;

a ninth execution unit 1202, configured to input the target image feature into a real second pixel quality image generator, so as to obtain a repaired second pixel quality image;

the image feature encoder and the real second pixel quality image generator are trained according to the training method of the image inpainting model described in any one of the above embodiments.

Based on the training method and the image inpainting method for the image inpainting model provided by the embodiment of the method, the application further provides electronic equipment, which comprises the following steps: one or more processors; a storage device, on which one or more programs are stored, which, when executed by the one or more processors, cause the one or more processors to implement the method for training an image inpainting model according to any of the embodiments, or the method for inpainting an image according to any of the embodiments

Referring now to FIG. 13, shown is a schematic diagram of an electronic device 1300 suitable for use in implementing embodiments of the present application. The terminal device in the embodiment of the present application may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a Digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Portable android device), a PMP (Portable multimedia Player), a car terminal (e.g., car navigation terminal), and the like, and a fixed terminal such as a Digital TV (television), a desktop computer, and the like. The electronic device shown in fig. 13 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 13, electronic device 1300 may include a processing device (e.g., central processing unit, graphics processor, etc.) 1301 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1302 or a program loaded from a storage device 1306 into a Random Access Memory (RAM) 1303. In the RAM1303, various programs and data necessary for the operation of the electronic apparatus 1300 are also stored. The processing device 1301, the ROM1302, and the RAM1303 are connected to each other via a bus 1304. An input/output (I/O) interface 1305 is also connected to bus 1304.

Generally, the following devices may be connected to the I/O interface 1305: input devices 1306 including, for example, touch screens, touch pads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, and the like; an output device 1307 including, for example, a Liquid Crystal Display (LCD), speaker, vibrator, etc.; storage devices 1306 including, for example, magnetic tape, hard disk, etc.; and a communication device 1309. The communications device 1309 may allow the electronic device 1300 to communicate wirelessly or by wire with other devices to exchange data. While fig. 13 illustrates an electronic device 1300 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may be alternatively implemented or provided.

In particular, according to embodiments of the present application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication device 1309, or installed from the storage device 1306, or installed from the ROM 1302. The computer program, when executed by the processing apparatus 1301, performs the above-described functions defined in the methods of the embodiments of the present application.

The electronic device provided by the embodiment of the present application and the training method and the image inpainting method of the image inpainting model provided by the embodiment of the present application belong to the same inventive concept, and technical details that are not described in detail in the embodiment of the present application can be referred to the embodiment of the present application, and the embodiment of the present application have the same beneficial effects.

Based on the training method of the image inpainting model and the image inpainting method provided by the above method embodiments, an embodiment of the present application provides a computer readable medium, on which a computer program is stored, where the program is executed by a processor to implement the training method of the image inpainting model according to any one of the above embodiments or the image inpainting method according to any one of the above embodiments.

It should be noted that the computer readable medium mentioned above in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the image inpainting model training method or the image inpainting method.

Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or hardware. Where the name of a unit/module does not in some cases constitute a limitation on the unit itself, for example, a voice data collection module may also be described as a "data collection module".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems on a chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present application, there is provided [ example one ] a method of training an image inpainting model, the method comprising:

inputting the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

According to one or more embodiments of the present application, [ example two ] there is provided a training method of an image inpainting model, the method further comprising:

inputting the pseudo-real first pixel quality image into the image feature encoder to obtain a third image feature;

calculating content consistency loss according to the second image characteristics and the third image characteristics;

the training the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator according to the domain alignment loss, the image generation loss, and the image reconstruction loss comprises:

the training the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator as a function of the domain alignment loss, the image generation loss, the image reconstruction loss, and the content consistency loss.

According to one or more embodiments of the present application, [ example three ] there is provided a training method of an image inpainting model, the method further comprising:

inputting the third image characteristic into the real second pixel quality image generator to obtain a pseudo-real reconstructed second pixel quality image;

said calculating an image reconstruction loss from said reconstructed second pixel quality image and said true second pixel quality image comprises:

calculating a first image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image;

calculating a second image reconstruction loss according to the pseudo-true reconstructed second pixel quality image and the true second pixel quality image; the first image reconstruction loss and the second image reconstruction loss constitute an image reconstruction loss.

According to one or more embodiments of the present application, there is provided [ example four ] a training method of an image inpainting model, the calculating an image generation loss from the regenerated real first pixel quality image, the pseudo-real first pixel quality image, and the real first pixel quality image, comprising:

calculating a first image generation loss from the regenerated true first pixel quality image and the true first pixel quality image;

calculating a second image generation loss from the pseudo-true first pixel quality image and the true first pixel quality image; the first image generation penalty and the second image generation penalty constitute an image generation penalty.

According to one or more embodiments of the present application, there is provided an image inpainting model training method, wherein the calculating of a domain alignment loss according to the first image feature and the second image feature includes:

inputting the first image characteristic into a characteristic discriminator to obtain a first probability value;

According to one or more embodiments of the present application, there is provided [ example six ] a method of training an image inpainting model, the calculating a content consistency loss from the second image feature and the third image feature, comprising:

According to one or more embodiments of the present application, example seven provides a training method of an image inpainting model, the calculating a first image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image, comprising:

According to one or more embodiments of the present application, [ example eight ] there is provided a training method of an image inpainting model, the calculating a second image reconstruction loss from the pseudo-true reconstructed second pixel quality image and the true second pixel quality image, comprising:

and calculating the reconstruction loss of the second image according to the false-real reconstructed second pixel quality image and the 1-norm between the real second pixel quality images.

According to one or more embodiments of the present application, there is provided [ example nine ] a training method of an image inpainting model, the calculating a first image generation loss from the regenerated true first pixel quality image and the true first pixel quality image, comprising:

calculating a first image generation loss from the regenerated true first pixel quality image and a 1-norm between the true first pixel quality images.

According to one or more embodiments of the present application, there is provided [ example ten ] a training method of an image inpainting model, the calculating a second image generation loss from the pseudo-true first pixel quality image and the true first pixel quality image, comprising:

According to one or more embodiments of the present application, [ example eleven ] there is provided a training method of an image inpainting model, the image feature encoder comprising at least one residual module;

According to one or more embodiments of the present application, [ example twelve ] there is provided an image inpainting method, the method comprising:

the image feature encoder and the real second pixel quality image generator are trained according to the training method of the image inpainting model described in any one of the above examples.

According to one or more embodiments of the present application, [ example thirteen ] there is provided an image inpainting model training apparatus, the apparatus including:

a training unit, configured to train the image feature encoder, the real first pixel quality image generator, and the real second pixel quality image generator according to the domain alignment loss, the image generation loss, and the image reconstruction loss, and repeatedly execute the step of inputting the real first pixel quality image into the image feature encoder to obtain a first image feature and subsequent steps until a preset condition is reached;

According to one or more embodiments of the present application, [ example fourteen ] there is provided an image inpainting model training apparatus, further comprising:

a sixth execution unit, configured to input the pseudo-true first pixel quality image into the image feature encoder to obtain a third image feature;

the training unit is specifically configured to train the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator according to the domain alignment loss, the image generation loss, the image reconstruction loss, and the content consistency loss.

According to one or more embodiments of the present application, [ example fifteen ] there is provided an image inpainting model training apparatus, further comprising:

the third calculation unit includes:

According to one or more embodiments of the present application, there is provided [ example sixteen ] a training apparatus of an image inpainting model, the second calculation unit including:

According to one or more embodiments of the present application, in an example seventeenth, there is provided a training apparatus for an image inpainting model, where the first computing unit is specifically configured to input the first image feature into a feature discriminator to obtain a first probability value;

According to one or more embodiments of the present application, in an eighteenth example, there is provided a training apparatus for an image inpainting model, where the fourth calculating unit is specifically configured to calculate a 1-norm between the second image feature and the third image feature;

According to one or more embodiments of the present application, the [ example nineteenth ] provides a training apparatus for an image inpainting model, wherein the first calculating subunit is specifically configured to calculate a 1-norm between the reconstructed second pixel quality image and the true second pixel quality image;

According to one or more embodiments of the present application, [ example twenty ] there is provided a training apparatus of an image inpainting model, the second calculating subunit being specifically configured to calculate a 1-norm between the pseudo-true reconstructed second pixel quality image and the true second pixel quality image;

According to one or more embodiments of the present application, example twenty-one provides a training apparatus for an image inpainting model, where the third computing subunit is specifically configured to compute the regenerated real first pixel quality image and a 1-norm between the real first pixel quality images;

According to one or more embodiments of the present application, an [ example twenty-two ] provides a training apparatus for an image inpainting model, where the fourth calculating subunit is specifically configured to input the pseudo-true first pixel quality image into a feature discriminator to obtain a third probability value;

According to one or more embodiments of the present application, an apparatus for training an image inpainting model is provided [ example twenty three ], the image feature encoder comprising at least one residual module;

According to one or more embodiments of the present application, there is provided [ example twenty-four ] an image inpainting apparatus, the apparatus comprising:

According to one or more embodiments of the present application, [ example twenty-five ] there is provided an electronic device comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement a method of training an image inpainting model as in any one of the examples above, or [ example twelve ].

According to one or more embodiments of the present application, an example twenty-six provides a computer-readable medium on which a computer program is stored, wherein the program, when executed by a processor, implements a training method of an image inpainting model according to any one of the examples above, or an image inpainting method according to [ example twelve ].

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the disclosure. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the application. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for training an image inpainting model, the method comprising:

2. The method of claim 1, further comprising:

the training the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator according to the domain alignment loss, the image generation loss, the image reconstruction loss, and the content consistency loss.

3. The method of claim 2, further comprising:

4. The method of claim 1, wherein said calculating an image generation penalty from said regenerated true first pixel quality image, said pseudo-true first pixel quality image, and said true first pixel quality image comprises:

calculating a second image generation loss from the pseudo-true first pixel quality image and the true first pixel quality image; the first image generation loss and the second image generation loss constitute an image generation loss.

5. The method of claim 1, wherein said calculating a domain alignment loss from the first image feature and the second image feature comprises:

inputting the first image feature into a feature discriminator to obtain a first probability value;

6. The method of claim 2, wherein the calculating a content consistency loss from the second image feature and the third image feature comprises:

7. The method of claim 3, wherein calculating a first image reconstruction loss from the reconstructed second pixel quality image and the true second pixel quality image comprises:

8. The method of claim 3, wherein calculating a second image reconstruction loss from the pseudo-true reconstructed second pixel quality image and the true second pixel quality image comprises:

9. The method of claim 4, wherein said calculating a first image generation penalty from said regenerated true first pixel quality image and said true first pixel quality image comprises:

10. The method of claim 4, wherein said computing a second image generation penalty from said pseudo-true first pixel quality image and said true first pixel quality image comprises:

11. The method according to any one of claims 1 to 10,

the image feature encoder comprises at least one residual module;

12. An image inpainting method, comprising:

the image feature encoder and the real second pixel quality image generator are trained according to the training method of the image inpainting model of any one of claims 1-11.

13. An apparatus for training an image inpainting model, the apparatus comprising:

14. An image restoration apparatus, characterized in that the apparatus comprises:

15. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement the method of training an image inpainting model according to any one of claims 1-11, or the method of image inpainting according to claim 12.

16. A computer-readable medium, on which a computer program is stored, wherein the program, when being executed by a processor, is adapted to carry out a method of training an image inpainting model as claimed in any one of claims 1 to 11, or a method of image inpainting as claimed in claim 12.