WO2022252883A1

WO2022252883A1 - Training method for image inpainting model and image inpainting method, apparatus, and device

Info

Publication number: WO2022252883A1
Application number: PCT/CN2022/089428
Authority: WO
Inventors: 王伟; 袁泽寰; 王长虎
Original assignee: 北京有竹居网络技术有限公司
Priority date: 2021-05-31
Filing date: 2022-04-27
Publication date: 2022-12-08
Also published as: CN113222144B; CN113222144A

Abstract

The present application discloses a training method for an image inpainting model, and an image inpainting method, apparatus, and device. By means of extracting image features from a real first pixel quality image and an artificially synthesized first pixel quality image, and by means of a real first pixel quality image generator and a real second pixel quality image generator, a regenerated real first pixel quality image, a pseudo-real first pixel quality image, and a reconstructed second pixel quality image are obtained. Domain alignment loss, image generation loss and image reconstruction loss are calculated respectively, and training is executed by using the obtained losses. An image inpainting model obtained after training has better generalization capabilities, has relatively accurate image inpainting, and exhibits relatively good model performance. An image feature encoder and the real second pixel quality image generator obtained by training are used to inpaint a first pixel quality image to be inpainted, so that a second pixel quality image having a relatively good inpainting effect can be obtained, and requirements for image use are met.

Description

Image restoration model training method, image restoration method, device and equipment

This application claims the priority of the Chinese patent application submitted to the State Intellectual Property Office of China on May 31, 2021, with the application number 202110604235.1, and the title of the invention is "the training method of the image restoration model and the image restoration method, device and equipment". The entire contents are incorporated by reference in this application.

technical field

The invention belongs to the technical field of mobile communication, and in particular relates to a training method, device and equipment for an image restoration model, and an image restoration method, device and equipment.

Background technique

In the process of image generation and processing, it may be affected by the equipment, resulting in low pixel quality of the image, which cannot meet the needs of image use. In order to improve the pixel quality of an image with lower pixel quality, image super-resolution reconstruction technology can be used to process the image with lower pixel quality, improve the pixel quality of the image with lower pixel quality, and obtain an image with higher pixel quality.

At present, the image super-resolution reconstruction technology can be realized through the image restoration model constructed by the convolutional neural network. The image inpainting model needs to be generated through training image training. The training images used to train the image inpainting model will have a large impact on the performance of the resulting image inpainting model. The training images generally use artificially constructed lower pixel quality images and higher pixel quality images. However, the artificially constructed lower pixel quality images have lower quality, resulting in poor performance of the trained image restoration model, which cannot meet the needs of image pixel quality restoration. Therefore, how to train and generate image inpainting models with better performance and generate inpainted images with higher quality is an urgent problem to be solved.

Contents of the invention

In view of this, the embodiment of the present application provides an image restoration model training method, device and equipment, and an image restoration method, device and equipment, which can train an image restoration model with better performance, and realize The lower pixel quality image is repaired to obtain a more accurate higher pixel quality image.

In order to solve the above problems, the technical solutions provided by the embodiments of the present application are as follows:

In the first aspect, the embodiment of the present application provides a method for training an image restoration model, the method comprising:

Inputting the real first pixel quality image into the image feature encoder to obtain the first image feature;

Inputting the artificially synthesized first pixel quality image into the image feature encoder to obtain the second image feature; the artificially synthesized first pixel quality image is obtained by blurring the real second pixel quality image;

Inputting the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

Inputting the second image feature into the real first pixel quality image generator to obtain a pseudo-true first pixel quality image;

Inputting the second image feature into a real second pixel quality image generator to obtain a reconstructed second pixel quality image;

calculating a domain alignment loss based on the first image feature and the second image feature;

calculating an image generation loss based on the regenerated true first pixel quality image, the pseudo true first pixel quality image, and the true first pixel quality image;

calculating an image reconstruction loss based on the reconstructed second pixel quality image and the true second pixel quality image;

Train the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator based on the domain alignment loss, the image generation loss, and the image reconstruction loss, repeating Execute said inputting the real first pixel quality image into the image feature encoder to obtain the first image feature and subsequent steps until reaching the preset condition;

Wherein, the image definition of the first pixel quality is lower than the image definition of the second pixel quality.

In a second aspect, an embodiment of the present application provides an image restoration method, the method comprising:

Input the first pixel quality image to be repaired into the image feature encoder to obtain the target image feature;

The target image feature is input into a real second pixel quality image generator to obtain a repaired second pixel quality image;

The image feature encoder and the real second pixel quality image generator are obtained through training according to the training method of the image inpainting model described in any one of the above embodiments.

In a third aspect, the embodiment of the present application provides a training device for an image restoration model, the device comprising:

The first execution unit is used to input the real first pixel quality image into the image feature encoder to obtain the first image feature;

The second execution unit is configured to input the artificially synthesized first pixel quality image into the image feature encoder to obtain the second image feature; the artificially synthesized first pixel quality image is obtained by blurring the real second pixel quality image of;

A third execution unit, configured to input the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

A fourth execution unit, configured to input the second image feature into the real first pixel quality image generator to obtain a pseudo-real first pixel quality image;

The fifth execution unit is configured to input the second image feature into the real second pixel quality image generator to obtain a reconstructed second pixel quality image;

A first calculation unit, configured to calculate a domain alignment loss according to the first image feature and the second image feature;

A second calculation unit, configured to calculate an image generation loss based on the regenerated real first pixel quality image, the pseudo real first pixel quality image, and the real first pixel quality image;

A third calculation unit, configured to calculate an image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image;

a training unit for training the image feature encoder, the true first pixel quality image generator, and the true second pixel quality based on the domain alignment loss, the image generation loss, and the image reconstruction loss The image generator repeatedly executes the steps of inputting the real first pixel quality image into the image feature encoder to obtain the first image feature and subsequent steps until the preset condition is reached;

In a fourth aspect, an embodiment of the present application provides an image restoration device, the device comprising:

The eighth execution unit is used to input the first pixel quality image to be repaired into the image feature encoder to obtain the target image feature;

A ninth execution unit, configured to input the features of the target image into a real second pixel quality image generator to obtain a repaired second pixel quality image;

In a fifth aspect, the embodiment of the present application provides an electronic device, including:

one or more processors;

a storage device on which one or more programs are stored,

When the one or more programs are executed by the one or more processors, the one or more processors implement the image repair model training method as described in any of the above embodiments, or any of the above implementations The image restoration method described in the example.

In a sixth aspect, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, wherein, when the program is executed by a processor, the method for training an image restoration model as described in any of the above-mentioned embodiments is implemented, Or the image restoration method described in any of the above-mentioned embodiments.

It can be seen that the embodiment of the present application has the following beneficial effects:

The embodiment of the present application provides a training method, device and equipment for an image inpainting model. By inputting the real first pixel quality image and the synthetic first pixel quality image into the image feature encoder respectively, the corresponding first image feature and the first pixel quality image are obtained. Two image features. Then input the first image feature and the second image feature into the real first pixel quality image generator respectively, and correspondingly obtain the regenerated real first pixel quality image and pseudo real first pixel quality image. Then, the second image feature is input into the real second pixel quality image generator to obtain the reconstructed second pixel quality image. Based on the obtained image features and pixel quality images, the domain alignment loss, image generation loss and image reconstruction loss are respectively calculated, and the image feature encoder, true first pixel quality image generator and true second pixel quality are trained based on the calculated losses image generator. Repeat the above training process until the preset condition is reached. Based on domain alignment loss, image generation loss and image reconstruction loss, it can better reduce the gap between real lower pixel quality images and artificially synthesized lower pixel quality images, and reduce the lower pixels in the generated simulated real scene The gap between the quality image and the lower pixel quality image in the actual real scene reduces the gap between the generated higher pixel quality image in the simulated real scene and the higher pixel quality image in the actual real scene, so that the calculation-based loss The trained image repair model has better generalization ability and repair ability, and has better model performance. The image restoration method, device, and equipment provided in the embodiments of the present application can obtain a good repair effect by using the image feature encoder generated by the above-mentioned image restoration model and the second pixel quality image generator to repair the first pixel quality image to be repaired. The second pixel quality image, which meets the needs of image usage.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the description, and are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the attached picture:

FIG. 1 is a schematic framework diagram of an exemplary application scenario provided by an embodiment of the present application;

FIG. 2 is a flow chart of a training method for an image restoration model provided in an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an image restoration model provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of another image restoration model provided by the embodiment of the present application;

FIG. 5 is a schematic structural diagram of another image restoration model provided by the embodiment of the present application;

FIG. 6 is a schematic structural diagram of a feature discriminator provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an image feature encoder provided in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a real second pixel quality image generator provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an RRDB unit provided in an embodiment of the present application;

FIG. 10 is a flow chart of an image restoration method provided by an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a training device for an image restoration model provided by an embodiment of the present application;

Fig. 12 is a schematic structural diagram of an image restoration device provided by an embodiment of the present application;

FIG. 13 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In order to make the above objects, features and advantages of the present application more obvious and understandable, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

In order to facilitate understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will first be described below.

After researching traditional image restoration methods, the inventor found that the images with lower pixel quality in the training images currently used to train the image restoration model are obtained by degrading the image quality of images with higher pixel quality of. By artificially simulating blur and noise, the process of obtaining a lower pixel quality image after processing a higher pixel quality image is difficult to simulate the process of producing a real lower pixel quality image. This results in a large distribution difference between the synthetic low-pixel-quality images and the real low-pixel-quality images. Using artificially synthesized images with lower pixel quality as training data to train the generated image inpainting model has poor generalization ability, and it is difficult to reconstruct higher-quality pixel quality images of lower pixel quality images to be repaired.

Based on this, the embodiment of the present application provides a training method, device and equipment for an image inpainting model. By inputting the real first pixel quality image and the synthetic first pixel quality image into the image feature encoder respectively, the corresponding first pixel quality image is obtained. image features and second image features. Then input the first image feature and the second image feature into the real first pixel quality image generator respectively, and correspondingly obtain the regenerated real first pixel quality image and pseudo real first pixel quality image. Then, the second image feature is input into the real second pixel quality image generator to obtain the reconstructed second pixel quality image. Based on the obtained image features and generated pixel-quality images, domain alignment loss, image generation loss, and image reconstruction loss are computed respectively, and based on the computed losses image feature encoder, true first pixel quality image generator, and true second pixel Quality image generator. Repeat the above training process until the preset condition is reached. Based on domain alignment loss, image generation loss and image reconstruction loss, it can better reduce the gap between real lower pixel quality images and artificially synthesized lower pixel quality images, and reduce the lower pixels in the generated simulated real scene The distribution gap between the quality image and the lower pixel quality image in the actual real scene reduces the gap between the generated higher pixel quality image in the simulated real scene and the higher pixel quality image in the actual real scene, so that the calculation-based The image inpainting model obtained by loss training has better generalization ability and inpainting ability, and has better model performance. The image inpainting method, device and equipment provided in the embodiments of the present application, by using the image feature encoder generated by the above-mentioned image inpainting model and the second pixel quality image generator to inpaint the first pixel quality image to be inpainted, can obtain a better inpainting effect. Good second-pixel quality images, suitable for image usage.

In order to facilitate understanding of the image restoration method provided by the embodiment of the present application, the following description will be made in conjunction with the scene example shown in FIG. 1 . Referring to FIG. 1 , the figure is a schematic framework diagram of an exemplary application scenario provided by an embodiment of the present application.

In practical applications, there is a first pixel quality image to be repaired with low image pixels that needs to be repaired, and the first pixel quality image is input into the image feature encoder, and the image feature of the first pixel quality image can be performed Extract the features of the target image. Then input the obtained target image features into the real second pixel quality image generator to obtain the repaired second pixel quality image. The second pixel quality image is an image with higher image pixels. Wherein, both the image feature encoder and the real second pixel quality image generator are pre-generated through the training method of the image restoration model.

Those skilled in the art can understand that the schematic frame diagram shown in FIG. 1 is only an example in which the embodiments of the present application can be implemented. The scope of applicability of the embodiments of the present application is not limited by any aspect of this framework.

In order to facilitate understanding of the present application, a method for training an image inpainting model provided by an embodiment of the present application will be described below with reference to the accompanying drawings.

Referring to Figure 2, which is a flow chart of a training method for an image repair model provided in an embodiment of the present application, as shown in Figure 2, the method may include S201-S209:

S201: Input a real first pixel quality image into an image feature encoder to obtain a first image feature.

Refer to FIG. 3 , which is a schematic structural diagram of an image inpainting model provided by an embodiment of the present application. The real first pixel quality image is an image with lower pixel quality generated in a real scene. The real image with the first pixel quality may be an image with a certain blur and/or noise and a relatively low pixel quality that is affected by equipment or data processing during image generation or image processing. Realistic first-pixel-quality images can suffer from multiple blurring and noise degradation problems.

An image feature encoder is an encoder used to extract features from an image. The real first pixel quality image is input into the image feature encoder, and feature extraction can be performed on the real first pixel quality image to obtain the first image feature corresponding to the real first pixel quality image.

S202: Input the artificially synthesized first pixel quality image into the image feature encoder to obtain the second image feature; the artificially synthesized first pixel quality image is obtained by blurring the real second pixel quality image.

Wherein, the image definition of the first pixel quality is lower than the image definition of the second pixel quality. Then the artificially synthesized first pixel quality image is an artificially constructed image with lower pixel quality. The artificially synthesized first pixel quality image may be obtained by blurring the real second pixel quality image. Wherein, the real second pixel quality image is an image of higher pixel quality generated in a real scene. By blurring the real second pixel quality image, the pixel quality of the real second pixel quality image can be degraded, and a lower pixel quality artificially synthesized first pixel quality image can be obtained. Blurring is an image processing method that reduces pixel quality. Specifically, blurring can add Gaussian noise and Gaussian blur to an image.

It should be noted that the embodiment of the present application does not limit whether the real first pixel quality image and the real second pixel quality image are images of the same content. The real image with the first pixel quality and the real image with the second pixel quality may have the same content, but only images with different pixel qualities, that is, images with different pixel qualities in pairs. The real image of the first pixel quality and the real image of the second pixel quality may also be images of different content and pixel quality, that is, images of different pixel qualities that are not paired.

The artificially synthesized first pixel quality image is input into the image feature encoder, and feature extraction is performed on the artificially synthesized first pixel quality image to obtain a second image feature corresponding to the artificially synthesized first pixel quality image.

S203: Input the first image feature into the real first pixel quality image generator to obtain a regenerated real first pixel quality image.

The real first pixel quality image generator is used to generate a corresponding lower pixel quality image in a real scene based on the input image features. The first image feature is input into the real first pixel quality image generator, and a regenerated real first pixel quality image can be obtained.

The regenerated true first pixel quality image is also a lower pixel quality image. The regenerated real first pixel quality image corresponds to the real first pixel quality image, simulating an image of lower pixel quality in a real scene.

S204: Input the second image feature into the real first pixel quality image generator to obtain a pseudo real first pixel quality image.

Inputting the second image feature into the real first pixel quality image generator can generate a pseudo real first pixel quality image. The pseudo-true first pixel quality image is an image of lower pixel quality. The pseudo-real first pixel quality image corresponds to the artificially synthesized first pixel quality image, simulating an image of lower pixel quality in a real scene.

S205: Input the second image feature into a real second pixel quality image generator to obtain a reconstructed second pixel quality image.

The real second pixel quality image generator is used to generate a corresponding higher pixel quality image in a real scene based on the input image features. The second image feature is input into the real second pixel quality image generator to obtain a reconstructed second pixel quality image.

The second pixel quality image is reconstructed as a higher pixel quality image. The reconstruction of the second pixel quality image corresponds to the artificial synthesis of the first pixel quality image, simulating an image of higher pixel quality in a real scene.

S206: Calculate a domain alignment loss according to the first image feature and the second image feature.

The first image feature is an image feature corresponding to the real first pixel quality image, and the second image feature is an image feature corresponding to the artificially synthesized first pixel quality image. Based on the first image feature and the second image feature, the real first pixel quality image and the artificially synthesized first pixel quality image may be compared.

From the obtained first image features as well as the second image features, a domain alignment loss can be calculated. The domain alignment loss can reflect the gap between real first pixel quality images and artificial first pixel quality images from the perspective of features.

In a possible implementation manner, the embodiment of the present application provides a specific implementation manner of calculating a domain alignment loss according to the first image feature and the second image feature, and please refer to the following for details.

S207: Calculate an image generation loss according to the regenerated real first pixel quality image, pseudo real first pixel quality image, and real first pixel quality image.

The regenerated real first pixel quality image is generated based on the first image feature by a real first pixel quality image generator. The pseudo real first pixel quality image is generated based on the second image feature by the real first pixel quality image generator. Based on the regenerated real first pixel quality, pseudo real first pixel quality image and real first pixel quality image, the loss of the real first pixel quality image generator in generating lower pixel quality images in simulated real scenes can be calculated , which is the image generation loss. The image generation loss measures the difference between the generated lower pixel quality image and the real lower pixel quality image.

In a possible implementation manner, this embodiment of the present application provides a specific implementation manner of calculating the image generation loss according to the regenerated real first pixel quality image, pseudo real first pixel quality image, and real first pixel quality image, See below for details.

S208: Calculate an image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image.

The reconstructed second pixel quality image is generated based on the second image features by the real second pixel quality image generator. The higher pixel quality image in the real scene corresponding to the reconstructed second pixel quality image is a real second pixel quality image. Based on the reconstructed second pixel quality image and the real second pixel quality image, the loss of the higher pixel quality image generated by the real second pixel quality image generator in the simulated real scene, that is, the image reconstruction loss, can be calculated. The image reconstruction loss measures the difference between the generated higher pixel quality image and the real higher pixel quality image.

In a possible implementation manner, this embodiment of the present application provides a specific implementation manner of calculating an image reconstruction loss according to the reconstructed second pixel-quality image and the real second pixel-quality image. For details, please refer to the following.

S209: According to the domain alignment loss, image generation loss and image reconstruction loss, train the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator, and repeatedly execute the real first pixel quality image input image The feature encoder obtains the features of the first image and performs subsequent steps until a preset condition is reached.

Based on the computed domain alignment loss, image generation loss, and image reconstruction loss, the image feature encoder, true first pixel quality image generator, and true second pixel quality image generator can be trained. The image feature encoder, the real-first pixel-quality image generator, and the real-second pixel-quality image generator are jointly tuned by exploiting the computed loss. And repeatedly execute the above S201-S209 to train the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator. Until the preset condition is satisfied, the training of the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator is stopped, so as to realize the training of the image restoration model. Specifically, the preset condition may be the number of repetitions of training, for example, the preset condition may be 500 repetitions of training.

In a possible implementation, the embodiment of the present application also provides a training image feature encoder, real first pixel quality image generator and real second pixel quality according to domain alignment loss, image generation loss and image reconstruction loss. For the specific implementation of the image generator, please refer to the following for details.

Based on the relevant content of S201-S209 above, it can be seen that by using the image feature encoder to extract image features, the corresponding lower pixel quality image is generated by the real first pixel quality image generator, and the corresponding lower pixel quality image is generated by the real second pixel quality image generator. Higher pixel quality images. Using the obtained image features, the generated lower pixel quality image, the generated higher pixel quality image, the corresponding lower pixel quality image in the real scene, and the corresponding higher pixel quality image in the real scene, the corresponding The image feature loss, image generation loss and image reconstruction loss. Based on the computed loss, an image feature encoder, a real first pixel quality image generator, and a real second pixel quality image generator are trained to reduce the real lower pixel quality image and the artificial lower pixel quality image Reduce the gap between the generated low pixel quality image in the simulated real scene and the low pixel quality image in the actual real scene, reduce the generated high pixel quality image in the simulated real scene and the actual real scene Lower the gap between images with higher pixel quality, obtain an image feature encoder with better generalization ability and better repair effect, and a real second pixel quality image generator, which can achieve better repair effect based on the trained image repair model Good higher pixel quality images.

In a possible implementation manner, when the performance of the real first pixel quality image generator is poor, the generated pseudo real first pixel quality image may also have differences in image content from the artificially synthesized first pixel quality image.

In view of the above problems, the embodiment of the present application provides a training method of an image repair model, in addition to the above S201-S209, the method also includes the following steps:

A1: Input the pseudo-real first pixel quality image into the image feature encoder to obtain the third image feature.

Refer to FIG. 4 , which is a schematic structural diagram of another image inpainting model provided by the embodiment of the present application.

Image features are extracted from the pseudo real first pixel quality image, and the pseudo real first pixel quality image is input into an image feature encoder to obtain a third image feature corresponding to the pseudo real first pixel quality image.

A2: Calculate the content consistency loss according to the second image feature and the third image feature.

The second image feature is an image feature of the artificially synthesized first pixel quality image. According to the second image feature and the third image feature, the content consistency loss can be calculated. The content consistency loss is used to measure the image gap between artificially synthesized first pixel quality images and fake real first pixel quality images. Based on the content consistency loss, the real first pixel quality image generator can be adjusted so that the image content of the generated lower pixel quality image in the simulated real scene is unchanged from that of the corresponding artificially synthesized first pixel quality image, Only the degradation mode of the image pixels is changed, so as to improve the image quality of the generated image with lower pixel quality in the simulated real scene, and ensure that the image content does not change.

Correspondingly, the embodiment of the present application provides a specific implementation of training the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator according to the domain alignment loss, image generation loss and image reconstruction loss ,include:

An image feature encoder, a true-first pixel-quality image generator, and a true-second pixel-quality image generator are trained on domain alignment loss, image generation loss, image reconstruction loss, and content consistency loss.

Based on the computed domain alignment loss, image generation loss, image reconstruction loss, and content consistency loss, the image feature encoder, true-first pixel-quality image generator, and true-second pixel-quality image generator are trained.

In a possible implementation manner, this embodiment of the present application provides a specific implementation manner of calculating the content consistency loss according to the second image feature and the third image feature, and please refer to the following for details.

In the embodiment of the present application, by calculating the content consistency loss and using the content consistency loss to train the image feature encoder, the real first pixel quality image generator and the real second pixel quality image generator, the generated The model performance of the image inpainting model ensures that the inpainted image is consistent with the image content of the image to be inpainted.

Further, the real second pixel quality image generator can also be used to obtain the corresponding higher pixel quality image in the simulated real scene by using the obtained third image features, and calculate the corresponding image reconstruction loss, and use the image reconstruction loss to train A more performant image inpainting model is obtained.

Correspondingly, a method for training an image inpainting model provided in an embodiment of the present application may further include the following steps:

The third image feature is input to the real second pixel quality image generator to obtain a pseudo real reconstructed second pixel quality image.

Refer to FIG. 5 , which is a schematic structural diagram of another image inpainting model provided by the embodiment of the present application.

The third image feature is an image feature of the pseudo-real first pixel quality image. By inputting the third image feature into the real second pixel quality image generator, a higher pixel quality image corresponding to the pseudo-real first pixel quality image in a simulated real scene can be obtained, that is, a pseudo-real reconstructed second pixel quality image.

Correspondingly, the embodiment of the present application provides a specific implementation manner of calculating the image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image, including the following two steps:

B1: Calculate the first image reconstruction loss based on the reconstructed second pixel quality image and the real second pixel quality image.

The reconstructed second pixel quality image is a generated higher pixel quality image in a simulated real scene corresponding to the real second pixel quality image. The first image reconstruction loss can be calculated according to the reconstructed second pixel quality image and the real second pixel quality image, and the first image reconstruction loss can be used to measure the gap between the reconstructed second pixel quality image and the real second pixel quality image.

In a possible implementation manner, this embodiment of the present application provides a specific implementation manner of calculating the reconstruction loss of the first image according to the reconstructed second pixel quality image and the real second pixel quality image. For details, please refer to the following.

B2: Calculating a second image reconstruction loss according to the pseudo real reconstructed second pixel quality image and the real second pixel quality image; the first image reconstruction loss and the second image reconstruction loss constitute an image reconstruction loss.

The pseudo-true reconstruction of the second pixel quality image also corresponds to the real second pixel quality image, which is a higher pixel quality image generated from a lower pixel quality image to simulate a real scene. The second image reconstruction loss can be calculated according to the reconstructed pseudo-real second pixel quality image, and the second image reconstruction loss can be used to measure the gap between the pseudo-real reconstructed second pixel quality image and the real second pixel quality image.

An image reconstruction loss is composed by using the obtained first image reconstruction loss and the second image reconstruction loss.

Based on the above content, it can be seen that through the real second pixel quality image, a pseudo-true reconstructed second pixel quality image is generated based on the third image feature, and based on the reconstructed second pixel quality image and the pseudo-real reconstructed second pixel quality image are respectively compared with the real second pixel quality image. Pixel Quality Image Computing image reconstruction loss. The obtained image reconstruction loss can more comprehensively measure the gap between the higher pixel quality image generated in the simulated real scene and the higher pixel quality image in the actual real scene, and realize the generation result of the real second pixel quality image generator constraints, so that an image inpainting model with better model performance can be obtained based on image reconstruction loss training.

In a possible implementation, the embodiment of the present application provides an image generation loss calculation method based on the regenerated real first pixel quality image, pseudo real first pixel quality image and real first pixel quality image, specifically including the following two steps:

C1: Calculate the first image generation loss from the regenerated real first pixel quality image and the real first pixel quality image.

The regenerated real first pixel quality image is a generated lower pixel quality image in a simulated real scene corresponding to the real first pixel quality image. According to the regenerated true first pixel quality image and the true first pixel quality image, the first image generation loss can be calculated, and the first image generation loss can be used to measure the regenerated true first pixel quality image and the true first pixel quality image The gap between quality images.

In a possible implementation manner, this embodiment of the present application provides a specific implementation manner of calculating the first image generation loss according to the regenerated real first pixel quality image and the real first pixel quality image, for details, please refer to the following.

C2: Calculate the second image generation loss according to the fake real first pixel quality image and the real first pixel quality image; the first image generation loss and the second image generation loss constitute the image generation loss.

The pseudo-real first pixel quality image is based on artificially synthesizing the first pixel quality image to generate a lower pixel quality image in a simulated real scene. The second image generation loss can be calculated according to the fake real first pixel quality image and the real first pixel quality image. The second image generation loss can be used to measure the gap between the fake real first pixel quality image and the real first pixel quality image.

Based on the obtained first image generation loss and the second image generation loss, an image generation loss can be composed.

In a possible implementation, the embodiment of the present application provides a method of calculating the second image generation loss according to the pseudo-true first pixel quality image and the real first pixel quality image; the first image generation loss and the second image generation loss are composed of For the specific implementation of the image generation loss, see below.

In this embodiment of the application, the first image generation loss is calculated by using the regenerated real first pixel quality image and the real first pixel quality image, and the first image generation loss is calculated by using the pseudo real first pixel quality image and the real first pixel quality image Two image generation loss. A more accurate image generation loss can be obtained based on the first image generation loss and the second image generation loss to implement constraints on the generation results of the real first pixel quality image generator. Based on the image generation loss, the gap between the generated lower pixel quality image and the lower pixel quality image in the real scene can be reduced, and the image inpainting model with better model performance can be obtained by training with the obtained image generation loss.

In a possible implementation, the embodiment of the present application provides a specific implementation of calculating the domain alignment loss according to the first image feature and the second image feature, including the following three steps:

D1: Input the first image feature into the feature discriminator to obtain the first probability value.

First of all, it needs to be explained that the real first pixel quality image corresponding to the first image feature and the synthetic first pixel quality image corresponding to the second image feature are obtained by processing different images. The image features can be input into the feature discriminator to obtain the corresponding probability value and calculate the domain alignment loss.

Referring to FIG. 6 , this figure is a schematic structural diagram of a feature discriminator provided by an embodiment of the present application. The feature discriminator can be composed of 6 modules, and each module consists of a 3*3 convolutional layer and an activation function layer. Wherein, the activation function layer may specifically adopt the LeakyRelu activation function.

Wherein, corresponding to the first image feature, the feature discriminator is configured to output a probability value that the generated first image feature is an image feature corresponding to an image in a real scene. By inputting the first image feature into the feature discriminator, a first probability value corresponding to the first image feature output by the feature discriminator can be obtained.

Specifically, the first image feature Fx can be expressed as Fx=E _C (I _X ), where I _X represents the real first pixel quality image, and E _C (I _X ) represents the real first pixel extracted by the image feature encoder The first image feature of a quality image, where E _C can represent an image feature encoder. The first probability value output by the feature discriminator can be expressed as logD _C (Fx). where _DC denotes the discriminator for features.

D2: Input the second image feature into the feature discriminator to obtain the second probability value.

By inputting the second image feature into the feature discriminator, a second probability value corresponding to the second image feature output by the feature discriminator can be obtained.

It should be noted that the second image feature is the image feature of the artificially synthesized image of the first pixel quality, and the probability value that the second image feature is an image feature of an image in a simulated real scene needs to be obtained through a feature discriminator.

Specifically, the second image feature F _Z can be expressed as F _Z =E _C (I _Z ), wherein, I _Z represents the artificial synthesis of the first pixel quality image, and E _C (I _Z ) represents the artificial Synthesizing a second image feature of the first pixel-quality image, where E _C may represent an image feature encoder. The second probability value output by the feature discriminator can be expressed as log[1-D _C (F _Z )].

D3: Calculate the domain alignment loss according to the first probability value and the second probability value.

Based on the first probability value and the second probability value output by the feature discriminator, the domain alignment loss can be calculated.

In one possible implementation, the domain alignment loss

It can be calculated by the following formula:

Among them, E _X [logD _C (Fx)] represents the expected value corresponding to the calculated first probability value, and E _Z {log[1-D _C (F _Z )]} represents the expected value corresponding to the calculated second probability value corresponding expected value. The domain alignment loss is obtained by computing the sum of the expectation for the first probability value and the expectation for the second probability value.

In the embodiment of the present application, the probability value corresponding to the first image feature and the second image feature is calculated by the feature discriminator, and then the calculated probability value is used to calculate the domain alignment loss, and the obtained domain alignment loss is relatively accurate. Through the feature discriminator, the corresponding loss value can be calculated based on the images obtained by different image processing, that is, unpaired images, so that the real second pixel quality image used when generating the artificially synthesized first pixel quality image can be compared with the real first pixel quality image. One-pixel quality images are not images of the same content, reducing the constraints on generating training images.

Further, the embodiment of the present application provides a specific implementation manner of calculating the content consistency loss according to the second image feature and the third image feature, including:

Calculate the 1-norm between the second image feature and the third image feature;

Based on the 1-norm between the second image feature and the third image feature, a content consistency loss is calculated.

Specifically, the 1-norm between the second image feature and the third image feature is calculated. Wherein, the second image feature may be represented as F _Z , and the third image feature may be represented as F _Z '. The 1-norm between the corresponding second image feature and the third image feature can be expressed as ||F _Z′ −F _Z || ₁ .

Based on the calculated 1-norm between the second image feature and the third image feature, a content consistency loss is calculated.

In a possible implementation, the content consistency loss L _pix can be calculated by the following formula:

L _pix (F _Z ′, F _Z )＝E _Z [||F _Z ′－F _Z || ₁ ] (2)

Among them, E _Z [||F _Z' －F _Z || ₁ ] represents the expected value corresponding to the calculated ||F _Z ′－F _Z || ₁ , specifically for the obtained ||F _Z ′－F _Z || ₁ computes the corresponding mean.

Based on the above content, it can be seen that by calculating the 1-norm between the second image feature and the third image feature, the constraints are implemented pixel by pixel, and the calculated content consistency loss can better represent the quality of the artificially synthesized first pixel The gap between the image and the pseudo-real first-pixel quality image can then be trained to obtain a better-performing image inpainting model.

Further, the embodiment of the present application provides a specific implementation method of calculating the reconstruction loss of the first image according to the reconstructed second pixel quality image and the real second pixel quality image, including:

calculating the 1-norm between the reconstructed second pixel quality image and the real second pixel quality image;

The first image reconstruction loss is calculated from the 1-norm between the reconstructed second pixel quality image and the real second pixel quality image.

Specifically, the reconstructed second pixel quality image may be expressed by I _Z→Y , and the real second pixel quality image may be expressed by I _Y. The 1-norm between the reconstructed second pixel quality image and the real second pixel quality image can be expressed as ||I _Z→Y −I _Y || ₁ .

A 1-norm between the reconstructed second pixel quality image and the real second pixel quality image is calculated.

The 1-norm between the reconstructed second pixel quality image and the real second pixel quality image can be expressed by the following formula:

L _pix (I _Z→Y ，I _Y )＝E _Y [||I _Z→Y －I _Y || ₁ ] (3)

Wherein, E _Y [||I _Z→Y －I _Y || ₁ ] represents the calculated expected value corresponding to ||I _Z→Y －I _Y || ₁ .

In the embodiment of the present application, by calculating the 1-norm between the reconstructed second pixel quality image and the real second pixel quality image, constraints can be performed pixel by pixel, and a more accurate first image reconstruction loss can be calculated, and then Image inpainting models with better performance can be obtained through training.

Further, in a possible implementation, the embodiment of the present application provides a specific implementation of calculating the reconstruction loss of the second image according to the reconstructed second pixel quality image of the pseudo-truth and the real second pixel quality image, including:

Computing the 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image;

A second image reconstruction loss is computed from the 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image.

The pseudo real reconstructed second pixel quality image is obtained by processing the pseudo real first pixel quality image, and is obtained by processing different images from the real second pixel quality image. The corresponding probability value can be obtained through the feature discriminator, and the domain alignment loss can be calculated.

Specifically, the pseudo-true reconstruction of the second pixel quality image may be represented by I _Z→X→Y , and the real second pixel quality image may be represented by I _Y. The 1-norm between the reconstructed second pixel quality image and the real second pixel quality image can be expressed as ||I _Z→X→Y −I _Y || ₁ .

A 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image is calculated.

The 1-norm between the pseudo real second pixel quality image and the real second pixel quality image can be represented by the following formula:

L _pix (I _Z→X→Y ，I _Y )＝E _Y [||I _Z→X→Y －I _Y || ₁ ] (4)

Among them, E _Y [||I _Z→X→Y －I _Y || ₁ ] represents the calculated expected value corresponding to ||I _Z→X→Y －I _Y || ₁ .

In the embodiment of the present application, by calculating the 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image, constraints can be performed pixel by pixel to further improve the accuracy of the second image reconstruction loss, In order to train an image restoration model with better performance.

In a possible implementation manner, this embodiment of the present application provides a calculation of the first image generation loss according to the regenerated real first pixel quality image and the real first pixel quality image, including:

Computing the 1-norm between the regenerated true first pixel quality image and the true first pixel quality image;

Compute the first image generation loss from the regenerated real first pixel quality image and the 1-norm between the real first pixel quality image.

Specifically, the regenerated real first pixel quality image may be represented by I _X→X , and the real first pixel quality image may be represented by I _X. The 1-norm between the regenerated true first pixel quality image and the true first pixel quality image can be expressed as ||I _X→X − I _X || ₁ .

The 1-norm between the regenerated true first pixel quality image and the true first pixel quality image can be expressed as:

L _pix (I _X→X ，I _X )＝E _X [||I _X→X －I _X || ₁ ] (5)

Wherein, E _X [||I _X→X - I _X || ₁ ] represents the calculated expected value corresponding to ||I _X→X - I _X || ₁ .

In the embodiment of the present application, by calculating the 1-norm between the regenerated real first pixel quality image and the real first pixel quality image, constraints can be performed pixel by pixel, further improving the accuracy of the first image generation loss , in order to train an image inpainting model with better performance.

In a possible implementation manner, this embodiment of the present application provides a method of calculating the second image generation loss according to the fake real first pixel quality image and the real first pixel quality image, specifically including:

Inputting the pseudo-true first pixel quality image into the feature discriminator to obtain a third probability value;

Input the real first pixel quality image into the feature discriminator to obtain the fourth probability value;

Based on the third probability value and the fourth probability value, a second image generation loss is calculated.

The pseudo real first pixel quality image is generated by artificially synthesizing the first pixel quality image, and belongs to a different image from the real first pixel quality image. The calculation of the generation loss of the second image can be realized by inputting the pseudo real first pixel quality image and the real first pixel quality image into the feature discriminator respectively to obtain corresponding probability values.

The pseudo-real first pixel quality image is input into the feature discriminator, and the third probability value output by the feature discriminator is obtained. The third probability value is used to represent the probability value that the pseudo real first pixel quality image is an image of lower pixel quality generated in a simulated real scene.

Specifically, the pseudo-real first pixel quality image can be expressed as I _Z→X . Correspondingly, the third probability value may be expressed as log[1-D _X (I _Z→X )]. Among them, D _X represents the discriminator for the image.

The real first pixel quality image is input into the feature discriminator, and the fourth probability value output by the feature discriminator is obtained. The fourth probability value is used to represent the probability value that the real first pixel quality image is a lower pixel quality image generated in a real scene.

Specifically, the true first pixel quality image can be expressed as I _X . Correspondingly, the fourth probability value may be expressed as logD _X (I _X ).

Based on the third probability value and the fourth probability value output by the feature discriminator, the second image generation loss can be calculated.

In a possible implementation, the second image loss L _adv can be calculated by the following formula:

L _adv (I _Z→X ，I _X )＝E _X [logD _X (I _X )]+E _Z {log[1-D _X (I _Z→X )]} (6)

Among them, E _X [logD _X (I _X )] represents the expected value corresponding to the calculated fourth probability value, E _Z {log[1-D _X (I _Z→X )]} represents the calculated third probability value The expected value corresponding to the probability value. The second image loss is obtained by computing the sum of the expectation for the third probability value and the expectation for the fourth probability value.

In the embodiment of the present application, the probability values corresponding to the pseudo-true first pixel quality image and the real first pixel quality image are calculated by the feature discriminator, and then the calculated probability value is used to calculate the second image loss, and the obtained second Image loss is more accurate. The images obtained by different image processing, that is, unpaired images, can be constrained by the feature discriminator.

In a possible implementation, in the image inpainting model provided in the embodiment of the present application, the image feature encoder includes at least one residual module;

The true first pixel quality image generator includes at least one residual module;

The real second pixel quality image generator includes a first convolutional layer, a basic unit, a second convolutional layer, an upsampling layer, a third convolutional layer, an activation function layer, and a fourth convolutional layer, and the basic unit includes at least one RRDB (Residual-in-Residual Dense Block) unit.

Referring to FIG. 7 , the figure is a schematic structural diagram of an image feature encoder provided by an embodiment of the present application. When training an image inpainting model, an image feature encoder is used to extract image features and can consist of multiple residual modules. Among them, each residual module can be composed of a 3*3 convolutional layer, an activation function layer, and a 3*3 convolutional layer.

The real first pixel quality image generator is used for auxiliary training and can be composed of multiple residual modules. The structure of the real first pixel quality image generator may be consistent with the structure of the image feature encoder, which will not be repeated here.

Refer to FIG. 8 , which is a schematic structural diagram of a real second pixel quality image generator provided by an embodiment of the present application. The real second pixel quality image generator performs image restoration after the training is completed, and requires a more complex structure to achieve more accurate image restoration. The true second pixel quality image generator consists of the first convolutional layer, the basic unit, the second convolutional layer, the upsampling layer, the third convolutional layer, the activation function layer, and the fourth convolutional layer. Wherein, the first convolutional layer and the fourth convolutional layer may be 1*1 convolutional layers, and the second convolutional layer and the third convolutional layer may be 3*3 convolutional layers. The activation function used in the activation function layer may specifically be a LeakyRelu activation function.

Specifically, the basic unit may be multiple RRDB units. Referring to FIG. 9 , the figure is a schematic structural diagram of an RRDB unit provided by an embodiment of the present application. The RRDB unit includes five modules, the first module is composed of a 3*3 convolutional layer and an activation function layer, and the second to fifth modules are composed of a 1*1 convolutional layer, It consists of a 3*3 convolutional layer and an activation function layer. The activation function used by the activation function layer in the RRDB unit may specifically be a LeakyRelu activation function.

In the embodiment of the present application, by adopting the relatively simple model structure of the image feature encoder and the real first pixel quality image generator, and the relatively complex model structure of the real second pixel quality image generator, the model structure can be ensured The model performance of the generated image inpainting model is guaranteed under the premise of being more streamlined.

Based on the method for training an image inpainting model provided in the foregoing embodiments, an embodiment of the present application further provides an image inpainting method.

Referring to Figure 10, which is a flow chart of an image restoration method provided in the embodiment of the present application, as shown in Figure 10, the method may include S1001-S1002:

S1001: Input the first pixel quality image to be repaired into an image feature encoder to obtain target image features.

The image with the first pixel quality to be repaired is an image with a lower pixel quality, and needs to be repaired to obtain a corresponding image with a higher pixel quality.

The first pixel quality image to be repaired is input into the image feature encoder, and the image features of the first pixel quality image to be repaired are extracted to obtain corresponding target image features.

Wherein, the image feature encoder is trained and generated according to the above image inpainting model training method. The image feature encoder trained by the above image inpainting model training method can better perform feature extraction on the first pixel quality image to be inpainted, so as to achieve more accurate image inpainting.

S1002: Input the features of the target image into a real second pixel quality image generator to obtain a repaired second pixel quality image.

The image feature encoder and the real second pixel quality image generator are obtained by training according to the training method of the image inpainting model in any of the above embodiments.

The resulting target image features are input into the real second pixel quality image generator. The real second pixel quality image generator is used to generate a corresponding higher pixel quality image based on the input target image features, that is, the repaired second pixel quality image.

The real second pixel quality image generator is trained and generated by the above-mentioned image inpainting model training method. The generator of the second pixel quality image trained by the above image restoration model training method can generate a repaired second pixel quality image corresponding to the first pixel quality image to be repaired based on the characteristics of the target image, and has a better repair effect.

Based on the relevant content of S1001-S1002 above, it can be seen that through the trained image feature encoder and the real second pixel quality image generator, the image repair of the first pixel quality image to be repaired can be better performed, and a repair with better repair effect can be obtained. After the second pixel quality image, to meet the needs of image usage.

Based on the method for training an image restoration model provided by the method embodiment above, the embodiment of the present application also provides a training device for an image restoration model. The training device for an image restoration model will be described below with reference to the accompanying drawings.

Refer to FIG. 11 , which is a schematic structural diagram of a training device for an image inpainting model provided by an embodiment of the present application. As shown in Figure 11, the training device of this image restoration model includes:

The first execution unit 1101 is configured to input the real first pixel quality image into the image feature encoder to obtain the first image feature;

The second execution unit 1102 is configured to input the artificially synthesized first pixel quality image into the image feature encoder to obtain the second image feature; the artificially synthesized first pixel quality image is blurred by the real second pixel quality image owned;

The third execution unit 1103 is configured to input the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

The fourth execution unit 1104 is configured to input the second image feature into the real first pixel quality image generator to obtain a pseudo-real first pixel quality image;

The fifth execution unit 1105 is configured to input the second image feature into a real second pixel quality image generator to obtain a reconstructed second pixel quality image;

A first calculation unit 1106, configured to calculate a domain alignment loss according to the first image feature and the second image feature;

The second calculation unit 1107 is configured to calculate an image generation loss according to the regenerated real first pixel quality image, the pseudo real first pixel quality image and the real first pixel quality image;

A third calculation unit 1108, configured to calculate an image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image;

A training unit 1109, configured to train the image feature encoder, the real first pixel quality image generator and the real second pixel according to the domain alignment loss, the image generation loss and the image reconstruction loss The quality image generator repeatedly executes the steps of inputting the real first pixel quality image into the image feature encoder to obtain the first image feature and subsequent steps until the preset condition is reached;

In a possible implementation manner, the device further includes:

A sixth execution unit, configured to input the pseudo real first pixel quality image into the image feature encoder to obtain a third image feature;

A fourth calculation unit, configured to calculate a content consistency loss according to the second image feature and the third image feature;

The training unit 1109 is specifically configured to train the image feature encoder, the real first A pixel quality image generator and said true second pixel quality image generator.

In a possible implementation manner, the device further includes:

A seventh execution unit, configured to input the third image feature into the real second pixel quality image generator to obtain a pseudo-real reconstructed second pixel quality image;

The third calculation unit 1108 includes:

A first calculation subunit, configured to calculate a first image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image;

A second calculation subunit, configured to calculate a second image reconstruction loss based on the pseudo real reconstructed second pixel quality image and the real second pixel quality image; the first image reconstruction loss and the second image reconstruction loss Compose the image reconstruction loss.

In a possible implementation manner, the second calculation unit 1107 includes:

A third calculation subunit, configured to calculate a first image generation loss according to the regenerated real first pixel quality image and the real first pixel quality image;

A fourth calculation subunit, configured to calculate a second image generation loss based on the pseudo-true first pixel quality image and the real first pixel quality image; the first image generation loss and the second image generation loss consist of Image generation loss.

In a possible implementation manner, the first calculation unit 1106 is specifically configured to input the first image feature into a feature discriminator to obtain a first probability value;

inputting the second image feature into the feature discriminator to obtain a second probability value;

A domain alignment loss is calculated according to the first probability value and the second probability value.

In a possible implementation manner, the fourth calculation unit is specifically configured to calculate a 1-norm between the second image feature and the third image feature;

A content consistency loss is calculated according to a 1-norm between the second image feature and the third image feature.

In a possible implementation manner, the first calculation subunit is specifically configured to calculate a 1-norm between the reconstructed second pixel quality image and the real second pixel quality image;

A first image reconstruction loss is calculated based on a 1-norm between the reconstructed second pixel quality image and the real second pixel quality image.

In a possible implementation manner, the second calculation subunit is specifically configured to calculate a 1-norm between the pseudo-real reconstructed second pixel quality image and the real second pixel quality image;

A second image reconstruction loss is calculated based on a 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image.

In a possible implementation manner, the third calculation subunit is specifically configured to calculate a 1-norm between the regenerated real first pixel quality image and the real first pixel quality image;

A first image generation loss is calculated based on the regenerated real first pixel quality image and a 1-norm between the real first pixel quality image.

In a possible implementation manner, the fourth calculation subunit is specifically configured to input the pseudo-real first pixel quality image into a feature discriminator to obtain a third probability value;

inputting the real first pixel quality image into the feature discriminator to obtain a fourth probability value;

A second image generation loss is calculated based on the third probability value and the fourth probability value.

In a possible implementation manner, the image feature encoder includes at least one residual module;

said true first pixel quality image generator comprising at least one residual module;

The real second pixel quality image generator includes a first convolutional layer, a basic unit, a second convolutional layer, an upsampling layer, a third convolutional layer, an activation function layer, and a fourth convolutional layer, and the basic unit Include at least one RRDB unit.

Based on the image restoration method provided by the foregoing method embodiments, the embodiment of the present application further provides an image restoration device, which will be described below with reference to the accompanying drawings.

Refer to FIG. 12 , which is a schematic structural diagram of an image restoration device provided by an embodiment of the present application. As shown in Figure 12, the image restoration device includes:

The eighth execution unit 1201 is used to input the first pixel quality image to be repaired into the image feature encoder to obtain the target image feature;

A ninth execution unit 1202, configured to input the features of the target image into a real second pixel quality image generator to obtain a repaired second pixel quality image;

Based on the image restoration model training method and the image restoration method provided by the above method embodiments, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored , when the one or more programs are executed by the one or more processors, so that the one or more processors implement the image restoration model training method described in any of the above embodiments, or any of the above implementations The image inpainting method described in the example

Referring now to FIG. 13 , it shows a schematic structural diagram of an electronic device 1300 suitable for implementing the embodiment of the present application. The terminal equipment in the embodiment of the present application may include but not limited to mobile phones, notebook computers, digital broadcast receivers, PDA (Personal Digital Assistant, personal digital assistant), PAD (portable android device, tablet computer), PMP (Portable Media Player, portable multimedia player), mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs (television, television sets), desktop computers, and the like. The electronic device shown in FIG. 13 is only an example, and should not limit the functions and scope of use of this embodiment of the present application.

As shown in FIG. 13, an electronic device 1300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1301, which may be randomly accessed according to a program stored in a read-only memory (ROM) 1302 or loaded from a storage device 1306. Various appropriate actions and processes are executed by programs in the memory (RAM) 1303 . In the RAM 1303, various programs and data necessary for the operation of the electronic device 1300 are also stored. The processing device 1301, ROM 1302, and RAM 1303 are connected to each other through a bus 1304. An input/output (I/O) interface 1305 is also connected to the bus 1304 .

Typically, the following devices can be connected to the I/O interface 1305: input devices 1306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 1307 such as a computer; a storage device 1306 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication means 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 13 shows electronic device 1300 having various means, it is to be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 1309 , or from storage means 1306 , or from ROM 1302 . When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

The electronic device provided by the embodiment of the present application belongs to the same inventive concept as the image repair model training method and the image repair method provided by the above-mentioned embodiment. For technical details not described in detail in this embodiment, please refer to the above-mentioned embodiment, and this embodiment It has the same beneficial effect as the above embodiment.

Based on the image restoration model training method and the image restoration method provided by the above method embodiments, the embodiment of the present application provides a computer-readable medium on which a computer program is stored, wherein the program is implemented when the program is executed by a processor. A method for training an image restoration model as described in any of the above embodiments, or an image restoration method as described in any of the above embodiments.

It should be noted that the computer-readable medium mentioned above in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable signal medium may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server can communicate using any currently known or future network protocols such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium The communication (eg, communication network) interconnections. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is made to execute the above-mentioned image restoration model training method or image restoration method.

Computer program code for carrying out the operations of this application may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present application may be implemented by means of software or by means of hardware. Wherein, the name of the unit/module does not constitute a limitation on the unit itself under certain circumstances, for example, the voice data collection module can also be described as a "data collection module".

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present application, [Example 1] provides a method for training an image repair model, the method comprising:

According to one or more embodiments of the present application, [Example 2] provides a training method for an image repair model, the method further includes:

Inputting the pseudo real first pixel quality image into the image feature encoder to obtain a third image feature;

calculating a content consistency loss based on the second image feature and the third image feature;

said training said image feature encoder, said true first pixel quality image generator, and said true second pixel quality image generator based on said domain alignment loss, said image generation loss, and said image reconstruction loss ,include:

The image feature encoder, the true first pixel quality image generator, and the true Second pixel quality image generator.

According to one or more embodiments of the present application, [Example 3] provides a training method for an image repair model, and the method further includes:

Inputting the third image feature into the real second pixel quality image generator to obtain a pseudo-real reconstructed second pixel quality image;

The calculating an image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image includes:

calculating a first image reconstruction loss based on the reconstructed second pixel quality image and the true second pixel quality image;

calculating a second image reconstruction loss based on the pseudo real reconstructed second pixel quality image and the real second pixel quality image; the first image reconstruction loss and the second image reconstruction loss constitute an image reconstruction loss.

According to one or more embodiments of the present application, [Example 4] provides a training method for an image inpainting model, according to the regenerated real first pixel quality image, the pseudo real first pixel quality image and calculating an image generation loss for the true first pixel quality image, comprising:

calculating a first image generation loss based on the regenerated true first pixel quality image and the true first pixel quality image;

A second image generation loss is calculated from the pseudo-true first pixel quality image and the real first pixel quality image; the first image generation loss and the second image generation loss constitute an image generation loss.

According to one or more embodiments of the present application, [Example 5] provides a training method for an image inpainting model, wherein the domain alignment loss is calculated according to the first image feature and the second image feature, including:

Inputting the first image feature into a feature discriminator to obtain a first probability value;

According to one or more embodiments of the present application, [Example 6] provides a training method for an image inpainting model, wherein the calculation of the content consistency loss according to the second image feature and the third image feature includes:

calculating a 1-norm between the second image feature and the third image feature;

According to one or more embodiments of the present application, [Example 7] provides an image restoration model training method, the first image is calculated according to the reconstructed second pixel quality image and the real second pixel quality image Reconstruction losses, including:

calculating a 1-norm between the reconstructed second pixel quality image and the true second pixel quality image;

According to one or more embodiments of the present application, [Example 8] provides a training method for an image inpainting model, the second pixel quality image is reconstructed according to the pseudo-truth and the second pixel quality image is calculated according to the real second pixel quality image Two image reconstruction losses, including:

calculating a 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image;

According to one or more embodiments of the present application, [Example 9] provides a training method for an image inpainting model, the calculation based on the regenerated real first pixel quality image and the real first pixel quality image The first image generation loss consists of:

calculating a 1-norm between said regenerated true first pixel quality image and said true first pixel quality image;

According to one or more embodiments of the present application, [Example 10] provides a training method for an image inpainting model, wherein the calculation of the second Image generation loss, including:

Inputting the pseudo real first pixel quality image into a feature discriminator to obtain a third probability value;

According to one or more embodiments of the present application, [Example 11] provides a training method for an image inpainting model, the image feature encoder includes at least one residual module;

According to one or more embodiments of the present application, [Example 12] provides an image restoration method, the method comprising:

The image feature encoder and the real second pixel quality image generator are obtained by training according to the image restoration model training method described in any of the above examples.

According to one or more embodiments of the present application, [Example 13] provides a training device for an image restoration model, the device comprising:

According to one or more embodiments of the present application, [Example Fourteen] provides a training device for an image repair model, the device further comprising:

The training unit is specifically configured to train the image feature encoder, the real first pixel according to the domain alignment loss, the image generation loss, the image reconstruction loss, and the content consistency loss quality image generator and said true second pixel quality image generator.

According to one or more embodiments of the present application, [Example 15] provides a training device for an image inpainting model, and the device further includes:

The third calculation unit includes:

According to one or more embodiments of the present application, [Example 16] provides a training device for an image inpainting model, and the second calculation unit includes:

According to one or more embodiments of the present application, [Example 17] provides a training device for an image inpainting model, the first computing unit is specifically configured to input the first image feature into a feature discriminator to obtain first probability value;

According to one or more embodiments of the present application, [Example 18] provides an image restoration model training device, the fourth calculation unit is specifically used to calculate the second image features and the third image 1-norm between features;

According to one or more embodiments of the present application, [Example 19] provides an image restoration model training device, the first calculation subunit is specifically used to calculate the reconstructed second pixel quality image and the 1-norm between real second pixel quality images;

According to one or more embodiments of the present application, [Example 20] provides a training device for an image inpainting model, and the second calculation subunit is specifically used to calculate the pseudo-real reconstructed second pixel-quality image and a 1-norm between said true second pixel quality images;

According to one or more embodiments of the present application, [Example 21] provides a training device for an image inpainting model, and the third calculation subunit is specifically configured to calculate the regenerated real first pixel quality a 1-norm between the image and said true first pixel quality image;

According to one or more embodiments of the present application, [Example 22] provides a training device for an image inpainting model, the fourth calculation subunit is specifically configured to input the pseudo-real first pixel quality image A feature discriminator to obtain a third probability value;

According to one or more embodiments of the present application, [Example 23] provides an image inpainting model training device, the image feature encoder includes at least one residual module;

According to one or more embodiments of the present application, [Example 24] provides an image restoration device, the device comprising:

The image feature encoder and the real second pixel quality image generator are obtained by training according to the image restoration model training method described in any one of the above examples.

According to one or more embodiments of the present application, [Example 25] provides an electronic device, including:

one or more processors;

a storage device on which one or more programs are stored,

When the one or more programs are executed by the one or more processors, so that the one or more processors implement the image restoration model training method as described in any of the above examples, or [Example 12] The image restoration method described above.

According to one or more embodiments of the present application, [Example 26] provides a computer-readable medium, on which a computer program is stored, wherein, when the program is executed by a processor, any of the above-mentioned examples can be implemented. The training method of the image restoration model described above, or the image restoration method described in [Example 12].

The above description is only a preferred embodiment of the present application and an illustration of the applied technical principle. Those skilled in the art should understand that the scope of disclosure involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features, but also covers the technical solutions made by the above technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features with similar functions disclosed in (but not limited to) this application.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the application. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

A training method for an image restoration model, characterized in that the method comprises:

Inputting the real first pixel quality image into the image feature encoder to obtain the first image feature;

Inputting the artificially synthesized first pixel quality image into the image feature encoder to obtain the second image feature; the artificially synthesized first pixel quality image is obtained by blurring the real second pixel quality image;

Inputting the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

Inputting the second image feature into the real first pixel quality image generator to obtain a pseudo-true first pixel quality image;

Inputting the second image feature into a real second pixel quality image generator to obtain a reconstructed second pixel quality image;

calculating a domain alignment loss based on the first image feature and the second image feature;

calculating an image generation loss based on the regenerated true first pixel quality image, the pseudo true first pixel quality image, and the true first pixel quality image;

calculating an image reconstruction loss based on the reconstructed second pixel quality image and the true second pixel quality image;

Train the image feature encoder, the true first pixel quality image generator, and the true second pixel quality image generator based on the domain alignment loss, the image generation loss, and the image reconstruction loss, repeating Execute said inputting the real first pixel quality image into the image feature encoder to obtain the first image feature and subsequent steps until reaching the preset condition;

Wherein, the image definition of the first pixel quality is lower than the image definition of the second pixel quality.
The method according to claim 1, further comprising:

Inputting the pseudo real first pixel quality image into the image feature encoder to obtain a third image feature;

calculating a content consistency loss based on the second image feature and the third image feature;

said training said image feature encoder, said true first pixel quality image generator, and said true second pixel quality image generator based on said domain alignment loss, said image generation loss, and said image reconstruction loss ,include:

The image feature encoder, the true first pixel quality image generator and the true second Pixel quality image generator.
The method according to claim 2, further comprising:

Inputting the third image feature into the real second pixel quality image generator to obtain a pseudo-real reconstructed second pixel quality image;

The calculating an image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image includes:

calculating a first image reconstruction loss based on the reconstructed second pixel quality image and the true second pixel quality image;

calculating a second image reconstruction loss based on the pseudo real reconstructed second pixel quality image and the real second pixel quality image; the first image reconstruction loss and the second image reconstruction loss constitute an image reconstruction loss.
The method according to claim 1, wherein said calculating an image generation loss based on said regenerated real first pixel quality image, said pseudo real first pixel quality image, and said real first pixel quality image ,include:

calculating a first image generation loss based on the regenerated true first pixel quality image and the true first pixel quality image;

A second image generation loss is calculated from the pseudo-true first pixel quality image and the real first pixel quality image; the first image generation loss and the second image generation loss constitute an image generation loss.
The method according to claim 1, wherein the calculating domain alignment loss according to the first image feature and the second image feature comprises:

Inputting the first image feature into a feature discriminator to obtain a first probability value;

inputting the second image feature into the feature discriminator to obtain a second probability value;

A domain alignment loss is calculated according to the first probability value and the second probability value.
The method according to claim 2, wherein the calculating the content consistency loss according to the second image feature and the third image feature comprises:

calculating a 1-norm between the second image feature and the third image feature;

A content consistency loss is calculated according to a 1-norm between the second image feature and the third image feature.
The method according to claim 3, wherein the calculating the first image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image comprises:

calculating a 1-norm between the reconstructed second pixel quality image and the true second pixel quality image;

A first image reconstruction loss is calculated based on a 1-norm between the reconstructed second pixel quality image and the real second pixel quality image.
The method according to claim 3, wherein the calculating a second image reconstruction loss according to the pseudo-true reconstructed second pixel quality image and the real second pixel quality image comprises:

calculating a 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image;

A second image reconstruction loss is calculated based on a 1-norm between the pseudo real reconstructed second pixel quality image and the real second pixel quality image.
The method according to claim 4, wherein the calculating the first image generation loss according to the regenerated real first pixel quality image and the real first pixel quality image comprises:

calculating a 1-norm between said regenerated true first pixel quality image and said true first pixel quality image;

A first image generation loss is calculated based on the regenerated real first pixel quality image and a 1-norm between the real first pixel quality image.
The method according to claim 4, wherein the calculating the second image generation loss according to the pseudo real first pixel quality image and the real first pixel quality image comprises:

Inputting the pseudo real first pixel quality image into a feature discriminator to obtain a third probability value;

inputting the real first pixel quality image into the feature discriminator to obtain a fourth probability value;

A second image generation loss is calculated based on the third probability value and the fourth probability value.
The method according to any one of claims 1-10, characterized in that,

The image feature encoder includes at least one residual module;

said true first pixel quality image generator comprising at least one residual module;

The real second pixel quality image generator includes a first convolutional layer, a basic unit, a second convolutional layer, an upsampling layer, a third convolutional layer, an activation function layer, and a fourth convolutional layer, and the basic unit Include at least one RRDB unit.
An image restoration method, characterized in that the method comprises:

Input the first pixel quality image to be repaired into the image feature encoder to obtain the target image feature;

The target image feature is input into a real second pixel quality image generator to obtain a repaired second pixel quality image;

The image feature encoder and the real second pixel quality image generator are obtained by training according to the training method of the image inpainting model described in any one of claims 1-11.
A training device for an image restoration model, characterized in that the device comprises:

The first execution unit is used to input the real first pixel quality image into the image feature encoder to obtain the first image feature;

The second execution unit is configured to input the artificially synthesized first pixel quality image into the image feature encoder to obtain the second image feature; the artificially synthesized first pixel quality image is obtained by blurring the real second pixel quality image of;

A third execution unit, configured to input the first image feature into a real first pixel quality image generator to obtain a regenerated real first pixel quality image;

A fourth execution unit, configured to input the second image feature into the real first pixel quality image generator to obtain a pseudo-real first pixel quality image;

The fifth execution unit is configured to input the second image feature into the real second pixel quality image generator to obtain a reconstructed second pixel quality image;

A first calculation unit, configured to calculate a domain alignment loss according to the first image feature and the second image feature;

A second calculation unit, configured to calculate an image generation loss based on the regenerated real first pixel quality image, the pseudo real first pixel quality image, and the real first pixel quality image;

A third calculation unit, configured to calculate an image reconstruction loss according to the reconstructed second pixel quality image and the real second pixel quality image;

a training unit for training the image feature encoder, the true first pixel quality image generator, and the true second pixel quality based on the domain alignment loss, the image generation loss, and the image reconstruction loss The image generator repeatedly executes the steps of inputting the real first pixel quality image into the image feature encoder to obtain the first image feature and subsequent steps until the preset condition is reached;

Wherein, the image definition of the first pixel quality is lower than the image definition of the second pixel quality.
An image restoration device, characterized in that the device comprises:

The eighth execution unit is used to input the first pixel quality image to be repaired into the image feature encoder to obtain the target image feature;

A ninth execution unit, configured to input the features of the target image into a real second pixel quality image generator to obtain a repaired second pixel quality image;

The image feature encoder and the real second pixel quality image generator are obtained by training according to the training method of the image inpainting model described in any one of claims 1-11.
An electronic device, characterized in that it comprises:

one or more processors;

a storage device on which one or more programs are stored,

When the one or more programs are executed by the one or more processors, the one or more processors implement the image repair model training method according to any one of claims 1-11, or the right The image restoration method described in claim 12.
A computer-readable medium, which is characterized in that a computer program is stored thereon, wherein, when the program is executed by a processor, the method for training an image restoration model according to any one of claims 1-11 is implemented, or the right The image restoration method described in claim 12.