CN116452903A

CN116452903A - Image restoration model training method, image restoration device and electronic equipment

Info

Publication number: CN116452903A
Application number: CN202111661640.3A
Authority: CN
Inventors: 韩如琪; 邓煜港
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Shanghai ICT Co Ltd; CM Intelligent Mobility Network Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Shanghai ICT Co Ltd; CM Intelligent Mobility Network Co Ltd
Priority date: 2021-12-31
Filing date: 2021-12-31
Publication date: 2023-07-18

Abstract

The invention discloses an image restoration model training method, an image restoration device and electronic equipment, relates to the technical field of image processing, and aims to solve the problem that an image restoration effect of a related image restoration scheme is poor. The method comprises the following steps: acquiring a lossless image sample set; determining a first gray scale map and a first contour map of a loss sample image corresponding to the loss sample image based on the loss sample image and the first mask in the loss image sample set; inputting the first gray level image, the first contour image, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network; determining a loss function value based on the repair image and the lossless sample image; based on the loss function values, parameters of the image restoration network are trained. The invention can ensure that the image restoration network obtained by training can accurately position the lost area in the image, and obtain better image restoration effect.

Description

Image restoration model training method, image restoration device and electronic equipment

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image restoration model training method, an image restoration device, and an electronic device.

Background

With advances in computer technology and advances in machine learning technology, a wide variety of repair approaches to image defects exist. Currently, image restoration technology based on deep learning is a mainstream trend, however, the image restoration scheme in the related art is poor in image restoration effect due to inaccurate positioning of a picture loss region.

Disclosure of Invention

The embodiment of the invention provides an image restoration model training method, an image restoration device and electronic equipment, which are used for solving the problem of poor image restoration effect of an image restoration scheme in the related technology.

In a first aspect, an embodiment of the present invention provides an image repair model training method, including:

acquiring a lossless image sample set;

determining a first gray scale map and a first contour map of a lost sample image corresponding to the lost sample image based on the lost sample image and a first mask in the lost sample image set;

inputting the first gray scale image, the first contour image, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network;

Determining a loss function value based on the repair image and the lossless sample image;

and training parameters of the image restoration network based on the loss function value.

Optionally, the first gray map is equal to a hadamard product of a matrix corresponding to the gray map of the lossless sample image and a first matrix, and the first contour map is equal to a hadamard product of a matrix corresponding to the contour map of the lossless sample image and the first matrix, where the first matrix is equal to a difference between a matrix with all element values of 1 and a matrix corresponding to the first mask, the matrix corresponding to the first mask uses an element value of 1 to represent a loss region, and uses an element value of 0 to represent a non-loss region.

Optionally, the image restoration network comprises a contour completion network and a color filling network;

inputting the first gray scale map, the first contour map, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network, wherein the restoration image comprises the following steps:

inputting the first gray scale map, the first contour map and the first mask into the contour completion network, and obtaining a second contour map of a loss region in the loss sample image output by the contour completion network;

Inputting the lost sample image and a third profile corresponding to a lost region in the lost sample image into the color filling network, and obtaining a repair image of the lost sample image output by the color filling network;

the determining a loss function value based on the repair image and the lossless sample image includes:

determining a contour loss function value based on the second contour map and the first contour map;

determining an image loss function value based on the repair image and the lossless sample image;

the training of the parameters of the image restoration network based on the loss function value includes:

training parameters of the profile completion network based on the profile loss function value;

and training parameters of the color filling network based on the image loss function value.

Optionally, the determining an image loss function value based on the repair image and the lossless sample image includes:

an L1 norm loss function value is determined based on a difference value of each pixel in the repair image and the lossless sample image, a loss region in the lossless sample image, and a number of pixels in the lossless sample image.

Optionally, the L1 norm loss function value includes an L1 loss value of a loss region of the loss sample image and an L1 loss value of a non-loss region of the loss sample image;

the L1 loss value of the non-loss area is equal to the difference between a matrix corresponding to a second mask and the Hadamard product L1 norm of the second matrix divided by the pixel number, the second matrix is equal to the difference between the matrix corresponding to the repair image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss area in the lossy sample image, the element value 0 is used for representing the loss area in the matrix corresponding to the second mask, and the element value 1 is used for representing the non-loss area;

the L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of a third matrix and the second matrix divided by the number of pixels, wherein the third matrix is equal to the difference between the matrix with the element values of all 1 and the matrix corresponding to the second mask.

calculating the L1 norm loss function value, the perceptual loss function value, and the style loss function value, respectively, based on the repair image and the lossless sample image;

And calculating a weighted sum of the L1 norm loss function value, the perception loss function value and the style loss function value to obtain the image loss function value.

Optionally, the image restoration network is a U-shaped network based on partial convolution.

Optionally, the image restoration network is a generating countermeasure network, and the generator for generating the countermeasure network is a U-shaped network based on partial convolution.

Optionally, the generator and the discriminator in the generation countermeasure network are each trained using spectral normalization.

In a second aspect, an embodiment of the present invention further provides an image restoration method, including:

acquiring a target image to be repaired;

inputting the target image into an image restoration network, and obtaining an image which is output by the image restoration network and is used for restoring the target image;

the image restoration network is a network trained by the image restoration model training method in the first aspect.

In a third aspect, an embodiment of the present invention further provides an image repair model training apparatus, including:

the first acquisition module is used for acquiring a lossless image sample set;

a first determining module, configured to determine a first gray scale map and a first contour map of a lost sample image corresponding to the lost sample image based on a lost sample image in the lost sample image set and a first mask;

The processing module is used for inputting the first gray level image, the first contour image, the first mask and the lost sample image into an image restoration network and obtaining a restoration image of the lost sample image output by the image restoration network;

a second determining module for determining a loss function value based on the repair image and the lossless sample image;

and the training module is used for training the parameters of the image restoration network based on the loss function value.

the processing module comprises:

the first processing unit is used for inputting the first gray level image, the first contour image and the first mask into the contour completion network and obtaining a second contour image of a loss region in the loss sample image output by the contour completion network;

The second processing unit is used for inputting the lost sample image and a third profile corresponding to a lost region in the lost sample image into the color filling network, and obtaining a repair image of the lost sample image output by the color filling network;

the second determining module includes:

a first determination unit configured to determine a contour loss function value based on the second contour map and the first contour map;

a second determination unit configured to determine an image loss function value based on the repair image and the lossless sample image;

the training module comprises:

the first training unit is used for training parameters of the contour completion network based on the contour loss function value;

and the second training unit is used for training the parameters of the color filling network based on the image loss function value.

Optionally, the second determining unit is configured to determine an L1 norm loss function value based on a difference value between each pixel in the repair image and the lossless sample image, a loss region in the lossless sample image, and the number of pixels in the lossless sample image.

Optionally, the second determining unit includes:

a first calculation subunit configured to calculate the L1 norm loss function value, the perceptual loss function value, and the style loss function value, respectively, based on the repair image and the lossless sample image;

and the second calculating subunit is used for calculating the weighted sum of the L1 norm loss function value, the perception loss function value and the style loss function value to obtain the image loss function value.

In a fourth aspect, an embodiment of the present invention further provides an image restoration apparatus, including:

the second acquisition module is used for acquiring a target image to be repaired;

the restoration module is used for inputting the target image into an image restoration network and acquiring an image which is output by the image restoration network and is used for restoring the target image;

In a fifth aspect, an embodiment of the present invention further provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the image restoration model training method as described above when the computer program is executed; or to implement the steps in the image restoration method as described above.

In a sixth aspect, embodiments of the present invention also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps in an image restoration model training method as described above; or to implement the steps in the image restoration method as described above.

In the embodiment of the invention, a lossless image sample set is obtained; determining a first gray scale map and a first contour map of a lost sample image corresponding to the lost sample image based on the lost sample image and a first mask in the lost sample image set; inputting the first gray scale image, the first contour image, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network; determining a loss function value based on the repair image and the lossless sample image; and training parameters of the image restoration network based on the loss function value. Therefore, the image restoration network is trained by using the gray level map, the loss outline map and the mask of the lossless original image, so that the image restoration network obtained by training can be ensured to accurately locate the loss area in the image, the accuracy of identifying the image edge is higher, and a better image restoration effect is further obtained.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.

FIG. 1 is a flowchart of an image restoration model training method provided by an embodiment of the present invention;

fig. 2 is a schematic structural diagram of a U-shaped network according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a model training process provided by an embodiment of the present invention;

FIG. 4 is a flowchart of an image restoration method provided by an embodiment of the present invention;

FIG. 5 is a block diagram of an image restoration model training apparatus provided by an embodiment of the present invention;

FIG. 6 is a block diagram of an image restoration device provided by an embodiment of the present invention;

fig. 7 is a block diagram of an electronic device according to an embodiment of the present invention;

fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1, fig. 1 is a flowchart of an image restoration model training method provided by an embodiment of the present invention, as shown in fig. 1, including the following steps:

and step 101, acquiring a lossless image sample set.

In the embodiment of the invention, in order to train an image restoration model which can accurately identify the image loss area and well restore the loss area, training is required to be carried out based on a loss-free image sample set and a corresponding loss image sample set and model. A loss-free image sample set may be acquired first at the time of model training in advance, which may be obtained by loading a mask onto a loss-free sample image in the loss-free image sample set in a manner that simulates a loss region. In one embodiment, a loss sample image of a loss region with random size and shape can be generated for each loss-free sample image by using masks with different sizes and shapes as a loss image sample set, so as to ensure the robustness of a trained image restoration model.

Step 102, determining a first gray level map and a first contour map of a lost sample image corresponding to the lost sample image based on the lost sample image and the first mask in the lost sample image set.

Before training, the lossless sample images in the lossless image sample set may be preprocessed, specifically, a corresponding lossy sample image may be determined based on the lossless sample images and the first mask, the lossy sample images may be converted into gray maps, so as to obtain a first gray map, and a contour map of a lossy region in the lossy sample images may be determined, so as to obtain a first contour map.

The first mask may be a mask for simulating a loss region, and the mask may be fixed or may be randomly generated, and specifically may be obtained by loading the first mask on the lossless sample image to obtain a corresponding lossy sample image, and converting the lossy sample image into a gray scale to obtain the first gray scale, or may be obtained by converting the lossless sample image into a gray scale first and then loading the first mask on the gray scale of the lossless sample image.

And step 103, inputting the first gray scale image, the first contour image, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network.

After determining a first gray scale map and a first contour map of a lost sample image corresponding to the lossless sample image, the first gray scale map, the first contour map, the first mask and the lost sample image can be used as input data to be input into an image restoration network, the input data is processed through the image restoration network, namely, a lost area in the lost sample image is identified and restored, a predicted image output by the image restoration network is obtained after the processing is finished, namely, a restored image after restoring the lost sample image is output, wherein the first contour map and the first mask can be used as limiting conditions for generating a lost area contour map which is as close as possible to the first contour map (real lost area contour) when the lost area is identified by a training network.

The initial training model of the image restoration network can be selected according to actual requirements, for example, the initial training model can be a neural network model such as a convolution network, a partial convolution network, a U-shaped network, a generation countermeasure network and the like, and can also be a combination of a plurality of neural networks.

In one embodiment, the lossless sample image may be converted into a gray scale image, and then the gray scale image of the lossless sample image and the first mask are subjected to hadamard product operation (i.e. multiplication of corresponding elements in two matrices) in a matrix form to obtain a gray scale image matrix of the lossless sample image corresponding to the lossless sample image, and the contour image of the lossless sample image may be calculated, and the hadamard product operation may be performed on the contour image of the lossless sample image and the first mask in a matrix form to obtain a contour image matrix of the lossless sample image corresponding to the lossless sample image, where the contour image calculation of the lossless sample image may be performed by using a Canny operator.

In particular, matrix I may be used _gt A gray scale and a contour map representing the lossless sample image using a matrix C respectively _gt Sum matrix I _gray Representing that the matrix M is the first mask, and then a first gray level map of the lost sample image corresponding to the lossless sample image First profile of the lost sample image +.>Wherein an element value of 1 in the matrix M represents a loss region and an element value of 0 represents a non-loss region.

The image restoration network may be based on the first gray scale mapSaid first profile->And the first mask M generates a contour map C of a loss region in the loss sample image _pred The specific relation may be, for example,the image restoration network may also be based on a generated outline map C of the lost region _pred And image features in the lost sample image, and may use the corresponding lossless sample image as a defining condition to cause the image restoration network to generate a signal that is identical to the lossless sample image (noLost artwork) as similar as possible.

In this way, the mode of calculating the gray level map and the outline map of the lost sample image by using the matrix Hadamard product is not a simple mapping relation, so that the accuracy of identifying the image edge can be ensured, and the positioning accuracy of the image lost region can be ensured.

Step 104, determining a loss function value based on the repair image and the lossless sample image.

After obtaining the repair image output by the image repair network, the repair image and the lossless sample image can be compared, and a loss function value is calculated by using a loss function, wherein the loss function can be selected according to actual needs, for example, an L1 norm loss function, a style loss function, a perception loss function and the like can be selected to calculate the loss function value, and a mode of combining various loss functions can be adopted to calculate the loss function value.

And step 105, training parameters of the image restoration network based on the loss function value.

After the loss function value is calculated, the parameters of the image restoration network can be trained based on the loss function value, namely, when the loss function value does not meet the requirement, such as the condition that the loss function value is larger, the parameters of the image restoration network are adjusted, and then the iterative training process of steps 102 to 105 is performed on the image restoration network again until the loss function value meets the requirement, so that the loss function value is minimized, and finally, the trained image restoration network is obtained.

the step 103 includes:

The step 104 includes:

the step 105 includes:

In one embodiment, the image restoration network may be formed by combining two parts of networks, including a contour completion network and a color filling network, where the contour completion network is used to identify a contour of a loss region in an input loss image, and the color filling network is used to restore the loss region output by the contour completion network, and finally output a restored image.

Thus, after the first gray scale map and the first contour map of the lost sample image are obtained, the first gray scale map, the first contour map, and the first mask may be input into the contour completion network to generate a second contour map of a lost region in the lost sample image through the contour completion network, e.g., the contour completion network may be based on the input first gray scale map, the first contour map, and the first mask, and the relational expression Outputting a contour map C of the loss region prediction in the loss sample image _pred 。

And then, calculating a contour loss function value by comparing a second contour map output by the contour completion network with a real original contour map, namely the first contour map, and training parameters of the contour completion network based on the contour loss function value, namely, after the parameters of the contour completion network are adjusted under the condition that the contour loss function value does not meet the requirement, if the contour loss function value is larger, training the contour completion network again until the contour loss function value meets the requirement, namely, the contour generated by the contour completion network is as close to the real original contour as possible, so that the trained contour completion network is finally obtained.

The color filling network may be trained together with the contour completion network, or may be trained separately, when training together, the second contour map output by the contour completion network and the loss sample image with a loss region may be input into the color filling network, and when training separately, the loss sample image labeled with the contour of the loss region may be input into the color filling network, i.e., the third contour map may be the second contour map output by the contour completion network, or may be a contour map labeled in the loss sample image. The color filling network may generate a repair image of a lost region in the lost sample image based on the lost sample image and the third profile.

And then, the image loss function value can be calculated by comparing the restored image output by the color filling network with the lossless original image, namely the lossless sample image, and the parameters of the color filling network can be trained based on the image loss function value, so that the color filling network can be trained again after the parameters of the color filling network are adjusted under the condition that the image loss function value is not satisfied, if the image loss function value is larger, the image loss function value satisfies the requirement, even if the color distribution of the restored image generated by the color filling network is as close to the lossless original image as possible, the trained color filling network is finally obtained.

When the contour completion network and the color filling network are trained together, parameters of the contour completion network and the color filling network can be adjusted based on the contour loss function value and the image loss function value respectively, and the parameters enter the next iterative training process after being adjusted.

Therefore, through the combined network with two different functions, the image loss area can be more accurately identified, and a better image restoration effect is obtained. And by using the contour loss function not only aims at improving the reconstruction accuracy of a single pixel, but also considers the factors of the overall structure, namely whether the repaired image pixel can be well blended into other surrounding environment pixels.

In one embodiment, for the color filling network, an L1 norm loss function may be used to calculate the output loss value of the network, where the L1 norm loss is also referred to as the minimum absolute deviation (Least Absolute Deviation, LAD) or minimum absolute error (Least Absolute Error, LAE), and the average absolute error (Mean Absolute Error, MAE) is obtained by dividing the L1 norm loss by the number of pixel values in a certain area, and in this embodiment, MAE may be used to calculate the L1 norm loss function value of the color filling network.

Specifically, the difference between each pixel in the repair image and the lossless sample image may be calculated, and then the L1 norm loss function value, that is, the average absolute error, is calculated in combination with the loss region in the lossless sample image and the number of pixels in the lossless sample image.

Therefore, the loss value of the repair image output by the color filling network is calculated from the image pixel level, the color filling network trained based on the loss value can be ensured to have higher precision, so that a better image repair effect is obtained, and the problems of large amount of blurring, deformity, noise and the like in repair are avoided.

That is, in one embodiment, the image loss value of the color filling network may be calculated from both lost and non-lost area losses, respectively.

Specifically, a single pixel reconstruction loss function in a loss region and a non-loss region may be defined, respectively, wherein the L1 loss function of the non-loss region of the loss sample image may be The L1 loss function of the loss region of the loss sample image may be +.>Wherein L is _valid L1 loss value, L, for the non-loss region _hole L1 loss value for the loss region, I _gt Representing a lossless original, i.e. the lossless sample image, i.e. I _out Representing network predicted outputThe color filling network outputs a repair image, M is a second mask representing a loss region of the lost sample image, wherein the value of the pixel point in the loss region is 0, and the value of the pixel point in the non-loss region is 1, < >>Representing the lossless sample image I _gt I.e., C x H x W, where C, H and W are the number of channels, image height, and image width, respectively.

Therefore, through the embodiment, the L1 loss value output by the network can be rapidly and accurately calculated, and the L1 loss values of the loss area and the non-loss area are calculated respectively, and the color filling network is trained based on the calculated loss values, so that the global repairing effect of the model can be ensured, and the structural disorder of the repaired image is avoided.

In yet another embodiment, a plurality of loss function combinations may be used to calculate an image loss function value, and in particular, an L1 norm loss function value, a perception loss function value, and a style loss function value may be calculated based on the repair image and the lossless sample image, respectively, and the image loss function value may be determined by weighted summing the L1 norm loss function value, the perception loss function value, and the style loss function value.

Wherein the perceptual loss function may be to calculate the perceptual loss value based on a first pooling layer, a second pooling layer and a third pooling layer (i.e. pool1 layer, pool2 layer and pool3 layer) in the color filling network.

The style loss function refers to a loss function in style migration, wherein the style loss and the perceived loss are similar, but each feature graph generated by the color filling network calculates an autocorrelation function value (i.e., a gram matrix) before calculating the L1 loss value, and the gram matrix includes relationships and relations between different features in the feature graph, i.e., the style of the feature graph. Based on this, the style loss between the feature map and the repair picture output by the color filling network can be calculated.

In addition, when the L1 norm loss function value, the perceptual loss function value, and the style loss function value are weighted, the weight coefficient of each loss function value may be determined through a plurality of image restoration tests.

Therefore, the color filling network is trained by combining the loss functions representing different image parameters, the repairing effect of the network can be further improved, the repairing image is guaranteed to have a better structure, and the visual cognition of a user is more met.

In one embodiment, a U-shaped network based on partial convolution can be used as an image restoration network, so that better learning of the model on the image structural features is ensured, the calculated amount can be reduced, and the model processing efficiency is improved.

The partial convolution based U-shaped network performs an image restoration process by using stacked partial convolution operations and mask updating steps, the partial convolution layer based U-shaped network replacing convolution layers in the U-shaped network with partial convolution layers (Partial Convolution). The U-shaped network structure is a variation of the full convolutional neural network, in which only convolutional layers and pooling layers are present in the network structure. In this embodiment, all the conventional convolution layers in the U-shaped network structure are replaced by partial convolution layers, so as to construct a variant U-shaped network.

The partial convolution operation and Mask (Mask) update function are jointly referred to as a partial convolution layer. Partial convolution means that the convolution is only performed in the active area of the image (i.e. the portion with Mask 0) and the Mask of the image is iterated and shrunk continuously as the number of layers of the network increases, i.e. both the image with Mask and Mask participate in the training. The partial convolution operation is formulated as:

wherein W is the weight of the convolution kernel, X is the feature (or pixel) value corresponding to the current convolution window, M is the binary Mask corresponding to X, b is the corresponding offset value, X' represents the output of the convolved input image, and the product of pixel by pixel is shown. The scaling factor sum (1)/sum (M) acts as an appropriate scaling to adjust the amount of change in the effective input. The matrices 1 and M have the same shape, e.g. are all 3 x 3 matrices, but the values in matrix 1 are all 1. After each partial convolution operation is completed, the Mask needs to be updated in a round, wherein the Mask updating function formula is as follows:

where m' represents the convolved output of the input Mask. After the partial convolution is performed, mask is updated, and the update rule may be: if at least one Mask value corresponding to each pixel point in the convolution window is 1, updating the Mask of the corresponding position after convolution to be 1, and if the network depth is enough, the Mask area size can be contracted to 0.

In the U-shaped network based on partial convolution, the size of an input image is the same as that of a Mask, the size of a convolution kernel is the same, the convolution kernel of the input image is updated continuously, the convolution kernel of the Mask is always 1, and no offset exists, so that training parameters which need to be input during convolution are reduced, resource consumption is reduced, and the training process is accelerated. In the Mask updating process, the value of each pixel of the Mask is also 1 or 0, and no decimal part exists.

Compared with the traditional convolution layer, the partial convolution layer in the U-shaped network based on the partial convolution has Mask, each pixel point in the traditional convolution layer needs to participate in convolution operation, and only the pixel points in the non-loss area of the partial convolution layer participate in convolution operation, so that the processing efficiency can be improved.

The structure of the U-shaped network can be shown in fig. 2, the whole network structure is in a U shape, the structure from the leftmost side to the middle of the bottom of the U-shaped network can be regarded as an encoder, and the structure from the middle of the bottom to the rightmost side can be regarded as a decoder. As shown in fig. 2, each layer has a jumper connection (connection) from the encoder side on the left to the decoder side on the right. The jumper connection is to cut the output result of the encoder and splice the output result as the input parameter with the deconvolution result of the decoder. Batch standardization (Batch Normalization) can be introduced into the network structure, so that scattered data are unified, and the network can learn under the optimal state, and is especially necessary for the neural network with more layers in the model. The number of network layers in fig. 2 is only schematic, and the actual number of U-shaped network layers may be designed according to requirements.

In the partial convolution-based U-shaped network, a leak Relu activation function can be adopted in a decoder stage, so that when data input is negative, the network can still learn, and neurons cannot die.

In the last partial convolution layer of the partial convolution-based U-shaped network, when the layer input value contains an original Image (Image) with a loss area and a Mask (Mask) which are input at first, the partial convolution-based U-shaped network outputs pixels of an uncorrupted part. Each jumper in fig. 2 is connected with the left and right feature images and the mask respectively, the two feature images and the mask are spliced into a feature image and the mask after being synthesized, then a new feature image is obtained through deconvolution operation, and the new feature image and the updated mask are used as two new input parameters and are provided for the next part of convolution layers, so that effective pixels in the image input at the beginning can provide reference information for restoration of the image in the decoding stage.

Therefore, by using the U-shaped network based on partial convolution as the image restoration network, the model processing efficiency can be improved, and better model restoration performance can be ensured.

In one embodiment, a generating countermeasure network may be employed as the image restoration network, and a partially convolution-based U-shaped network may be employed as the generator of the generating countermeasure network for identifying image loss regions and generating restoration images, and an output of the partially convolution-based U-shaped network may be employed as an input to the discriminator of the generating countermeasure network for determining whether the generated loss region profile is authentic and whether the generated restoration images are sufficiently similar to lossless originals.

Wherein the discriminator network may be trained using objective evaluation metrics such as contrast loss and feature matching loss, which compares the feature patterns at the intermediate layers of the discriminator, forcing the generator to produce similar results for the features and the real images by defining the data generated at the intermediate layers, thereby stabilizing the training process to allow the model to converge.

It should be noted that, when the image restoration network includes two parts of a contour complement network and a color filling network, the contour complement network and the color filling network may both use a generating countermeasure network, and may use a U-shaped network based on partial convolution as a generator. Thus, the process of training the outline completing network and the color filling network can be as shown in fig. 3.

Therefore, the embodiment adds the U-shaped network of the partial convolution layer into the network structure for generating the countermeasure network, so that the model can learn the structural characteristics of the image better, abstract of invalid and even harmful areas is avoided, and the repairing effect of the model can be further improved.

That is, in one embodiment, the generator and the discriminator in the generated countermeasure network may use spectral normalization to train, so that the respective maximum values of the countermeasure loss, the feature matching loss, and the like are reduced by the weight matrix in proportion, so that the training process is more stable, and the liphatz constant of the network is effectively limited to 1.

Thus, by using spectral normalization in the generator and the discriminator, the parameters and gradient variation in a short time can be effectively limited, so that the countermeasures loss can play the maximum role in the variance calculation of the discriminator, and the feature matching loss can effectively constrain the feature map trained by the generator.

According to the image restoration model training method, a lossless image sample set is obtained; determining a first gray scale map and a first contour map of a lost sample image corresponding to the lost sample image based on the lost sample image and a first mask in the lost sample image set; inputting the first gray scale image, the first contour image, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network; determining a loss function value based on the repair image and the lossless sample image; and training parameters of the image restoration network based on the loss function value. Therefore, the image restoration network is trained by using the gray level map, the loss outline map and the mask of the lossless original image, so that the image restoration network obtained by training can be ensured to accurately locate the loss area in the image, the accuracy of identifying the image edge is higher, and a better image restoration effect is further obtained.

Referring to fig. 4, fig. 4 is a flowchart of an image restoration method according to an embodiment of the present invention, as shown in fig. 4, including the following steps:

step 401, acquiring a target image to be repaired.

When the lost image needs to be repaired, a target image to be repaired, namely, a lost area exists in the target image, can be acquired.

Step 402, inputting the target image into an image restoration network, and obtaining an image which is output by the image restoration network and is used for restoring the target image;

the image restoration network is a network trained by the image restoration model training method in the embodiment shown in fig. 1.

In the step, the obtained target image to be repaired can be input into an image repairing network so as to identify and repair the lost area of the target image through the image repairing network, and further, the image which is output by the image repairing network and is used for repairing the target image is obtained.

The image restoration network is a network trained by the image restoration model training method in the embodiment shown in fig. 1, and specifically, the related description in the foregoing embodiment can be referred to, and the same technical effects can be obtained, so that repetition is avoided, and no further description is provided herein.

The image restoration method of the embodiment of the invention acquires the target image to be restored; inputting the target image into an image restoration network, and obtaining an image which is output by the image restoration network and is used for restoring the target image; the image restoration network is a network trained by the image restoration model training method in the embodiment shown in fig. 1. Therefore, the used network restoration model is obtained by training the image restoration network by using the gray level diagram, the loss outline diagram and the mask of the lossless original diagram, so that a better image restoration effect can be ensured.

The embodiment of the invention also provides an image restoration model training device. Referring to fig. 5, fig. 5 is a block diagram of an image restoration model training apparatus according to an embodiment of the present invention. Because the principle of solving the problem of the image restoration model training device is similar to that of the image restoration model training method in the embodiment of the invention, the implementation of the image restoration model training device can be referred to the implementation of the method, and the repetition is not repeated.

As shown in fig. 5, the image restoration model training apparatus 500 includes:

a first obtaining module 501, configured to obtain a lossless image sample set;

A first determining module 502, configured to determine a first gray scale map and a first contour map of a lost sample image corresponding to the lost sample image based on the lost sample image in the lost sample image set and a first mask;

a processing module 503, configured to input the first gray scale map, the first contour map, the first mask, and the lost sample image into an image restoration network, and obtain a restoration image of the lost sample image output by the image restoration network;

a second determination module 504 for determining a loss function value based on the repair image and the lossless sample image;

a training module 505, configured to train parameters of the image restoration network based on the loss function value.

the processing module 503 includes:

the second determination module 504 includes:

the training module 505 includes:

Optionally, the second determining unit includes:

The image restoration model training device provided by the embodiment of the invention can execute the method embodiment, the implementation principle and the technical effect are similar, and the embodiment is not repeated here.

The image restoration model training device 500 of the embodiment of the invention acquires a lossless image sample set; determining a first gray scale map and a first contour map of a lost sample image corresponding to the lost sample image based on the lost sample image and a first mask in the lost sample image set; inputting the first gray scale image, the first contour image, the first mask and the lost sample image into an image restoration network, and obtaining a restoration image of the lost sample image output by the image restoration network; determining a loss function value based on the repair image and the lossless sample image; and training parameters of the image restoration network based on the loss function value. Therefore, the image restoration network is trained by using the gray level map, the loss outline map and the mask of the lossless original image, so that the image restoration network obtained by training can be ensured to accurately locate the loss area in the image, the accuracy of identifying the image edge is higher, and a better image restoration effect is further obtained.

The embodiment of the invention also provides an image restoration device. Referring to fig. 6, fig. 6 is a block diagram of an image restoration apparatus according to an embodiment of the present invention. Because the principle of solving the problem of the image restoration device is similar to that of the image restoration method in the embodiment of the invention, the implementation of the image restoration device can be referred to the implementation of the method, and the repetition is omitted.

As shown in fig. 6, the image restoration apparatus 600 includes:

a second acquiring module 601, configured to acquire a target image to be repaired;

the restoration module 602 is configured to input the target image into an image restoration network, and obtain an image output by the image restoration network after restoration of the target image;

The image restoration device provided by the embodiment of the invention can execute the method embodiment, the implementation principle and the technical effect are similar, and the embodiment is not repeated here.

The image restoration device 600 of the embodiment of the invention acquires a target image to be restored; inputting the target image into an image restoration network, and obtaining an image which is output by the image restoration network and is used for restoring the target image; the image restoration network is a network trained by the image restoration model training method in the embodiment shown in fig. 1. Therefore, the used network restoration model is obtained by training the image restoration network by using the gray level diagram, the loss outline diagram and the mask of the lossless original diagram, so that a better image restoration effect can be ensured.

The embodiment of the invention also provides electronic equipment. Because the principle of the electronic device for solving the problem is similar to that of the image restoration model training method in the embodiment of the invention, the implementation of the electronic device can be referred to the implementation of the method, and the repetition is omitted. As shown in fig. 7, an electronic device according to an embodiment of the present invention includes:

the processor 700 is configured to read the program in the memory 720, and execute the following procedures:

acquiring a lossless image sample set;

Wherein in fig. 7, a bus architecture may comprise any number of interconnected buses and bridges, and in particular one or more processors represented by processor 700 and various circuits of memory represented by memory 720, linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. The processor 700 is responsible for managing the bus architecture and general processing, and the memory 720 may store data used by the processor 700 in performing operations.

Optionally, the processor 700 is further configured to read the program in the memory 720, and perform the following steps:

the first gray level map is equal to the Hadamard product of a matrix corresponding to the gray level map of the lossless sample image and a first matrix, the first contour map is equal to the Hadamard product of the matrix corresponding to the contour map of the lossless sample image and the first matrix, the first matrix is equal to the difference between the matrix with the element value of 1 and the matrix corresponding to the first mask, the matrix corresponding to the first mask uses the element value of 1 to represent a loss area, and uses the element value of 0 to represent a non-loss area.

the processor 700 is further configured to read the program in the memory 720, and perform the following steps:

The electronic device provided by the embodiment of the invention can execute the embodiment of the image restoration model training method, and the implementation principle and the technical effect are similar, and the embodiment is not repeated here.

The embodiment of the invention also provides another electronic device. Because the principle of solving the problem of the electronic device is similar to that of the image restoration method in the embodiment of the invention, the implementation of the electronic device can be referred to the implementation of the method, and the repetition is omitted. As shown in fig. 8, an electronic device according to an embodiment of the present invention includes:

processor 800, for reading the program in memory 820, performs the following processes:

acquiring a target image to be repaired;

Wherein in fig. 8, a bus architecture may comprise any number of interconnected buses and bridges, and in particular, one or more processors represented by processor 800 and various circuits of memory represented by memory 820, linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. The processor 800 is responsible for managing the bus architecture and general processing, and the memory 820 may store data used by the processor 800 in performing operations.

The electronic device provided by the embodiment of the invention can execute the embodiment of the image restoration method, and the implementation principle and the technical effect are similar, and the embodiment is not repeated here.

Furthermore, a computer readable storage medium of an embodiment of the present invention is used to store a computer program, where the computer program may be executed by a processor to implement the steps of the method embodiment shown in fig. 1, or to implement the steps of the method embodiment shown in fig. 4.

In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may be physically included separately, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in hardware plus software functional units.

The integrated units implemented in the form of software functional units described above may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform part of the steps of the transceiving method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

While the foregoing is directed to the preferred embodiments of the present invention, it will be appreciated by those skilled in the art that various modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.

Claims

1. An image restoration model training method, comprising the steps of:

acquiring a lossless image sample set;

2. The method of claim 1, wherein the first gray scale map is equal to a hadamard product of a matrix corresponding to a gray scale map of the lossless sample image and a first matrix, the first contour map is equal to a hadamard product of a matrix corresponding to a contour map of the lossless sample image and the first matrix, wherein the first matrix is equal to a difference between a matrix having all element values of 1 and a matrix corresponding to the first mask, wherein a loss region is represented by an element value of 1 and a non-loss region is represented by an element value of 0 in the matrix corresponding to the first mask.

3. The method of claim 1, wherein the image restoration network comprises a contour completion network and a color filling network;

4. The method of claim 3, wherein the determining an image loss function value based on the repair image and the lossless sample image comprises:

5. The method of claim 4, wherein the L1 norm loss function value comprises an L1 loss value for a loss region of the lost sample image and an L1 loss value for a non-loss region of the lost sample image;

6. The method of claim 4, wherein the determining an image loss function value based on the repair image and the lossless sample image comprises:

7. An image restoration method, comprising:

acquiring a target image to be repaired;

wherein the image restoration network is a network trained by the image restoration model training method according to any one of claims 1 to 6.

8. An image restoration model training apparatus, characterized by comprising:

the first acquisition module is used for acquiring a lossless image sample set;

9. An image restoration device, comprising:

10. An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor; -characterized in that the processor is adapted to read a program in a memory for implementing the steps in the image restoration model training method according to any one of claims 1 to 6; or to implement the steps in the image restoration method as defined in claim 7.

11. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps in the image restoration model training method according to any one of claims 1 to 6; or to implement the steps in the image restoration method as defined in claim 7.