CN111242874A

CN111242874A - Image restoration method and device, electronic equipment and storage medium

Info

Publication number: CN111242874A
Application number: CN202010087216.1A
Authority: CN
Inventors: 朱曼瑜; 刘霄; 文石磊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2020-02-11
Filing date: 2020-02-11
Publication date: 2020-06-05
Anticipated expiration: 2040-02-11
Also published as: CN111242874B

Abstract

The application discloses a method, a device, electronic equipment and a storage medium for image restoration, wherein the method comprises the following steps: receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired; the image restoration method comprises the steps of inputting an image to be restored and information of the image to be restored into an image restoration model to obtain a restored image after the image to be restored is restored, wherein the image restoration model comprises an encoder, a restoration decoder and a refinement decoder, the encoder is used for encoding the image to be restored to obtain a feature map of the image to be restored, the restoration decoder is used for restoring the image to be restored according to the feature map and the information of the image to be restored to obtain an initial restored image, and the refinement decoder is used for refining and restoring the initial restored image to obtain a restored image and outputting the restored image. According to the image restoration method, the image restoration model restores the image from multiple stages, and the accuracy of image restoration is improved.

Description

Image restoration method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image restoration method and apparatus, an electronic device, and a storage medium.

Background

In order to reduce bandwidth resources, in the process of transmitting an image, the image is often compressed, the resolution is reduced, and the like, and the image may also be subjected to non-artificial disturbance noise in the process of transmitting the image, and both the processing of the image and the noise disturbance may cause image damage. In order to enable a user to obtain a clear, high-quality image, it is necessary to perform image restoration on the above-described damaged image or low-quality image.

In the prior art, an image is usually repaired based on a deep learning convolutional neural network, specifically, semantic information of the image is learned from large-scale image data, and pixel blocks with similar semantics are used to fill in defective parts. However, the convolutional neural network in the prior art has few stages of deep learning, resulting in low image restoration accuracy.

Disclosure of Invention

The application provides an image restoration method, an image restoration device, an electronic device and a storage medium, which can restore an image from multiple stages and improve the accuracy of image restoration.

The application provides a method for repairing an image, which comprises the following steps:

receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired; inputting the image to be restored and the information of the image to be restored into an image restoration model to obtain a restored image obtained after restoring the image to be restored, wherein the image restoration model comprises an encoder, a restoration decoder and a refinement decoder, the encoder is used for encoding the image to be restored to obtain a feature map of the image to be restored, the restoration decoder is used for restoring the image to be restored according to the feature map and the information of the image to be restored to obtain an initial restored image, and the refinement decoder is used for refining and restoring the initial restored image to obtain the restored image; and outputting the repaired image.

The image restoration model in this embodiment includes a plurality of restoration decoders, and can perform restoration processing on an image from a plurality of stages to improve the restoration accuracy of the image.

In one possible design, the network depths of the encoder and the repair decoder are both multi-layered, and the number of layers of the network depths of the encoder and the repair decoder is equal; each layer of the encoder is used for encoding the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and the feature map corresponding to each layer is output to the corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder.

Each layer of the restoration decoder is configured to restore the image to be restored according to the feature map of the corresponding layer of the encoder and the information of the image to be restored, to obtain an initial restoration image corresponding to each layer of the restoration decoder, and output the initial restoration image corresponding to each layer to the corresponding layer of the refinement decoder, where the initial restoration image includes the initial restoration image corresponding to each layer of the restoration decoder.

In one possible design, the network depths of the refinement decoder are all multiple layers, and the number of layers of the network depths of the refinement decoder and the repair decoder are equal; each layer of the refinement decoder is used for performing refinement restoration processing on the initial restoration image of the corresponding layer of the restoration decoder to obtain the restoration image corresponding to each layer of the refinement decoder so as to obtain the restoration image, and the restoration image comprises the restoration image corresponding to each layer of the refinement decoder.

In this design, the network depth of each refinement decoder is multi-layered, and the number of layers of network depth of each refinement decoder and repair decoder is equal. That is, in this embodiment, the initial repair image may be subjected to a plurality of thinning repair processes to improve the accuracy of image repair.

In one possible design, the number of refinement decoders is multiple, the network depth of each refinement decoder is multiple layers, and the number of layers of the network depth of each refinement decoder and the number of layers of the network depth of each repair decoder are equal.

a. Each layer of the kth thinning decoder is used for thinning and repairing the initial repaired image of the corresponding layer of the repairing decoder to obtain the repaired image corresponding to each layer of the kth thinning decoder, and the repaired image corresponding to each layer of the kth thinning decoder is output to the corresponding layer of the (k + 1) th thinning decoder, wherein k is an integer greater than or equal to 1;

b. and b, adding 1 to k, and executing the step a until the k is a first preset value to obtain the repaired image, wherein the repaired image comprises the repaired image of each layer of each thinning decoder, and the first preset value is equal to the number of the thinning decoders.

In the design, a plurality of thinning and repairing models can be set, further thinning and repairing of the repaired image can be realized, and the accuracy of image repairing can be further improved.

In one possible design, each layer of the encoder is configured to perform encoding processing on the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and output the feature map corresponding to each layer to a corresponding layer of the repair decoder, including:

A. coding the feature map of the i-1 th layer of the encoder to obtain the feature map of the i-th layer, and outputting the feature map of the i-th layer to the i +1 th layer of the encoder and the i +1 th layer of the repair decoder, wherein the size of the feature map of the i-th layer is a times of the size of the feature map of the i-1 th layer, i is an integer which is greater than or equal to 2 and smaller than a second preset value, a is greater than 0 and smaller than 1, and the second preset value is equal to the number of layers of the network depth of the repair decoder;

B. adding 1 to i, executing the step A until the value of the i plus 1 is equal to the second preset value, and executing the step C;

C. and carrying out coding processing on the feature map of the ith layer of the coder to obtain the feature map of the (i + 1) th layer, and outputting the feature map of the (i + 1) th layer to the (i + 1) th layer of the repair decoder.

In one possible design, each layer of the repair decoder is configured to perform repair processing on the image to be repaired according to the feature map of the corresponding layer of the encoder and the information of the image to be repaired, to obtain an initial repair image corresponding to each layer of the repair decoder, and output the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder, including:

a', on the ith layer of the restoration decoder, performing first restoration processing on the image to be restored according to the feature map of the ith-1 layer of the encoder, the first initial restoration image of the (i + 1) th layer of the restoration decoder and the information of the image to be restored to obtain the first initial restoration image of the ith layer of the encoder, wherein i is an integer which is greater than or equal to 1 and smaller than the second preset value;

b', splicing the feature map of the first initial repair image of the ith layer with the feature map of the ith-1 layer of the encoder to obtain a second initial repair image of the ith layer of the repair decoder, wherein the size of the feature map of the first initial repair image of the ith layer is 1/a times of the size of the first initial repair image of the ith-1 layer;

c', performing second restoration processing on a second initial restoration image of the ith layer of the restoration decoder to obtain an initial restoration image of the ith layer of the restoration decoder, and outputting the initial restoration image of the ith layer to the (i-1) th layer of the restoration decoder and the (i + 1) th layer of the refinement decoder;

d ', subtracting 1 from i, and executing the steps A ' -C ' until i is 1.

In one possible design, each layer of the refinement decoder is configured to perform refinement and restoration processing on an initial restoration image of a corresponding layer of the restoration decoder to obtain a restoration image corresponding to each layer of the refinement decoder, so as to obtain the restoration image, and the method includes:

a', on the ith layer of the refinement decoder, carrying out refinement and restoration processing on the initial restoration image of the ith-1 layer of the restoration decoder and the restoration image of the ith +1 layer of the refinement decoder to obtain the restoration image of the ith layer of the refinement decoder, and outputting the restoration image of the ith layer to the ith-1 layer of the refinement decoder, wherein i is an integer which is greater than or equal to 2 and smaller than the second preset value;

b ', subtracting 1 from i, and executing the step A' until i is 1.

In one possible design, the encoding process includes a partial convolution process, a batch normalization process, and a collation process, the first repair process includes a transposed convolution process, the batch normalization process, and the collation process, the second repair process includes the partial convolution process and the collation process, and the refined repair process includes a convolution process.

In a possible design, the information of the image to be repaired is a mask of the image to be repaired.

A second aspect of the present application provides an apparatus for image restoration, comprising:

the image restoration device comprises a receiving and sending module, a restoration processing module and a restoration processing module, wherein the receiving and sending module is used for receiving an image to be restored and information of the image to be restored, and the information of the image to be restored indicates a region to be restored of the image to be restored.

The processing module is used for inputting the image to be repaired and the information of the image to be repaired into an image repairing model, obtaining a repaired image obtained after the image to be repaired is repaired, and outputting the repaired image, the image repairing model comprises an encoder, a repairing decoder and a thinning decoder, the encoder is used for encoding the image to be repaired to obtain a feature map of the image to be repaired, the repairing decoder is used for repairing the image to be repaired according to the feature map and the information of the image to be repaired to obtain an initial repaired image, and the thinning decoder is used for thinning and repairing the initial repaired image to obtain the repaired image.

Optionally, the network depths of the encoder and the repair decoder are both multiple layers, and the number of layers of the network depths of the encoder and the repair decoder is equal.

Each layer of the encoder is used for encoding the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and the feature map corresponding to each layer is output to the corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder.

Optionally, the network depths of the refinement decoder are all multiple layers, and the number of layers of the network depths of the refinement decoder and the repair decoder is equal.

Each layer of the refinement decoder is used for performing refinement restoration processing on the initial restoration image of the corresponding layer of the restoration decoder to obtain the restoration image corresponding to each layer of the refinement decoder so as to obtain the restoration image, and the restoration image comprises the restoration image corresponding to each layer of the refinement decoder.

Optionally, the number of refinement decoders is multiple, the network depth of each refinement decoder is multiple layers, and the number of layers of the network depth of each refinement decoder is equal to that of the repair decoder.

Optionally, the encoding processing manner of the encoder includes:

Optionally, the repair processing mode of the repair decoder includes:

d ', subtracting 1 from i, and executing the steps A ' -C ' until i is 1.

Optionally, the refinement and repair processing method of the refinement decoder includes:

b ', subtracting 1 from i, and executing the step A' until i is 1.

Optionally, the encoding process includes a partial convolution process, a batch normalization layer process, and a proofreading process, the first repair process includes a transposed convolution process, the batch normalization process, and the proofreading process, the second repair process includes the partial convolution process and the proofreading process, and the refined repair process includes a convolution process.

Optionally, the information of the image to be repaired is a mask of the image to be repaired.

The beneficial effects of the image restoration apparatus provided by the second aspect and each possible design can be referred to the beneficial effects of the first aspect and each possible design, which are not described herein again.

A third aspect of the present application provides an electronic device comprising: at least one processor and memory; the memory stores computer-executable instructions; the at least one processor executes computer-executable instructions stored by the memory to cause the electronic device to perform the method of image inpainting of the first aspect described above.

A fourth aspect of the present application provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed by a processor, implement the method of image inpainting of the first aspect described above.

Other effects of the above-described alternative will be described below with reference to specific embodiments.

Drawings

The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a method for image restoration provided herein;

fig. 2 is a schematic view of a scene to which the image inpainting method provided in the present application is applied;

FIG. 3 is a first schematic structural diagram of an image restoration model provided in the present application;

FIG. 4 is a schematic diagram illustrating an encoding process of an i-th layer of an encoder provided in the present application;

fig. 5 is a schematic diagram of a repair process of an i-th layer of a repair decoder provided in the present application;

FIG. 6 is a schematic diagram of a refinement and repair process of the ith layer of the refinement decoder provided in the present application;

FIG. 7 is a second schematic structural diagram of an image restoration model provided in the present application;

FIG. 8 is a schematic structural diagram of an image restoration apparatus provided in the present application;

fig. 9 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

For convenience of description of the image restoration method provided in the present application, a method of image restoration in the prior art is first described.

In the prior art, an image restoration method based on pixel blocks with low-dimensional features can be used for restoring an image, and specifically, a pixel block with the highest feature similarity is searched in a low-dimensional feature space of the image to fill up a defect part in the image. However, the method has the disadvantages that semantic information in the image cannot be learned, and the repairing effect on the picture with strong structuralization, such as a human face, a scene and an object, is poor.

In order to solve the above technical problems, the prior art further provides an image restoration method based on a deep learning convolutional network, which can restore an image in a feature space at a high altitude, but the deep learning based on the deep learning convolutional network in the prior art has fewer stages, resulting in low image restoration accuracy.

In order to solve the problems in the prior art, the application provides an image restoration method, which restores an image in a multi-stage mode on the basis of a traditional architecture based on a deep learning convolutional network so as to improve the restoration precision of the image.

It should be understood that the main execution subject of the method for performing image restoration in the present application is an apparatus for image restoration, and the apparatus for image restoration may be an electronic device with processing capability, such as a server or a terminal, and the electronic device may be implemented by any software and/or hardware. Alternatively, the terminal may include, but is not limited to, a mobile terminal or a fixed terminal. The mobile terminal devices include, but are not limited to, a mobile phone, a Personal Digital Assistant (PDA), a tablet computer, a portable device (e.g., a portable computer, a pocket computer, or a handheld computer), and the like. Fixed terminals include, but are not limited to, desktop computers and the like.

The following describes the image restoration method provided by the present application with reference to specific embodiments. Fig. 1 is a schematic flowchart of an embodiment of an image inpainting method provided in the present application. As shown in fig. 1, the method for image restoration provided by this embodiment may include:

s101, receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired.

In this embodiment, a user may input the image to be repaired and information of the image to be repaired to the image repairing apparatus, so that the image repairing apparatus can repair the image to be repaired according to the information of the image to be repaired. The information of the image to be repaired indicates the area to be repaired of the image to be repaired. Optionally, the information of the image to be repaired is a mask of the image to be repaired (the mask marks a defective pixel block in the image to be repaired, 0 indicates a defect, and 1 indicates no defect).

Optionally, the user may input the image to be restored and information of the image to be restored to the image restoration apparatus through the terminal device. Correspondingly, the image restoration device receives the image to be restored and the information of the image to be restored from the terminal equipment. It should be understood that the terminal device may specifically refer to the related description of the terminal device described above. It should be understood that, in this scenario, fig. 2 is a schematic view of a scenario to which the image inpainting method provided in this application is applicable. As shown in fig. 2, the scene includes the terminal device and the apparatus for image restoration. Fig. 2 illustrates an example in which an image restoration apparatus is used as a server.

S102, inputting the image to be repaired and information of the image to be repaired into an image repairing model to obtain a repaired image after the image to be repaired is repaired, wherein the image repairing model comprises an encoder, a repairing decoder and a thinning decoder, the encoder is used for encoding the image to be repaired to obtain a feature map of the image to be repaired, the repairing decoder is used for repairing the image to be repaired according to the feature map and the information of the image to be repaired to obtain an initial repaired image, and the thinning decoder is used for thinning and repairing the initial repaired image to obtain the repaired image.

In this embodiment, an image restoration model is stored in the image restoration apparatus. The image to be restored and the information of the image to be restored can be input into the image restoration model, and the restored image after the image to be restored is obtained. It should be noted that a plurality of decoders are adopted in the image restoration model in the present embodiment to restore an image from a plurality of stages to improve the accuracy of image restoration.

The image restoration model in this embodiment includes an Encoder (Encoder), a restoration Decoder (RecoveryDecoder), and a Refinement Decoder (Refinement Decoder). Specifically, the encoder is configured to perform encoding processing on the image to be restored to obtain a feature map of the image to be restored. The encoder may also be understood as encoding the image to be repaired into an abstract representation, for example, converting the image into a vector, and extracting a feature vector of the vector to represent the image to be repaired, i.e., the feature map in this embodiment.

And the restoration decoder is used for restoring the image to be restored according to the characteristic diagram and the information of the image to be restored to obtain an initial restoration image. It should be understood that the purpose of the repair decoder in this embodiment is to repair the image to be repaired, and the specific repair manner may be the same as that of the decoder in the prior art. Different from the prior art, in the embodiment, the image is repaired in multiple stages, and besides the repair decoder, the image repair model in the embodiment further includes a refinement decoder, where the refinement decoder is configured to perform refinement and repair processing on the initial repair image to obtain the repair image. It should be understood that the above-mentioned refinement and restoration processing may be convolution processing on the initial restored image to implement refinement and restoration on the initial restored image, thereby obtaining a restored image.

And S103, outputting the repaired image.

In this embodiment, the image restoration model may output the restored image, and the image restoration apparatus may output the restored image. Optionally, the outputting of the repaired image by the image repairing apparatus may be displaying the repaired image or sending the displayed repaired image to the terminal device, so that the terminal device displays the repaired image.

The image restoration method provided in this embodiment may input the image to be restored and information of the image to be restored to the image restoration model, and different from the image restoration model in the prior art, the image restoration model in this embodiment includes a plurality of restoration decoders, and the image restoration process may be performed on the image from a plurality of stages, so as to improve the image restoration accuracy.

On the basis of the above embodiments, in order to further improve the accuracy of image restoration, in this embodiment, the network depths of the encoder, the restoration decoder, and the refinement decoder are all set to be multiple layers, and the number of layers of the network depths of the encoder, the restoration decoder, and the refinement decoder is equal. The structure of the image restoration model employed in the present application, as well as the encoder, the restoration decoder, and the refinement decoder are explained below with reference to fig. 3. Fig. 3 is a schematic structural diagram of an image restoration model provided in the present application.

As shown in fig. 3, each layer of the encoder is used for encoding the image to be repaired, and a feature map corresponding to each layer of the encoder is obtained. It should be understood that layer 1 of the encoder may perform encoding processing on the image to be repaired to obtain a feature map of layer 1; layer 2 of the encoder may perform an encoding process on the feature map of layer 1 to obtain a feature map … … of layer 2, and so on, and may obtain a corresponding feature map for each layer of the encoder. It should be understood that the feature map of the image to be repaired in the present embodiment includes a feature map corresponding to each layer of the encoder. Optionally, the feature map of the upper layer of the encoder in this embodiment may be input to the lower layer, so that the lower layer of the encoder performs encoding processing on the feature map of the upper layer of the encoder to obtain the feature map of the lower layer of the encoder.

In this embodiment, the encoder may output the feature map corresponding to each layer to the corresponding layer of the repair decoder. Optionally, the layer corresponding to the layer 1 of the encoder and corresponding to the repair decoder may be the layer 1 or other layers of the repair decoder.

In a possible implementation manner, the i-th layer of the encoder in this embodiment corresponds to the i + 1-th layer of the repair decoder. In this scenario, the processing for each layer of the encoder is as follows:

A. and in the ith layer of the encoder, encoding the characteristic diagram of the ith-1 layer of the encoder to obtain the characteristic diagram of the ith layer, and outputting the characteristic diagram of the ith layer to the (i + 1) th layer of the encoder and the (i + 1) th layer of the repair decoder, wherein the size of the characteristic diagram of the ith layer is a times of the size of the characteristic diagram of the ith-1 layer, i is an integer which is greater than or equal to 2 and less than a preset value, and a is greater than 0 and less than 1.

In step a, at the 2 nd layer of the encoder, the feature map of the 1 st layer of the encoder may be encoded to obtain the feature map of the 2 nd layer, and the feature map of the 2 nd layer of the encoder may be output to the 3 rd layer of the encoder, so that the 3 rd layer of the encoder performs the step in a to obtain the feature map of the 3 rd layer of the encoder. In this embodiment, the feature map of the layer 2 may also be output to the layer 3 of the repair decoder, and the following description of the repair decoder is provided for how the repair decoder performs processing according to the feature map of the layer 2.

In steps a-C of this embodiment, i is an integer greater than or equal to 2 and less than the second predetermined value. Wherein the second preset value is equal to the number of layers of the network depth of the repair decoder. If the second preset value is 7, the number of layers of the network depth of the encoder is 7, and when i is 2 to 6, the method can be executed according to the above step a. In addition, when i is 1, in the present embodiment, in the layer 1 of the encoder, the image to be repaired may be encoded to obtain the feature map of the layer 1, and the feature map of the layer 1 may be output to the layer 2 of the encoder and the layer 2 of the repair decoder.

It is to be understood that the size of the feature map of the i-th layer in this embodiment is a times the size of the feature map of the i-1 th layer, and a is greater than 0 and less than 1. Illustratively, if a is 1/2, then the size of the eigenmap of layer 3 of the encoder is 1/2 of the size of the eigenmap of layer 2 of the encoder.

Optionally, the encoding processing in this embodiment may be partial convolution (PConv), Batch Normalization (BN), and layer correction (Relu) performed in sequence.

B. And adding 1 to i, executing the step A until the value of the i added with 1 is equal to a second preset value, and executing the step C.

In the case where i in the above step a is an integer greater than or equal to 1 and smaller than the second preset value, in this case, if i is 6, i plus 1 is 7, that is, the second preset value, the following step C may be performed.

C. And carrying out coding processing on the feature map of the ith layer of the coder to obtain the feature map of the (i + 1) th layer, and outputting the feature map of the (1) th layer to the (i + 1) th layer of the repair decoder.

In step C, when i is 6, i plus 1 is 7, i is the second preset value. The feature map of layer 6 of the encoder may be encoded to obtain the feature map of layer 7. Since layer 7 is the last layer of the encoder, the feature map of this layer 7 can be output to layer 7 of the repair decoder. It should be understood that the feature map of the layer 7 of the encoder in this embodiment is the feature map of the image to be repaired in the above embodiment.

Fig. 4 is a schematic diagram of an encoding process of an i-th layer of an encoder provided in the present application. As shown in fig. 4, the characteristic diagram of the i-1 th layer of the encoder is sequentially processed by PConv, BN and Relu to obtain the characteristic diagram (m) of the i-th layer of the encoder_e ^l) Output to the i +1 th layer of the encoder and the i +1 th layer of the repair decoder.

Corresponding to the encoder, each layer of the repair decoder in this embodiment is configured to perform repair processing on the image to be repaired according to the feature map of the layer corresponding to the encoder and the information of the image to be repaired, so as to obtain an initial repair image corresponding to each layer of the repair decoder. For example, if the layer corresponding to the encoder corresponding to the layer 2 of the repair decoder is the layer 1 of the encoder, the layer 2 of the repair decoder may perform repair processing on the image to be repaired according to the feature map output by the layer 1 of the encoder and the information of the image to be repaired, specifically, perform repair processing on the image to be repaired at the layer 3 of the repair decoder, so as to obtain an initial repair image corresponding to the layer 2 of the repair decoder. It should be understood that the initial repair picture in the above embodiments includes a corresponding initial repair picture for each layer of the repair decoder.

Further, the repair decoder may output an initial repair image corresponding to each layer to a corresponding layer of the refinement decoder, the initial repair image including the initial repair image corresponding to each layer of the repair decoder. Alternatively, the layer corresponding to the refinement decoder corresponding to layer 1 of the repair decoder may be layer 1 of the refinement decoder or another layer. Optionally, the repair image of the previous layer of the repair decoder in this embodiment may be input to the next layer, so that the next layer of the repair decoder performs repair processing on the feature map of the corresponding layer of the encoder and the initial repair image of the previous layer of the repair decoder to obtain the initial repair image of the next layer of the repair decoder.

In one possible implementation, the i-th layer of the repair decoder in this embodiment corresponds to the i + 1-th layer of the refinement decoder. In this scenario, the processing of each layer of the repair decoder is as follows:

and A', on the ith layer of the restoration decoder, performing first restoration processing on the image to be restored according to the characteristic diagram of the ith-1 layer of the encoder, the first initial restoration image of the (i + 1) th layer of the restoration decoder and the information of the image to be restored to obtain the first initial restoration image of the ith layer of the encoder, wherein i is an integer which is greater than or equal to 1 and smaller than a second preset value.

As shown in fig. 3, the feature map of the layer 6 and the feature map of the layer 7 of the encoder in the present embodiment are both input to the layer 7 of the repair decoder, and the repair decoder performs repair processing in order of the layer 7 to the layer 1.

In the steps a '-D' of the present embodiment, i is an integer greater than or equal to 1 and less than the second preset value. The second preset value is equal to the second preset value of i in the encoder, and is 7 if the second preset value is equal to the number of layers of the network depth of the repair decoder. For example, when i is 6, the first restoration process may be performed on the image to be restored according to the feature map output by the 5 th layer of the encoder, the first initial restoration image of the 7 th layer of the restoration decoder, and the information of the image to be restored, so as to obtain the first initial restoration image of the 6 th layer of the restoration decoder. Optionally, the first repair process in this embodiment is to sequentially perform a transpose convolution (DConv), a batch normalization process, and a proofreading process. It should be understood that when i is 1, the feature map output by the layer 0 of the encoder may be an image to be repaired.

When i is 7, considering that the 7 th layer of the repair decoder inputs the feature maps of the 6 th layer and the 7 th layer from the encoder, at the 7 th layer of the repair decoder, the first repair process may be performed according to the feature map output by the 6 th layer of the encoder, the feature map output by the 7 th layer of the encoder, and the information of the image to be repaired, so as to obtain a first initial repair image of the 7 th layer of the repair decoder.

B', splicing the characteristic diagram of the first initial repaired image of the ith layer with the characteristic diagram of the ith-1 layer of the encoder to obtain a second initial repaired image of the ith layer of the repair decoder, wherein the size of the characteristic diagram of the first initial repaired image of the ith layer is 1/a times of the size of the first initial repaired image of the ith-1 layer.

The feature map of the first initial repair image of the i-th layer of the repair decoder in this embodiment has a size 1/a times the size of the first initial repair image of the i-1-th layer. Illustratively, the feature map of the first initial repair image of layer 6 of the repair decoder has a size 2 times the size of the first initial repair image of layer 7. In this embodiment, the size of the feature map of the first initial repair image of the ith layer of the repair decoder is the same as the size of the feature map of the ith-1 layer of the encoder, so that in this embodiment, the feature map of the first initial repair image of the ith layer of the repair decoder and the feature map of the ith-1 layer of the encoder may be spliced to obtain the second initial repair image of the ith layer of the repair decoder.

For example, in this embodiment, the size of the feature map of the first initial repair image of the 6 th layer of the repair decoder is the same as the size of the feature map of the 5 th layer of the encoder, so in this embodiment, the feature map of the first initial repair image of the 6 th layer of the repair decoder and the feature map of the 5 th layer of the encoder may be spliced to obtain the second initial repair image of the 6 th layer of the repair decoder.

And C', performing second restoration processing on the second initial restoration image of the ith layer of the restoration decoder to obtain the initial restoration image of the ith layer of the restoration decoder, and outputting the initial restoration image of the ith layer to the (i-1) th layer of the restoration decoder and the (i + 1) th layer of the refinement decoder.

Optionally, the second repair process in this embodiment may be a partial convolution process and a proofreading process performed in sequence. In this embodiment, the second initial repair image of the ith layer of the repair decoder may be subjected to the second repair process to obtain the initial repair image of the ith layer of the repair decoder. It should be understood that, in this embodiment, the second initial repair image of each layer of the repair decoder may be subjected to the second repair process, so as to obtain the initial repair image of each layer of the repair decoder.

Further, the initial repair image of the ith layer may be output to the (i-1) th layer of the repair decoder and the (i + 1) th layer of the refinement decoder in this embodiment. Since i in the present embodiment is an integer greater than or equal to 1 and less than the second preset value, the initial repair image of the 7 th layer of the repair encoder may be output to the 6 th layer of the repair decoder, and the 7 th layer of the refinement decoder.

D ', subtracting 1 from i, and executing the steps A ' -C ' until i is 1.

It should be understood that, according to the above-described related explanation, the repair decoder sequentially performs the repair process in the order of layer 7 to layer 1. Therefore, in the present embodiment, according to the above description, the above steps a '-C' may be sequentially performed after obtaining the initial repair image of the 7 th layer of the repair decoder, to obtain the initial repair image of the 6 th layer, the initial repair image … … of the 5 th layer, and the initial repair image of the 1 st layer. It should be understood that the initial repair image of the layer 1 of the repair decoder in this embodiment is the initial repair image of the image to be repaired in the above embodiment.

Fig. 5 is a schematic diagram of a repair process of an i-th layer of a repair decoder according to the present application. As shown in fig. 5, the initial repair image (r) for the i-th layer of the repair decoder^l) After DConv, BN and Relu in sequence, a first initial repair image (m) of the i-th layer of the repair decoder is obtained_r ^l) After the feature of the first initial repair image is fused with the feature map of the i-1 layer of the encoder, a second initial repair image (m) of the i-layer of the repair decoder can be obtained_r ^l-1) And further the second initial repair image (m)_r ^l-1) After PConv, Relu, an initial repair image (r) of layer i-1 of the repair decoder can be obtained^l-1)。

Corresponding to the encoder and the repair decoding, each layer of the refinement decoder in this embodiment is configured to perform refinement and repair processing on the initial repair image of the corresponding layer of the repair decoder, so as to obtain a repair image corresponding to each layer of the refinement decoder, so as to obtain a repair image. Wherein the repair picture comprises a repair picture corresponding to each layer of the refinement decoder. Optionally, the repaired image of the previous layer of the refinement decoder in this embodiment may be input to the next layer, so that the next layer of the refinement decoder performs refinement and repair processing on the initial repaired image of the corresponding layer of the refinement decoder and the repaired image of the previous layer of the refinement decoder to obtain the repaired image of the next layer of the refinement decoder.

In one possible implementation, the i-th layer of the repair decoder in this embodiment corresponds to the i + 1-th layer of the refinement decoder. In this scenario, the processing of each layer of the refinement decoder is as follows:

a', thinning and repairing the initial repaired image of the i-1 th layer of the repaired decoder and the repaired image of the i +1 th layer of the thinned decoder at the i-th layer of the thinned decoder to obtain the repaired image of the i-th layer of the thinned decoder, and outputting the repaired image of the i-th layer to the i-1 th layer of the thinned decoder, wherein i is an integer which is more than or equal to 2 and less than a second preset value.

As shown in fig. 3, the feature map of the 6 th layer of the repair decoder and the initial repair image of the 7 th layer in the present embodiment are both input to the 7 th layer of the refinement decoder, and the refinement decoder performs the repair process in order of the 7 th layer to the 1 st layer.

In step a "-B" of this embodiment, i is an integer greater than or equal to 2 and smaller than the second preset value. The second preset value is equal to the second preset value of i in the encoder, and is equal to 7 if the number of layers of the network depth of the refinement decoder is equal to the second preset value. For example, when i is 6, the refinement and restoration processing may be performed on the initial restored image output by the 5 th layer of the restoration decoder to obtain the restored image of the 6 th layer of the refinement decoder. It is to be understood that when i is 7, the refinement restoration process may be performed on the initial restoration image output by the layer 6 of the restoration decoder and the initial restoration image output by the layer 7 of the restoration decoder to obtain the restoration image of the layer 7 of the refinement decoder.

When i is 1, in this embodiment, a refinement restoration process may be performed on a restoration image output by the layer 2 of the refinement decoder to obtain a restoration image of the layer 1 of the refinement decoder, where the restoration image of the layer 1 of the refinement decoder is the restoration image of the image to be restored in the above embodiment, that is, the restoration image output by the image restoration model.

Optionally, the refinement and repair processing in this embodiment is convolution (Conv).

B ', subtracting 1 from i, and executing the step A' until i is 1.

It should be understood that, according to the above-described related explanation, the repair decoder sequentially performs the repair process in the order of layer 7 to layer 1. Therefore, in the present embodiment, according to the above description, after the restored image of the 7 th layer of the refinement decoder is obtained, the above steps a '-C' are sequentially performed to obtain the restored image of the 6 th layer, the restored image … … of the 5 th layer, and the restored image of the 1 st layer.

In this embodiment, each stage of the image restoration model, for example, the neural networks of the encoder, the restoration decoder, and the refinement decoder are all multilayer, so that the image restoration model in this embodiment can restore an image on multiple scales, and the accuracy of image restoration is further improved.

Fig. 6 is a schematic diagram of a refinement and repair process of the ith layer of the refinement decoder provided in the present application. As shown in fig. 6, the restored image (f) of the i-th layer of the refinement decoder^l) After Conv, a restored image (f) of the i-1 th layer of the refinement decoder is obtained^l-1)。

On the basis of the above-described embodiment, in order to further improve the accuracy of image inpainting, a plurality of refinement decoders may be provided in the present embodiment. The network depth of each refinement decoder is multilayer, and the number of layers of the network depth of each refinement decoder is equal to that of the repair decoder. That is, in this embodiment, the initial repair image may be subjected to a plurality of thinning repair processes to improve the accuracy of image repair.

Fig. 7 is a schematic structural diagram of an image restoration model provided in the present application. In this embodiment, when there are a plurality of refinement decoders, the repair decoder outputs an initial repair image corresponding to each layer to a corresponding layer of the 1 st refinement decoder. The specific way in which the repair decoder outputs the initial repair image corresponding to each layer to the corresponding layer of the 1 st refinement decoder may refer to the above-mentioned description about the way in which the repair decoder outputs the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder.

The processing mode of each layer of each refinement decoder is specifically as follows:

a. each layer of the kth thinning decoder is used for thinning and repairing the initial repairing image of the corresponding layer of the repairing decoder to obtain the repairing image corresponding to each layer of the kth thinning decoder, the repairing image corresponding to each layer of the kth thinning decoder is output to the corresponding layer of the (k + 1) th thinning decoder, and k is an integer greater than or equal to 1.

In step a, for example, when k is 1, each layer of the 1 st refinement decoder processes the initial repair image in the same way as each layer of the refinement decoder processes the initial repair image, and specifically, reference may be made to the above-mentioned description about a "-B". Unlike in the above, the repaired image corresponding to each layer of the kth refinement decoder may be output to the corresponding layer of the (k + 1) th refinement decoder in the present embodiment.

It is to be understood that the i-th layer of the kth refinement decoder and the corresponding layer of the (k + 1) -th refinement decoder are the i + 1-th layer of the (k + 1) -th refinement decoder. Illustratively, the restored image corresponding to the 1 st layer of the kth refinement decoder may be output to the 2 nd layer of the (k + 1) th refinement decoder as in the present embodiment. Wherein, the processing mode of each layer of the (k + 1) th refinement decoder on the initial repair image can be as follows:

and the 7 th layer of the (k + 1) th refinement decoder performs refinement and restoration processing on the restored images output by the 6 th layer and the 7 th layer of the (k + 1) th refinement decoder to obtain a restored image of the 7 th layer of the (k + 1) th refinement decoder, and outputs the restored image of the 7 th layer to the 6 th layer of the (k + 1) th refinement decoder. And the 6 th layer carries out thinning and repairing treatment on the repaired images output by the 5 th layer of the kth thinning decoder and the 7 th layer of the k +1 th thinning decoder to obtain the repaired image … … of the 5 th layer of the k +1 th thinning decoder, and the like to obtain the repaired image of the 2 nd layer of the k +1 th thinning decoder. It is worth noting that the layer 1 of the (k + 1) th refinement decoder performs refinement and repair processing on the repaired images output by the layer 0 of the (k + 1) th refinement decoder and the layer 2 of the (k + 1) th refinement decoder to obtain the repaired image of the layer 1 of the (k + 1) th refinement decoder, wherein the repaired image of the layer 0 of the (k + 1) th refinement decoder is the repaired image of the (k) th refinement decoder.

b. And b, adding 1 to k, and executing the step a until k is a first preset value to obtain a repaired image, wherein the repaired image comprises the repaired image of each layer of each thinning decoder.

In this embodiment, the first preset value of k may be different from the second preset value of i. As illustrated in fig. 7 by taking k as 2, i.e. the refinement decoder as two examples. It should be understood that, in this embodiment, when k is a first preset value, the repaired image output by the layer 1 of the kth refinement decoder is the repaired image output by the image repair model.

In this embodiment, a plurality of refinement restoration models may be set, so that further refinement restoration of the restored image may be realized, and the accuracy of image restoration may be further improved.

Fig. 8 is a schematic structural diagram of an image restoration apparatus provided in the present application. As shown in fig. 8, the image restoration apparatus 800 includes: a transceiver module 801 and a processing module 802.

The transceiver module 801 is configured to receive an image to be repaired and information of the image to be repaired, where the information of the image to be repaired indicates a region to be repaired of the image to be repaired.

The processing module 802 is configured to input the image to be restored and information of the image to be restored into an image restoration model, obtain a restored image after the image to be restored is restored, and output the restored image, where the image restoration model includes an encoder, a restoration decoder, and a refinement decoder, the encoder is configured to perform encoding processing on the image to be restored, obtain a feature map of the image to be restored, the restoration decoder is configured to perform restoration processing on the image to be restored according to the feature map and the information of the image to be restored, obtain an initial restored image, and the refinement decoder is configured to perform refinement restoration processing on the initial restored image, and obtain a restored image.

Each layer of the encoder is used for encoding an image to be repaired to obtain a feature map corresponding to each layer of the encoder, and the feature map corresponding to each layer is output to a corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder;

and each layer of the restoration decoder is used for restoring the image to be restored according to the feature map of the corresponding layer of the encoder and the information of the image to be restored to obtain an initial restoration image corresponding to each layer of the restoration decoder, and outputting the initial restoration image corresponding to each layer to the corresponding layer of the refinement decoder, wherein the initial restoration image comprises the initial restoration image corresponding to each layer of the restoration decoder.

Each layer of the refinement decoder is used for carrying out refinement and restoration processing on the initial restoration image of the corresponding layer of the restoration decoder to obtain the restoration image corresponding to each layer of the refinement decoder so as to obtain the restoration image, and the restoration image comprises the restoration image corresponding to each layer of the refinement decoder.

Optionally, the number of refinement decoders is multiple, the network depth of each refinement decoder is multiple layers, and the number of layers of the network depth of each refinement decoder and the number of layers of the network depth of the repair decoder are equal.

b. and b, adding 1 to k, and executing the step a until k is a first preset value to obtain a repaired image, wherein the repaired image comprises the repaired image of each layer of each thinning decoder, and the number of the first preset values is equal to that of the thinning decoders.

Optionally, the encoding processing manner of the encoder includes:

A. coding the characteristic diagram of the i-1 th layer of the encoder to obtain the characteristic diagram of the i-th layer, and outputting the characteristic diagram of the i-th layer to the i +1 th layer of the encoder and the i +1 th layer of the repair decoder, wherein the size of the characteristic diagram of the i-th layer is a times of the size of the characteristic diagram of the i-1 th layer, i is an integer which is greater than or equal to 2 and smaller than a second preset value, a is greater than 0 and smaller than 1, and the second preset value is equal to the number of layers of the network depth of the repair decoder;

B. adding 1 to i, executing the step A until the value of the added 1 to i is equal to a second preset value, and executing the step C;

Optionally, the repair processing mode of the repair decoder includes:

a', on the ith layer of a repair decoder, performing first repair processing on an image to be repaired according to a feature map of the (i-1) th layer of an encoder, a first initial repair image of the (i + 1) th layer of the repair decoder and information of the image to be repaired to obtain the first initial repair image of the ith layer of the encoder, wherein i is an integer which is greater than or equal to 1 and smaller than a second preset value;

b', splicing the characteristic diagram of the first initial repaired image of the ith layer with the characteristic diagram of the ith-1 layer of the encoder to obtain a second initial repaired image of the ith layer of the repair decoder, wherein the size of the characteristic diagram of the first initial repaired image of the ith layer is 1/a times of that of the first initial repaired image of the ith-1 layer;

d ', subtracting 1 from i, and executing the steps A ' -C ' until i is 1.

a', on the ith layer of a thinning decoder, thinning and repairing the initial repaired image of the ith-1 layer of the repairing decoder and the repaired image of the (i + 1) th layer of the thinning decoder to obtain the repaired image of the ith layer of the thinning decoder, and outputting the repaired image of the ith layer to the ith-1 layer of the thinning decoder, wherein i is an integer which is more than or equal to 2 and less than a second preset value;

b ', subtracting 1 from i, and executing the step A' until i is 1.

Optionally, the encoding processing includes partial convolution processing, batch normalization processing, and collation processing, the first repair processing includes transposition convolution processing, batch normalization processing, and collation processing, the second repair processing includes partial convolution processing and collation processing, and the refinement repair processing includes convolution processing.

The image restoration apparatus provided in this embodiment is similar to the principle and the technical effect of the image restoration method, and is not described herein again.

According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.

Fig. 9 is a schematic structural diagram of an electronic device provided in the present application, and as shown in fig. 9, is a block diagram of an electronic device of an image repairing method according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.

As shown in fig. 9, the electronic apparatus includes: one or more processors 901, memory 902, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 9 illustrates an example of a processor 901.

Memory 902 is a non-transitory computer readable storage medium as provided herein. Wherein the memory stores instructions executable by at least one processor to cause the at least one processor to perform the method of image inpainting provided herein. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method of image inpainting provided herein.

The memory 902, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the method of image inpainting in the embodiments of the present application. The processor 901 executes various functional applications of the server and sample processing, i.e., a method of implementing image inpainting in the above-described method embodiments, by executing non-transitory software programs, instructions, and modules stored in the memory 902.

The memory 902 may include a storage program area and a storage sample area, wherein the storage program area may store an operating system, an application program required for at least one function; the stored sample area may store a sample created according to use of an electronic device for performing the method for image restoration, or the like. Further, the memory 902 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 902 may optionally include memory located remotely from the processor 901, which may be connected via a network to an electronic device for performing the method of image inpainting. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device of the method of image inpainting may further include: an input device 903 and an output device 904. The processor 901, the memory 902, the input device 903 and the output device 904 may be connected by a bus or other means, and fig. 9 illustrates the connection by a bus as an example.

The input device 903 may receive input numeric or character information and generate key signal inputs related to user settings and function control of an electronic apparatus for performing the method of image inpainting, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or the like. The output devices 904 may include a display device, auxiliary lighting devices (e.g., LEDs), tactile feedback devices (e.g., vibrating motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving samples and instructions from, and transmitting samples and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or samples to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or samples to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a sample server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital sample communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.

The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of image inpainting, comprising:

receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired;

inputting the image to be restored and the information of the image to be restored into an image restoration model to obtain a restored image obtained after restoring the image to be restored, wherein the image restoration model comprises an encoder, a restoration decoder and a refinement decoder, the encoder is used for encoding the image to be restored to obtain a feature map of the image to be restored, the restoration decoder is used for restoring the image to be restored according to the feature map and the information of the image to be restored to obtain an initial restored image, and the refinement decoder is used for refining and restoring the initial restored image to obtain the restored image;

and outputting the repaired image.

2. The method of claim 1, wherein the network depths of the encoder and the repair decoder are both multi-layered, and the number of layers of the network depths of the encoder and the repair decoder are equal;

each layer of the encoder is used for encoding the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and the feature map corresponding to each layer is output to the corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder;

3. The method of claim 2, wherein the network depth of the refinement decoder is multi-layered, and the number of layers of network depths of the refinement decoder and the repair decoder are equal;

4. The method according to claim 2, wherein the refinement decoder is plural, the network depth of each refinement decoder is plural layers, and the number of layers of the network depth of each refinement decoder and the repair decoder is equal;

5. The method according to claim 3 or 4, wherein each layer of the encoder is configured to perform encoding processing on the image to be repaired, obtain a feature map corresponding to each layer of the encoder, and output the feature map corresponding to each layer to a corresponding layer of the repair decoder, and the method includes:

6. The method according to claim 5, wherein each layer of the repair decoder is configured to perform repair processing on the image to be repaired according to the feature map of the corresponding layer of the encoder and the information of the image to be repaired, to obtain an initial repair image corresponding to each layer of the repair decoder, and output the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder, and the method includes:

d ', subtracting 1 from i, and executing the steps A ' -C ' until i is 1.

7. The method according to claim 6, wherein each layer of the refinement decoder is configured to perform refinement and restoration processing on an initial restoration image of a corresponding layer of the restoration decoder to obtain a restoration image corresponding to each layer of the refinement decoder to obtain the restoration image, and the method comprises:

b ', subtracting 1 from i, and executing the step A' until i is 1.

8. The method of claim 6, wherein the encoding process comprises a partial convolution process, a batch normalization layer process, and a collation process, wherein the first repair process comprises a transposed convolution process, the batch normalization process, and the collation process, wherein the second repair process comprises the partial convolution process and the collation process, and wherein the refined repair process comprises a convolution process.

9. The method according to any one of claims 1 to 8, wherein the information of the image to be repaired is a mask of the image to be repaired.

10. An apparatus for image restoration, comprising:

the image restoration device comprises a receiving and sending module, a restoration processing module and a restoration processing module, wherein the receiving and sending module is used for receiving an image to be restored and information of the image to be restored, and the information of the image to be restored indicates a region to be restored of the image to be restored;

11. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-9.