CN111242874B

CN111242874B - Image restoration method, device, electronic equipment and storage medium

Info

Publication number: CN111242874B
Application number: CN202010087216.1A
Authority: CN
Inventors: 朱曼瑜; 刘霄; 文石磊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2020-02-11
Filing date: 2020-02-11
Publication date: 2023-08-29
Anticipated expiration: 2040-02-11
Also published as: CN111242874A

Abstract

The application discloses an image restoration method, an image restoration device, electronic equipment and a storage medium, wherein the method comprises the following steps: receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates an area to be repaired of the image to be repaired; inputting the image to be repaired and the information of the image to be repaired into an image repairing model to obtain a repaired image after the image to be repaired is repaired, wherein the image repairing model comprises an encoder, a repairing decoder and a thinning decoder, the encoder is used for carrying out encoding processing on the image to be repaired to obtain a characteristic map of the image to be repaired, the repairing decoder is used for carrying out repairing processing on the image to be repaired according to the characteristic map and the information of the image to be repaired to obtain an initial repairing image, the thinning decoder is used for carrying out thinning repairing processing on the initial repairing image to obtain a repairing image, and the repairing image is output. According to the image restoration method, the image restoration model restores the image from a plurality of stages, so that the accuracy of image restoration is improved.

Description

Image restoration method, device, electronic equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method and apparatus for repairing an image, an electronic device, and a storage medium.

Background

In order to reduce bandwidth resources, in the process of transmitting an image, the image is often subjected to processing such as compression and resolution reduction, and the image may also be subjected to non-artificial disturbance noise in the process of transmitting, and the image damage is caused by the processing and noise disturbance of the image. In order for a user to acquire a clear, high quality image, it is necessary to perform image restoration of the damaged image or the low quality image.

In the prior art, an image is usually repaired based on a deep learning convolutional neural network, specifically, semantic information of the image is learned from large-scale image data, and a defect part is filled by using pixel blocks with similar semantics. However, the convolutional neural network in the prior art has few stages of deep learning, resulting in low restoration accuracy of images.

Disclosure of Invention

The application provides an image restoration method, an image restoration device, electronic equipment and a storage medium, which can restore images from a plurality of stages and improve the accuracy of image restoration.

The first aspect of the present application provides a method for image restoration, comprising:

receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired; inputting the image to be repaired and the information of the image to be repaired into an image repairing model to obtain a repaired image after repairing the image to be repaired, wherein the image repairing model comprises an encoder, a repairing decoder and a thinning decoder, the encoder is used for encoding the image to be repaired to obtain a characteristic diagram of the image to be repaired, the repairing decoder is used for repairing the image to be repaired according to the characteristic diagram and the information of the image to be repaired to obtain an initial repairing image, and the thinning decoder is used for thinning the initial repairing image to obtain the repairing image; and outputting the repair image.

The image restoration model in this embodiment includes a plurality of restoration decoders, and restoration processing can be performed on an image from a plurality of stages to improve restoration accuracy of the image.

In one possible design, the network depths of the encoder and the repair decoder are all multiple layers, and the layers of the network depths of the encoder and the repair decoder are equal; each layer of the encoder is used for encoding the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and outputting the feature map corresponding to each layer to the corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder.

Each layer of the repair decoder is used for performing repair processing on the image to be repaired according to the characteristic diagram of the corresponding layer of the encoder and the information of the image to be repaired to obtain an initial repair image corresponding to each layer of the repair decoder, and outputting the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder, wherein the initial repair image comprises the initial repair image corresponding to each layer of the repair decoder.

In one possible design, the network depths of the refinement decoder are all multiple layers, and the number of layers of the network depths of the refinement decoder and the repair decoder are equal; each layer of the refinement decoder is used for performing refinement repair processing on an initial repair image of a corresponding layer of the refinement decoder to obtain a repair image corresponding to each layer of the refinement decoder so as to obtain the repair image, wherein the repair image comprises the repair image corresponding to each layer of the refinement decoder.

In this design, the network depth of each refinement decoder is multi-layered, and the number of layers of the network depth of each refinement decoder and repair decoder is equal. That is, in this embodiment, the initial repair image may be subjected to multiple refinement repair processes to improve the accuracy of image repair.

In one possible design, the refinement decoder is a plurality of, the network depth of each of the refinement decoders is a plurality of layers, and the number of layers of the network depth of each of the refinement decoders and the repair decoder is equal.

a. Each layer of the kth thinning decoder is used for carrying out thinning repair processing on an initial repair image of a corresponding layer of the repair decoder to obtain a repair image corresponding to each layer of the kth thinning decoder, and outputting the repair image corresponding to each layer of the kth thinning decoder to a corresponding layer of the kth+1th thinning decoder, wherein k is an integer greater than or equal to 1;

b. and d, adding 1 to k, and executing the step a until the k is a first preset value, so as to obtain the repair image, wherein the repair image comprises repair images of each layer of each refinement decoder, and the first preset value is equal to the number of the refinement decoders.

In the design, a plurality of refinement repair models can be arranged, further refinement repair of the repair image can be realized, and the accuracy of image repair can be further improved.

In one possible design, each layer of the encoder is configured to encode the image to be repaired, obtain a feature map corresponding to each layer of the encoder, and output the feature map corresponding to each layer to a corresponding layer of the repair decoder, where the method includes:

A. coding the characteristic diagram of the ith layer-1 layer of the encoder in the ith layer of the encoder to obtain the characteristic diagram of the ith layer, outputting the characteristic diagram of the ith layer to the (i+1) th layer of the encoder and the (i+1) th layer of the repair decoder, wherein the size of the characteristic diagram of the ith layer is a times of the size of the characteristic diagram of the ith layer-1 layer, i is an integer which is more than or equal to 2 and less than a second preset value, a is more than 0 and less than 1, and the second preset value is equal to the number of layers of the network depth of the repair decoder;

B. adding 1 to i, executing the step A until the i is added with 1 to be equal to the second preset value, and executing the step C;

C. and carrying out coding processing on the characteristic diagram of the ith layer of the coder to obtain the characteristic diagram of the (i+1) th layer, and outputting the characteristic diagram of the (i+1) th layer to the (i+1) th layer of the repair decoder.

In one possible design, each layer of the repair decoder is configured to perform repair processing on the image to be repaired according to the feature map of the corresponding layer of the encoder and the information of the image to be repaired, to obtain an initial repair image corresponding to each layer of the repair decoder, and output the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder, where the method includes:

a', carrying out first restoration processing on the image to be restored according to a feature map of the ith layer-1 of the encoder, a first initial restoration image of the (i+1) th layer of the restoration decoder and information of the image to be restored in the ith layer of the restoration decoder to obtain a first initial restoration image of the ith layer of the encoder, wherein i is an integer which is greater than or equal to 1 and smaller than a second preset value;

b', splicing the characteristic diagram of the first initial repair image of the ith layer and the characteristic diagram of the i-1 th layer of the encoder to obtain a second initial repair image of the ith layer of the repair decoder, wherein the size of the characteristic diagram of the first initial repair image of the ith layer is 1/a times of that of the first initial repair image of the i-1 th layer;

C', performing second restoration processing on a second initial restoration image of an ith layer of the restoration decoder to obtain an initial restoration image of the ith layer of the restoration decoder, and outputting the initial restoration image of the ith layer to an ith-1 layer of the restoration decoder and an (i+1) th layer of the refinement decoder;

d ', subtracting 1 from i, and performing the steps a ' -C ' until i is 1.

In one possible design, each layer of the refinement decoder is configured to perform a refinement repair process on an initial repair image of a corresponding layer of the repair decoder, to obtain a repair image corresponding to each layer of the refinement decoder, so as to obtain the repair image, where the refinement decoder includes:

a', at the ith layer of the thinning decoder, performing thinning repair processing on an initial repair image of the ith-1 layer of the thinning decoder and a repair image of the (i+1) th layer of the thinning decoder to obtain a repair image of the ith layer of the thinning decoder, and outputting the repair image of the ith layer to the ith-1 layer of the thinning decoder, wherein i is an integer which is more than or equal to 2 and less than the second preset value;

b ", subtracting 1 from i, and performing the step a" until i is 1.

In one possible design, the encoding process includes a partial convolution process, a batch normalization layer process, and a collation process, the first repair process includes a transpose convolution process, the batch normalization process, and the collation process, the second repair process includes the partial convolution process and the collation process, and the refinement repair process includes a convolution process.

In one possible design, the information of the image to be repaired is a mask of the image to be repaired.

A second aspect of the present application provides an apparatus for image restoration, comprising:

the receiving and transmitting module is used for receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired.

The processing module is used for inputting the image to be repaired and the information of the image to be repaired into an image repairing model to obtain a repaired image after repairing the image to be repaired, and outputting the repaired image, wherein the image repairing model comprises an encoder, a repairing decoder and a thinning decoder, the encoder is used for carrying out encoding processing on the image to be repaired to obtain a characteristic diagram of the image to be repaired, the repairing decoder is used for carrying out repairing processing on the image to be repaired according to the characteristic diagram and the information of the image to be repaired to obtain an initial repairing image, and the thinning decoder is used for carrying out thinning repairing processing on the initial repairing image to obtain the repairing image.

Optionally, the network depths of the encoder and the repair decoder are all multiple layers, and the layers of the network depths of the encoder and the repair decoder are equal.

Each layer of the encoder is used for encoding the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and outputting the feature map corresponding to each layer to the corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder.

Optionally, the network depth of the refinement decoder is multi-layered, and the layers of the network depths of the refinement decoder and the repair decoder are equal.

Each layer of the refinement decoder is used for performing refinement repair processing on an initial repair image of a corresponding layer of the refinement decoder to obtain a repair image corresponding to each layer of the refinement decoder so as to obtain the repair image, wherein the repair image comprises the repair image corresponding to each layer of the refinement decoder.

Optionally, the refinement decoder is multiple, the network depth of each refinement decoder is multiple layers, and the layers of the network depth of each refinement decoder and the repair decoder are equal.

Optionally, the encoding processing mode of the encoder includes:

Optionally, the repair processing manner of the repair decoder includes:

D ', subtracting 1 from i, and performing the steps a ' -C ' until i is 1.

Optionally, the refinement repair processing mode of the refinement decoder includes:

b ", subtracting 1 from i, and performing the step a" until i is 1.

Optionally, the encoding process includes a partial convolution process, a batch normalization layer process, and a collation process, the first repair process includes a transpose convolution process, the batch normalization process, and the collation process, the second repair process includes the partial convolution process and the collation process, and the refinement repair process includes a convolution process.

Optionally, the information of the image to be repaired is a mask of the image to be repaired.

The advantages of the image restoration device provided by the second aspect and the possible designs can be seen from the first aspect and the advantages of the possible designs, which are not described herein.

A third aspect of the present application provides an electronic apparatus comprising: at least one processor and memory; the memory stores computer-executable instructions; the at least one processor executes computer-executable instructions stored by the memory to cause the electronic device to perform the method of image restoration of the first aspect described above.

A fourth aspect of the present application provides a computer readable storage medium having stored thereon computer executable instructions which, when executed by a processor, implement the method of image restoration of the first aspect described above.

Other effects of the above alternative will be described below in connection with specific embodiments.

Drawings

The drawings are included to provide a better understanding of the present application and are not to be construed as limiting the application. Wherein:

FIG. 1 is a flow chart of an embodiment of a method for image restoration provided by the present application;

FIG. 2 is a schematic diagram of a scenario in which the image restoration method provided by the present application is applicable;

FIG. 3 is a schematic diagram of an image restoration model according to the present application;

FIG. 4 is a schematic diagram of an encoding process of an ith layer of the encoder according to the present application;

FIG. 5 is a schematic diagram of a repair process of an ith layer of the repair decoder according to the present application;

FIG. 6 is a schematic diagram of a refinement repair process of an ith layer of a refinement decoder provided by the present application;

FIG. 7 is a schematic diagram II of an image restoration model according to the present application;

FIG. 8 is a schematic diagram of an apparatus for image restoration according to the present application;

fig. 9 is a schematic structural diagram of an electronic device provided by the present application.

Detailed Description

Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In order to facilitate the description of the image restoration method provided by the application, the image restoration method in the prior art will be described first.

In the prior art, an image can be repaired based on an image repairing method of a pixel block with low dimension characteristics, specifically, in a low dimension characteristic space of the image, the pixel block with the highest characteristic similarity is searched for to fill a defect part in the image. However, the method has the defect that semantic information in the image cannot be learned, and the repair effect on pictures with strong structures, such as faces, scenes and objects, is poor.

In order to solve the technical problems, the prior art also provides an image restoration method based on a deep learning convolutional network, which can restore images in a feature space with high latitude, but has few stages of deep learning based on the deep learning convolutional network in the prior art, so that the restoration accuracy of the images is low.

In order to solve the problems in the prior art, the application provides an image restoration method, which restores images in a multi-stage mode on the traditional architecture based on a deep learning convolution network so as to improve the restoration accuracy of the images.

It should be understood that the main execution body of the method for executing image restoration in the present application is an image restoration device, and the image restoration device may be an electronic device with processing capability, such as a server, a terminal, etc., and the electronic device may be implemented by any software and/or hardware. Alternatively, the terminal may include, but is not limited to being, a mobile terminal or a fixed terminal. Mobile terminal devices include, but are not limited to, cell phones, personal digital assistants (Personal Digital Assistant, abbreviated to: PDAs), tablet computers, portable devices (e.g., portable computers, pocket computers, or hand-held computers), and the like. Fixed terminals include, but are not limited to, desktop computers and the like.

The method of image restoration provided by the application is described below with reference to specific examples. Fig. 1 is a flowchart of an embodiment of a method for image restoration provided by the present application. As shown in fig. 1, the method for repairing an image provided in this embodiment may include:

s101, receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates an area to be repaired of the image to be repaired.

In this embodiment, a user may input an image to be repaired and information of the image to be repaired to an image repairing device, so that the image repairing device may repair the image to be repaired according to the information of the image to be repaired. Wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired. Optionally, the information of the image to be repaired is a mask of the image to be repaired (mask marks a defective pixel block in the image to be repaired, 0 indicates a defect, and 1 indicates no defect).

Optionally, the user may input the image to be repaired and information of the image to be repaired to the image repairing device through the terminal device. Correspondingly, the image restoration device receives the image to be restored and the information of the image to be restored from the terminal equipment. It will be appreciated that the terminal device may be referred to in particular in relation to the description of the terminal device described above. It should be understood that, in this scenario, fig. 2 is a schematic diagram of a scenario where the method for image restoration provided by the present application is applicable. As shown in fig. 2, the scene includes the terminal device and the image restoration apparatus. The image restoration device is illustrated as a server in fig. 2.

S102, inputting the image to be repaired and information of the image to be repaired into an image repair model to obtain a repaired image after the image to be repaired is repaired, wherein the image repair model comprises an encoder, a repair decoder and a refinement decoder, the encoder is used for carrying out encoding processing on the image to be repaired to obtain a feature map of the image to be repaired, the repair decoder is used for carrying out repair processing on the image to be repaired according to the feature map and the information of the image to be repaired to obtain an initial repair image, and the refinement decoder is used for carrying out refinement repair processing on the initial repair image to obtain the repair image.

In this embodiment, the image restoration device stores an image restoration model. The image to be repaired and the information of the image to be repaired can be input into an image repairing model to obtain a repaired image after the image to be repaired is repaired. It should be noted that a plurality of decoders are employed in the image restoration model in the present embodiment to restore an image from a plurality of stages to improve the accuracy of image restoration.

The image restoration model in the present embodiment includes an Encoder (Encoder), a restoration Decoder (restoration Decoder), and a refinement Decoder (Refinement Decoder), among others. Specifically, the encoder is used for encoding the image to be repaired to obtain a characteristic diagram of the image to be repaired. An encoder may also be understood as encoding the image to be repaired into an abstract representation, such as converting the image into vectors, to extract feature vectors of the vectors to represent the image to be repaired, i.e. the feature map in this embodiment.

And the restoration decoder is used for carrying out restoration processing on the image to be restored according to the information of the feature map and the image to be restored to obtain an initial restoration image. It should be understood that the purpose of the repair decoder in this embodiment is to repair the image to be repaired, where the specific repair manner may be the same as that of the decoder in the prior art. Unlike the prior art, in this embodiment, the image is restored by using multiple stages, and in addition to the restoration decoder described above, the image restoration model in this embodiment further includes a refinement decoder, where the refinement decoder is configured to perform a refinement restoration process on the initial restoration image, so as to obtain a restoration image. It should be understood that the above thinning repair process may be a convolution process performed on the initial repair image, so as to implement thinning repair on the initial repair image, thereby obtaining the repair image.

S103, outputting a repair image.

In this embodiment, the image restoration model may output a restoration image, and the image restoration device may output the restoration image. Alternatively, the means for image restoration may output the restoration image as the display restoration image or send the display restoration image to the terminal device, so that the terminal device displays the restoration image.

In the method for repairing an image provided in this embodiment, an image to be repaired and information of the image to be repaired may be input into an image repairing model, unlike the image repairing model in the prior art, the image repairing model in this embodiment includes a plurality of repairing decoders, and repairing processing may be performed on the image from a plurality of stages, so as to improve repairing accuracy of the image.

On the basis of the above embodiment, in order to further improve the accuracy of image restoration, the network depths of the encoder, the restoration decoder, and the refinement decoder in this embodiment are all set to be multiple layers, and the number of layers of the network depths of the encoder, the restoration decoder, and the refinement decoder are equal. The structure of the image restoration model employed in the present application, and the encoder, restoration decoder, and refinement decoder will be described below with reference to fig. 3. Fig. 3 is a schematic structural diagram of an image restoration model provided by the present application.

As shown in fig. 3, each layer of the encoder is used for encoding the image to be repaired, so as to obtain a feature map corresponding to each layer of the encoder. It should be understood that layer 1 of the encoder may encode the image to be repaired to obtain a layer 1 feature map; the layer 2 of the encoder may encode the layer 1 feature map to obtain a layer 2 feature map … …, and so on, may obtain a feature map corresponding to each layer of the encoder. It should be understood that the feature map of the image to be repaired in this embodiment includes a feature map corresponding to each layer of the encoder. Alternatively, the feature map of the upper layer of the encoder in this embodiment may be input to the lower layer, so that the lower layer of the encoder performs encoding processing on the feature map of the upper layer of the encoder to obtain the feature map of the lower layer of the encoder.

In this embodiment, the encoder may output the feature map corresponding to each layer to the corresponding layer of the repair decoder. Alternatively, the corresponding layer of the repair decoder corresponding to layer 1 of the encoder may be layer 1 or other layers of the repair decoder.

In one possible implementation, the i-th layer of the encoder in this embodiment corresponds to the i+1-th layer of the repair decoder. In this scenario, the processing manner of each layer of the encoder is as follows:

A. and (3) coding the characteristic diagram of the ith layer-1 of the encoder in the ith layer of the encoder to obtain the characteristic diagram of the ith layer, outputting the characteristic diagram of the ith layer to the ith+1 layer of the encoder and the ith+1 layer of the repair decoder, wherein the size of the characteristic diagram of the ith layer is a times of the size of the characteristic diagram of the ith layer-1, i is an integer which is more than or equal to 2 and less than a preset value, and a is more than 0 and less than 1.

In step a, in layer 2 of the encoder, the feature map of layer 1 of the encoder may be encoded to obtain a feature map of layer 2, and the feature map of layer 2 of the encoder may be output to layer 3 of the encoder, so that layer 3 of the encoder performs step a to obtain the feature map of layer 3 of the encoder. In this embodiment, the layer 2 feature map may also be output to the layer 3 of the repair decoder, and the following description of how the repair decoder processes according to the layer 2 feature map is specifically described below.

In steps A-C of the present embodiment, i is an integer greater than or equal to 2 and less than a second preset value. The second preset value is equal to the layer number of the network depth of the repair decoder. If the second preset value is 7, the number of layers of the network depth of the encoder is 7, and if i is 2-6, the method can be performed according to the mode in the step A. In addition, when i is 1, in the layer 1 of the encoder in this embodiment, the image to be repaired may be encoded to obtain a layer 1 feature map, and the layer 1 feature map may be output to the layer 2 of the encoder and the layer 2 of the repair decoder.

It is to be understood that the size of the feature pattern of the i-th layer in this embodiment is a times the size of the feature pattern of the i-1 th layer, and a is greater than 0 and less than 1. Illustratively, if a is 1/2, then the size of the layer 3 profile of the encoder is 1/2 of the size of the layer 2 profile of the encoder.

Alternatively, the encoding process in this embodiment may be a partial convolution process (Partial convolution, PConv), a batch normalization layer process (Batch Normalization, BN), and a collation layer process (Relu) performed in this order.

B. And adding 1 to i, executing the step A until the added 1 to i is equal to a second preset value, and executing the step C.

In the case where i in the above step a is an integer greater than or equal to 1 and less than the second preset value, in this case, if i is 6,i plus 1 is 7, i.e., the second preset value, the following step C may be performed.

C. And carrying out coding processing on the characteristic diagram of the ith layer of the coder to obtain the characteristic diagram of the (i+1) th layer, and outputting the characteristic diagram of the (+1i) th layer to the (i+1) th layer of the repair decoder.

In the step C, when i is 6, i is added with 1 to be 7, i.e. a second preset value. The layer 6 feature map of the encoder may be encoded to obtain a layer 7 feature map. Since layer 7 is the last layer of the encoder, the 7 th layer feature map can be output to layer 7 of the repair decoder. It should be understood that the 7 th layer of the encoder in this embodiment is the feature map of the image to be repaired in the above embodiment.

Fig. 4 is a schematic diagram of an encoding process of an ith layer of the encoder according to the present application. As shown in FIG. 4, the feature map of the ith layer of the encoder is obtained by sequentially passing PConv, BN and Relu through the feature map of the ith layer of the encoder (m _e ^l ) Output to the i+1 layer of the encoder and repair the i+1 layer of the decoder.

Corresponding to the encoder, each layer of the repair decoder in this embodiment is configured to perform repair processing on the image to be repaired according to the feature map of the corresponding layer of the encoder and the information of the image to be repaired, so as to obtain an initial repair image corresponding to each layer of the repair decoder. For example, if the corresponding layer of the encoder corresponding to the layer 2 of the repair decoder is the layer 1 of the encoder, the layer 2 of the repair decoder may perform repair processing on the image to be repaired according to the feature map output by the layer 1 of the encoder and the information of the image to be repaired, specifically perform repair processing on the image to be repaired of the layer 3 of the repair decoder, so as to obtain an initial repair image corresponding to the layer 2 of the repair decoder. It should be understood that the initial repair image in the above embodiment includes an initial repair image corresponding to each layer of the repair decoder.

Further, the repair decoder may output an initial repair image corresponding to each layer to a corresponding layer of the refinement decoder, the initial repair image including the initial repair image corresponding to each layer of the repair decoder. Alternatively, the corresponding layer of the refinement decoder corresponding to layer 1 of the repair decoder may be layer 1 or other layers of the refinement decoder. Alternatively, the repair image of the upper layer of the repair decoder in this embodiment may be input to the lower layer, so that the lower layer of the repair decoder performs repair processing on the feature map of the corresponding layer of the encoder and the initial repair image of the upper layer of the repair decoder, so as to obtain the initial repair image of the lower layer of the repair decoder.

In one possible implementation, the ith layer of the repair decoder in this embodiment corresponds to the (i+1) th layer of the refinement decoder. In this scenario, the processing manner of each layer of the repair decoder is as follows:

a', in the ith layer of the restoring decoder, according to the characteristic diagram of the ith-1 layer of the encoder, the first initial restoring image of the (i+1) th layer of the restoring decoder and the information of the image to be restored, carrying out first restoring treatment on the image to be restored to obtain the first initial restoring image of the ith layer of the encoder, wherein i is an integer which is more than or equal to 1 and less than a second preset value.

As shown in fig. 3, the feature map of the 6 th layer and the feature map of the 7 th layer of the encoder in this embodiment are input to the 7 th layer of the repair decoder, which sequentially performs repair processing in the order of 7 th layer to 1 st layer.

In the steps a '-D', i is an integer greater than or equal to 1 and less than a second preset value. The second preset value is equal to the second preset value of i in the encoder, for example, the number of layers equal to the network depth of the repair decoder is 7. For example, when i is 6, the first repair process may be performed on the image to be repaired according to the feature map output by the 5 th layer of the encoder, the first initial repair image of the 7 th layer of the repair decoder, and the information of the image to be repaired, to obtain the first initial repair image of the 6 th layer of the repair decoder. Optionally, the first repair process in this embodiment is a transpose convolution process (transpose convolution, DConv), a batch normalization process, and a collation process performed sequentially. It should be appreciated that when i is 1, the feature map of the layer 0 output of the encoder may be the image to be repaired.

When i is 7, in view of the fact that the 7 th layer of the repair decoder inputs the characteristic diagrams from the 6 th layer and the 7 th layer of the encoder, a first repair process can be performed on the 7 th layer of the repair decoder according to the characteristic diagram output by the 6 th layer of the encoder, the characteristic diagram output by the 7 th layer of the encoder and the information of the image to be repaired, so as to obtain a first initial repair image of the 7 th layer of the repair decoder.

B', splicing the characteristic diagram of the first initial repair image of the ith layer and the characteristic diagram of the i-1 th layer of the encoder to obtain a second initial repair image of the ith layer of the repair decoder, wherein the size of the characteristic diagram of the first initial repair image of the ith layer is 1/a times of that of the first initial repair image of the i-1 th layer.

The feature map of the first initial restoration image of the i-th layer of the restoration decoder in this embodiment is 1/a times the size of the first initial restoration image of the i-1-th layer. Illustratively, the feature map of the first initial repair image of layer 6 of the repair decoder has a size that is 2 times the size of the first initial repair image of layer 7. The size of the feature map of the first initial repair image of the ith layer of the repair decoder in this embodiment is the same as the size of the feature map of the ith-1 layer of the encoder, so in this embodiment, the feature map of the first initial repair image of the ith layer of the repair decoder and the feature map of the ith-1 layer of the encoder may be spliced to obtain the second initial repair image of the ith layer of the repair decoder.

Illustratively, the size of the feature map of the first initial repair image of the 6 th layer of the repair decoder is the same as the size of the feature map of the 5 th layer of the encoder in this embodiment, so in this embodiment, the feature map of the first initial repair image of the 6 th layer of the repair decoder and the feature map of the 5 th layer of the encoder may be spliced to obtain the second initial repair image of the 6 th layer of the repair decoder.

C', performing second restoration processing on a second initial restoration image of the ith layer of the restoration decoder to obtain an initial restoration image of the ith layer of the restoration decoder, and outputting the initial restoration image of the ith layer to the (i-1) th layer of the restoration decoder and the (i+1) th layer of the refinement decoder.

Alternatively, the second repair process in this embodiment may be a partial convolution process and a collation process performed in order. In this embodiment, a second repair process may be performed on a second initial repair image of the ith layer of the repair decoder, to obtain an initial repair image of the ith layer of the repair decoder. It should be understood that in this embodiment, the second repair process may be performed on the second initial repair image of each layer of the repair decoder, so as to obtain an initial repair image of each layer of the repair decoder.

Further, the initial repair image of the ith layer may be output to the ith-1 layer of the repair decoder and the (i+1) th layer of the refinement decoder in this embodiment. Since i in the present embodiment is an integer greater than or equal to 1 and less than the second preset value, the initial repair image of the 7 th layer of the repair encoder may be output to the 6 th layer of the repair decoder, and the 7 th layer of the refinement decoder.

D ', subtracting 1 from i, and performing the steps a ' -C ' until i is 1.

It should be understood that the repair decoder sequentially performs repair processing in the order of layer 7 to layer 1 according to the above-described related explanation. Therefore, in this embodiment, according to the above description, the above steps a '-C' may be sequentially performed after the initial repair image of the 7 th layer of the repair decoder is obtained, to obtain the initial repair image of the 6 th layer, the initial repair image … … of the 5 th layer, and the initial repair image of the 1 st layer. It should be understood that the initial repair image of layer 1 of the repair decoder in this embodiment is the initial repair image of the image to be repaired in the above embodiment.

Fig. 5 is a schematic diagram of a repair process of an ith layer of the repair decoder according to the present application. As shown in fig. 5, for an initial repair image (r ^l ) After DConv, BN and Relu in this order, a first initial repair image (m _r ^l ) After the features of the first initial restoration image are fused with the feature map of the ith layer-1 of the encoder, a second initial restoration image (m _r ^l-1 ) And further the second initial restoration image (m _r ^l-1 ) After PConv, relu, an initial repair image (r) of the i-1 th layer of the repair decoder can be obtained ^l-1 )。

Corresponding to the above-mentioned encoder and repair decoding, each layer of the refinement decoder in this embodiment is configured to perform refinement repair processing on an initial repair image of a corresponding layer of the repair decoder, so as to obtain a repair image corresponding to each layer of the refinement decoder, so as to obtain a repair image. Wherein the repair image includes a repair image corresponding to each layer of the refinement decoder. Alternatively, the repair image of the upper layer of the refinement decoder in this embodiment may be input to the lower layer, so that the lower layer of the refinement decoder performs a refinement repair process on the initial repair image of the corresponding layer of the refinement decoder and the repair image of the upper layer of the refinement decoder to obtain the repair image of the lower layer of the refinement decoder.

In one possible implementation, the ith layer of the repair decoder in this embodiment corresponds to the (i+1) th layer of the refinement decoder. In this scenario, the processing manner of each layer of the refinement decoder is as follows:

a', at the ith layer of the thinning decoder, carrying out thinning repair processing on the initial repair image of the ith-1 layer of the thinning decoder and the repair image of the (i+1) th layer of the thinning decoder to obtain the repair image of the ith layer of the thinning decoder, and outputting the repair image of the ith layer to the ith-1 layer of the thinning decoder, wherein i is an integer which is more than or equal to 2 and less than a second preset value.

As shown in fig. 3, the feature map of the 6 th layer and the initial repair image of the 7 th layer of the repair decoder in this embodiment are input to the 7 th layer of the refinement decoder, and the refinement decoder sequentially performs repair processing in the order of 7 th layer to 1 st layer.

In step a "-B" of the present embodiment, i is an integer greater than or equal to 2 and less than a second preset value. The second preset value is equal to the second preset value of i in the encoder, for example, the number of layers equal to the network depth of the refinement decoder is 7. Illustratively, when i is 6, the initial repair image output from the 5 th layer of the repair decoder may be subjected to a refinement repair process to obtain the 6 th layer repair image of the refinement decoder. It should be understood that when i is 7, the initial repair image output by the 6 th layer of the repair decoder and the initial repair image output by the 7 th layer of the repair decoder may be subjected to a thinning repair process to obtain the 7 th layer repair image of the thinning decoder.

When i is 1, in this embodiment, a refinement repair process may be performed on the repair image output by the layer 2 of the refinement decoder to obtain the repair image of the layer 1 of the refinement decoder, where the repair image of the layer 1 of the refinement decoder is the repair image of the image to be repaired in the above embodiment, that is, the repair image output by the image repair model.

Alternatively, the refinement repair process in the present embodiment is a convolution process (Conv).

B ", subtracting 1 from i, and performing the step a" until i is 1.

It should be understood that the repair decoder sequentially performs repair processing in the order of layer 7 to layer 1 according to the above-described related explanation. Therefore, in this embodiment, according to the above description, after obtaining the repair image of the 7 th layer of the refinement decoder, the above steps a '-C' may be sequentially performed to obtain the repair image of the 6 th layer, the repair image … … of the 5 th layer, and the repair image of the 1 st layer.

In this embodiment, the neural network of each stage of the image restoration model, such as the encoder, the restoration decoder and the refinement decoder, is multi-layered, so that the image restoration model in this embodiment can restore the image on multiple scales, and further improves the accuracy of image restoration.

Fig. 6 is a schematic diagram of refinement repair processing of an ith layer of the refinement decoder provided by the present application. As shown in fig. 6, for the restored image (f ^l ) After Conv, a restored image (f) of the ith-1 layer of the refinement decoder is obtained ^l-1 )。

On the basis of the above embodiment, in order to further improve the accuracy of image restoration, a plurality of refinement decoders may be provided in the present embodiment. Wherein the network depth of each refinement decoder is multi-layered and the number of layers of the network depth of each refinement decoder and repair decoder is equal. That is, in this embodiment, the initial repair image may be subjected to multiple refinement repair processes to improve the accuracy of image repair.

Fig. 7 is a schematic structural diagram of an image restoration model provided by the present application. In this embodiment, when there are a plurality of refinement decoders, the repair decoder outputs an initial repair image corresponding to each layer to the corresponding layer of the 1 st refinement decoder. The specific manner in which the repair decoder outputs the initial repair image corresponding to each layer to the corresponding layer of the 1 st refinement decoder may refer to the above description that the repair decoder outputs the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder.

The processing mode of each layer of each refinement decoder is specifically as follows:

a. each layer of the kth thinning decoder is used for carrying out thinning repair processing on the initial repair image of the corresponding layer of the repair decoder to obtain a repair image corresponding to each layer of the kth thinning decoder, and outputting the repair image corresponding to each layer of the kth thinning decoder to the corresponding layer of the kth+1th thinning decoder, wherein k is an integer greater than or equal to 1.

In step a, for example, when k is 1, the processing manner of each layer of the 1 st refinement decoder on the initial repair image is the same as that of each layer of the refinement decoder on the initial repair image, and specific reference may be made to the related description of a "-B" above. Unlike the above, the repair image corresponding to each layer of the kth refinement decoder may be output to the corresponding layer of the kth+1th refinement decoder in the present embodiment.

It should be understood that the i-th layer of the kth refinement decoder and the corresponding layer of the k+1-th refinement decoder are the i+1-th layer of the k+1-th refinement decoder. For example, as in the present embodiment, the repair image corresponding to the 1 st layer of the kth refinement decoder may be output to the 2 nd layer of the (k+1) th refinement decoder. The processing mode of each layer of the k+1th refinement decoder on the initial repair image may be:

and the 7 th layer of the (k+1) th refinement decoder performs refinement repair processing on the 6 th layer of the (k+1) th refinement decoder and the repair image output by the 7 th layer to obtain the 7 th layer repair image of the (k+1) th refinement decoder, and outputs the 7 th layer repair image to the 6 th layer of the (k+1) th refinement decoder. The 6 th layer performs thinning repair processing on the repair image output by the 5 th layer of the k thinning decoder and the 7 th layer of the k+1 thinning decoder to obtain the repair image … … of the 5 th layer of the k+1 thinning decoder, and the like to obtain the repair image of the 2 nd layer of the k+1 thinning decoder. It is noted that, the 1 st layer of the k+1th refinement decoder performs a refinement repair process on the repair image output by the 0 th layer of the k+1th refinement decoder and the 2 nd layer of the k+1th refinement decoder to obtain the 1 st layer repair image of the k+1th refinement decoder, where the repair image of the 0 th layer of the k refinement decoder is the repair image of the k refinement decoder.

b. And adding 1 to k, and executing the step a until k is a first preset value, so as to obtain a repair image, wherein the repair image comprises the repair image of each layer of each refinement decoder.

In this embodiment, the first preset value of k may be different from the second preset value of i described above. As illustrated in fig. 7 by taking k as 2, i.e. two refinement decoders as examples. It should be understood that, in this embodiment, when k is a first preset value, the repair image output by the 1 st layer of the kth refinement decoder is the repair image output by the image repair model.

In this embodiment, a plurality of refinement repair models may be set, so that further refinement repair of the repair image may be realized, and accuracy of image repair may be further improved.

Fig. 8 is a schematic structural diagram of an image restoration device according to the present application. As shown in fig. 8, the apparatus 800 for image restoration includes: a transceiver module 801 and a processing module 802.

The transceiver module 801 is configured to receive an image to be repaired and information of the image to be repaired, where the information of the image to be repaired indicates a region to be repaired of the image to be repaired.

The processing module 802 is configured to input an image to be repaired and information of the image to be repaired into an image repair model to obtain a repaired image after the image to be repaired is repaired, and output the repaired image, where the image repair model includes an encoder, a repair decoder, and a refinement decoder, the encoder is configured to perform encoding processing on the image to be repaired to obtain a feature map of the image to be repaired, the repair decoder is configured to perform repair processing on the image to be repaired according to the feature map and the information of the image to be repaired to obtain an initial repaired image, and the refinement decoder is configured to perform refinement repair processing on the initial repaired image to obtain the repaired image.

Each layer of the encoder is used for encoding an image to be repaired to obtain a feature map corresponding to each layer of the encoder, and the feature map corresponding to each layer is output to a corresponding layer of the repair decoder, wherein the feature map of the image to be repaired comprises the feature map corresponding to each layer of the encoder;

each layer of the repair decoder is used for performing repair processing on the image to be repaired according to the characteristic diagram of the corresponding layer of the encoder and the information of the image to be repaired, so as to obtain an initial repair image corresponding to each layer of the repair decoder, and outputting the initial repair image corresponding to each layer to the corresponding layer of the refinement decoder, wherein the initial repair image comprises the initial repair image corresponding to each layer of the repair decoder.

Optionally, the network depth of the refinement decoder is multi-layered, and the number of layers of the network depths of the refinement decoder and the repair decoder are equal.

Each layer of the thinning decoder is used for carrying out thinning repair processing on the initial repair image of the corresponding layer of the thinning decoder to obtain a repair image corresponding to each layer of the thinning decoder so as to obtain a repair image, wherein the repair image comprises the repair image corresponding to each layer of the thinning decoder.

Optionally, the refinement decoder is multiple, the network depth of each refinement decoder is multiple layers, and the number of layers of the network depth of each refinement decoder and the repair decoder is equal.

b. and d, adding 1 to k, and executing the step a until k is a first preset value, so as to obtain a repair image, wherein the repair image comprises the repair image of each layer of each refinement decoder, and the first preset value is equal to the number of the refinement decoders.

Optionally, the encoding processing mode of the encoder includes:

A. coding the characteristic diagram of the ith layer of the encoder in the ith layer of the encoder to obtain the characteristic diagram of the ith layer, outputting the characteristic diagram of the ith layer to the (i+1) th layer of the encoder and the (i+1) th layer of the repair decoder, wherein the size of the characteristic diagram of the ith layer is a times of the size of the characteristic diagram of the ith layer of the encoder, i is an integer which is more than or equal to 2 and less than a second preset value, a is more than 0 and less than 1, and the second preset value is equal to the number of layers of the network depth of the repair decoder;

B. Adding 1 to i, executing the step A until the adding 1 of i is equal to a second preset value, and executing the step C;

Optionally, the repair processing manner of the repair decoder includes:

a', in the ith layer of the repair decoder, according to the characteristic diagram of the ith-1 layer of the encoder, a first initial repair image of the (i+1) th layer of the repair decoder and information of an image to be repaired, performing first repair processing on the image to be repaired to obtain a first initial repair image of the ith layer of the encoder, wherein i is an integer which is greater than or equal to 1 and smaller than a second preset value;

b', splicing the characteristic diagram of the first initial repair image of the ith layer with the characteristic diagram of the i-1 th layer of the encoder to obtain a second initial repair image of the ith layer of the repair decoder, wherein the size of the characteristic diagram of the first initial repair image of the ith layer is 1/a times of that of the first initial repair image of the i-1 th layer;

c', performing second restoration processing on a second initial restoration image of the ith layer of the restoration decoder to obtain an initial restoration image of the ith layer of the restoration decoder, and outputting the initial restoration image of the ith layer to the (i-1) th layer of the restoration decoder and the (i+1) th layer of the refinement decoder;

D ', subtracting 1 from i, and performing the steps a ' -C ' until i is 1.

a', at the ith layer of the thinning decoder, carrying out thinning repair processing on an initial repair image of the ith layer-1 of the thinning decoder and a repair image of the (i+1) th layer of the thinning decoder to obtain a repair image of the ith layer of the thinning decoder, and outputting the repair image of the ith layer to the ith layer-1 of the thinning decoder, wherein i is an integer which is more than or equal to 2 and less than a second preset value;

b ", subtracting 1 from i, and performing the step a" until i is 1.

Optionally, the encoding process includes a partial convolution process, a batch normalization layer process, and a collation process, the first repair process includes a transpose convolution process, a batch normalization process, and a collation process, the second repair process includes a partial convolution process and a collation process, and the refinement repair process includes a convolution process.

The image restoration device provided in this embodiment is similar to the principle and technical effects achieved by the above image restoration method, and will not be described herein.

According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application, and as shown in fig. 9, is a block diagram of the electronic device according to a method for repairing an image according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.

As shown in fig. 9, the electronic device includes: one or more processors 901, memory 902, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). In fig. 9, a processor 901 is taken as an example.

Memory 902 is a non-transitory computer readable storage medium provided by the present application. Wherein the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of image restoration provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method of image restoration provided by the present application.

The memory 902 is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the methods of image restoration in embodiments of the present application. The processor 901 performs various functional applications of the server and sample processing, i.e., a method of implementing image restoration in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 902.

Memory 902 may include a storage program area and a storage sample area, wherein the storage program area may store an operating system, at least one application program required for a function; the stored sample area may store samples or the like created according to the use of the electronic device for performing the method of image restoration. In addition, the memory 902 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 902 optionally includes memory remotely located relative to processor 901, which may be connected to an electronic device for performing the method of image restoration via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device of the method of image restoration may further include: an input device 903 and an output device 904. The processor 901, memory 902, input devices 903, and output devices 904 may be connected by a bus or other means, for example in fig. 9.

The input device 903 may receive input numeric or character information and generate key signal inputs related to user settings and function controls of the electronic device for performing the method of image restoration, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, etc. input devices. The output means 904 may include a display device, auxiliary lighting means (e.g., LEDs), tactile feedback means (e.g., vibration motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive samples and instructions from, and transmit samples and instructions to, a storage system, at least one input device, and at least one output device.

These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or samples to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or samples to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a sample server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital sample communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed embodiments are achieved, and are not limited herein.

The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.

Claims

1. A method of image restoration, comprising:

receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired;

inputting the image to be repaired and the information of the image to be repaired into an image repairing model to obtain a repaired image after repairing the image to be repaired, wherein the image repairing model comprises encoders and repairing decoders with multiple layers of network depths and equal layers, each layer of the encoders is used for encoding the image to be repaired to obtain a characteristic diagram corresponding to each layer of the encoders, each layer of the repairing decoders is used for repairing the image to be repaired according to the characteristic diagram of the corresponding layer of the encoders and the information of the image to be repaired to obtain an initial repairing image corresponding to each layer of the repairing decoders, and the image repairing model further comprises a thinning decoder used for thinning and repairing the initial repairing image to obtain the repairing image;

Outputting the repair image;

each layer of the encoder is further configured to output the feature map corresponding to each layer to a corresponding layer of the repair decoder.

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

each layer of the repair decoder is further configured to output an initial repair image corresponding to the each layer to a corresponding layer of the refinement decoder.

3. The method according to claim 2, wherein the network depth of the refinement decoder is multi-layered and the number of layers of the network depths of the refinement decoder and the repair decoder are equal;

4. The method of claim 2, wherein the refinement decoder is a plurality of, each of the refinement decoders has a plurality of layers of network depth, and each of the refinement decoder and the repair decoder has an equal number of layers of network depth;

5. The method according to claim 3 or 4, wherein each layer of the encoder is configured to perform encoding processing on the image to be repaired to obtain a feature map corresponding to each layer of the encoder, and the method includes:

C. performing coding processing on the characteristic diagram of the ith layer of the coder to obtain the characteristic diagram of the (i+1) th layer,

correspondingly, outputting the feature map corresponding to each layer to the corresponding layer of the repair decoder, including:

and outputting the characteristic diagram of the i+1th layer to the i+1th layer of the repair decoder.

6. The method according to claim 5, wherein each layer of the repair decoder is configured to perform repair processing on the image to be repaired according to the feature map of the corresponding layer of the encoder and the information of the image to be repaired, to obtain an initial repair image corresponding to each layer of the repair decoder, and the method includes:

a', at a j-th layer of the repair decoder, performing first repair processing on the image to be repaired according to a feature map of the j-1-th layer of the encoder, a first initial repair image of the j+1-th layer of the repair decoder and information of the image to be repaired to obtain a first initial repair image of the j-th layer of the encoder, wherein j is an integer greater than or equal to 1 and smaller than the second preset value;

b', splicing the characteristic diagram of the first initial repair image of the j-th layer with the characteristic diagram of the j-1-th layer of the encoder to obtain a second initial repair image of the j-th layer of the repair decoder, wherein the size of the characteristic diagram of the first initial repair image of the j-th layer is 1/a times of that of the first initial repair image of the j-1-th layer;

C', carrying out second restoration processing on a second initial restoration image of the j-th layer of the restoration decoder to obtain an initial restoration image of the j-th layer of the restoration decoder, and correspondingly, outputting the initial restoration image corresponding to each layer to a corresponding layer of the refinement decoder comprises the following steps: outputting the initial repair image of the j-th layer to the j-1-th layer of the repair decoder and the j+1-th layer of the refinement decoder;

d ', subtracting 1 from j, and executing the steps A ' -C ' until j is 1.

7. The method of claim 6, wherein each layer of the refinement decoder is configured to perform a refinement repair process on an initial repair image of a corresponding layer of the repair decoder to obtain a repair image corresponding to each layer of the refinement decoder, so as to obtain the repair image, including:

a', at a first layer of the refinement decoder, performing refinement restoration processing on an initial restoration image of a first-1 layer of the refinement decoder and a restoration image of a first+1 layer of the refinement decoder to obtain a restoration image of the first layer of the refinement decoder, and outputting the restoration image of the first layer to the first-1 layer of the refinement decoder, wherein l is an integer which is greater than or equal to 2 and smaller than the second preset value;

B ", subtracting 1 from l, and performing the step a" until l is 1.

8. The method of claim 6, wherein the encoding process comprises a partial convolution process, a batch normalization layer process, and a collation process, wherein the first repair process comprises a transpose convolution process, the batch normalization process, and the collation process, wherein the second repair process comprises the partial convolution process and the collation process, and wherein the refinement repair process comprises a convolution process.

9. The method according to any one of claims 1-4, 6-8, wherein the information of the image to be repaired is a mask of the image to be repaired.

10. An apparatus for image restoration, comprising:

the receiving and transmitting module is used for receiving an image to be repaired and information of the image to be repaired, wherein the information of the image to be repaired indicates a region to be repaired of the image to be repaired;

the processing module is used for inputting the image to be repaired and the information of the image to be repaired into an image repairing model to obtain a repaired image after repairing the image to be repaired, and outputting the repaired image, wherein the image repairing model comprises encoders and repairing decoders with multiple layers of network depths and equal layers, each layer of the encoders is used for encoding the image to be repaired to obtain a characteristic image corresponding to each layer of the encoders, each layer of the repairing decoders is used for repairing the image to be repaired according to the characteristic image of the corresponding layer of the encoders and the information of the image to be repaired to obtain an initial repairing image corresponding to each layer of the repairing decoders, and the image repairing model further comprises a thinning decoder used for thinning the initial repairing image to obtain the repairing image;

11. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.