CN115965561A

CN115965561A - Image restoration method and device, readable medium and electronic equipment

Info

Publication number: CN115965561A
Application number: CN202210064498.2A
Authority: CN
Inventors: 陈坤
Original assignee: Petal Cloud Technology Co Ltd
Current assignee: Petal Cloud Technology Co Ltd
Priority date: 2021-10-11
Filing date: 2022-01-20
Publication date: 2023-04-14

Abstract

The application relates to the field of image processing and discloses an image restoration method, an image restoration device, a readable medium and electronic equipment. The method comprises the following steps: repairing at least one first reference area outside the area to be repaired in the image to be repaired by adopting a first repairing method to obtain first reference repairing areas corresponding to the first reference areas; and determining whether the area to be repaired in the image to be repaired is repaired by adopting a first repairing method and whether the obtained repairing result of the area to be repaired meets the repairing requirement or not based on the first similarity of each first reference area and each corresponding first reference repairing area. By the image restoration method, the electronic device can quickly and effectively evaluate whether the restoration result of the image to be restored by the first restoration method can meet the requirement, and restore the area to be restored by other methods under the condition that the restoration result of the first restoration method cannot meet the requirement.

Description

Image restoration method and device, readable medium and electronic equipment

The present application claims priority of chinese patent application filed on 11/10/2021 under the name "image repair device" under the application number 202122450788.4 by the national intellectual property office, which is incorporated herein by reference in its entirety.

Technical Field

The present application relates to the field of image processing, and in particular, to an image restoration method, an image restoration apparatus, a readable medium, and an electronic device.

Background

The Image Inpainting technology is to fill up a missing area in an Image to be repaired through an algorithm, so that the repaired Image looks very natural and is difficult to distinguish from an undamaged Image, and the Image Inpainting technology can be applied to various Image/video editing scenes such as removal of a specified object, removal of marks and subtitles in the Image, and the like. With the rapid development of the computer vision industry, the image restoration technology is more mature, and the quality requirement of the image output by the restoration algorithm is higher and higher.

However, since the missing region of the image has no correct reference answer, it is not easy to evaluate the quality of the image restoration. At present, the image restoration quality is generally evaluated by a manual evaluation method, for example, a plurality of workers Score the restored image to obtain a Mean Opinion Score (MOS) value, and then determine the image restoration quality according to the MOS value. However, the evaluation of the image restoration quality by adopting a manual evaluation mode has large subjectivity of an evaluation result due to the lack of a uniform evaluation standard, and the effectiveness of the evaluation result is influenced. In addition, the efficiency of evaluating the image restoration quality by adopting a manual evaluation mode is lower.

Disclosure of Invention

In view of this, embodiments of the present application provide an image restoration method, an apparatus, a readable medium, and an electronic device. The method comprises the steps of repairing a region outside a region to be repaired in an image to be repaired and/or at least one reference region in an adjacent image of the image to be repaired by adopting a first repairing method to obtain a reference repairing region of each reference region, and determining whether a repairing result of the region to be repaired obtained by repairing the region to be repaired in the image to be repaired by adopting the first repairing method meets requirements or not according to the similarity between each reference repairing region and the corresponding reference region. Therefore, the quality of image restoration can be quickly and effectively evaluated, so that the area to be restored is restored by adopting other methods under the condition that the restoration result of the first restoration method does not meet the requirement.

In a first aspect, an embodiment of the present application provides an image repairing method, which is applied to an electronic device, and the method includes: repairing at least one first reference area outside the area to be repaired in the image to be repaired by adopting a first repairing method to obtain first reference repairing areas corresponding to the first reference areas; and determining whether the area to be repaired in the image to be repaired is repaired by adopting a first repairing method and whether the obtained repairing result of the area to be repaired meets the repairing requirement or not based on the first similarity between each first reference area and each corresponding first reference repairing area.

That is to say, the electronic device repairs a partial region (first reference region) in another region except for the region to be repaired in the image to be repaired by the first repairing method to obtain a repairing result (first reference repairing region) of the region, calculates a similarity between the repairing result and a corresponding image (first reference region) of the region in the image to be repaired, and determines whether the repairing result of the region to be repaired obtained by repairing the region to be repaired in the image to be repaired by the first repairing method meets the repairing requirement according to the similarity.

By the image restoration method provided by the embodiment of the application, the quality of image restoration can be evaluated quickly and effectively, so that the electronic equipment can perform the next operation conveniently. For example, after the image to be repaired is repaired by the first repairing method, whether the repairing of the image to be repaired by the first repairing method can meet the repairing requirement can be determined by the method provided by the embodiment of the application, and the influence on the user experience of image repairing due to the output of the repairing result which does not meet the requirement is avoided. For another example, before the image to be repaired is repaired by the first repairing method, whether the repairing of the image to be repaired by the first repairing method can meet the repairing requirement is determined by the method provided by the embodiment of the application, and the image to be repaired by the first repairing method is still repaired under the condition that the repairing result of the image to be repaired by the first repairing method does not meet the requirement, so that the computing resource of the electronic device is saved.

In a possible implementation of the first aspect, it is determined whether the repair result of the area to be repaired meets the repair requirement by: and under the condition that the average value of the first similarities is larger than the similarity threshold, determining that the repair result of the area to be repaired meets the repair requirement.

That is, in the embodiment of the present application, it may be determined that the repair result of the region to be repaired satisfies the repair requirement by comparing the average value of the first similarities of each first reference region and each corresponding first reference repair region with the preset similarity threshold.

In a possible implementation of the first aspect, it is determined whether the repair result of the area to be repaired meets the repair requirement by: repairing at least one second reference area in at least one adjacent image adjacent to the image to be repaired in the video by adopting a first repairing method to obtain second reference repairing areas corresponding to the second reference areas; and calculating second similarity of each second reference area and each corresponding second reference repairing area, and determining that the repairing result of the area to be repaired meets the repairing requirement under the condition that the average value of each second similarity and/or each first similarity is larger than a preset value.

That is to say, in the embodiment of the present application, if the image to be repaired is an image in a video, the second reference repair area may be set in another image in the video (for example, an adjacent frame described below), so that accuracy of determining whether a result of repairing the image to be repaired by using the first repair method satisfies a repair requirement may be improved.

In a possible implementation of the first aspect, a frame difference between each of the neighboring images and the image to be repaired in the video is smaller than a preset value.

That is to say, the frame difference between the adjacent image in the video and the image to be repaired is smaller than the preset value, for example, the position of the adjacent image in the video is within the range of 25 frames before and after the image to be repaired, so that the similarity between the adjacent image and the image to be repaired can be ensured, and the accuracy of determining whether the result of repairing the image to be repaired by using the first repairing method meets the repairing requirement is improved.

In a possible implementation of the first aspect, each second reference area is a partial area in an image with the largest similarity to the image to be repaired in each adjacent image.

That is to say, in the embodiment of the present application, each second reference region is set in an image with the largest similarity to the image to be restored (i.e., a similar frame described below) in an adjacent image of the image to be restored in the video, so that the restoration quality of the second reference region restored by using the first restoration method can more accurately reflect the restoration quality of the image to be restored by using the first restoration method, and the accuracy of determining whether the result of restoring the image to be restored by using the first restoration method meets the restoration requirement is improved.

In the embodiment of the present application, the image with the largest similarity to the image to be repaired in each adjacent image may be determined based on at least one of the following parameters: the structural similarity between each adjacent image and the image to be restored; similarity of histograms of adjacent images and images to be restored; and the cosine similarity between each adjacent image and the image to be restored.

In a possible implementation of the first aspect, the position of each second reference area in the image corresponding to each second reference area is the same as the position of the area to be repaired in the image to be repaired.

That is to say, in the embodiment of the present application, the positions of the second reference regions in the images corresponding to the second reference regions and the positions of the to-be-repaired regions in the to-be-repaired images are set to be the same, so that the repair quality of repairing the second reference regions by using the first repair method can more accurately reflect the repair quality of repairing the to-be-repaired regions by using the first repair method, and the accuracy of determining whether the result of repairing the to-be-repaired images by using the first repair method meets the repair requirement is improved. In addition, in some embodiments, the shapes and sizes of the second reference areas and the to-be-repaired areas may also be set to be the same, so that the repair quality of repairing the second reference areas by using the first repair method can more accurately reflect the repair quality of repairing the to-be-repaired areas by using the first repair method, and the accuracy of determining whether the result of repairing the to-be-repaired image by using the first repair method meets the repair requirement is further improved.

In a possible implementation of the first aspect, the first reference regions and the region to be repaired have the same shape and size.

In a possible implementation of the first aspect, a distance between each first reference region and the region to be repaired is less than a preset value.

In one possible implementation of the first aspect, the method further includes: under the condition that the repairing result of the to-be-repaired area meets the repairing requirement, repairing the to-be-repaired area in the to-be-repaired image by adopting a first repairing method; and under the condition that the repairing result of the region to be repaired is determined not to meet the repairing requirement, repairing the region to be repaired in the image to be repaired by adopting a second repairing method.

That is to say, in the embodiment of the present application, the electronic device first determines whether a repair result of repairing the image to be repaired by using the first repair method satisfies a repair requirement, and repairs the area to be repaired in the image to be repaired by using the first repair method when it is determined that the repair result satisfies the repair requirement. Therefore, the electronic equipment can avoid that before the first repairing method is used for repairing the image to be repaired, whether the repairing of the image to be repaired by using the first repairing method can meet the repairing requirement is determined, and the electronic equipment can avoid that the image to be repaired by using the first repairing method is still repaired under the condition that the repairing result of the image to be repaired by using the first repairing method does not meet the requirement, so that the computing resource is saved.

And the electronic equipment repairs the image to be repaired by adopting the second repairing method under the condition that the repairing result of repairing the image to be repaired by adopting the first repairing method does not meet the repairing requirement, so that the image repairing quality can be improved. It is understood that the second repairing method may be a method completely different from the first repairing method, or may be a method in which parameters of the first repairing method are adjusted.

In a second aspect, an embodiment of the present application provides an image restoration apparatus, including: the image restoration module is used for restoring at least one first reference area outside the area to be restored in the image to be restored by adopting a first restoration method to obtain a first reference restoration area corresponding to each first reference area; and the repair quality evaluation module is used for determining whether the region to be repaired in the image to be repaired is repaired by adopting a first repair method and whether the obtained repair result of the region to be repaired meets the repair requirement or not based on the first similarity between each first reference region and each corresponding first reference repair region.

That is, the image repairing apparatus repairs a partial region (first reference region) in another region except for the region to be repaired in the image to be repaired by a first repairing method (for example, a preset image repairing method hereinafter) to obtain a repairing result (first reference repairing region) of the region, calculates a similarity between the repairing result and a corresponding image (first reference region) of the region in the image to be repaired, and determines whether the repairing result of the region to be repaired obtained by repairing the region to be repaired in the image to be repaired by the first repairing method satisfies the repairing requirement according to the similarity.

The image restoration device provided by the embodiment of the application can quickly and effectively evaluate the image restoration quality so as to facilitate the next operation and improve the image restoration quality. For example, after the image to be repaired is repaired by the first repairing method, the device may determine whether the repairing of the image to be repaired by the first repairing method can meet the repairing requirement, so as to avoid outputting the repairing result that does not meet the requirement and affecting the user experience of repairing the image. For another example, before the image to be repaired is repaired by the first repairing method, the device determines whether the repairing of the image to be repaired by the first repairing method can meet the repairing requirement, so that the image to be repaired by the first repairing method is prevented from being repaired under the condition that the repairing result of the image to be repaired by the first repairing method does not meet the requirement, and the computing resource is saved.

It can be understood that the first similarity between each first reference region and each corresponding first reference repair region may be determined by calculating the structural similarity and the cosine similarity between each first reference region and each corresponding first reference repair region, and comparing the similarity of histograms between each first reference region and each corresponding first reference repair region.

In a possible implementation of the second aspect, the repair quality evaluation module determines whether the repair result of the area to be repaired meets the repair requirement by: and under the condition that the average value of the first similarities is larger than the similarity threshold, determining that the repair result of the area to be repaired meets the repair requirement.

That is, in the embodiment of the present application, it is determined that the repair result of the to-be-repaired area meets the repair requirement by comparing the average value of the first similarities of each first reference area and each corresponding first reference repair area with the preset similarity threshold.

In a possible implementation of the second aspect, the repair quality evaluation module determines whether the repair result of the area to be repaired meets the repair requirement by: the image restoration module restores at least one second reference area in at least one adjacent image adjacent to the image to be restored in the video by adopting a first restoration method to obtain second reference restoration areas corresponding to the second reference areas; and the repair quality evaluation module calculates second similarity of each second reference area and each corresponding second reference repair area, and determines that the repair result of the area to be repaired meets the repair requirement under the condition that the average value of each second similarity and/or each first similarity is greater than a preset value.

In a possible implementation of the second aspect, a frame difference between each of the adjacent images and the image to be repaired in the video is smaller than a preset value.

That is to say, the frame difference between the adjacent image and the image to be restored in the video is smaller than the preset value, for example, the position of the adjacent image in the video is within 25 frames before and after the image to be restored, so that the similarity between the adjacent image and the image to be restored can be ensured, and the accuracy of determining whether the result of restoring the image to be restored by using the first restoring method meets the restoring requirement is improved.

In a possible implementation of the second aspect, the second reference areas are partial areas in an image with the largest similarity to the image to be restored in adjacent images.

In some embodiments, the above apparatus further comprises: the preprocessing module is used for determining the image with the maximum similarity with the image to be repaired in each adjacent image based on at least one of the following parameters: the structural similarity between each adjacent image and the image to be restored; similarity of histograms of adjacent images and images to be restored; and the cosine similarity between each adjacent image and the image to be restored.

In a possible implementation of the second aspect, the position of each second reference area in the image corresponding to each second reference area is the same as the position of the area to be repaired in the image to be repaired.

That is to say, in the embodiment of the present application, the positions of the second reference regions in the image corresponding to each second reference region are set to be the same as the positions of the regions to be repaired in the image to be repaired, so that the repair quality of the second reference regions to be repaired by using the first repair method can more accurately reflect the repair quality of the regions to be repaired by using the first repair method, and the accuracy of determining whether the result of repairing the regions to be repaired by using the first repair method meets the repair requirement is improved.

In addition, in some embodiments, the shape and size of each second reference region may be set to be the same as those of the region to be repaired, so that the repair quality of the second reference region repaired by using the first repair method can more accurately reflect the repair quality of the region to be repaired by using the first repair method, and the accuracy of determining whether the result of repairing the image to be repaired by using the first repair method meets the repair requirement is improved.

In one possible implementation of the second aspect, the first reference regions and the region to be repaired have the same shape and size.

That is to say, in the embodiment of the present application, the shape and size of each first reference region are set to be the same as those of the region to be repaired, so that the repair quality of the first reference region repaired by using the first repair method can more accurately reflect the repair quality of the region to be repaired by using the first repair method, and the accuracy of determining whether the result of repairing the image to be repaired by using the first repair method meets the repair requirement is improved.

In a possible implementation of the second aspect, a distance between each of the first reference regions and the region to be repaired is less than a preset value.

That is to say, in the embodiment of the present application, the position of each first reference region is set near the region to be repaired, so that the repair quality of repairing the second reference region by using the first repair method can more accurately reflect the repair quality of repairing the region to be repaired by using the first repair method, and the accuracy of determining whether the result of repairing the image to be repaired by using the first repair method meets the repair requirement is improved.

In a possible implementation of the second aspect, the image repairing module repairs the to-be-repaired area in the to-be-repaired image by using the first repairing method when the repairing quality evaluation module determines that the repairing result of the to-be-repaired area meets the repairing requirement; and the image repairing module repairs the area to be repaired in the image to be repaired by adopting a second repairing method under the condition that the repairing quality evaluation module determines that the repairing result of the area to be repaired does not meet the repairing requirement.

That is to say, in the embodiment of the present application, the image repairing apparatus first determines whether a repairing result of the image to be repaired by using the first repairing method satisfies a repairing requirement, and repairs the region to be repaired in the image to be repaired by using the first repairing method when it is determined that the repairing result satisfies the repairing requirement. Therefore, the device can be prevented from determining whether the repair of the image to be repaired by using the first repairing method can meet the repair requirement or not before the image to be repaired by using the first repairing method is repaired, the image to be repaired by using the first repairing method is prevented from being repaired under the condition that the repair result of the image to be repaired by using the first repairing method does not meet the requirement, and computing resources are saved.

And the image restoration device adopts the second restoration method to restore the image to be restored under the condition that the restoration result obtained by adopting the first restoration method to restore the image to be restored does not meet the restoration requirement, so that the quality of image restoration can be improved. It is understood that the second repairing method may be a method completely different from the first repairing method, or may be a method in which parameters of the first repairing method are adjusted.

In a third aspect, an embodiment of the present application provides a readable medium, where instructions are stored on the readable medium, and when executed on an electronic device, the instructions cause the electronic device to perform any one of the image repairing methods described in the first aspect and various possible implementations of the first aspect.

In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory to store instructions for execution by one or more processors of an electronic device; and a processor, which is one of the processors of the electronic device, and is configured to execute instructions to control the electronic device to implement any one of the image inpainting methods described in the first aspect and the various possible implementations of the first aspect of the claims.

Drawings

FIG. 1A illustrates a scene schematic of image inpainting, according to some embodiments of the present application;

fig. 1B to 1D show the repair results of image 0 using different methods;

FIG. 2 illustrates a flow diagram of a method of image inpainting, according to some embodiments of the present application;

FIG. 3A illustrates a schematic view of a region to be repaired in an image, according to some embodiments of the present application;

FIG. 3B illustrates a schematic view of a region to be repaired in yet another image, in accordance with some embodiments of the present application;

FIG. 4A illustrates a schematic diagram of a reference region and a reference region, according to some embodiments of the present application;

FIG. 4B illustrates a schematic view of a reference repair area of the reference area shown in FIG. 4A, according to some embodiments of the present application;

FIG. 5 illustrates a flow diagram of a method of image inpainting, according to some embodiments of the present application;

FIG. 6A illustrates a schematic diagram of an image to be repaired and similar frames of the image to be repaired, according to some embodiments of the application;

FIG. 6B illustrates a schematic diagram of a reference region and a reference region, according to some embodiments of the present application;

FIG. 6C illustrates a schematic view of a reference repair area of the reference area shown in FIG. 6B, according to some embodiments of the present application;

FIG. 7 illustrates a flow diagram of a method of image inpainting, according to some embodiments of the present application;

FIG. 8 is a schematic diagram illustrating an image restoration quality evaluation apparatus, according to some embodiments of the present application;

fig. 9 illustrates a schematic structural diagram of an electronic device 100, according to some embodiments of the present application.

Detailed Description

The technical scheme provided by the embodiment of the application is described below with reference to the accompanying drawings.

Fig. 1A illustrates a scene schematic of image inpainting, according to some embodiments of the present application. As shown in fig. 1, the image 0 includes the subtitle 01, and in order to change the language type of the subtitle, and obtain an image without the subtitle, the electronic device 100 may generally repair the image 0 by using a preset image repair method, for example, by using an image repair method based on a Generative Adaptive Network (GAN) to repair a region where the subtitle 01 is located in the image 0 as a missing region, so as to remove the subtitle 01, thereby obtaining a repaired image 0A, and after the repaired image 0A meets the repair requirement, performing subsequent operations, for example, adding subtitles in other languages, sharing an image without subtitles, and the like. That is, when the image 0 is repaired by using the preset image repairing method, the electronic device 100 determines the area where the subtitle 01 is located as the missing area, and completes the missing area by using the preset image repairing method, so that the repaired image, for example, the image 0A looks natural and the repairing trace is not easily observed from the image 01A.

As described above, currently, the repaired image is usually evaluated in a manual evaluation manner to determine whether the repaired image 0A meets the repair requirement, but the result of evaluating the image repair quality in the manual evaluation manner has a high subjectivity, which is not favorable for objectively evaluating whether the result of repairing the image 0 by the GAN-based image repair method meets the repair requirement, and the efficiency of evaluating the image repair quality in the manual evaluation manner is low, which is not favorable for the electronic device 100 to adjust the parameters of the image repair method according to the repair result or take other remedial measures.

For example, fig. 1B to 1D show images obtained by repairing an area of the image 0 where the subtitle 01 is located by different image repairing methods, where: the caption trace of the image 0B shown in FIG. 1B is still clear, and the repair result can be considered to not meet the repair requirement during manual evaluation; the caption trace of the image 0C shown in FIG. 1C is hardly visible, and the repairing result can be considered to meet the repairing requirement during manual evaluation; however, the image 0D shown in fig. 1D has less subtitle traces, but needs to be identified by careful observation, and the evaluation result of different people for the repair requirement satisfaction of fig. 1D may be different in the manual evaluation.

In view of this, an embodiment of the present application provides an image repairing method, where an electronic device may set a reference region in an image to be repaired in a region other than a region to be repaired, repair the reference region based on a preset image repairing method, generate a reference repairing region (i.e., an image generated by repairing the reference region by the preset image repairing method), and determine whether repairing the region to be repaired in the image to be repaired by the preset image repairing method satisfies a repairing requirement according to a similarity between the reference repairing region and the reference region. By the image restoration method provided by the embodiment of the application, whether the quality of the image restored by the electronic equipment meets the restoration requirement or not can be determined rapidly and objectively, and the efficiency and effectiveness of evaluating the image restoration quality in the process of restoring the image by the electronic equipment are improved, so that the electronic equipment can perform subsequent operations, such as adjusting the preset parameters of the image restoration method and restoring the image to be restored; the image restoration method different from the preset image restoration method is adopted to restore the image to be restored, and the like, so that the accuracy of the electronic equipment for restoring the image can be improved, and the electronic equipment is prevented from outputting a restoration result which does not meet the restoration requirement.

It is understood that the preset image restoration method may be an image restoration method based on a Neural network, such as an image restoration method based on GAN, an image restoration method based on a Convolutional Neural Network (CNN), or an image restoration method based on image Decomposition, such as an image restoration method based on a Singular Value Decomposition (SVD) similarity matrix, or other types of image restoration methods, which is not limited herein.

The following describes the technical solution of the embodiment of the present application in detail with reference to the scenario shown in fig. 1A.

Fig. 2 is a schematic flow chart illustrating evaluation of a repair result by the electronic device 100 after an image to be repaired is repaired by a preset image repairing method according to some embodiments of the present application. The main execution body of the process is the electronic device 100, as shown in fig. 2, the process includes the following steps:

s201: and acquiring the area to be repaired in the image to be repaired. The electronic device 100 acquires the area to be repaired in the image to be repaired so as to set a reference area in an area other than the area to be repaired.

For example, referring to fig. 3A, the electronic device 100 repairs a rectangular area 01A in an image 0 based on an image repairing method of GAN to obtain an image 0A, and the area 01A is an area to be repaired of the image 0.

For another example, in another embodiment, referring to fig. 3B, when the electronic device repairs the image 0 by using a preset image repairing method to remove the subtitle 01, only the region 01B formed by the edges of the characters of the subtitle 01 may be repaired, and at this time, the region 01B is the region to be repaired of the image 0.

S202: and setting at least one reference area in the area of the image to be repaired except the area to be repaired. In the process of evaluating the repair result, the electronic device 100 sets at least one reference region in a region other than the region to be repaired in the image to be repaired, so as to repair the reference region by using a preset image repair method, and compares the repaired reference repair region with the reference region to determine whether the repair result meets the repair requirement.

For example, referring to fig. 3A and 4A, the area to be repaired in the image 0 is a rectangular area 01A, the reference area may be set to have the same size and shape as the rectangular area 01A, and the rectangular area 02 is located above the rectangular area 01A by a certain distance (for example, 100 pixels above the rectangular area 01A), so as to repair the rectangular area 02 by the GAN-based image repairing method to generate a reference repairing area, and the reference repairing area is compared with the rectangular area 02 to determine whether the repairing result of the image repairing of the rectangular area 01A of the image 0 by the GAN-based image repairing method meets the repairing requirement.

It is understood that the shape and size of the reference area 02 are set to be the same as those of the rectangular area 01A in order to more accurately evaluate the repairing effect of the preset image method on the image to be repaired, and in other embodiments, the reference area 02 may also be in other shapes and sizes, for example, an area with the same shape as the rectangular area 01A but different size, an area with other shapes larger or smaller than the rectangular area 01A, and the like, which are not limited herein.

It can be understood that, in some embodiments, the distance between the reference region 02 and the region to be repaired is smaller than a preset value, for example, the distance between the reference region 02 and the center of the region to be repaired is smaller than 200 pixels, so that the similarity between the reference region 02 and the region to be repaired can be ensured, and the repairing effect of the preset image method on the image to be repaired can be evaluated more accurately.

S203: and repairing each reference area by adopting a preset image repairing method to generate a reference repairing area corresponding to each reference area. The electronic device 100 repairs the reference areas set in step S202 by using a preset image repairing method, and generates reference repairing areas corresponding to the reference areas, that is, the electronic device sets the reference areas in the image to be repaired as missing areas, and completes the reference areas, and generates reference repairing areas corresponding to the reference areas.

It can be understood that the preset image restoration method is the same as the method for restoring the image to be restored in the restoration result to be evaluated. For example, referring to the scene shown in fig. 1, if the repair result to be evaluated is an image 0A generated by repairing the image 0 by the GAN-based image repair method, the GAN-based image repair method is also used to repair each reference region, for example, the reference region 02 shown in fig. 4A is repaired, and a reference repair region 02A corresponding to the reference region 02 is generated.

S204: and determining whether the restoration result of the image to be restored meets the restoration requirement or not according to the similarity of each reference area and the reference restoration area corresponding to each reference area. For example, the electronic device 100 may determine that the repair result of the image to be repaired meets the repair requirement when the similarity between each reference region and the reference repair region corresponding to each reference region is greater than the similarity threshold, for example, 0.9.

Specifically, in some embodiments, the electronic device 100 may calculate a Structural Similarity (SSIM) value of the reference region 02 and the reference repair region 02A by the following formula (1) to determine the Similarity of the reference region 02 and the reference repair region 02A.

Wherein, the SSIM (X, Y) values of the SSIM reference region 02 and the reference repair region 02A; x is a reference area 02; y is a reference repair area 02A; mu.s _X Is the average of the pixel values of the reference area 02; mu.s _Y Is an average value of the pixel values of the reference repair region 02A; sigma _XY Covariance as pixel values of the reference region 02 and the reference repair region 02A;

is the variance of the pixel value of the reference area 02, < > is>

Variance of pixel values that are reference repair area 02A; c. C ₁ ＝(0.01L) ² ，c ₂ ＝(0.03L) ² L is a dynamic range of pixel values of the reference region 02 and the reference repair region 02A (for example, when the image 02 and the image 02A are 8-bit images, L is 2 ⁸ -1＝255)。

It can be understood that the value range of the SSIM value calculated by the formula (1) is [ -1,1], the larger the value is, the higher the similarity of the image is, and when X and Y are the same, the SSIM value is 1.

It is understood that, in other embodiments, the SSIMs of the reference region 02 and the reference repair region 02A may be calculated in other manners, which is not limited herein.

It is understood that in other embodiments, the similarity between the reference region 02 and the reference repair region 02A may be determined in other manners, for example, by comparing the similarities between the reference region 02 and the reference repair region 02A and the histogram, calculating the cosine similarity between the reference region 02 and the reference repair region 02A, and the like, which is not limited herein.

It can be understood that, in some embodiments, when the number of the reference areas is multiple, the similarity between each reference area and the reference repair area corresponding to each reference area may be calculated, and the average value of the calculated similarities is compared with the similarity threshold value to determine whether the repair result of the image to be repaired meets the repair requirement.

It can be understood that, in some embodiments, in a case that the electronic device 100 determines that the repair result of the image to be repaired does not meet the repair requirement, the electronic device 100 may adjust a parameter of a preset image repair method, and repair the image to be repaired again, or repair the image to be repaired by using an image repair method different from the preset image repair method. In other embodiments, in a case that the electronic device 100 determines that the repair results of the images to be repaired by the multiple repair methods do not meet the repair requirement, it may further prompt the user that a better repair result cannot be obtained, so that the user may take other measures.

It is to be understood that, in other embodiments, the execution sequence of the steps S201 to S204 may also adopt other sequences, which is not limited herein.

The image restoration method provided by the embodiment of the application can be used for rapidly and objectively evaluating the quality of image restoration, and the efficiency and effectiveness of evaluating the image restoration quality are improved, so that the electronic equipment can conveniently perform subsequent operations, such as adjusting the preset parameters of the image restoration method and restoring the image to be restored; the image restoration method different from the preset image restoration method is adopted to restore the image to be restored, and the like, so that the quality of the image restored by the electronic equipment can be improved, and the electronic equipment is prevented from outputting a restoration result which does not meet the restoration requirement.

The foregoing embodiment introduces the implementation process of the single image restoration method, and if the restored image is a frame image in a video, the reference area may also be set in a video frame other than the video frame (hereinafter referred to as an original frame) where the image to be restored is located, so as to improve accuracy of the evaluation result, further improve accuracy of the image restored by the electronic device, and prevent the electronic device from outputting a restoration result that does not meet the restoration requirement.

The following description is continued with reference to the scene shown in fig. 1 to describe a technical solution for evaluating the restoration quality of an image in a video.

Specifically, fig. 5 shows a flowchart illustrating that the electronic device 100 evaluates a repair result after repairing an image to be repaired in a video by a preset image repairing method according to some embodiments of the present application. The main execution body of the process is the electronic device 100, and referring to fig. 5, the process includes the following steps:

s501: and acquiring a to-be-repaired area in the to-be-repaired image and a plurality of adjacent frames of the video frame corresponding to the to-be-repaired image. The electronic device 100 obtains an area to be repaired in an image to be repaired and images of a plurality of adjacent frames of a video frame corresponding to the image to be repaired, so as to set a reference area.

For example, referring to fig. 1A, the repair result to be evaluated is an image 0A generated by image repair of the image 0 by the GAN-based image processing method shown in fig. 1A. That is, the image to be repaired is image 0, and image 0 is a frame in the video. Referring to fig. 3A, the electronic device 100 may acquire an area to be repaired 01A in the image 0; and referring to fig. 6A, the electronic apparatus 100 may further acquire k frame images after the video frame corresponding to image 0 as adjacent frames of the original frame, such as adjacent frame 1, adjacent frame 2, … … adjacent frame k, so that the electronic apparatus 100 may set a reference area based on area 01A, image 0 and each adjacent frame.

It is understood that, in some embodiments, the plurality of adjacent frames of the original frame may include m frames before and/or m frames after the original frame, which is not limited in this application,

it is understood that, in some embodiments, the frame difference between the adjacent frames of the original frame and the original frame (i.e. the number of video frames spaced between two video frames) does not exceed a preset value, for example, does not exceed 25 frames, i.e. the adjacent frames of the original frame are within the first 25 frames or the last 25 frames of the original frame, so as to ensure the similarity between the adjacent frames and the original frame, and further improve the accuracy of the evaluation result.

S502: at least one reference area is set in an area outside the area to be repaired in the image to be repaired and/or in an adjacent frame of the original frame.

For example, in some embodiments, a reference area having the same shape and size as the area to be repaired may be set in each of an area above the area to be repaired in the image to be repaired and a frame (similar frame) having the greatest similarity to the image to be repaired among a plurality of adjacent frames.

Specifically, in some embodiments, referring to fig. 3A, electronic device 100 may set reference region 02 100 pixels above region 01A in image 0. Referring to fig. 6B, the electronic device 100 may further determine a frame (similar frame) with the largest similarity to the image to be repaired among the adjacent frames 1 to k, for example, when the similarity between the adjacent frame k and the image to be repaired is determined to be the largest, the adjacent frame k is the similar frame, and a reference region 03 is set in the similar frame at the same position as the position of the region 01A in the image 0.

It is understood that, in some embodiments, the similar frame may be determined by calculating the SSIM value between the original frame and the adjacent frame based on the aforementioned formula (1), for example, taking the frame with the largest SSIM value with respect to the original frame in the adjacent frame of the original frame as the similar frame. In other embodiments, the similarity may also be determined by the similarity, cosine similarity, and the like of the histogram between the original frame and each adjacent frame, which is not limited in this embodiment.

It is to be understood that the reference region 03 is set at the same position in the image 0 as the position of the region 01A in the similar frame, so as to more accurately determine whether the repair result obtained by using the preset image repair method to repair the image to be repaired meets the repair requirement, and in other embodiments, the reference region 03 may also be set at other positions in the similar frame, which is not limited herein

It is to be understood that, in some embodiments, the number of the reference areas in the image to be repaired and the number of the reference areas in the adjacent frame may be other, or only the reference areas in the image to be repaired or the adjacent frame may be set, which is not limited herein.

It can be understood that, in some embodiments, the similar frame may be a frame without subtitles, and the electronic device 100 may first identify whether there is a text in multiple adjacent frames of the original frame, and use a frame with the greatest similarity to the image to be repaired in the adjacent frames without texts as the similar frame, so as to more accurately determine whether a repair result obtained by repairing the image to be repaired by using a preset image repairing method satisfies a repair requirement.

S503: and repairing each reference area by adopting a preset image repairing method to generate a reference repairing area corresponding to each reference area. The electronic device 100 repairs each reference area according to a preset image repairing method, and generates a reference repair area corresponding to each reference area, which may specifically refer to step S203 and is not described herein again.

For example, referring to the scenario shown in fig. 1, if the repair result to be evaluated is an image 0A generated by repairing an image 0 by a GAN-based image repair method, the GAN-based image repair method is also used to repair each reference area. For example, repairing the reference area 02 shown in fig. 4A using a GAN-based image repairing method generates a reference repaired area 02A as shown in fig. 4B; for another example, the reference region 03 shown in fig. 6C is repaired by a GAN-based image repairing method to generate a reference repair region 03A.

S504: and calculating the similarity of each reference area and the reference repair area corresponding to each reference area. The electronic device 100 calculates the similarity between each reference region and the reference repair region corresponding to each reference region, and is configured to evaluate whether the repair result of the image to be repaired meets the repair requirement, which may specifically refer to step S204 and is not described herein again.

For example, it can be determined by the aforementioned formula (1) that the SSIM values of the reference region 02 and the reference repair region 02A are 0.97, and the SSIM values of the reference region 03 and the reference repair region 03A are 0.93.

S505: and judging whether the similarity meets a preset condition. For example, the electronic device 100 determines whether the repair result of the image to be repaired meets the repair requirement according to whether the average value of the similarities of the reference areas is greater than the similarity threshold. When the average value of the similarity is larger than the similarity threshold value, determining that the repairing result of the image to be repaired meets the repairing requirement, and turning to the step S506 to output the evaluation result that the repairing result meets the repairing requirement; otherwise, determining that the repairing result of the image to be repaired does not meet the repairing requirement, and turning to the step S507 to output the evaluation result that the repairing result does not meet the repairing requirement.

For example, as described above, if the SSIM values of the reference region 02 and the reference repair region 02A are 0.97, the SSIM values of the reference region 03 and the reference repair region 03A are 0.93, the average value is 0.95, and the similarity is greater than the similarity threshold value of 0.9, it can be determined that the repair result of the image 0 satisfies the repair requirement.

It is understood that, in the case of 1 reference region, the average value of the similarity is the similarity between the reference region and the reference repair region.

S506: and determining that the repairing result of the image to be repaired meets the repairing requirement. The electronic device 100 determines that the repair result of the image to be repaired satisfies the repair requirement when the average value of the aforementioned similarities is greater than the similarity threshold.

In some embodiments, the electronic device 100 may prompt the user when determining that the repair result of the image to be repaired meets the repair requirement, so that the user may share the repair result of the image to be repaired with other electronic devices.

S507: and determining that the repairing result of the image to be repaired does not meet the repairing requirement. The electronic device 100 determines that the repair result of the image to be repaired does not meet the repair requirement when the average value of the aforementioned similarities is less than or equal to the similarity threshold.

In some embodiments, when the electronic device 100 determines that the repair result of the image to be repaired does not meet the repair requirement, the preset parameter of the image repair method may be adjusted to repair the image to be repaired again, or an image repair method different from the preset image repair method may be used to repair the image to be repaired. In other embodiments, in a case that the electronic device 100 determines that none of the repair results of the images to be repaired by the multiple repair methods satisfies the repair requirement, the user may be prompted that a better repair result cannot be obtained, so that the user may take other measures.

It is to be understood that, in other embodiments, the execution sequence of the steps S501 to S507 may also adopt other sequences, which is not limited herein.

By the method provided by the embodiment of the application, the image restoration quality in the video can be evaluated rapidly and objectively, the efficiency and effectiveness of evaluating the image restoration quality are improved, and the evaluation accuracy can be improved because the reference area can be arranged in the video frame with the highest similarity to the image to be restored. The electronic equipment can perform subsequent operations according to the evaluation result, for example, adjusting the parameters of a preset image restoration method and restoring the image to be restored; the image restoration method different from the preset image restoration method is adopted to restore the image to be restored, and the like, so that the accuracy of restoring the image by the electronic equipment can be improved, and the electronic equipment is prevented from outputting a restoration result which does not meet the restoration requirement.

The embodiment of the application further provides an image restoration method, which can firstly determine whether the restoration of the image to be restored by the preset image restoration method meets the restoration requirement, and restore the image to be restored by the method under the condition that the restoration result obtained by restoring the image to be restored by the method can meet the restoration requirement, so that the computing resources of the electronic device are saved.

In particular, fig. 7 shows a flow diagram of an image inpainting method, according to some embodiments of the present application. The process is executed by the electronic device 100, and as shown in fig. 7, the process includes the following steps.

S701: and acquiring a region to be repaired in the image to be repaired.

In the case that the image to be repaired is a single image, reference may be specifically made to step S201, which is not described herein again.

In a case that the image to be restored is one frame of image in the video, the electronic device 100 may further obtain a plurality of adjacent frames of the video frame where the image to be restored is located, which may specifically refer to step S501, and is not described herein again.

S702: and setting at least one reference area in the area of the image to be repaired except the area to be repaired.

In the case that the image to be repaired is a single image, the reference area may be set in step S202, which is not described herein again.

In the case that the image to be repaired is one frame of image in the video, the reference area may be set with reference to step S502, which is not described herein again.

S703: and repairing each reference area by adopting a preset image repairing method to generate a reference repairing area corresponding to each reference area. The specific process may refer to step S203, which is not described herein.

S704: and calculating the similarity of each reference area and the reference repair area corresponding to each reference area. For a specific calculation method, refer to step S204, which is not described herein.

S705: and judging whether the similarity meets a preset condition. When determining that the similarity between the reference region and the reference repair region corresponding to each reference region meets the preset condition, the electronic device 100 indicates that the repair of the image to be repaired by using the preset image repair method can meet the repair requirement, and then the process goes to step S706; otherwise, it indicates that the image to be repaired by the preset image repairing method cannot meet the repairing requirement, and the process goes to step S707. For a specific determination method, refer to step S505, which is not described herein again.

S706: and repairing the image to be repaired by adopting a preset image repairing method. That is, the electronic device 100 repairs the image to be repaired by the method when it is determined that the image to be repaired by the preset image repairing method can meet the repairing requirement.

S707: and repairing the image to be repaired by adopting an image repairing method different from the preset image method. That is, under the condition that it is determined that the image to be repaired by using the preset image repairing method cannot meet the repairing requirement, the electronic device 100 repairs the image to be repaired by using an image repairing method different from the preset image repairing method. For example, referring to the scenario shown in fig. 1A, in a case that it is determined that the repair result obtained by repairing the image to be repaired by the GAN-based image repairing method cannot meet the repair requirement, the image to be repaired may be repaired by using a CNN-based image repairing method, an SVD-similarity-matrix-based image repairing method, or the like.

It is understood that, in other embodiments, the execution sequence of the steps S701 to S707 may also adopt other sequences, and is not limited herein.

By the image restoration method provided by the embodiment of the application, the situation that the image to be restored is restored by a preset image restoration method and cannot meet the restoration result can be avoided, and the computing resources of the electronic equipment are saved.

The embodiment of the application also provides an image restoration device, which can quickly and objectively evaluate the restoration effect of image restoration to be performed by adopting a preset method, so as to improve the objectivity and effectiveness of evaluating the image restoration quality, for example, adjusting the preset parameters of the image restoration method and restoring the image to be restored; the image restoration method different from the preset image restoration method is adopted to restore the image to be restored, and the like, so that the accuracy of the electronic equipment for restoring the image can be improved, and the electronic equipment is prevented from outputting a restoration result which does not meet the restoration requirement.

Specifically, fig. 8 shows a schematic structural diagram of an image restoration apparatus 800 according to some embodiments of the present application, and as shown in fig. 8, the image restoration apparatus 800 includes: a preprocessing module 801, a reference region acquisition module 802, an image restoration module 803, and a restoration quality evaluation module 804.

The preprocessing module 801 is configured to acquire a region to be repaired in an image to be repaired, so that the reference region acquiring module 802 may set a reference region according to the region to be repaired. In addition, when the image to be repaired is one frame of image in the video, the preprocessing module 801 is further configured to obtain an adjacent frame of the video frame where the image to be repaired is located. Specifically, reference may be made to the embodiments shown in fig. 2, fig. 5 and fig. 7, which are not described herein again.

In addition, referring to the embodiment shown in fig. 5, the preprocessing module 801 may also be configured to determine a similar frame of the video frame where the image to be repaired is located.

The reference region acquiring module 802 is configured to set a reference region outside the to-be-repaired region according to the to-be-repaired region in the to-be-repaired image acquired by the preprocessing module 801, so that the image repairing module 803 can repair the reference region according to a preset image repairing method, and generate a reference repaired region corresponding to the reference region. For example, referring to the embodiment shown in fig. 2, the reference region acquiring module 802 may set a reference region outside the region to be repaired in the image to be repaired. For another example, referring to the embodiment shown in fig. 5, the reference region obtaining module 802 may set the reference region in the adjacent frame of the video frame where the image to be repaired is located and the similar frame of the video frame where the image to be repaired is located.

The image restoration module 803 is configured to perform image restoration on the reference area set by the reference area acquisition module 802 according to a preset image restoration method, and generate a reference restoration area corresponding to each reference area, so as to evaluate a restoration result of an image to be restored by the preset image restoration method.

In some embodiments, referring to the embodiment shown in fig. 7, in a case that the repair quality evaluation module 804 determines that the result of repairing the image to be repaired by using the preset image repairing method does not satisfy the preset condition, the image repairing module 803 may also repair the image to be repaired by using another method.

For example, referring to the scenario shown in fig. 1 and the embodiment shown in fig. 7, in a case that the repair quality evaluation module 804 determines that a repair result obtained by repairing the image to be repaired by using the GAN-based image repairing method cannot meet the repair requirement, the image repairing module 803 may repair the image to be repaired by using a CNN-based image repairing method, an SVD-similarity-matrix-based image repairing method, or may adjust parameters of the GAN-based image repairing method to repair the image to be repaired again.

Repair quality evaluation module 804: the image restoration method is used for calculating the similarity between each reference area and the corresponding reference restoration area of each reference area, and determining whether the restoration requirement can be met by adopting a preset image restoration method to restore the image to be restored according to the calculated similarity.

In some embodiments, referring to the embodiments shown in fig. 2, fig. 5, and fig. 7, the repair quality evaluation module 804 may determine the similarity between each reference region and the reference repair region corresponding to each reference region based on the SSIM value and the like between each reference region and the reference repair region corresponding to each reference region generated by the image repair module 803, and determine that the repair requirement can be met by repairing the image to be repaired by using a preset image repair method when the average value of the similarities between each reference region and the reference repair region corresponding to each reference region is greater than a preset similarity threshold.

It is understood that the structure of the image evaluation apparatus 800 shown in fig. 8 is only an example, in other embodiments, the image evaluation apparatus 800 may also include more or fewer modules, and may also merge or split some modules, and the embodiments of the present application are not limited thereto.

It is understood that the electronic device 100 in the foregoing embodiments may be any electronic device, including but not limited to a mobile phone, a smart screen, a desktop computer, a tablet computer, a laptop computer, a wearable device, a head-mounted display, a mobile email device, a portable game console, a Personal Digital Assistant (PDA), a Virtual Reality (VR) or Augmented Reality (AR) device, an ultra-mobile personal computer (UMPC), a netbook, a television with one or more processors embedded or coupled therein, and so on.

Fig. 9 shows a schematic structural diagram of an electronic device 100, according to some embodiments of the present application. As shown in fig. 9, the electronic device 100 may include: one or more processors 101, a system Memory 102, a Non-Volatile Memory (NVM) 103, an input/output (I/O) device 104, a communication interface 105, and system control logic 106 for coupling the processors 101, the system Memory 102, the NVM 103, the I/O device 104, and the communication interface 105. Wherein:

the processor 101 may include a Central Processing Unit (CPU), an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. Wherein, the different processing units may be independent devices or may be integrated in one or more processors. In some embodiments, the processor 101 may be configured to execute instructions to implement the image quality evaluation method provided by the above embodiments. In other implementations, the NPU may also repair the reference area based on a preset image repair method.

The system Memory 102 is a volatile Memory, such as a Random-Access Memory (RAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), and the like. The system memory is used for temporarily storing data and/or instructions, for example, in some embodiments, the system memory 102 may be used for temporarily storing a reference region, a reference region corresponding to the reference region, a reference repair region of the reference region, and the like; and the method can also be used for storing instructions and the like corresponding to the preset image restoration method.

Non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and/or any suitable non-volatile storage device, such as a Hard Disk Drive (HDD), compact Disc (CD), digital Versatile Disc (DVD), solid-State Drive (SSD), and the like. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a Secure Digital (SD) memory card or the like. In some embodiments, the nonvolatile memory 103 may be configured to store an instruction corresponding to the image repairing method provided in each of the above embodiments, may also be configured to store an instruction corresponding to a preset image repairing method, and may also be configured to store an image to be repaired corresponding to a repairing result to be evaluated.

In particular, system memory 102 and non-volatile storage 103 may each include: a temporary copy and a permanent copy of instruction 107. The instructions 107 may include: when executed by at least one of the processors 101, cause the electronic device 100 to implement the image inpainting methods provided by the embodiments of the present application.

Input/output (I/O) device 104 may include a user interface to enable a user to interact with electronic device 100. For example, in some embodiments, input/output (I/O) devices 104 may include output devices such as a display, and may also include input devices such as a keyboard, a mouse, a touch screen, and so forth. A user may interact with the electronic device 100 through a display, a keyboard, a mouse, a touch screen, etc., in order to determine an area to be repaired in an image to be repaired.

The communication interface 105 may include a transceiver to provide a wired or wireless communication interface for the electronic device 100 to communicate with any other suitable device over one or more networks. In some embodiments, the communication interface 105 may be integrated with other components of the electronic device 100, for example, the communication interface 105 may be integrated in the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 105, for example, the electronic device 100 may obtain an image to be repaired from other electronic devices through the communication interface 105, and may also transmit the evaluation result of the image repairing quality to other electronic devices through the communication interface 105.

System control logic 106 may include any suitable interface controllers to provide any suitable interfaces to the other modules of electronic device 100. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

In some embodiments, at least one of the processors 101 may be packaged together with logic for one or more controllers of the System control logic 106 to form a System In Package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same Chip with logic for one or more controllers of the System control logic 106 to form a System-on-Chip (SoC).

Embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of these implementations. Embodiments of the application may be implemented as computer programs or program code executing on programmable systems comprising at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.

Program code may be applied to input instructions to perform the functions described herein and generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

The program code may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. Program code may also be implemented in assembly or machine language, if desired. Indeed, the mechanisms described in this application are not limited in scope to any particular programming language. In any case, the language may be a compiled or interpreted language.

In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy diskettes, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random Access Memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared digital signals, etc.) using the internet in an electrical, optical, acoustical or other form of propagated signal. Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

In the drawings, some features of structures or methods may be shown in a particular arrangement and/or order. However, it is to be understood that such specific arrangement and/or ordering may not be required. Rather, in some embodiments, the features may be arranged in a manner and/or order different from that shown in the illustrative figures. In addition, the inclusion of a structural or methodical feature in a particular figure is not meant to imply that such feature is required in all embodiments, and in some embodiments, may not be included or may be combined with other features.

It should be noted that, in the embodiments of the apparatuses in the present application, each unit/module is a logical unit/module, and physically, one logical unit/module may be one physical unit/module, or may be a part of one physical unit/module, and may also be implemented by a combination of multiple physical units/modules, where the physical implementation manner of the logical unit/module itself is not the most important, and the combination of the functions implemented by the logical unit/module is the key to solve the technical problem provided by the present application. Furthermore, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units/modules which are not so closely related to solve the technical problems presented in the present application, which does not indicate that no other units/modules exist in the above-mentioned device embodiments.

It is noted that, in the examples and descriptions of this patent, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the verb "comprise a" to define an element does not exclude the presence of another, same element in a process, method, article, or apparatus that comprises the element.

While the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. An image restoration method applied to an electronic device, comprising:

repairing at least one first reference region outside the region to be repaired in the image to be repaired by adopting a first repairing method to obtain a first reference repairing region corresponding to each first reference region;

and determining whether the first repairing method is adopted to repair the to-be-repaired area in the to-be-repaired image and whether the obtained repairing result of the to-be-repaired area meets the repairing requirement or not based on the first similarity between each first reference area and each corresponding first reference repairing area.

2. The method according to claim 1, characterized by determining whether the repair result of the area to be repaired meets the repair requirement by:

and under the condition that the average value of the first similarities is greater than a similarity threshold, determining that the repair result of the area to be repaired meets the repair requirement.

3. The method according to claim 2, characterized by determining whether the repair result of the area to be repaired meets the repair requirement by:

repairing at least one second reference area in at least one adjacent image adjacent to the image to be repaired in the video by adopting a first repairing method to obtain second reference repairing areas corresponding to the second reference areas;

and calculating second similarity of each second reference area and each corresponding second reference repair area, and determining that the repair result of the area to be repaired meets the repair requirement under the condition that the average value of each second similarity and/or each first similarity is greater than a preset value.

4. The method according to claim 3, wherein a frame difference between each of the neighboring images and the image to be repaired in the video is smaller than a preset value.

5. The method according to claim 3, wherein each of the second reference regions is a partial region in an image having a maximum similarity with the image to be repaired in each of the neighboring images.

6. The method according to any one of claims 3, wherein the position of each second reference region in the corresponding image of each second reference region is the same as the position of the region to be repaired in the image to be repaired.

7. The method according to any one of claims 1 to 6, wherein each of the first reference region and the region to be repaired has the same shape and size.

8. The method according to claim 7, wherein the distance between each first reference region and the region to be repaired is less than a preset value.

9. The method of any of claim 8, further comprising:

under the condition that the repairing result of the area to be repaired meets the repairing requirement, repairing the area to be repaired in the image to be repaired by adopting the first repairing method;

and under the condition that the repairing result of the area to be repaired is determined not to meet the repairing requirement, repairing the area to be repaired in the image to be repaired by adopting a second repairing method.

10. An image restoration device, characterized by comprising:

the image restoration module is used for restoring at least one first reference area outside the area to be restored in the image to be restored by adopting a first restoration method to obtain a first reference restoration area corresponding to each first reference area;

and the repair quality evaluation module is used for determining whether the repair of the area to be repaired in the image to be repaired by adopting the first repair method is performed based on the first similarity between each first reference area and each corresponding first reference repair area, and the obtained repair result of the area to be repaired meets the repair requirement.

11. The apparatus according to claim 10, wherein the repair quality evaluation module determines whether the repair result of the area to be repaired meets the repair requirement by:

12. The apparatus according to claim 10, wherein the repair quality evaluation module determines whether the repair result of the area to be repaired meets the repair requirement by:

the image restoration module restores at least one second reference area in at least one adjacent image adjacent to the image to be restored in the video by adopting a first restoration method to obtain second reference restoration areas corresponding to the second reference areas;

the repair quality evaluation module calculates second similarity of each second reference region and each corresponding second reference repair region, and determines that the repair result of the region to be repaired meets the repair requirement under the condition that each second similarity and/or the average value of each first similarity is larger than a preset value.

13. The apparatus according to claim 12, wherein a frame difference between each of the neighboring pictures and the picture to be repaired in the video is smaller than a preset value.

14. The apparatus according to claim 12, wherein each of the second reference regions is a partial region in an image having a largest similarity with the image to be restored in each of the neighboring images.

15. The apparatus according to claim 12, wherein the position of each second reference region in the corresponding image of each second reference region is the same as the position of the region to be repaired in the image to be repaired.

16. The apparatus according to any one of claims 10 to 15, wherein each of the first reference region and the region to be repaired is the same in shape and size.

17. The apparatus of claim 16, wherein each of the first reference regions is spaced from the area to be repaired by a distance less than a predetermined value.

18. The apparatus according to any one of claims 17, wherein the image restoration module restores the area to be restored in the image to be restored by using the first restoration method when the restoration quality evaluation module determines that the restoration result of the area to be restored meets the restoration requirement; and the image repairing module adopts a second repairing method to repair the to-be-repaired area in the to-be-repaired image under the condition that the repairing quality evaluation module determines that the repairing result of the to-be-repaired area does not meet the repairing requirement.

19. A readable medium having stored thereon instructions that, when executed on an electronic device, cause the electronic device to implement the image inpainting method of any one of claims 1 to 9.

20. An electronic device, comprising:

a memory to store instructions for execution by one or more processors of an electronic device;

and a processor, which is one of the processors of the electronic device, for executing the instructions to cause the electronic device to implement the image inpainting method of any one of claims 1 to 9.