CN117830102A - Image super-resolution restoration method, device, computer equipment and storage medium - Google Patents
Image super-resolution restoration method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117830102A CN117830102A CN202311661012.4A CN202311661012A CN117830102A CN 117830102 A CN117830102 A CN 117830102A CN 202311661012 A CN202311661012 A CN 202311661012A CN 117830102 A CN117830102 A CN 117830102A
- Authority
- CN
- China
- Prior art keywords
- frequency
- low
- resolution image
- band
- super
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 239000012633 leachable Substances 0.000 claims abstract description 18
- 238000001228 spectrum Methods 0.000 claims description 39
- 238000004590 computer program Methods 0.000 claims description 24
- 230000008447 perception Effects 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 14
- 230000003042 antagnostic effect Effects 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 10
- 230000000694 effects Effects 0.000 abstract description 5
- 230000003044 adaptive effect Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 5
- 230000009466 transformation Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008485 antagonism Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012886 linear function Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Abstract
The application relates to the technical field of image processing, and provides an image super-resolution restoration method, an image super-resolution restoration device, computer equipment and a storage medium. The method comprises the following steps: inputting the low-resolution image sample into a generator to obtain a super-resolution image; inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample; obtaining first low-frequency information and first high-frequency information of the super-resolution image, and second low-frequency information and second high-frequency information of the high-resolution image sample; and reconstructing the high-resolution image according to the super-resolution image, the high-resolution image sample and the high-low frequency information. By adopting the method, the high-frequency information and the low-frequency information of the image can be effectively separated, the effect of super-resolution processing is enhanced, and a more real and clear high-resolution image is reconstructed.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image super-resolution restoration method, an apparatus, a computer device, a storage medium, and a computer program product.
Background
In practical situations, the acquired digital image is often subject to the influence of conditions such as acquisition environment and equipment, and the detail information is lost, so that the resolution ratio of the acquired digital image is lower. In order to solve the problems brought by the low-resolution image in practical application, the requirements on the image quality and definition are met, an image super-resolution technology is developed, the quality and definition of the image can be improved, and the image can better play a role in various application scenes.
The existing image super-resolution technology is based on an algorithm of a generation type countermeasure network, and separates high-frequency and low-frequency information of an image so as to better utilize different properties of the high-frequency and low-frequency information of the image and convert the low-resolution image into a high-resolution image. However, the existing image super-resolution technology has poor effect of separating high-frequency and low-frequency information of the image, and affects the effect of super-resolution processing.
Disclosure of Invention
In view of the foregoing, it is desirable to provide an image super-resolution restoration method, apparatus, computer device, computer-readable storage medium, and computer program product.
In a first aspect, the present application provides an image super-resolution restoration method, including:
acquiring a low-resolution image sample and a corresponding high-resolution image sample;
inputting the low-resolution image sample into a generator to obtain a super-resolution image;
inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample;
obtaining the first low-frequency information and the first high-frequency information of the super-resolution image according to the first low-frequency characteristic band and the first high-frequency characteristic band;
obtaining the second low-frequency information and the second high-frequency information of the high-resolution image sample according to the second low-frequency characteristic band and the second high-frequency characteristic band;
and reconstructing the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information to obtain a high-resolution image.
In one embodiment, the inputting the super-resolution image and the high-resolution image sample into a binary-based filter and a learnable filter to obtain a first low-frequency feature band and a first high-frequency feature band of the super-resolution image, and a second low-frequency feature band and a second high-frequency feature band of the high-resolution image sample includes:
inputting the super-resolution image and the high-resolution image sample into a binary basis filter to obtain a first low frequency band and a first high frequency band of the super-resolution image and a second low frequency band and a second high frequency band of the high-resolution image sample;
inputting the first low frequency band and the first high frequency band, the second low frequency band and the second high frequency band into a leachable filter to obtain a first low frequency learning band and a first high frequency learning band of the super-resolution image, and a second low frequency learning band and a second high frequency learning band of the high-resolution image sample;
obtaining a first low-frequency characteristic band and a first high-frequency characteristic band according to the first low-frequency band, the first high-frequency band, the first low-frequency learning band and the first high-frequency learning band;
and obtaining a second low-frequency characteristic band and a second high-frequency characteristic band according to the second low-frequency band, the second high-frequency band, the second low-frequency learning band and the second high-frequency learning band.
In one embodiment, the obtaining the first low-frequency information and the first high-frequency information of the super-resolution image according to the first low-frequency feature band and the first high-frequency feature band includes:
performing Fourier transform on the super-resolution image to obtain a first frequency spectrum of the super-resolution image;
performing pixel-level multiplication on the first low-frequency characteristic band and the first frequency spectrum to obtain a first low-frequency product result;
performing pixel-level multiplication on the first high-frequency characteristic band and the first frequency spectrum to obtain a first high-frequency multiplication result;
performing inverse Fourier transform on the first low-frequency product result to obtain the first low-frequency information;
and performing inverse Fourier transform on the first high-frequency product result to obtain the first high-frequency information.
In one embodiment, the obtaining the second low frequency information and the second high frequency information of the high resolution image sample according to the second low frequency feature band and the second high frequency feature band includes:
performing Fourier transform on the high-resolution image sample to obtain a second frequency spectrum of the high-resolution image sample;
performing pixel-level product on the second low-frequency characteristic band and the second frequency spectrum to obtain a second low-frequency product result;
Performing pixel-level product on the second high-frequency characteristic band and the second frequency spectrum to obtain a second high-frequency product result;
performing inverse Fourier transform on the second low-frequency product result to obtain second low-frequency information;
and performing inverse Fourier transform on the second high-frequency product result to obtain the second high-frequency information.
In one embodiment, the reconstructing the high resolution image according to the super resolution image, the high resolution image sample, the first low frequency information, the first high frequency information, the second low frequency information, and the second high frequency information includes:
using low-frequency reconstruction loss for the first low-frequency information and the second low-frequency information to obtain a low-frequency reconstruction loss result;
using an antagonistic loss to the first high frequency information to obtain an antagonistic loss result;
calculating the perception loss of the super-resolution image and the high-resolution image sample to obtain a perception loss result;
and reconstructing to obtain a high-resolution image according to the low-frequency reconstruction loss result, the contrast loss result and the perception loss result.
In one embodiment, after reconstructing the high resolution image, the method further comprises:
Comparing the high-resolution image with the high-resolution image sample to obtain a prediction similarity;
when the predicted similarity is smaller than a threshold value, inputting the low-resolution image sample into a generator again, and continuing training;
and stopping training when the predicted similarity is greater than the threshold value to obtain an image super-resolution restoration algorithm.
In a second aspect, the present application further provides an image super-resolution restoration apparatus, including:
the image sample acquisition module is used for acquiring a low-resolution image sample and a corresponding high-resolution image sample;
the super-resolution image generation module is used for inputting the low-resolution image sample into a generator to obtain a super-resolution image;
the characteristic band acquisition module is used for inputting the super-resolution image and the high-resolution image sample into a binary base filter and a leachable filter to obtain a first low-frequency characteristic band, a first high-frequency characteristic band, a second low-frequency characteristic band and a second high-frequency characteristic band;
the high-low frequency information acquisition module is used for acquiring the first low frequency information and the first high frequency information of the super-resolution image according to the super-resolution image, the first low frequency characteristic band and the first high frequency characteristic band; obtaining the second low-frequency information and the second high-frequency information of the high-resolution image sample according to the high-resolution image sample, the second low-frequency characteristic band and the second high-frequency characteristic band;
And the high-resolution image reconstruction module is used for reconstructing and obtaining a high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor executing the method described above.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which is executed by a processor to perform the above method.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which is executed by a processor to perform the above method.
The above image super-resolution restoration method, apparatus, computer device, storage medium and computer program product acquire a low-resolution image sample and a corresponding high-resolution image sample; inputting the low-resolution image sample into a generator to obtain a super-resolution image; inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample; obtaining first low-frequency information and first high-frequency information of the super-resolution image according to the first low-frequency characteristic band and the first high-frequency characteristic band; obtaining second low-frequency information and second high-frequency information of the high-resolution image sample according to the second low-frequency characteristic band and the second high-frequency characteristic band; and reconstructing the high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information. Inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image, and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample; obtaining first low-frequency information and first high-frequency information of the super-resolution image, and second low-frequency information and second high-frequency information of the high-resolution image sample; and reconstructing the high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information to obtain the high-resolution image, effectively separating the super-resolution image from the high-low frequency information of the high-resolution image, and better utilizing the frequency domain information of the super-resolution image and the high-resolution image to enhance the effect of super-resolution processing, so as to reconstruct a more real and clear high-resolution image.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for a person having ordinary skill in the art.
FIG. 1 is an application environment diagram of an image super-resolution restoration method in one embodiment;
FIG. 2 is a flow chart of a method for restoring super resolution of an image according to an embodiment;
FIG. 3 is a flow diagram of reconstructing a high resolution image in one embodiment;
FIG. 4 is a flow chart of a method for obtaining high and low frequency information in one embodiment;
FIG. 5 is a block diagram illustrating an apparatus for restoring super resolution of an image according to an embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.
The embodiment of the application provides an image Super-resolution restoration method, which can be used for improving an ESRGAN (Enhanced Super-Resolution Generative Adversarial Network, enhanced Super-resolution generation countermeasure network) algorithm, and the embodiment can be executed by computer equipment, as shown in fig. 1, the computer equipment can acquire a low-resolution image sample and a corresponding high-resolution image sample, and the low-resolution image sample is input into a generator to obtain a Super-resolution image; inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample; obtaining first low-frequency information and first high-frequency information of the super-resolution image according to the first low-frequency characteristic band and the first high-frequency characteristic band; obtaining second low-frequency information and second high-frequency information of the high-resolution image sample according to the second low-frequency characteristic band and the second high-frequency characteristic band; and reconstructing the high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information. It will be appreciated that the computer device may be implemented by a server, a terminal, or an interactive system between the terminal and the server. In this embodiment, the method includes the steps shown in fig. 2:
In step S201, a low resolution image sample and a corresponding high resolution image sample are acquired.
Because the training of super-resolution algorithms is mostly dependent on supervised training, high Resolution (HR) and corresponding Low Resolution (LR) image pairs are required.
In the high-low resolution image pair for model training, a high resolution image can be obtained through a camera shooting method and the like, and a corresponding low resolution image is obtained through degradation of high resolution image downsampling (bicubic interpolation downsampling in general), so that a data set obtaining method is simple and convenient.
Step S202, inputting the low-resolution image sample into a generator to obtain a super-resolution image.
The generator consists of a shallow feature extraction module, a depth feature extraction module, an up-sampling module and a reconstruction module, can learn fine details and textures of an image, and converts a low-resolution image sample into a super-resolution image (G (x)).
Step S203, inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image, and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample.
The binary base filter can separate high and low frequency information of the image, and analysis accuracy is improved. The learnable filter may extract characteristic information of the frequency.
The method comprises the steps of inputting a super-resolution image and a high-resolution image sample into a binary basis filter, specifically, carrying out Fourier transform on frequency domain information of the super-resolution image to obtain first frequency domain information, carrying out Fourier transform on the frequency domain information of the high-resolution image sample to obtain second frequency domain information, obtaining corresponding high-low frequency bands of the super-resolution image and the high-resolution image sample according to the first frequency domain information and the second frequency domain information, adding a corresponding leachable filter respectively, adding a combination of the binary basis filter and the corresponding leachable filter to be called an adaptive filter, and when the adaptive filter is used for outputting high-frequency information of the image, obtaining the first low-frequency characteristic band and the first high-frequency characteristic band of the super-resolution image and the second low-frequency characteristic band and the second high-frequency characteristic band of the high-resolution image sample when the adaptive filter is used for outputting low-frequency information of the image, namely the adaptive high-pass filter.
In the step, corresponding high-low frequency bands of the super-resolution image and the high-resolution image sample with higher precision are obtained through a binary basis filter, and the characteristic information of the high-low frequency bands is extracted through a learned filter to a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample.
Step S204, obtaining first low-frequency information and first high-frequency information of the super-resolution image according to the first low-frequency characteristic band and the first high-frequency characteristic band; and obtaining second low-frequency information and second high-frequency information of the high-resolution image sample according to the second low-frequency characteristic band and the second high-frequency characteristic band.
And respectively carrying out pixel-level multiplication on the first low-frequency characteristic band and the first high-frequency characteristic band and the frequency spectrum of the super-resolution image, and then carrying out Fourier transformation to convert the frequency spectrum back into a space domain to obtain the first low-frequency information and the first high-frequency information of the super-resolution image, so that the high-frequency information and the low-frequency information of the super-resolution image can be effectively separated.
And respectively carrying out pixel-level multiplication on the second low-frequency characteristic band and the second high-frequency characteristic band and the frequency spectrum of the high-resolution image sample, and then carrying out Fourier transformation to convert back to a space domain to obtain first low-frequency information and first high-frequency information of the high-resolution image sample, so that the high-frequency information and the low-frequency information of the high-resolution image sample are effectively separated.
Step S205, reconstructing the high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information.
After the high-low frequency information of the super-resolution image and the high-resolution image sample is effectively separated, different loss functions can be used according to the characteristics of different frequency band information, and the high-resolution image can be obtained by means of reconstruction by utilizing different properties of the high-low frequency information.
In the above image super-resolution restoration method, a super-resolution image and a high-resolution image sample are input into a binary basis filter and a learnable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image, and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample; obtaining first low-frequency information and first high-frequency information of the super-resolution image, and second low-frequency information and second high-frequency information of the high-resolution image sample; and reconstructing the high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information to obtain the high-resolution image, effectively separating the super-resolution image from the high-low frequency information of the high-resolution image, and better utilizing the frequency domain information of the super-resolution image and the high-resolution image to enhance the effect of super-resolution processing, so as to reconstruct a more real and clear high-resolution image.
In one embodiment, the super-resolution image and the high-resolution image sample are input into a binary-base filter and a learnable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image, and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample, which specifically comprise the following steps: inputting the super-resolution image and the high-resolution image sample into a binary basis filter to obtain a first low frequency band and a first high frequency band of the super-resolution image and a second low frequency band and a second high frequency band of the high-resolution image sample; inputting the first low frequency band and the first high frequency band, the second low frequency band and the second high frequency band into a leachable filter to obtain a first low frequency learning band and a first high frequency learning band of the super-resolution image, and a second low frequency learning band and a second high frequency learning band of the high-resolution image sample; obtaining a first low-frequency characteristic band and a first high-frequency characteristic band according to the first low-frequency band, the first high-frequency band, the first low-frequency learning band and the first high-frequency learning band; and obtaining a second low-frequency characteristic band and a second high-frequency characteristic band according to the second low-frequency band, the second high-frequency band, the second low-frequency learning band and the second high-frequency learning band.
ExampleThe super-resolution image is input to two binary basis filters w L base And w H base Explicitly dividing the frequency band of the super-resolution image into a first low frequency band and a first high frequency band to obtain the first low frequency band and the first high frequency band of the super-resolution image; then respectively adding a corresponding leachable filter w L learnable And w H learnable Obtaining a first low-frequency learning band and a first high-frequency learning band of the super-resolution image, and finally obtaining the self-adaptive high-pass filter w H And an adaptive low pass filter w L In the form of w respectively H ＝w H base +σ(w H learnable ) And w L ＝w L base +σ(w L learnable ) Wherein the non-linear functionThe variable x is compressed between intervals (-1, 1). Obtaining a first low-frequency characteristic band according to the first low-frequency band and the first low-frequency learning band; and obtaining a first high-frequency characteristic band according to the first high-frequency band and the first high-frequency learning band.
Inputting high resolution image samples into two binary basis filters w L base And w H base Explicitly dividing the frequency band of the high-resolution image sample into a second low frequency band and a second high frequency band to obtain a second low frequency band and a second high frequency band of the high-resolution image sample; then respectively adding a corresponding leachable filter w L learnable And w H learnable Obtaining a second low-frequency learning band and a second high-frequency learning band of the high-resolution image sample, and finally obtaining the self-adaptive high-pass filter w H And an adaptive low pass filter w L In the form of w respectively H ＝w H base +σ(w H learnable ) And w L ＝w L base +σ(w L learnable ) Wherein the non-linear functionThe variable x is compressed between intervals (-1, 1). Obtaining a second low-frequency characteristic band according to the second low-frequency band and the second low-frequency learning band; and obtaining a second high-frequency characteristic band according to the second high-frequency band and the second high-frequency learning band.
In one embodiment, according to the first low-frequency characteristic band and the first high-frequency characteristic band, first low-frequency information and first high-frequency information of the super-resolution image are obtained, and the specific steps are as follows: performing Fourier transform on the super-resolution image to obtain a first frequency spectrum of the super-resolution image; performing pixel level multiplication on the first low-frequency characteristic band and the first frequency spectrum to obtain a first low-frequency multiplication result; performing pixel level multiplication on the first high-frequency characteristic band and the first frequency spectrum to obtain a first high-frequency multiplication result; performing inverse Fourier transform on the first low-frequency product result to obtain first low-frequency information; and performing inverse Fourier transform on the first high-frequency product result to obtain first high-frequency information.
The spectrum of the image may be obtained by fourier transformation. Fourier transform is a method commonly used in signal processing to convert a signal from the time domain to the frequency domain. For an image, one-dimensional Discrete Fourier Transform (DFT) may be performed on the horizontal direction and the vertical direction of the image, respectively, or two-dimensional discrete fourier transform may be performed directly on the entire image.
The spectrum of the image introduces a study of the signal from the time domain to the frequency domain, leading to a more intuitive understanding.
And respectively carrying out pixel-level multiplication on the first low-frequency characteristic band and the first high-frequency characteristic band and the frequency spectrum of the super-resolution image, and then carrying out Fourier transformation to convert the frequency spectrum back into a space domain to obtain the first low-frequency information and the first high-frequency information of the super-resolution image, so that the high-frequency information and the low-frequency information of the super-resolution image can be effectively separated.
Thus for an input super resolution image, an adaptive high pass filter w H The output high frequency information is:
self-adaptingShould low pass filter w L The output low frequency information is:
wherein the method comprises the steps ofRepresenting element-level product, ++>Representing a discrete fourier transform.
In one embodiment, according to the second low-frequency characteristic band and the second high-frequency characteristic band, second low-frequency information and second high-frequency information of the high-resolution image sample are obtained, and the specific steps are as follows: performing Fourier transform on the high-resolution image sample to obtain a second frequency spectrum of the high-resolution image sample; performing pixel-level product on the second low-frequency characteristic band and the second frequency spectrum to obtain a second low-frequency product result; performing pixel-level product on the second high-frequency characteristic band and the second frequency spectrum to obtain a second high-frequency product result; performing inverse Fourier transform on the second low-frequency product result to obtain second low-frequency information; and performing inverse Fourier transform on the second high-frequency product result to obtain second high-frequency information.
And respectively carrying out pixel-level multiplication on the second low-frequency characteristic band and the second high-frequency characteristic band and the frequency spectrum of the high-resolution image sample, and then carrying out Fourier transformation to convert back to a space domain to obtain first low-frequency information and first high-frequency information of the high-resolution image sample, so that the high-frequency information and the low-frequency information of the high-resolution image sample are effectively separated.
In one embodiment, the high resolution image is reconstructed according to the super resolution image, the high resolution image sample, the first low frequency information, the first high frequency information, the second low frequency information and the second high frequency information, and the specific steps are as shown in fig. 3: using low-frequency reconstruction loss for the first low-frequency information and the second low-frequency information to obtain a low-frequency reconstruction loss result; using the resistive loss to the first high frequency information to obtain a resistive loss result; calculating the perception loss of the super-resolution image and the high-resolution image sample to obtain a perception loss result; and reconstructing to obtain a high-resolution image according to the low-frequency reconstruction loss result, the resistance loss result and the perception loss result.
The method provided by the application comprises a generator G, an adaptive frequency domain separation module (AFSM, adaptive Frequency Separation Module) and a discriminator D. Wherein the AFSM is used to separate high and low frequency information of the SR image, comprising a binary basis filter and a learnable filter, also called an adaptive high-pass filter and an adaptive low-pass filter.
Assuming that the input low-resolution image is x, the super-resolution image obtained by the generator G is G (x), the real high-resolution image is y, and the low-frequency information obtained after filtering by using the AFSM is G (x) respectively L And y L The obtained high-frequency information is G (x) H And y H 。
G (x) is a target for keeping the low frequency information before and after super resolution of the image unchanged L By optimizing it with y L L at pixel level in between 1 The loss to reconstruct, i.e. the low frequency reconstruction loss is:
where m represents the batch size (batch size).
In order to make G (x) H The distribution of high-frequency information of the real high-resolution image is close, the perceived quality of the image is further improved, and the use resistance loss is avoided:
further, the high frequency part G (x) of G (x) can be formed H And carrying out Fourier transform to obtain corresponding amplitude spectrum and phase spectrum which are used as the input of the frequency domain discriminator. Since the fourier spectrum does not meet the translational invariance assumption of spatial convolution, the general convolution arbiter structure is no longer applicable. Can flatten the frequency spectrumIs one-dimensional vector, is input into a fully connected network for discrimination after connection, and outputs G (x) H For computing the antagonistic losses of the generator and the frequency domain arbiter, providing guidance on the frequency domain features.
To ensure that by L adv And optimizing the generated high frequency details and the output of L 1 The optimally generated low frequency content is perceptually matched, requiring the computation of the perceptual loss L over the complete super-resolution image G (x) per
Wherein phi is j Representing a characteristic spectrum obtained after convolution operation and before activation of a function by inputting G (x) or y into a pretrained VGG19 network; w (w) j Representing the corresponding weights.
And reconstructing to obtain a high-resolution image according to the low-frequency reconstruction loss result, the resistance loss result and the perception loss result.
In order to facilitate the direct guidance of the generator to generate high frequency information conforming to the spectral distribution of natural images, the reconstruction accuracy and visual perception quality are further improved, and frequency domain antagonism loss can be further introduced on the basis of the airspace antagonism lossAnd reconstructing the high-resolution image according to the low-frequency reconstruction loss result, the contrast loss result, the perception loss result and the frequency domain contrast loss result.
In one embodiment, the method provided by the present application further comprises: comparing the high-resolution image with the high-resolution image sample to obtain a prediction similarity; when the predicted similarity is smaller than the threshold value, inputting the low-resolution image sample into the generator again, and continuing training; and stopping training when the predicted similarity is greater than the threshold value to obtain an image super-resolution restoration algorithm.
And evaluating whether the reconstructed high-resolution image is real or not by using a discriminator, namely comparing the high-resolution image with a high-resolution image sample to obtain a prediction similarity.
When the predicted similarity is smaller than the threshold value, inputting the low-resolution image sample into the generator again, and continuing training until the predicted similarity is larger than the threshold value, stopping training; and stopping training when the predicted similarity is greater than the threshold value to obtain an image super-resolution restoration algorithm.
In order to better understand the above method, an application embodiment of obtaining high-low frequency information of a super-resolution image or a high-resolution image sample in the image super-resolution restoration method of the present application is described in detail below, and the method provided by the application embodiment may be used to improve the ESRGAN algorithm, as shown in fig. 4 in particular.
Taking a super-resolution image as an example, carrying out Fourier transform on frequency domain information of the super-resolution image to obtain first frequency domain information, and inputting the first frequency domain information into a binary base filter and a leachable filter, namely, an adaptive high-pass filter and an adaptive low-pass filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image; performing Fourier transform on the super-resolution image to obtain a first frequency spectrum of the super-resolution image; performing pixel level multiplication on the first low-frequency characteristic band and the first frequency spectrum to obtain a first low-frequency multiplication result; performing pixel level multiplication on the first high-frequency characteristic band and the first frequency spectrum to obtain a first high-frequency multiplication result; performing inverse Fourier transform on the first low-frequency product result to obtain first low-frequency information; and performing inverse Fourier transform on the first high-frequency product result to obtain first high-frequency information. The second low frequency information and the second high frequency information of the high resolution image sample can be obtained in the same way.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides an image super-resolution restoration device for realizing the above related image super-resolution restoration method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation in the embodiments of the apparatus for restoring super-resolution of an image provided below may be referred to the limitation of the method for restoring super-resolution of an image in the above description, which is not repeated here.
In one exemplary embodiment, as shown in fig. 5, there is provided an image super-resolution apparatus, wherein:
an image sample acquisition module 501, configured to acquire a low resolution image sample and a corresponding high resolution image sample;
the super-resolution image generation module 502 is configured to input the low-resolution image sample into a generator to obtain a super-resolution image;
a feature band obtaining module 503, configured to input the super-resolution image and the high-resolution image sample into a binary basis filter and a learnable filter, to obtain a first low-frequency feature band, a first high-frequency feature band, a second low-frequency feature band, and a second high-frequency feature band;
a high-low frequency information obtaining module 504, configured to obtain the first low frequency information and the first high frequency information of the super-resolution image according to the super-resolution image, the first low frequency feature band and the first high frequency feature band; obtaining the second low-frequency information and the second high-frequency information of the high-resolution image sample according to the high-resolution image sample, the second low-frequency characteristic band and the second high-frequency characteristic band;
the high resolution image reconstruction module 505 is configured to reconstruct a high resolution image according to the super resolution image, the high resolution image sample, the first low frequency information, the first high frequency information, the second low frequency information, and the second high frequency information.
In one embodiment, the feature band acquisition module 503 is further configured to: inputting the super-resolution image and the high-resolution image sample into a binary basis filter to obtain a first low frequency band and a first high frequency band of the super-resolution image and a second low frequency band and a second high frequency band of the high-resolution image sample; inputting the first low frequency band and the first high frequency band, the second low frequency band and the second high frequency band into a leachable filter to obtain a first low frequency learning band and a first high frequency learning band of the super-resolution image, and a second low frequency learning band and a second high frequency learning band of the high-resolution image sample; obtaining a first low-frequency characteristic band and a first high-frequency characteristic band according to the first low-frequency band, the first high-frequency band, the first low-frequency learning band and the first high-frequency learning band; and obtaining a second low-frequency characteristic band and a second high-frequency characteristic band according to the second low-frequency band, the second high-frequency band, the second low-frequency learning band and the second high-frequency learning band.
In one embodiment, the high-low frequency information acquisition module 504 is further configured to: performing Fourier transform on the super-resolution image to obtain a first frequency spectrum of the super-resolution image; performing pixel-level multiplication on the first low-frequency characteristic band and the first frequency spectrum to obtain a first low-frequency product result; performing pixel-level multiplication on the first high-frequency characteristic band and the first frequency spectrum to obtain a first high-frequency multiplication result; performing inverse Fourier transform on the first low-frequency product result to obtain the first low-frequency information; and performing inverse Fourier transform on the first high-frequency product result to obtain the first high-frequency information.
In one embodiment, the high-low frequency information acquisition module 504 is further configured to: performing Fourier transform on the high-resolution image sample to obtain a second frequency spectrum of the high-resolution image sample; performing pixel-level product on the second low-frequency characteristic band and the second frequency spectrum to obtain a second low-frequency product result; performing pixel-level product on the second high-frequency characteristic band and the second frequency spectrum to obtain a second high-frequency product result; performing inverse Fourier transform on the second low-frequency product result to obtain second low-frequency information; and performing inverse Fourier transform on the second high-frequency product result to obtain the second high-frequency information.
In one embodiment, the high resolution image reconstruction module 505 is further configured to: using low-frequency reconstruction loss for the first low-frequency information and the second low-frequency information to obtain a low-frequency reconstruction loss result; using an antagonistic loss to the first high frequency information to obtain an antagonistic loss result; calculating the perception loss of the super-resolution image and the high-resolution image sample to obtain a perception loss result; and reconstructing to obtain a high-resolution image according to the low-frequency reconstruction loss result, the contrast loss result and the perception loss result.
In one embodiment, the apparatus further includes a prediction similarity determination module configured to: comparing the high-resolution image with the high-resolution image sample to obtain a prediction similarity; when the predicted similarity is smaller than a threshold value, inputting the low-resolution image sample into a generator again, and continuing training; and stopping training when the predicted similarity is greater than the threshold value to obtain an image super-resolution restoration algorithm.
The above-mentioned modules in the image super-resolution restoration device may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one exemplary embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data of the image super-resolution restoration method. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of image super resolution restoration.
It will be appreciated by those skilled in the art that the structure shown in fig. 6 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components. In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use, and processing of the related data are required to meet the related regulations.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method for restoring super resolution of an image, the method comprising:
acquiring a low-resolution image sample and a corresponding high-resolution image sample;
inputting the low-resolution image sample into a generator to obtain a super-resolution image;
inputting the super-resolution image and the high-resolution image sample into a binary basis filter and a leachable filter to obtain a first low-frequency characteristic band and a first high-frequency characteristic band of the super-resolution image and a second low-frequency characteristic band and a second high-frequency characteristic band of the high-resolution image sample;
Obtaining the first low-frequency information and the first high-frequency information of the super-resolution image according to the first low-frequency characteristic band and the first high-frequency characteristic band;
obtaining the second low-frequency information and the second high-frequency information of the high-resolution image sample according to the second low-frequency characteristic band and the second high-frequency characteristic band;
and reconstructing the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information to obtain a high-resolution image.
2. The method of claim 1, wherein said inputting the super-resolution image and the high-resolution image samples into a binary basis filter and a learnable filter results in a first low-frequency feature band and a first high-frequency feature band of the super-resolution image and a second low-frequency feature band and a second high-frequency feature band of the high-resolution image samples, comprising:
inputting the super-resolution image and the high-resolution image sample into a binary basis filter to obtain a first low frequency band and a first high frequency band of the super-resolution image and a second low frequency band and a second high frequency band of the high-resolution image sample;
Inputting the first low frequency band and the first high frequency band, the second low frequency band and the second high frequency band into a leachable filter to obtain a first low frequency learning band and a first high frequency learning band of the super-resolution image, and a second low frequency learning band and a second high frequency learning band of the high-resolution image sample;
obtaining a first low-frequency characteristic band and a first high-frequency characteristic band according to the first low-frequency band, the first high-frequency band, the first low-frequency learning band and the first high-frequency learning band;
and obtaining a second low-frequency characteristic band and a second high-frequency characteristic band according to the second low-frequency band, the second high-frequency band, the second low-frequency learning band and the second high-frequency learning band.
3. The method according to claim 1, wherein the obtaining the first low frequency information and the first high frequency information of the super resolution image from the first low frequency feature band and the first high frequency feature band includes:
performing Fourier transform on the super-resolution image to obtain a first frequency spectrum of the super-resolution image;
performing pixel-level multiplication on the first low-frequency characteristic band and the first frequency spectrum to obtain a first low-frequency product result;
Performing pixel-level multiplication on the first high-frequency characteristic band and the first frequency spectrum to obtain a first high-frequency multiplication result;
performing inverse Fourier transform on the first low-frequency product result to obtain the first low-frequency information;
and performing inverse Fourier transform on the first high-frequency product result to obtain the first high-frequency information.
4. The method of claim 1, wherein the obtaining the second low frequency information and the second high frequency information of high resolution image samples from the second low frequency feature band and the second high frequency feature band comprises:
performing Fourier transform on the high-resolution image sample to obtain a second frequency spectrum of the high-resolution image sample;
performing pixel-level product on the second low-frequency characteristic band and the second frequency spectrum to obtain a second low-frequency product result;
performing pixel-level product on the second high-frequency characteristic band and the second frequency spectrum to obtain a second high-frequency product result;
performing inverse Fourier transform on the second low-frequency product result to obtain second low-frequency information;
and performing inverse Fourier transform on the second high-frequency product result to obtain the second high-frequency information.
5. The method of claim 1, wherein reconstructing the high resolution image from the super resolution image, the high resolution image sample, the first low frequency information, the first high frequency information, the second low frequency information, and the second high frequency information comprises:
using low-frequency reconstruction loss for the first low-frequency information and the second low-frequency information to obtain a low-frequency reconstruction loss result;
using an antagonistic loss to the first high frequency information to obtain an antagonistic loss result;
calculating the perception loss of the super-resolution image and the high-resolution image sample to obtain a perception loss result;
and reconstructing to obtain a high-resolution image according to the low-frequency reconstruction loss result, the contrast loss result and the perception loss result.
6. The method of claim 1, wherein after reconstructing the high resolution image, the method further comprises:
comparing the high-resolution image with the high-resolution image sample to obtain a prediction similarity;
when the predicted similarity is smaller than a threshold value, inputting the low-resolution image sample into a generator again, and continuing training;
And stopping training when the predicted similarity is greater than the threshold value to obtain an image super-resolution restoration algorithm.
7. An image super-resolution restoration apparatus, characterized in that the apparatus comprises:
the image sample acquisition module is used for acquiring a low-resolution image sample and a corresponding high-resolution image sample;
the super-resolution image generation module is used for inputting the low-resolution image sample into a generator to obtain a super-resolution image;
the characteristic band acquisition module is used for inputting the super-resolution image and the high-resolution image sample into a binary base filter and a leachable filter to obtain a first low-frequency characteristic band, a first high-frequency characteristic band, a second low-frequency characteristic band and a second high-frequency characteristic band;
the high-low frequency information acquisition module is used for acquiring the first low frequency information and the first high frequency information of the super-resolution image according to the super-resolution image, the first low frequency characteristic band and the first high frequency characteristic band; obtaining the second low-frequency information and the second high-frequency information of the high-resolution image sample according to the high-resolution image sample, the second low-frequency characteristic band and the second high-frequency characteristic band;
And the high-resolution image reconstruction module is used for reconstructing and obtaining a high-resolution image according to the super-resolution image, the high-resolution image sample, the first low-frequency information, the first high-frequency information, the second low-frequency information and the second high-frequency information.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311661012.4A CN117830102A (en) | 2023-12-05 | 2023-12-05 | Image super-resolution restoration method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311661012.4A CN117830102A (en) | 2023-12-05 | 2023-12-05 | Image super-resolution restoration method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117830102A true CN117830102A (en) | 2024-04-05 |
Family
ID=90518042
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311661012.4A Pending CN117830102A (en) | 2023-12-05 | 2023-12-05 | Image super-resolution restoration method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117830102A (en) |
-
2023
- 2023-12-05 CN CN202311661012.4A patent/CN117830102A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ilesanmi et al. | Methods for image denoising using convolutional neural network: a review | |
| CN111369440B (en) | Model training and image super-resolution processing method, device, terminal and storage medium | |
| Yu et al. | A unified learning framework for single image super-resolution | |
| Li et al. | FilterNet: Adaptive information filtering network for accurate and fast image super-resolution | |
| Yokota et al. | Simultaneous visual data completion and denoising based on tensor rank and total variation minimization and its primal-dual splitting algorithm | |
| CN111105352A (en) | Super-resolution image reconstruction method, system, computer device and storage medium | |
| Zuo et al. | Convolutional neural networks for image denoising and restoration | |
| CN111415323B (en) | Image detection method and device and neural network training method and device | |
| CN111476719A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN110809126A (en) | Video frame interpolation method and system based on adaptive deformable convolution | |
| Jiang et al. | Fast and high quality image denoising via malleable convolution | |
| Wen et al. | Learning flipping and rotation invariant sparsifying transforms | |
| CN110782398B (en) | Image processing method, generative countermeasure network system and electronic device | |
| CN110298806A (en) | Infrared image enhancing method and system | |
| CN111223046B (en) | Image super-resolution reconstruction method and device | |
| Fang et al. | Learning explicit smoothing kernels for joint image filtering | |
| Zhao et al. | Saliency map-aided generative adversarial network for raw to rgb mapping | |
| CN116630152A (en) | Image resolution reconstruction method and device, storage medium and electronic equipment | |
| CN117830102A (en) | Image super-resolution restoration method, device, computer equipment and storage medium | |
| CN116071279A (en) | Image processing method, device, computer equipment and storage medium | |
| Cho et al. | Example-based super-resolution using self-patches and approximated constrained least squares filter | |
| Liu et al. | CNN-Enhanced graph attention network for hyperspectral image super-resolution using non-local self-similarity | |
| CN113128583A (en) | Medical image fusion method and medium based on multi-scale mechanism and residual attention | |
| CN111199574A (en) | Holographic image generation method and equipment | |
| CN113902618B (en) | Image super-resolution algorithm based on multi-modal spatial filtering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |