CN113160056A - Deep learning-based noisy image super-resolution reconstruction method - Google Patents

Abstract

The invention discloses a super-resolution reconstruction method of a noisy image based on deep learning, which comprises the following steps of carrying out down-sampling and noise-adding processing on a high-resolution image in a data set to generate a simulated low-resolution image and a simulated camera image; constructing an integral network to perform hyper-resolution reconstruction on the simulation sample; adding a Patch-GAN structure to the output end of the whole network, thereby reducing the noise after reconstruction and completing the primary training of the whole network; and extracting an edge part of the low-resolution image without noise by using an edge extraction operator, selecting a part of the edge image block with the modulo sum larger than a threshold value to enter a new training data set, and finely tuning the network by using the image block with rich edges. According to the method, the Patch-GAN can be utilized to enable the network to be more concentrated on the image detail part, meanwhile, in order to avoid the influence of random picture cutting, pixel blocks in the edge area are selected in the training process, a new data set is added, the network model is finely adjusted, and a good visual effect can be obtained for the noisy picture super-resolution reconstruction.

Description

Deep learning-based noisy image super-resolution reconstruction method

Technical Field

The invention relates to the technical field of computer vision image processing, in particular to a noisy image super-resolution reconstruction method based on deep learning.

Background

In recent years, with the development of technology, the way in which people acquire information has been diversified. The image is used as a main carrier and medium for information transmission, and is a great important source for the world to be recognized. The higher the resolution of the image, the more information is contained therein. However, in real life, the reduction of visual experience and related device performance is often caused by the too low resolution of images or videos, for example, the entertainment is reduced by the too low resolution of watching television series, and the low resolution in a security scene causes the waste of police strength. Therefore, it is important to improve the resolution of the picture.

On one hand, the limited performance of the acquisition equipment in physical reality can restrict the quality effect of the image; on the other hand, the storage space also directly affects the pixel range of the picture. From a hardware perspective, improving the quality of the acquisition and storage device will improve the visualization of the image from the source, but with a substantial increase in hardware costs. From the software perspective, the imaging quality can be improved by adopting various image super-resolution algorithms, and the process of clarifying the low-resolution images is realized by learning the mapping relation from low resolution to high resolution. The image on the software level is over-divided, so that the cost is reduced and the feasibility is higher.

Early on there were frequency domain based, interpolation based, reconstruction based image super-resolution algorithms. However, since these processes are relatively simple, the recovered image has various disadvantages, such as blurring of the super-divided picture due to simple interpolation, because no additional information is introduced by interpolation, and noise is amplified during the super-division. With the development of computer performance, deep learning becomes the mainstream method of many problems in the field of computer vision. In the super-resolution field of image video, the deep learning related algorithm refreshes the highest performance (SOTA) of the field again and again.

Hinton and Alex in 2012 applied convolutional neural networks to image classification with great success, and subsequently proposed in the industry that excellent networks such as SRCNN, FSRCNN, DRCN, VDSR and the like are gradually improved in image reconstruction performance. Inspired by the excellent generation effect of the antagonistic generation network GAN, the image over-partitioning task is gradually converted into two parts of image generation and authenticity judgment, which typically represent SRGAN, ESRGAN and the like, so that the texture detail part of the picture is further improved.

However, the current super-resolution reconstruction algorithm is usually directed to super-resolution reconstruction of a noise-free picture, and a picture taken by a camera in a real situation contains large environmental noise. How to perform super-resolution reconstruction on noisy pictures has greater application significance.

Disclosure of Invention

The purpose of the invention is as follows: in order to overcome the defects in the prior art, the invention provides a noisy image super-resolution reconstruction method based on deep learning, and the method can better remove noise in the process of carrying out super-resolution reconstruction on noisy images.

The technical scheme is as follows: in order to achieve the above object, the present invention provides a method for super-resolution reconstruction of a noisy image based on deep learning, comprising the steps of,

step 1, carrying out down-sampling and noise-adding processing on a high-resolution image in a public data set to generate a simulated low-resolution image and a simulated camera image, taking the low-resolution image as input image data of an integral network, forming the input image data and an original high-resolution image into an image data pair, training a model, and simultaneously carrying out data enhancement in the training process;

and 2, adopting a generated confrontation network series structure as a main body of the whole network and carrying out hyper-resolution reconstruction on the low-resolution analog camera noise image obtained in the step 1. In order to obtain a better generation effect, an enhanced super-resolution generation countermeasure network is adopted, network parameters are initialized, supervision information is high-resolution image data, end-to-end training is carried out on the model, and the network model is stored when the training loss is reduced and converged and the PSNR (visual image index) is stably raised;

step 3, adding a Patch-GAN structure as an integral network to the output end of the enhanced super-resolution generation countermeasure network, thereby reducing the noise after reconstruction and completing the primary training of the integral network;

and 4, extracting an edge part of the low-resolution image without noise by using an edge extraction operator, selecting a module sum of an edge image block which is larger than a threshold value part to enter a new training data set, and finely adjusting the whole network by using image blocks with rich edges, so that the whole network can pay attention to the detail texture.

Further, in the present invention: said step 1 further comprises the step of,

step 1-1, selecting a public data set DIV2K, and performing down-sampling on a high-resolution image in the data set to obtain a low-resolution image;

step 1-2, three kinds of noises of shot noise, dark noise and read noise existing in a camera are added on a low-resolution image, and a simulated camera noise image which is closer to the camera generation is simulated;

step 1-3: further adding noise to the analog camera noise image to obtain an analog camera noise image, wherein the added noise is Gaussian noise;

step 1-4: and performing data enhancement on the noise-added image of the low-resolution analog camera so as to increase the training data amount, wherein the data enhancement mode comprises turning and cutting.

Further, in the present invention: said step 2 further comprises the step of,

step 2-1: initializing network parameters;

step 2-2, in the training process of the network, if the network is too large and the output is 4 times of the input size, the whole sample image cannot be directly input into the whole network for processing due to too large calculated amount, and the image data pair obtained in the step 1 is cut to obtain a Patch block pair with smaller pixels;

step 2-3: generating a super-resolution picture by using a generator of a super-resolution network, and judging whether the reconstructed picture is more real or not by the whole network;

step 2-4: training the whole network by using a random gradient descent method, wherein supervision information is a high-resolution image of a data set, when the training loss is reduced and converged and the signal-to-noise ratio of the image visual index peak value is increased and stabilized, the model is stored, and the training is finished, otherwise, pixel blocks in an edge area are selected and added into the data set, and the training is continued;

step 2-5: in the forward reasoning stage, the test picture set is cut regularly from top to bottom and from left to right, the cut test picture set is respectively input into the network generator part for prediction and then spliced to obtain a high-resolution image, and the image is output after being spliced.

Further, in the present invention: said step 3 further comprises the step of,

step 3-1: the arbiter of the enhanced super-resolution generation countermeasure network introducing Patch-GAN maps the output to a real number representing the probability that the sample is real, and replaces the probability value with a Patch of N × N size, where each value in the Patch represents the probability that the input corresponding block is real. The nxn block of values may be decided through the step 2-3;

step 3-2: the whole network obtained after the Patch-GAN structure is added at the output end outputs each block to predict the authenticity of a part of input areas, and the detail part influenced by noise can be paid more attention in the reconstruction process, so that the noise after reconstruction is reduced, and the detail is improved.

Further, in the present invention: said step 4 further comprises the step of,

step 4-1: the large size image is cropped to a picture Patch. During each training, a certain area block is randomly cut from the image and input into the network, the problem of overlarge calculated amount in the training process can be avoided by using the method, the training is accelerated, and meanwhile, the training data set is expanded, so that the training is more sufficient;

step 4-2: after extracting edge parts of the low-resolution image without noise by using an edge extraction operator, selecting a module of the edge image and a part larger than a threshold value to enter a new training data set for model fine adjustment, and extracting a picture block by using a second-order differential operator Laplacian operator.

Has the advantages that: compared with the prior art, the invention has the beneficial effects that:

(1) in order to deeply fit the over-resolution reconstruction of the image with noise, the invention simulates the noise of a camera according to the scene requirement, and adds shot noise, dark noise and read noise to generate a low-resolution picture with noise;

(2) in order to reduce the influence of noise on the hyper-resolution result, the method provided by the invention utilizes Patch-GAN to enable the network to focus more on the image detail part;

(3) in order to avoid the influence of random clipping of the picture, the method provided by the invention utilizes a Laplacian operator to select the edge of the noiseless picture in the training process, and adds a pixel block with slightly larger edge area information into a new data set to be used as an input fine-tuning over-division network.

Gaussian noise is the most common type of noise, and in many scenarios we use gaussian noise to model real noise. In the invention, the simulation process of the picture of the to-be-photographed camera is more complex, unequal noises are added in sequence, and the model can be more robust by training the sample simulated by the complex noises;

the generation of the confrontation network can have better detailed information so as to improve the visual effect of the picture. In the patent, a decision maker is replaced by a Patch-GAN structure, so that a model focuses on a detail part further;

randomly clipped Patch may be a simple background, i.e., Patch is relatively simple, training with such a dataset may make it difficult for the network to focus on the details, fitting purely colored backgrounds. The patent proposes to use an edge extraction convolution kernel to extract pixel blocks with obvious edge changes (rich edge information) so as to fine tune the network, thereby enabling the network to pay attention to details and reducing the degree of overfitting of the network to the background.

Drawings

FIG. 1 is a schematic overall flow chart of the deep learning-based noisy image super-resolution reconstruction method.

Detailed Description

The technical scheme of the invention is further explained in detail by combining the attached drawings:

the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

As shown in fig. 1, an overall flow diagram of a method for super-resolution reconstruction of a noisy image based on deep learning according to the present invention is provided, which specifically includes the following steps,

step 1, carrying out down-sampling and noise-adding processing on a high-resolution image in a public data set to generate a simulated low-resolution image and a simulated camera image, taking the low-resolution image as input image data of an integral network, forming the input image data and an original high-resolution image into an image data pair, training a model, and simultaneously carrying out data enhancement in the training process;

specifically, the step 1 further comprises the following steps,

step 1-1, selecting a public data set DIV2K, and performing down-sampling on a high-resolution image in the data set to obtain a low-resolution image;

step 1-2, three kinds of noises of shot noise, dark noise and read noise existing in a camera are added on a low-resolution image, and a simulated camera noise image which is closer to the camera generation is simulated; wherein, the shot noise is the statistical fluctuation caused by the photon dispersion characteristic, the dark noise is the statistical fluctuation caused by the electron dispersion characteristic, and the readout noise is the statistical fluctuation caused by the circuit amplification or the digital-to-analog conversion; analog camera noise images are more blurred than low resolution definition images.

Step 1-3: further adding noise to the analog camera noise image to obtain an analog camera noise image, wherein the added noise is Gaussian noise; according to the noise adding strategy, the simulated noise-added low-resolution image has more fuzzy detail textures than an image obtained by simply adding Gaussian noise, and is more consistent with a real scene;

step 1-4: and performing data enhancement on the noise-added image of the low-resolution analog camera so as to increase the training data amount, wherein the data enhancement mode comprises turning and cutting. More training data is obtained so that the model is fully trained.

And 2, adopting a generated confrontation network series structure as a main body of the whole network and carrying out hyper-resolution reconstruction on the low-resolution analog camera noise image obtained in the step 1. In order to obtain a better generation effect, an enhanced super-resolution generation countermeasure network is adopted, network parameters are initialized, supervision information is high-resolution image data, end-to-end training is carried out on the model, and the model is saved when the training loss is reduced and converged and the PSNR (visual image index) is raised and stabilized;

specifically, the step 2 further comprises the following steps:

step 2-1: initializing network parameters;

step 2-2, in the training process of the network, if the network is too large and the output is 4 times of the input size, the whole sample image cannot be directly input into the whole network for processing due to too large calculated amount, and the image data pair obtained in the step 1 is cut to obtain a Patch block pair with smaller pixels;

step 2-3: generating a super-resolution picture by using a generator of a super-resolution network, and judging whether the reconstructed picture is more real or not by the whole network; specifically, the probability that the true picture is more true relative to the hyper-resolution picture is judged by the countermeasure network judger through the enhanced super-resolution generation;

step 2-4: and training the whole network by using a random gradient descent method, wherein the supervision information is a high-resolution image of the data set. And when the training loss is reduced and converged and the image visual index peak signal-to-noise ratio is increased and stabilized, storing the model, finishing the training, and otherwise, selecting pixel blocks in the edge area to add into the data set and continuing the training.

Step 2-5: in the forward reasoning stage, the test picture set is cut from top to bottom and from left to right regularly, the test picture set is respectively input into a network generator part for prediction and then spliced to obtain a high-resolution image, and the image is output after being spliced;

specifically, the initial enhanced super-resolution generation countermeasure network is introduced into Patch-GAN and is added with an improved network obtained by edge fine tuning, and compared with the initial enhanced super-resolution generation countermeasure network, an image obtained by reconstruction is closer to a real image, and a better generation effect can be obtained.

Step 3, adding a Patch-GAN structure as an integral network to the output end of the enhanced super-resolution generation countermeasure network, thereby reducing the noise after reconstruction and completing the primary training of the integral network;

specifically, the step 3 further includes the following steps:

step 3-1: the arbiter of the enhanced super-resolution generation countermeasure network introducing Patch-GAN maps the output to a real number representing the probability that the sample is real, and replaces the probability value with a Patch of N × N size, where each value in the Patch represents the probability that the input corresponding block is real. The nxn block of values may be decided through the step 2-3;

step 3-2: the whole network obtained after the Patch-GAN structure is added at the output end outputs each block to predict the authenticity of a part of input areas, and the detail part influenced by noise can be paid more attention in the reconstruction process, so that the noise after reconstruction is reduced, and the detail is improved.

And 4, extracting an edge part of the low-resolution image without noise by using an edge extraction operator, selecting a module sum of an edge image block which is larger than a threshold value part to enter a new training data set, and finely adjusting the whole network by using image blocks with rich edges, so that the whole network can pay attention to the detail texture.

Specifically, the step 4 further includes the following steps:

step 4-1: the noiseless low resolution sharpness image is cropped to a picture Patch Patch. During each training, a certain area block is randomly cut from the image and input into the network, the problem of overlarge calculated amount in the training process can be avoided by using the method, the training is accelerated, and meanwhile, the training data set is expanded, so that the training is more sufficient;

step 4-2: after extracting edge parts of the low-resolution image without noise by using an edge extraction operator, selecting a module of the edge image and a part larger than a threshold value to enter a new training data set for model fine adjustment, and extracting a picture block by using a second-order differential operator Laplacian operator. The Laplacian operator is sensitive to noise, so that the Laplacian operator is used for extracting edges from low-resolution pictures without noise, the same data set contains images with larger edge information difference, for picture blocks with smaller edges, the absolute value sum of the values of the extracted edges is smaller by using the edge extraction operator, the picture blocks can be filtered in a fine adjustment stage, and for picture blocks with larger edges, the fine adjustment data set is added and input into a network which is preliminarily trained in the network fine adjustment step 2-4, so that the network pays more attention to detail textures, and the overfitting degree of the network to a background is reduced.

It should be noted that the above-mentioned examples only represent some embodiments of the present invention, and the description thereof should not be construed as limiting the scope of the present invention. It should be noted that, for those skilled in the art, various modifications can be made without departing from the spirit of the present invention, and these modifications should fall within the scope of the present invention.

Claims (5)

1. A method for reconstructing super-resolution of a noisy image based on deep learning is characterized by comprising the following steps: comprises the following steps of (a) carrying out,

step 1, carrying out down-sampling and noise-adding processing on a high-resolution image in a public data set to generate a simulated low-resolution image and a simulated camera image, taking the low-resolution image as input image data of an integral network, forming the input image data and an original high-resolution image into an image data pair, training a model, and simultaneously carrying out data enhancement in the training process;

step 2, adopting a series of structures of a generated countermeasure network as a main body of the whole network and carrying out hyper-resolution reconstruction on the low-resolution analog camera noise image obtained in the step 1; in order to obtain a better generation effect, an enhanced super-resolution generation countermeasure network is adopted, network parameters are initialized, supervision information is high-resolution image data, end-to-end training is carried out on the model, and the network model is stored when the training loss is reduced and converged and the PSNR (visual image index) is stably raised;

step 3, adding a Patch-GAN structure as an integral network to the output end of the enhanced super-resolution generation countermeasure network, thereby reducing the noise after reconstruction and completing the primary training of the integral network;

and 4, extracting an edge part of the low-resolution image without noise by using an edge extraction operator, selecting a module sum of an edge image block which is larger than a threshold value part to enter a new training data set, and finely adjusting the whole network by using image blocks with rich edges, so that the whole network can pay attention to the detail texture.

2. The deep learning-based noisy image super-resolution reconstruction method according to claim 1, characterized in that: said step 1 further comprises the step of,

step 1-1, selecting a public data set DIV2K, and performing down-sampling on a high-resolution image in the data set to obtain a low-resolution image;

step 1-2, three kinds of noises of shot noise, dark noise and read noise existing in a camera are added on a low-resolution image, and a simulated camera noise image which is closer to the camera generation is simulated;

step 1-3: further adding noise to the analog camera noise image to obtain an analog camera noise image, wherein the added noise is Gaussian noise;

step 1-4: and performing data enhancement on the noise-added image of the low-resolution analog camera so as to increase the training data amount, wherein the data enhancement mode comprises turning and cutting.

3. The deep learning-based noisy image super-resolution reconstruction method according to claim 1 or 2, characterized in that: said step 2 further comprises the step of,

step 2-1: initializing network parameters;

step 2-2, in the training process of the network, if the network is too large and the output is 4 times of the input size, the whole sample image cannot be directly input into the whole network for processing due to too large calculated amount, and the image data pair obtained in the step 1 is cut to obtain a Patch block pair with smaller pixels;

step 2-3: generating a super-resolution picture by using a generator of a super-resolution network, and judging whether the reconstructed picture is more real or not by the whole network;

step 2-4: training the whole network by using a random gradient descent method, wherein supervision information is a high-resolution image of a data set, when the training loss is reduced and converged and the signal-to-noise ratio of the image visual index peak value is increased and stabilized, the model is stored, and the training is finished, otherwise, pixel blocks in an edge area are selected and added into the data set, and the training is continued;

step 2-5: in the forward reasoning stage, the test picture set is cut regularly from top to bottom and from left to right, the cut test picture set is respectively input into the network generator part for prediction and then spliced to obtain a high-resolution image, and the image is output after being spliced.

4. The deep learning-based noisy image super-resolution reconstruction method according to claim 3, wherein: said step 3 further comprises the step of,

step 3-1: the arbiter of the enhanced super-resolution generation countermeasure network introducing Patch-GAN maps the output to a real number representing the probability that the sample is real, and replaces the probability value with a Patch of N × N size, where each value in the Patch represents the probability that the input corresponding block is real. The nxn block of values may be decided through the step 2-3;

step 3-2: the whole network obtained after the Patch-GAN structure is added at the output end outputs each block to predict the authenticity of a part of input areas, and the detail part influenced by noise can be paid more attention in the reconstruction process, so that the noise after reconstruction is reduced, and the detail is improved.

5. The deep learning-based noisy image super-resolution reconstruction method according to claim 4, wherein: said step 4 further comprises the step of,

step 4-1: the large size image is cropped to a picture Patch. During each training, a certain area block is randomly cut from the image and input into the network, the problem of overlarge calculated amount in the training process can be avoided by using the method, the training is accelerated, and meanwhile, the training data set is expanded, so that the training is more sufficient;

step 4-2: after extracting edge parts of the low-resolution image without noise by using an edge extraction operator, selecting a module of the edge image and a part larger than a threshold value to enter a new training data set for model fine adjustment, and extracting a picture block by using a second-order differential operator Laplacian operator.

Images