CN115456908A - A Robust Self-Supervised Image Denoising Method - Google Patents

Abstract

The invention belongs to the technical field of digital image processing, and particularly relates to a robust self-supervision image denoising method based on a pair noise sample. The method comprises the following steps: acquiring a rough denoising image of a noisy image through a pre-denoising network; taking the group as a unit to make a difference between the original noisy image and the rough de-noising image to obtain noise which is approximately truly distributed; performing intra-group cyclic shift operation on the grouped noise twice, and adding the result of the intra-group cyclic shift operation to the rough de-noised image to obtain two grouped noisy samples; a constructed paired noisy sample and an uncertainty perception loss function are utilized to train a dual-branch denoising network, so that the denoising performance and robustness of the network are improved. Experimental results show that the method overcomes the defects of the prior self-supervision image denoising method, effectively improves the definition of a denoised image, and has strong practical value in the method for constructing the denoised sample.

Description

A Robust Self-Supervised Image Denoising Method

technical field

The invention belongs to the technical field of digital image processing, and in particular relates to a robust self-supervised image denoising method.

Background technique

Image denoising aims to recover clean signals from noisy observations, and it is one of the important tasks in image processing and low-level computer vision. Recently, with the rapid development of neural networks, learning-based supervised denoising models have achieved satisfactory performance. However, these methods rely heavily on noise-clean or noise-noise paired images. In practical applications, collecting such paired images is complex and expensive, and even in tasks such as dynamic scenes and medical imaging, due to the constraints of real-world conditions, qualified paired images cannot be obtained at all, which leads to supervised image de- The denoising method is difficult to adapt to the real denoising scene, or it is difficult to achieve the ideal denoising effect.

Compared with supervised image denoising methods, self-supervised image denoising methods are more practical because they do not require clean images and paired images as references. Most of the current self-supervised methods can realize the training of the denoising model only by using a single noisy image. The core idea of these methods is to construct pairs of noisy samples from a single noisy image. However, this process is extremely challenging, and the quality of the constructed pairs of noisy samples is closely related to the performance of the denoising model. Blind convolution and subsampling, two strategies built to learn from samples, are widely used in existing self-supervised denoising methods, such as N2V ^[1] , NBR ^[4] and B2UB ^[5] . These strategies achieve the goal of learning denoising results from a single noisy image, but the performance improvement of denoising models is hindered due to the problems of underutilization of information and pixel misalignment in the strategies.

In addition, most self-supervised image denoising methods focus on improving the denoising performance of the model, while paying little attention to the robustness of the model, resulting in these denoising models being very sensitive to unseen noise and severely degraded. In practice, since the noise distribution and intensity of the collected noisy images cannot be guaranteed to be strictly compliant, the denoising network must remain robust in the face of complex scenes. Using an uncertainty-aware loss function has been shown to be effective in improving model stability in previous studies ^[3] , but this idea has not been discussed for denoising tasks.

Contents of the invention

The purpose of the present invention is to overcome the shortcomings of the existing self-supervised image denoising tasks, propose a robust self-supervised image denoising method, and significantly improve the performance of the denoising network.

The robust self-supervised image denoising method provided by the present invention is based on the method of constructing paired noise samples; the noisy samples synthesized by this method have the same "clean" scene and independent and identically distributed noise, which can effectively alleviate information The problem of low utilization rate and image misalignment has significantly improved the performance of the denoising network; at the same time, an uncertainty-aware loss function is introduced to improve the robustness of the denoising model.

The robust self-supervised image denoising method provided by the present invention has the following steps:

(1) Pre-denoising: use the N2V denoising model proposed by Krull et al. [1] to perform rough denoising on noisy images;

(2) Construct pairs of noisy samples: use the original noisy image and rough denoised image, and construct paired noise samples through simple addition, subtraction and shift operations;

(3) Training a dual-branch denoising network: use the paired noise samples constructed in step (2) to train a dual-branch denoising network. During the training process, an uncertainty-aware loss function is used to constrain the network, thereby significantly improving the denoising of the network. performance and robustness.

In step (1), use the N2V denoising model, denoted as f( ), to pre-denoise a set of noisy images Y={y ₁ , y ₂ , y ₃ }, and obtain the corresponding rough denoising result X ′={x′ ₁ , x′ ₂ , x′ ₃ }, where the numerical subscripts mark different noisy images; namely:

X'=f(Y)

This process does not involve updating of network parameters.

In step (2), first obtain the noise group N={n ₁ , n ₂ , n ₃ } close to the real distribution by calculating the difference between the noisy image and the rough denoising result, namely:

N=Y-X'

Note that the noise in the noise group is independent and identically distributed at this time.

Secondly, perform two cyclic shift operations on the noise group to obtain two groups of noise groups, N ₁ and N ₂ , which differ only in sequence; the specific operation is to move the first noise in the noise group to the last, and repeat two times; that is:

N ₁ ={n ₂ ,n ₃ ,n ₁ }

N ₂ ={n ₃ ,n ₁ ,n ₂ }

Finally, add the two groups of noise and rough denoising results respectively to construct a pair of noisy sample groups, Y′ _A and Y′ _B ; namely:

Y′ _A =X′+N ₁ ={y′ _A1 , y′ _A2 , y′ _A3 }

Y′ _B =X′+N ₂ ={y′ _B1 , y′ _B2 , y′ _B3 }

Furthermore, two pairs of noisy images constructed through the above process, for example: y′ _A1 and y′ _B1 , have the same “clean” scene x′ _* and independent and identically distributed noise, where * represents any number subscript.

In the step (3), the paired noisy samples constructed by the step (2) are used to train a double-branch denoising network; specifically, one noisy sample in the paired noisy samples is used as the input of the double-branched network, and the other The noisy samples serve as the target for the network to learn.

The bipartite denoising branch network is constructed on the basis of DnCNN ^[2] . The first two layers of DnCNN are convolutional layers and ReLU activation layers, followed by 15 convolutional layers, batch normalization layers, and ReLU activation layers. Constructed modules, the last layer is a convolutional layer. In the dual-branch network, a network module composed of convolutional layer, batch normalization layer and ReLU activation layer modules stacked 7 times is connected in the middle of the DnCNN network as a denoising branch of the dual-branch network; in the original DnCNN The second half is another branch, which is the uncertainty branch; the two branch networks share the first half of DnCNN. The kernel size of the convolution layer in the branch network is 3×3. The output of the denoising branch is the denoised image, and the output of the uncertainty branch is the uncertainty image.

In the process of dual-branch network training, an uncertainty-aware loss function ^[3] is used:

Among them, K represents the total number of samples contained in the noisy sample data set, f ₁ (·) represents the branch that outputs the denoising result in the dual-branch network, and u _i is the output result of the branch that predicts uncertainty in the dual-branch network.

The uncertainty map output by the uncertainty branch measures the confidence level of the denoising result output by the f ₁ (·) branch at the pixel level, that is, in u _i , the pixel region with a larger value has a higher uncertainty, Correspondingly, the region in the denoising result output by f ₁ (·) has a lower confidence level, that is, the pixel values with high uncertainty are more deviated from the pixel values of the real clean image.

Compared with the MSE loss function commonly used in denoising tasks:

The dual-branch network trained with the uncertainty loss function has the ability to perceive the difficulty of denoising in different regions of the image. When denoising a noisy image, it does not treat each pixel region equally, but makes more for each pixel. A wise choice, this enables the dual-branch network to distinguish the impact of unseen noise on the difficulty of denoising when dealing with unseen noise, and thus has better denoising performance and robustness.

Description of drawings

Fig. 1 is the flow chart of the present invention.

Fig. 2 is a flow chart of paired noisy samples constructed in the present invention.

Fig. 3 is a schematic diagram of the structure of the dual-branch denoising network of the present invention.

Fig. 4 is a diagram of denoising results of a noisy image using the present invention. Among them, (a) is a noisy image, and (b) is a denoised image.

detailed description

The embodiments of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the examples.

The specific steps are:

(1) When obtaining rough denoising results, the noisy image input to the network is randomly cropped into 128×128 noisy image blocks, and the size of the image remains unchanged when the pair of noisy samples is subsequently constructed. Construct a pair of noisy samples with a number of 40,000 pairs as a training set;

(2) During training, one of the constructed pairs of noisy samples is used as input, and the other is used as the learning target of the dual-branch network. The training cycle of the dual-branch network is 100, the initial learning rate of the network is set to 0.0003, and the decay rate is 0.5, which decays once every 20 cycles. The training process uses the method of small batch stochastic gradient descent to minimize the loss function, and the batch size is set to 4;

(3) During the test, the entire test image keeps its original size and is input to the network, and only the result of the branch that outputs the denoised image is used as the final denoising result.

references:

[1]Krull A, Buchholz T O, Jug F.Noise2Void-Learning Denoising FromSingle Noisy Images[C]//2019IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2019.Mount J.The equivalence of logistic regression and maximum entropymodels[J].2011.

[2] Zhang, Kai, et al."Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising."IEEE transactions on image processing 26.7(2017):3142-3155.

[3]Kendall, Alex, and Yarin Gal."What uncertainties do we need inbayesian deep learning for computer vision?."Advances in neural informationprocessing systems 30(2017).

[4]Huang T, Li S, Jia X, et al.Neighbor2neighbor: Self-supervised denoising from single noisy images[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.2021:14781-14790.

[5]Wang Z, Liu J, Li G, et al.Blind2Unblind:Self-Supervised Image Denoising with Visible Blind Spots[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:2027-2036.

Claims (4)

1. a robust self-supervision image denoising method based on a constructed pair noise sample is characterized by comprising the following specific steps:

(1) Pre-denoising: carrying out rough denoising on the image with noise by adopting an N2V denoising model;

(2) Configured to provide a noisy sample: constructing a pair of noise samples by adopting an original noisy image and a rough de-noised image through addition and subtraction and shift operations;

(3) Training a two-branch denoising network: and (3) training a dual-branch denoising network by using the pair of noise samples constructed in the step (2), and adopting a loss function of uncertainty perception to constrain the network in the training process, thereby remarkably improving the denoising performance and robustness of the network.

2. The method of claim 1, wherein in step (1), an N2V denoising model, denoted as f (-) is used to perform denoising on a set of noisy images: y = { Y ₁ ，y ₂ ，y ₃ Carrying out pre-denoising to obtain a corresponding rough denoising result: x '= { X' ₁ ，x′ ₂ ，x′ ₃ -wherein the digital subscripts mark different noisy images; namely:

X′＝f(Y)。

3. the self-supervised denoising method of claim 2, wherein the process configured to denoise the sample in step (2) is:

firstly, obtaining a noise group which is approximately distributed in a real way by solving the difference between a noisy image and a rough denoising result: n = { N = ₁ ，n ₂ ，n ₃ And that is:

N＝Y-X′

the noise in the noise group is independently and equally distributed;

secondly, performing two cyclic shift operations on the noise groups to obtain two groups of noise groups with different sequences, and marking the two groups of noise groups as N ₁ And N ₂ (ii) a Specifically, the operation is to move the noise in the first bit of the noise group to the last bit, and repeat the operation twice, that is, there are:

N ₁ ＝{n ₂ ，n ₃ ，n ₁ }

N ₂ ＝{n ₃ ，n ₁ ，n ₂ }

and finally, respectively adding the two groups of noise and the rough denoising result to construct a paired noisy sample group: y' _A And Y' _B ；

Y′ _A ＝X′+N ₁ ＝{y′ _A1 ，y′ _A2 ，y′ _A3 }

Y′ _B ＝X′+N ₂ ＝{y′ _B1 ，y′ _B2 ，y′ _B3 }

The two pairs of noisy images constructed by the above process have the same "clean" scene x' _* And independently identically distributed noise, where x represents any numerical subscript.

4. The self-supervised denoising method of claim 3, wherein the pair of noisy samples constructed in step (2) in step (3) is used to train a dual-branch denoising network, specifically: one of the noise samples is used as the input of the double-branch network, and the other noise sample is used as the target to be learned by the network; the double-branch denoising network is constructed on the basis of DnCNN, the first two layers of the DnCNN are a convolution layer and a ReLU active layer, the last two layers of the DnCNN are 15 modules consisting of the convolution layer, a batch normalization layer and the ReLU active layer, and the last layer is the convolution layer; a network module formed by stacking a convolution layer, a batch normalization layer and a ReLU active layer module for 7 times is accessed in the middle of the DnCN network and is used as a denoising branch of the double-branch network; the second half of the original DnCNN is another branch which is an uncertain branch; the two branch networks share the first half of the DnCNN; the convolution kernel size of the convolution layer in the branch network is 3 multiplied by 3; the denoising branch output is a denoised image, and the uncertainty branch output is an uncertainty image;

in the process of training the double-branch network, an uncertainty perception loss function is adopted:

wherein K represents the total number of samples contained in the noise sample data set, f ₁ (. Represents the branch of the output denoised result in a two-branch network, u _i Is a branch output result of prediction uncertainty in a dual-branch network;

uncertainty branch output uncertainty map measures at pixel level f ₁ Confidence level of denoised result of branch output, i.e. at u _i In the pixel region with larger value, f corresponding to f with higher uncertainty ₁ The area in the output denoised result has lower confidence, i.e. the pixel values with high uncertainty deviate more from the pixel values of the real clean image.

Images