CN115456908A - Robust self-supervision image denoising method - Google Patents
Robust self-supervision image denoising method Download PDFInfo
- Publication number
- CN115456908A CN115456908A CN202211192386.1A CN202211192386A CN115456908A CN 115456908 A CN115456908 A CN 115456908A CN 202211192386 A CN202211192386 A CN 202211192386A CN 115456908 A CN115456908 A CN 115456908A
- Authority
- CN
- China
- Prior art keywords
- denoising
- noise
- branch
- image
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000006870 function Effects 0.000 claims abstract description 10
- 230000008447 perception Effects 0.000 claims abstract description 6
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 4
- 238000012549 training Methods 0.000 claims description 13
- 238000010606 normalization Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 3
- 230000007547 defect Effects 0.000 abstract description 2
- 230000004913 activation Effects 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention belongs to the technical field of digital image processing, and particularly relates to a robust self-supervision image denoising method based on a pair noise sample. The method comprises the following steps: acquiring a rough denoising image of a noisy image through a pre-denoising network; taking the group as a unit to make a difference between the original noisy image and the rough de-noising image to obtain noise which is approximately truly distributed; performing intra-group cyclic shift operation on the grouped noise twice, and adding the result of the intra-group cyclic shift operation to the rough de-noised image to obtain two grouped noisy samples; a constructed paired noisy sample and an uncertainty perception loss function are utilized to train a dual-branch denoising network, so that the denoising performance and robustness of the network are improved. Experimental results show that the method overcomes the defects of the prior self-supervision image denoising method, effectively improves the definition of a denoised image, and has strong practical value in the method for constructing the denoised sample.
Description
Technical Field
The invention belongs to the technical field of digital image processing, and particularly relates to a robust self-supervision image denoising method.
Background
Image denoising aims at recovering a clean signal from noisy observations, which is one of the important tasks in image processing and low-level computer vision. Recently, with the rapid development of neural networks, a learning-based supervised denoising model has achieved satisfactory performance. However, these methods rely heavily on noise-clean or noise-noise paired images. In practical application, it is complex and expensive to collect such paired images, and even in tasks such as dynamic scenes and medical imaging, due to the limitation of practical conditions, the paired images meeting requirements cannot be obtained at all, which results in that the supervised image denoising method is difficult to adapt to real denoising scenes or to achieve ideal denoising effect.
Compared with a supervised image denoising method, the self-supervised image denoising method has higher practical value because the self-supervised image denoising method does not need a clean picture and a matched image as references. At present, most of self-supervision methods can realize the training of a denoising model only by using a single noisy image, and the core idea of the methods is to construct paired noisy samples from the single noisy image. However, this process is extremely challenging, and the quality of the constructed noisy samples is closely related to the performance of the denoising model. The existing self-supervision denoising method widely adopts two strategies of blind spot convolution and sub-sampling to construct and learn samples, such as N2V [1] 、NBR [4] And B2UB [5] . These strategies achieve the goal of learning the denoising result from a single noise image, but the performance improvement of the denoising model is hindered due to the problems of insufficient information utilization and pixel dislocation in the strategies.
In addition, most of the self-supervision image denoising methods aim at improving the denoising performance of the model, and pay little attention to the robustness of the model, so that the denoising models are very sensitive to unseen noise and generate serious degradation. In practice, the denoising network must be robust in the face of complex scenes because it cannot be guaranteed that the noise distribution and intensity of the collected noisy images are strictly satisfactory. Using uncertainty perceptual loss function in previous studies [3] It has been proven to be effective in improving the stability of the model, but this idea has not been discussed in the denoising task.
Disclosure of Invention
The invention aims to overcome the defects of the existing self-supervision image denoising task, provides a robust self-supervision image denoising method, and obviously improves the performance of a denoising network.
The robust self-supervision image denoising method provided by the invention is based on a method for constructing a pair noise sample; the noisy samples synthesized by the method have the same 'clean' scene and independent and same-distributed noise, so that the problems of low information utilization rate and image misalignment can be effectively relieved, and the performance of a denoising network is remarkably improved; meanwhile, a loss function of uncertain perception is introduced to improve the robustness of the denoising model.
The invention provides a robust self-supervision image denoising method, which comprises the following specific steps:
(1) Pre-denoising: carrying out rough denoising on the image with noise by adopting an N2V denoising model proposed by Krull et al [1 ];
(2) Configured to provide a noisy sample: constructing a pair of noise samples by adopting an original noisy image and a rough de-noised image through simple addition and subtraction and shift operation;
(3) Training a two-branch denoising network: and (3) training a dual-branch denoising network by using the pair of noise samples constructed in the step (2), and adopting a loss function of uncertainty perception to constrain the network in the training process, thereby remarkably improving the denoising performance and robustness of the network.
In the step (1), an N2V denoising model is used and recorded as f (·), and a group of noisy images Y = { Y = is subjected to 1 ，y 2 ，y 3 Pre-denoising to obtain a corresponding rough denoising result X '= { X' 1 ，x′ 2 ，x′ 3 H, where the numerical indices mark different noisy images; namely:
X′＝f(Y)
the process does not involve updating of network parameters.
In the step (2), firstly, the difference between the noisy image and the rough denoising result is obtainedNoise group N = { N) taking near true distribution 1 ，n 2 ，n 3 And that is:
N＝Y-X′
note that the noise within the noise group is independently and equally distributed at this time.
Second, performing two cyclic shift operations on the noise groups results in two sets of noise groups, N, that are only sequentially different 1 And N 2 (ii) a Specifically, the noise in the first bit of the noise group is moved to the last bit, and the operation is repeated twice; namely, the method comprises the following steps:
N 1 ＝{n 2 ，n 3 ，n 1 }
N 2 ＝{n 3 ，n 1 ，n 2 }
finally, the two groups of noise and the rough denoising result are respectively added to form a pair of noisy sample group, Y' A And Y' B (ii) a Namely:
Y′ A ＝X′+N 1 ＝{y′ A1 ，y′ A2 ，y′ A3 }
Y′ B ＝X′+N 2 ＝{y′ B1 ，y′ B2 ，y′ B3 }
further, two pairs of noisy images are constructed by the above process, for example: y' A1 And y' B1 All having the same "clean" scene x' * And independently identically distributed noise, where x represents any numerical subscript.
In the step (3), the paired noisy samples constructed in the step (2) are utilized to train a dual-branch denoising network; specifically, one of the noise samples in the pair of noise samples is used as an input of the dual branch network, and the other noise sample is used as a target to be learned by the network.
The double-branch de-noising network is in DnCNN [2] The structure is obtained on the basis that the first two layers of DnCNN are a convolution layer and a ReLU activation layer, the last layer is a convolution layer, and the last layer is 15 modules consisting of the convolution layer, a batch normalization layer and the ReLU activation layer. In the double-branch network, a convolution layer, a batch normalization layer and a ReLU excitation are accessed in the middle of the DnCN networkThe active layer module is stacked 7 times to form a network module which is used as a denoising branch of the double-branch network; the second half of the original DnCNN is another branch which is an uncertain branch; the two branch networks share the first half of the DnCNN. The convolution kernel size of the convolutional layer in the branched network is 3 × 3. The denoising branch output is a denoised image, and the uncertainty branch output is an uncertainty image.
In the process of training the double branch network, an uncertainty perception loss function is adopted [3] ：
Wherein, K represents the total number of samples contained in the noise sample data set, f 1 (. C) represents a branch of the output denoised result in a two-branch network, u i Is the branch output result of prediction uncertainty in the two-branch network.
Uncertainty branch output uncertainty map measures at pixel level f 1 Confidence level of denoised result of branch output, i.e. at u i In the pixel region having a larger value, f corresponding to f having a higher uncertainty 1 The regions in the output denoised result have lower confidence, i.e. the pixel values with high uncertainty deviate more from those of the true clean image.
Compared with the MSE loss function commonly used in the denoising task:
the dual-branch network trained by the uncertainty loss function has the capability of sensing the denoising difficulty of different regions in an image, each pixel region is not treated equally when the image with noise is denoised, and each pixel is selected more wisely, so that the dual-branch network can distinguish the influence of the unseen noise on the denoising difficulty when the unseen noise is treated, and further has better denoising performance and robustness.
Drawings
FIG. 1 is a flow chart of the present invention.
Fig. 2 the present invention is configured as a flow chart for noisy samples.
FIG. 3 is a schematic diagram of a dual-branch denoising network structure according to the present invention.
FIG. 4 is a graph of the denoising result of a noisy image using the present invention. Wherein, (a) is a noisy image, and (b) is a noise-removed image.
Detailed Description
The embodiments of the present invention will be described in detail below, but the scope of the present invention is not limited to the examples.
The method comprises the following specific steps:
(1) When a rough denoising result is obtained, a noisy image input into a network is randomly cut into 128 x 128 noisy image blocks, and the size of the image is kept unchanged when a noisy sample is constructed subsequently. The number of the samples with noise is 40000 pairs, and the samples are used as a training set;
(2) During training, one of the constructed paired noisy samples serves as an input, and the other serves as a target of the two-branch network learning. The training period of the double-branch network is 100, the initial learning rate of the network is set to be 0.0003, the attenuation rate is set to be 0.5, and the attenuation is carried out once every 20 periods. In the training process, a small batch random gradient descent method is adopted to minimize a loss function, and the batch size is set to be 4;
(3) During testing, the whole test image keeps the original size of the input network, and only the branch result of the output denoised image is used as the final denoising result.
Reference documents:
[1]Krull A,Buchholz T O,Jug F.Noise2Void-Learning Denoising From Single Noisy Images[C]//2019IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2019.Mount J.The equivalence of logistic regression and maximum entropymodels[J].2011.
[2]Zhang,Kai,et al."Beyond a gaussian denoiser:Residual learning of deep cnn for image denoising."IEEE transactions on image processing 26.7(2017):3142-3155.
[3]Kendall,Alex,and Yarin Gal."What uncertainties do we need in bayesian deep learning for computer vision？."Advances in neural information processing systems 30(2017).
[4]Huang T,Li S,Jia X,et al.Neighbor2neighbor:Self-supervised denoising from single noisy images[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.2021:14781-14790.
[5]Wang Z,Liu J,Li G,et al.Blind2Unblind:Self-Supervised Image Denoising with Visible Blind Spots[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:2027-2036。
Claims (4)
1. a robust self-supervision image denoising method based on a constructed pair noise sample is characterized by comprising the following specific steps:
(1) Pre-denoising: carrying out rough denoising on the image with noise by adopting an N2V denoising model;
(2) Configured to provide a noisy sample: constructing a pair of noise samples by adopting an original noisy image and a rough de-noised image through addition and subtraction and shift operations;
(3) Training a two-branch denoising network: and (3) training a dual-branch denoising network by using the pair of noise samples constructed in the step (2), and adopting a loss function of uncertainty perception to constrain the network in the training process, thereby remarkably improving the denoising performance and robustness of the network.
2. The method of claim 1, wherein in step (1), an N2V denoising model, denoted as f (-) is used to perform denoising on a set of noisy images: y = { Y 1 ，y 2 ，y 3 Carrying out pre-denoising to obtain a corresponding rough denoising result: x '= { X' 1 ，x′ 2 ，x′ 3 -wherein the digital subscripts mark different noisy images; namely:
X′＝f(Y)。
3. the self-supervised denoising method of claim 2, wherein the process configured to denoise the sample in step (2) is:
firstly, obtaining a noise group which is approximately distributed in a real way by solving the difference between a noisy image and a rough denoising result: n = { N = 1 ，n 2 ，n 3 And that is:
N＝Y-X′
the noise in the noise group is independently and equally distributed;
secondly, performing two cyclic shift operations on the noise groups to obtain two groups of noise groups with different sequences, and marking the two groups of noise groups as N 1 And N 2 (ii) a Specifically, the operation is to move the noise in the first bit of the noise group to the last bit, and repeat the operation twice, that is, there are:
N 1 ＝{n 2 ，n 3 ，n 1 }
N 2 ＝{n 3 ，n 1 ，n 2 }
and finally, respectively adding the two groups of noise and the rough denoising result to construct a paired noisy sample group: y' A And Y' B ；
Y′ A ＝X′+N 1 ＝{y′ A1 ，y′ A2 ，y′ A3 }
Y′ B ＝X′+N 2 ＝{y′ B1 ，y′ B2 ，y′ B3 }
The two pairs of noisy images constructed by the above process have the same "clean" scene x' * And independently identically distributed noise, where x represents any numerical subscript.
4. The self-supervised denoising method of claim 3, wherein the pair of noisy samples constructed in step (2) in step (3) is used to train a dual-branch denoising network, specifically: one of the noise samples is used as the input of the double-branch network, and the other noise sample is used as the target to be learned by the network; the double-branch denoising network is constructed on the basis of DnCNN, the first two layers of the DnCNN are a convolution layer and a ReLU active layer, the last two layers of the DnCNN are 15 modules consisting of the convolution layer, a batch normalization layer and the ReLU active layer, and the last layer is the convolution layer; a network module formed by stacking a convolution layer, a batch normalization layer and a ReLU active layer module for 7 times is accessed in the middle of the DnCN network and is used as a denoising branch of the double-branch network; the second half of the original DnCNN is another branch which is an uncertain branch; the two branch networks share the first half of the DnCNN; the convolution kernel size of the convolution layer in the branch network is 3 multiplied by 3; the denoising branch output is a denoised image, and the uncertainty branch output is an uncertainty image;
in the process of training the double-branch network, an uncertainty perception loss function is adopted:
wherein K represents the total number of samples contained in the noise sample data set, f 1 (. Represents the branch of the output denoised result in a two-branch network, u i Is a branch output result of prediction uncertainty in a dual-branch network;
uncertainty branch output uncertainty map measures at pixel level f 1 Confidence level of denoised result of branch output, i.e. at u i In the pixel region with larger value, f corresponding to f with higher uncertainty 1 The area in the output denoised result has lower confidence, i.e. the pixel values with high uncertainty deviate more from the pixel values of the real clean image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211192386.1A CN115456908A (en) | 2022-09-28 | 2022-09-28 | Robust self-supervision image denoising method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211192386.1A CN115456908A (en) | 2022-09-28 | 2022-09-28 | Robust self-supervision image denoising method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115456908A true CN115456908A (en) | 2022-12-09 |
Family
ID=84306759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211192386.1A Pending CN115456908A (en) | 2022-09-28 | 2022-09-28 | Robust self-supervision image denoising method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115456908A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118212153A (en) * | 2024-05-22 | 2024-06-18 | 成都信息工程大学 | Ultrasonic image denoising method based on unsupervised learning model |
-
2022
- 2022-09-28 CN CN202211192386.1A patent/CN115456908A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118212153A (en) * | 2024-05-22 | 2024-06-18 | 成都信息工程大学 | Ultrasonic image denoising method based on unsupervised learning model |
| CN118212153B (en) * | 2024-05-22 | 2024-09-13 | 成都信息工程大学 | Ultrasonic image denoising method based on unsupervised learning model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tian et al. | Deep learning on image denoising: An overview | |
| CN109410127B (en) | Image denoising method based on deep learning and multi-scale image enhancement | |
| CN106204467B (en) | Image denoising method based on cascade residual error neural network | |
| CN111161178A (en) | Single low-light image enhancement method based on generation type countermeasure network | |
| CN110148088B (en) | Image processing method, image rain removing method, device, terminal and medium | |
| CN110189260B (en) | Image noise reduction method based on multi-scale parallel gated neural network | |
| CN112819705B (en) | Real image denoising method based on mesh structure and long-distance correlation | |
| CN112991199A (en) | Image high-low frequency decomposition noise removing method based on residual error dense network | |
| Zhao et al. | A simple and robust deep convolutional approach to blind image denoising | |
| CN115345768A (en) | Robust watermark attack method and system based on neural network | |
| Saleem et al. | A non-reference evaluation of underwater image enhancement methods using a new underwater image dataset | |
| CN115456908A (en) | Robust self-supervision image denoising method | |
| CN116823627A (en) | Image complexity evaluation-based oversized image rapid denoising method | |
| CN117392036A (en) | Low-light image enhancement method based on illumination amplitude | |
| Mafi et al. | Deep convolutional neural network for mixed random impulse and Gaussian noise reduction in digital images | |
| Xiao et al. | Effective PRNU extraction via densely connected hierarchical network | |
| CN111047537A (en) | System for recovering details in image denoising | |
| Chen et al. | Image denoising via generative adversarial networks with detail loss | |
| Satti et al. | An efficient noise separation technique for removal of Gaussian and mixed noises in monochrome and color images | |
| CN112927151A (en) | Image denoising method based on edge enhancement and convolutional neural network | |
| Latifi et al. | Image denoising using convolutional neural network | |
| Li et al. | SE–RWNN: an synergistic evolution and randomly wired neural network‐based model for adaptive underwater image enhancement | |
| CN113379641A (en) | Single image rain removing method and system based on self-coding convolutional neural network | |
| Dey et al. | Median filter aided cnn model for removal of gaussian noise from images | |
| CN110415190A (en) | Method, apparatus and processor based on deep learning removal compression of images noise |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |