CN112801897B - Image denoising method based on wide convolution neural network - Google Patents
Image denoising method based on wide convolution neural network Download PDFInfo
- Publication number
- CN112801897B CN112801897B CN202110071024.6A CN202110071024A CN112801897B CN 112801897 B CN112801897 B CN 112801897B CN 202110071024 A CN202110071024 A CN 202110071024A CN 112801897 B CN112801897 B CN 112801897B
- Authority
- CN
- China
- Prior art keywords
- network
- sub
- image
- wavelet
- wcnn
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 15
- 238000012549 training Methods 0.000 claims abstract description 55
- 230000006870 function Effects 0.000 claims abstract description 33
- 230000009466 transformation Effects 0.000 claims abstract description 7
- 238000013527 convolutional neural network Methods 0.000 claims description 19
- 238000012360 testing method Methods 0.000 claims description 13
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 238000010200 validation analysis Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 230000004913 activation Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 3
- 230000000007 visual effect Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 11
- 230000000694 effects Effects 0.000 description 6
- 238000009826 distribution Methods 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 230000001965 increasing effect Effects 0.000 description 3
- 238000002679 ablation Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000004904 shortening Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000452 restraining effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20064—Wavelet transform [DWT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image denoising method based on a wide convolution neural network, which is characterized in that a sub-network is deployed on each wavelet sub-band and is responsible for capturing and learning image features in a specific direction and in a specific scale, each sub-network not only has an independent convolution neural network structure and comprises fewer convolution layers, but also has own loss functions, the loss functions are used for supervising the learning process of each sub-network, so that WCNN can process noise in a certain range by using a set of learning parameters, and when all the loss functions reach an optimal value, clean images are acquired from a coarse mode to a fine mode by utilizing wavelet inverse transformation, thereby improving the denoising performance of the convolution neural network and reducing training time.
Description
Technical Field
The invention belongs to the technical field of neural network image denoising, and relates to an image denoising method based on a wide convolution neural network.
Background
At present, a Convolutional Neural Network (CNN) -based image denoising method is endless, and the existing CNN structure for image denoising captures image features by deepening the CNN network structure and improves denoising performance. However, for CNN models employing a single flow structure, it is difficult to acquire image detail features from multiple directions, and if the depth of the network is increased blindly, the information flow of the image features tends to be weakened, which makes network training very difficult. Because the CNN of image denoising requires that the feature map generated in the network has the same size as the input image and the output image, the network running these large-size feature maps must consume a large amount of memory and time of the GPU even though the number of convolution layers is small, limiting the depth of the CNN network of image denoising, i.e. the optimal balance between the image denoising performance and the network training time cannot be achieved by increasing the number of convolution layers on a single chain.
Disclosure of Invention
The invention aims to provide an image denoising method based on a wide convolution neural network, which solves the problems of low denoising performance and long training time of the existing neural network image denoising method.
The technical scheme adopted by the invention is that the image denoising method based on the wide convolution neural network is characterized by comprising the following steps of;
step 1, constructing a network WCNN;
step 2, training a network WCNN;
step 2.1, setting a data set comprising a training set, a verification set and a test set;
step 2.2, setting parameters of a training WCNN network;
and 2.3, setting a training platform of the network WCNN.
The present invention is also characterized in that,
the network WCNN constructed in step 1 includes 10 subnets, respectively res net1, res net2, res net3, res net4, res net5, res net6, UNet1, UNet2, UNet3 and DenseNet1; ten wavelet subbands are obtained through wavelet three-layer decomposition of the image, the ten wavelet subbands are HH1, LH1, HL1, HH2, LH2, HL2, HH3, LH3, HL3 and LL3 respectively, and 10 subnets are corresponding to the feature mapping of the ten wavelet subbands respectively responsible for learning the image.
The specific steps of each subnet in the network WCNN constructed in the step 1 are as follows:
step 1.1, six subnets of ResNet1, resNet2, resNet3, resNet4, resNet5 and ResNet6 are designed firstly; the six sub-networks are correspondingly responsible for training the HH1, LH1, HL1, HH2, LH2 and HL2 fine sub-bands obtained by decomposing the first layer and the second layer of the wavelet; adopting a ResNet structure, directly estimating noise by residual error learning, and estimating a denoised wavelet sub-band by jump connection; wherein, three subnets of ResNet1, resNet2 and ResNet3 are composed of 6 standard convolution layers, and ResNet4, resNet5 and ResNet6 are composed of 8 standard convolution layers;
step 1.2, then designing three subnets of UNet1, UNet2 and UNet 3; the three sub-networks are responsible for training HH3, LH3 and HL3 fine sub-bands obtained by decomposing a third layer of the wavelet, adopting a UNet structure, and totally comprising 6 convolution layers, wherein 4 convolution layers are formed by convolution obtained by the operation of extended convolution and standard convolution;
step 1.3, designing a DenseNet subnet; the training method is responsible for training a LL3 coarse sub-band obtained by decomposing a wavelet third layer, adopts a DenseNet structure and consists of 4 dense blocks containing 3 layers of convolution;
step 1.4, designing a loss function of each subnet;
and 1.5, carrying out wavelet inverse transformation on ten wavelet subbands processed by each subnet when the loss function of each subband reaches an optimal value, and obtaining an image with clear details and cleanness.
The step 1.4 specifically comprises the following steps:
step 1.4.1 the loss function of the wavelet transformed coarse subband uses the mean square error measure MSE l ；
Wherein x (i, j) and y (i, j) represent the estimated image and the corresponding wavelet coefficient values of the net image, respectively, and c, w and h represent the channel, width and height of the input subband pair, respectively;
step 1.4.2, calculating the loss function of the wavelet transformation fine sub-band, introducing a weight factor delta and an adjustment factor beta into the mean square error measurement index (1), and calculating the loss function MSE of the fine sub-band h The following are provided:
wherein the weight factor δ is calculated by:
here, ave represents an average value of wavelet coefficients of each fine subband, each subband coefficient average value ave is calculated by formula (4), and the adjustment factor β is calculated by formula (5):
where σ represents the noise strength.
When the noise level increases in step 1.4, the amplitude of the noise in the subband increases and may be greater than the average of the subband coefficients; to prevent these noise figures, which are greater than the average of the subband coefficients, from being enhanced, an adjustment factor β is used to intervene; if the variance sigma of the noise level is above 45, then the subband coefficient value is not less than 1.2 times the average value, and is considered as the image detail coefficient, and a delta=1.1 weight is given; thus suppressing coefficients less than 1.2 times the average value, which are considered to represent noise information; the ave of each fine subband is different and is closely related to the noise figure and the characteristic coefficient of each subband.
The training set in step 2.1 consists of 800 images of dataset DIV2K, 200 images of dataset BSD, and 4744 images of dataset WED.
The validation set in step 2.1 consists of the image in dataset RNI5 and 300 images of dataset DIV 2K.
The test Set in step 2.1 consists of the image in the data Set CSet8 and the image in Set 12.
The images in the training set in step 2.2 are sized 256×256, gaussian noise with a certain noise level, i.e. σ=5, 15, 25, 35, 45, 55, 65 and 75, are added to the clean image, 256×8000 image pairs are generated, the network model obtained by training the network WCNN using noise images with superimposed low noise intensities, i.e. σ.ltoreq.45, and using noise images with superimposed high noise levels, i.e. 45< σ.ltoreq.75, the network model being denoted WCNN1 and the latter being denoted WCNN2, respectively; when the noise intensity variance of the test noise image is not more than 45, the network WCNN1 is used for denoising; if the variance of the noise intensity of the test noise image is greater than 45, the WCNN2 network is used to denoise.
Step (a)2.3 setting up WCNN network in TensorFlow framework, and updating with Adam optimizer, wherein the activation function is ReLU, and learning rate of all subnets is initially set to 9×10 -4 The learning rate was reduced by one third after every 16 cycles, training the WCNN network with NVIDIA RTX 2080 Ti.
The beneficial effects of the invention are as follows:
1. according to the invention, the CNN is placed on the sub-bands of different scales and directions of the image, so that the image characteristics and details of each scale and each direction can be fully learned, and the high resolution of the image can be maintained while the speckle noise is restrained.
2. Each of the subnets constituting the network WCNN has its own structure and loss function, ensuring that each wavelet subband of the noise image is network trained to be most similar to the corresponding subband of the clean image.
3. Each wavelet sub-band concentrates image features of a particular scale in a particular direction, so having a few convolution layers or simple sub-networks is sufficient to capture and learn the image features contained in each wavelet sub-band, effectively suppressing noise in each sub-band.
4. The sub-networks used for training the wavelet sub-bands are independent of each other, and the sub-networks can run in parallel on a single computer or multiple computers, thereby shortening the network training time.
Drawings
FIG. 1 is a diagram of a WCNN network framework in accordance with the present invention;
FIG. 2 is a network structure diagram of first and second layer wavelet fine subbands according to the present invention;
FIG. 3 is a network structure diagram of a third layer wavelet fine sub-band of the present invention;
FIG. 4 is a network structure diagram of a third layer wavelet coarse subband of the present invention;
FIG. 5 (a) is a graph comparing the effect of different subnetwork structures on PSNR network performance;
FIG. 5 (b) is a graph comparing the impact of different subnet structures on SSIM network performance;
FIG. 5 (c) is a graph comparing the impact of different subnetwork structures on IFC network performance;
FIG. 6 (a) is a graph comparing the effect of different loss functions on PSNR network performance;
FIG. 6 (b) is a graph comparing the impact of different loss functions on SSIM network performance;
FIG. 6 (c) is a graph comparing the impact of different loss functions on IFC network performance;
FIG. 7 (a) is a view showing the visual effect obtained by denoising a gray image;
fig. 7 (b) is a view of visual effect obtained by denoising a gray image and BM 3D;
FIG. 7 (c) is a view of the visual effect obtained by denoising a gray image, dnCNN;
FIG. 7 (d) is a visual effect diagram obtained by denoising a gray image, MWCNN;
fig. 7 (e) is a view of the visual effect obtained by the UDNet for denoising the gray image;
FIG. 7 (f) is a visual effect diagram obtained by denoising a gray image and FFDNet;
FIG. 7 (g) is a visual effect diagram obtained by the WCNN for denoising the gray image;
FIG. 8 (a) is a view showing the visual effect obtained by denoising a color image;
FIG. 8 (b) is a visual effect diagram obtained by BM3D for denoising color images;
FIG. 8 (c) is a view of the visual effect obtained by denoising a color image, dnCNN;
FIG. 8 (d) is a visual effect diagram obtained by denoising a color image, MWCNN;
FIG. 8 (e) is a visual effect diagram obtained by UDNet for denoising color images;
FIG. 8 (f) is a visual effect diagram obtained by denoising a color image, FFDNet;
fig. 8 (g) is a view of the visual effect obtained by the WCNN for denoising color images.
Detailed Description
The invention will be described in detail below with reference to the drawings and the detailed description.
The invention relates to an image denoising method based on a wide convolution neural network, which is characterized in that when an image is denoised, the image is converted into a plurality of wavelet sub-bands through wavelet decomposition, each wavelet sub-band denoising is realized by learning characteristic mapping of a wavelet sub-band with a specific direction and a specific dimension and smaller dimension by using a CNN and restraining noise coefficients contained in the wavelet sub-band, so that a plurality of CNNs with different independent structures are arranged on each wavelet sub-band of the image, the detail characteristics of the image with a certain dimension in a certain direction in each sub-band are captured, and noise with a certain intensity range is removed by using a set of learning parameters, so that the best balance between the image denoising performance and the network training time is obtained.
The invention discloses an image denoising method based on a wide convolution neural network, which is implemented according to the following steps:
step 1, constructing a network WCNN;
the network WCNN constructed in step 1 includes 10 subnets, as shown in fig. 1, which are respectively res net1, res net2, res net3, res net4, res net5, res net6, UNet1, UNet2, UNet3 and DenseNet1; ten wavelet sub-bands are obtained through wavelet three-layer decomposition of the image, the ten wavelet sub-bands are HH1, LH1, HL1, HH2, LH2, HL2, HH3, LH3, HL3 and LL3 respectively, and 10 sub-networks are corresponding to the feature mapping of the ten wavelet sub-bands respectively responsible for learning the image;
the specific steps of each subnet in the network WCNN constructed in the step 1 are as follows:
step 1.1, six subnets of ResNet1, resNet2, resNet3, resNet4, resNet5 and ResNet6 are designed firstly; the six sub-networks are correspondingly responsible for training the HH1, LH1, HL1, HH2, LH2 and HL2 fine sub-bands obtained by decomposing the first layer and the second layer of the wavelet; adopting a ResNet structure, directly estimating noise by residual error learning, and estimating a denoised wavelet sub-band by jump connection; wherein three subnetworks ResNet1, resNet2, and ResNet3 consist of 6 standard convolutional layers, resNet4, resNet5, and ResNet6 consist of 8 standard convolutional layers, as shown in FIG. 2;
step 1.2, then designing three subnets of UNet1, UNet2 and UNet 3; the three sub-networks are responsible for training HH3, LH3 and HL3 fine sub-bands obtained by decomposing a third layer of the wavelet, adopting a UNet structure, and totally adopting 6 convolution layers, wherein 4 convolution layers obtained by the operation of extended convolution and standard convolution form a mixed convolution layer, as shown in figure 3;
step 1.3, designing a DenseNet subnet; the training wavelet is responsible for training a LL3 coarse sub-band obtained by decomposing a wavelet third layer, adopts a DenseNet structure and consists of 4 dense blocks containing 3 layers of convolution, as shown in figure 4;
step 1.4, designing a loss function of each subnet.
Step 1.4.1 the loss function of the wavelet transformed coarse subband uses the mean square error measure MSE l ；
Wherein x (i, j) and y (i, j) represent the estimated image and the corresponding wavelet coefficient values of the net image, respectively, and c, w and h represent the channel, width and height of the input subband pair, respectively;
step 1.4.2, calculating the loss function of the wavelet transformation fine sub-band, introducing a weight factor delta and an adjustment factor beta into the mean square error measurement index (1), and calculating the loss function MSE of the fine sub-band h The following are provided:
wherein the weight factor δ is calculated by:
here, ave represents an average value of wavelet coefficients of each fine subband, each subband coefficient average value ave is calculated by formula (4), and the adjustment factor β is calculated by formula (5):
wherein σ represents the noise intensity;
as the noise level increases, the amplitude of the noise in the subband increases and may be greater than the average of the subband coefficients; to prevent these noise figures, which are greater than the average of the subband coefficients, from being enhanced, an adjustment factor β is used to intervene; if the variance sigma of the noise level is above 45, then the subband coefficient value is not less than 1.2 times the average value, and is considered as the image detail coefficient, and a delta=1.1 weight is given; thus suppressing coefficients less than 1.2 times the average value, which are considered to represent noise information; it should be noted that the ave of each fine subband is different and is closely related to the noise figure and the characteristic coefficient of each subband.
And 1.5, carrying out wavelet inverse transformation on ten wavelet subbands processed by each subnet when the loss function of each subband reaches an optimal value, and obtaining an image with clear details and cleanness.
In the step 1, each subnet of the WCNN network is a wavelet sub-set of the independent training image, and the convolution layer number of each subnet is small, so that the network WCNN has 10 independent characteristic extraction and learning channels. Thus, the performance of the network WCNN to suppress noise and capture image features is improved by expanding the network width rather than deepening the network depth; because each sub-network of the WCNN network is mutually independent, the sub-networks can be trained on a plurality of computers in parallel, so that the training time of the WCNN network can be obviously shortened under the condition that the network performance is not influenced, the loss function of each sub-network is related to the characteristics of the wavelet sub-band coefficients trained by the sub-network, and the contradiction between the detail reservation of the coordinated image and the noise removal is facilitated.
Step 2, training a network WCNN;
step 2.1, a data set comprising a training set, a validation set and a test set is set.
The training set consists of 800 images of dataset DIV2K, 200 images of dataset BSD, and 4744 images of dataset WED;
the verification set is composed of images in the data set RNI5 and 300 images of the data set DIV 2K;
test set: consists of images in dataset CSet8 and images in Set 12;
step 2.2, setting parameters of a training WCNN network;
the images in the training set are sized 256×256, gaussian noise with a certain noise level, i.e., σ=5, 15, 25, 35, 45, 55, 65, and 75, are added to the clean images, 256×8000 image pairs are generated, the network WCNN is trained using noise images with superimposed low noise intensities, i.e., σ+.45, and using noise images with superimposed high noise levels, i.e., 45< σ+.75, respectively, the network model obtained by the former being labeled WCNN1 and the latter being labeled WCNN2;
when the noise intensity variance of the test noise image is not more than 45, the network WCNN1 is used for denoising; if the noise intensity variance of the test noise image is greater than 45, denoising by using the WCNN2 network;
step 2.3, setting a training platform of the network WCNN;
setting up WCNN network in TensorFlow framework, updating with Adam optimizer, activating function to be ReLU, and setting learning rate of all sub-networks to 9×10 initially -4 The learning rate decreases by one third after every 16 cycles, taking about 7 hours to train the WCNN network with NVIDIA RTX 2080 Ti.
Examples
The invention aims to provide an image denoising method based on a wide convolutional neural network, which aims to improve the denoising performance of the convolutional neural network and reduce training time, so that the performance of WCNN is tested and verified by performing experiments. Firstly, examining advantages brought by independent training of each subnet; secondly, researching the influence of different sub-networks forming the WCNN on the denoising quality of the image; third, the effect of the loss function on WCNN performance was investigated.
The effect of the fundamental component on WCNN performance was demonstrated by ablation studies of three experiments. Finally, five representative denoising methods are selected as comparison baselines, and the denoising effect of the proposed WCNN method is compared and comprehensively analyzed: one is wavelet based CNN denoising method (MWCNN [1 ]), three CNN based methods (DnCNN [2], UDNet [3] and FFDNet [4 ]), and a representative conventional method (BM 3D [5 ]).
1. Ablation experiment
1. Independent training subnet study
In this section, verifying that each subnet is trained independently not only shortens training time, but also ensures denoising quality. One advantage of WCNN is that it can be divided into several sub-networks that learn the feature mapping of sub-bands in parallel on respectively different computers, the trained sub-bands are integrated by Haar inverse wavelet transform to obtain a clear and clean image, and WCNN trained in this way is denoted as WCNN-1; WCNN is a model for learning all wavelet subband feature maps on a single computer where WCNN is compared to BM3D, dnCNN, MWCNN, FFDNet and UDNet denoising benchmark methods, the watermark transparency performance index results of which are shown in table 1.
TABLE 1 different method run times and PSNR (dB)/SSIM/IFC index comparison
Table 1 shows that when WCNN and the comparison method are applied to 200 grayscale images from the DIV2K dataset, the GPU run time and PSNRs/SSIMs/IFCs when sigma=25 noise is added, both WCNN-1 and WCNN achieve optimal performance at a relatively low execution time compared to the most advanced denoising method, thus it can be seen that the training time and execution time of WCNN are slightly more than WCNN-1, because each subnet of WCNN-1 can learn their feature maps in parallel on multiple computers, while each subnet of WCNN can only run on one computer; due to the multi-scale multi-directional decomposition of the wavelet, the subbands of the wavelet have not only a common directional characteristic, e.g., the coefficients of the horizontal directional edges in the LH subband are larger, but also smaller sizes, each subband does not lose any detail features due to clipping. Thus, without the need for a deep CNN with multiple convolution layers to capture the characteristics of these subbands, and with each subnet having its own loss function, the parameters of the subnets can be controlled and adjusted to ensure that each estimated subband is very similar to the subband of the clean image, which measures ensure that the WCNN can obtain higher PSNRs/SSIM at a relatively faster rate than other comparison methods, even if the WCNN is running on a computer.
2. Subnet Structure research
As shown in fig. 5 (a) -5 (c), the positive impact of the subnet structure on WCNN performance is shown; WCNN-2 represents a variant of WCNN in which all subnets have a structural design of res net, i.e., the structure shown in fig. 2, each subnet having ten convolutional layers. Using the performance of the MWCNN as a base line, the image is subjected to multi-layer wavelet decomposition, and all obtained wavelet subbands are input into one CNN for feature learning and training; FIGS. 5 (a) -5 (c) show a proposed comparison of the WCNN method, the WCNN-2 method and the MWCNN method in terms of PSNRs/SSIMs/IFCs; these data are from the average of 200 denoised images; comparing the figures 5 (a) -5 (c), the performance of the WCNN is obviously better than that of the MWCNN, and the performance of the WCNN-2 is slightly higher than that of the MWCNN method; this shows that the strategy of training different subbands with different CNN structures can significantly improve the denoising performance of WCNN.
3. Loss function study
The following experiments were performed: each subnet learns its feature map using the same loss function, equation (1), and represents this WCNN as WCNN-3; here, UDNet is considered a comparison baseline because it is able to process images with a range of noise levels using a single network; the three networks WCNN-3, WCNN and UDNet used 1000 images from BSD and WED datasets with image plus noise variance intensity σ=45 for training; the three networks were then used to test 200 images, with a noise variance of 40 images plus σ=5, a noise variance of 40 images plus σ=15, a noise variance of 40 images plus σ=25, a noise variance of 40 images plus σ=35, and a noise variance of 40 images plus σ=45.
As shown in fig. 6 (a) -6 (c), probability distributions of PSNR, SSIM, and IFC gains of the 200 images are shown; the white bar graph represents the distribution of index values obtained by the method WCNN-3 compared with the index value obtained by the method UDNet, wherein a part of index values obtained from 200 images are lower than the value obtained by the method UDNet, and the distribution is distributed at the left half part of the abscissa 0; the black histogram represents the distribution of index values obtained by the method WCNN compared with the standard UDNet method, the index values obtained from 200 images are substantially higher than the values obtained by UDNet, and almost all values are distributed in the right half of the abscissa 0; these gain values are obtained from WCNN and WCNN-3 relative to UDNet baseline, and the PSNR/SSIM/IFC gains in fig. 6 (a) -6 (c) illustrate that WCNN performance is far beyond UDNet and WCNN-3 performance is slightly below UDNet, which results indicate that different loss functions can significantly improve WCNN performance when WCNN uses a trained set of parameters to handle a range of noise.
2. Comparison of Performance with comparison method
To fully verify the performance of WCNN, the quality of WCNN and other methods when dealing with images of σ=5, 15, 25, 35, 45, 55, 65, or 75 noise were studied. In addition, the performance of wcnn+ which is another variation of WCNN whose number of convolution layers in each subnet is increased to 15, three evaluation values of PSNR, SSIM, and IFC obtained by these methods, that is, an average value of 20 gray images and an average value of 20 color images, as shown in table 2, the visual quality of the denoised image, gray image part, and color image Comic, respectively, as shown in fig. 7 (a) -7 (g) and fig. 8 (a) -8 (g), are examined, effect comparison is performed, and a target region of interest (ROI) enlarged by bicubic interpolation (×2) is displayed at the corner for comparing the detail features of the denoised image.
TABLE 2 PSNR (dB)/SSIM/IFC indicators obtained by different methods
The numerical results obtained by WCNN in table 2 are optimal, and the method not only has the highest average PSNRs, but also has relatively high SSIM and IFCs; high PSNRs indicate that the denoised image is closest to the original clean image, and higher SSIM and IFC values indicate that: these methods can restore edge and texture details, as shown in fig. 7 (a) -7 (g) and fig. 8 (a) -8 (g), the visual quality obtained from WCNN is quite excellent, some minor artifacts only appear at some edges, and in addition, when each subnet of WCNN uses more convolution layers, the evaluation result based on PSNRs/SSIMs/IFCs is better, and the method of the present invention is superior to the current most advanced denoising method.
According to the image denoising method based on the wide convolution neural network, the wavelet sub-bands are learned by training each sub-network in parallel, so that the WCNN network improves the image denoising performance by expanding the network width instead of the depth; each subnet can run on different computers, thereby shortening the network training time; each subnet captures image characteristics and noise with specific scale and specific direction, so that each subnet has a simple structure and needs fewer convolution layers; each sub-network has a loss function adapted to itself, so that it can be ensured that the noise sub-band and the clean image sub-band of each training are most similar; and calculating a loss function of the fine sub-band, and enhancing the influence of the image characteristic coefficient, so that the image characteristic details of the denoising image are more reserved.
Claims (6)
1. The image denoising method based on the wide convolution neural network is characterized by comprising the following steps of;
step 1, constructing a network WCNN;
the network WCNN constructed in the step 1 includes 10 subnets, which are respectively res net1, res net2, res net3, res net4, res net5, res net6, UNet1, UNet2, UNet3 and DenseNet1; ten wavelet sub-bands are obtained through wavelet three-layer decomposition of the image, the ten wavelet sub-bands are HH1, LH1, HL1, HH2, LH2, HL2, HH3, LH3, HL3 and LL3 respectively, and 10 sub-networks are corresponding to the feature mapping of the ten wavelet sub-bands respectively responsible for learning the image;
the specific steps of each subnet in the network WCNN constructed in the step 1 are as follows:
step 1.1, six subnets of ResNet1, resNet2, resNet3, resNet4, resNet5 and ResNet6 are designed firstly; the six sub-networks are correspondingly responsible for training the HH1, LH1, HL1, HH2, LH2 and HL2 fine sub-bands obtained by decomposing the first layer and the second layer of the wavelet; adopting a ResNet structure, directly estimating noise by residual error learning, and estimating a denoised wavelet sub-band by jump connection; wherein, three subnets of ResNet1, resNet2 and ResNet3 are composed of 6 standard convolution layers, and ResNet4, resNet5 and ResNet6 are composed of 8 standard convolution layers;
step 1.2, then designing three subnets of UNet1, UNet2 and UNet 3; the three sub-networks are responsible for training HH3, LH3 and HL3 fine sub-bands obtained by decomposing a third layer of the wavelet, adopting a UNet structure, and totally comprising 6 convolution layers, wherein 4 convolution layers are formed by convolution obtained by the operation of extended convolution and standard convolution;
step 1.3, designing a DenseNet subnet; the training method is responsible for training a LL3 coarse sub-band obtained by decomposing a wavelet third layer, adopts a DenseNet structure and consists of 4 dense blocks containing 3 layers of convolution;
step 1.4, designing a loss function of each subnet;
step 1.5, carrying out wavelet inverse transformation on ten wavelet sub-bands processed by each sub-network when the loss function of each sub-band reaches an optimal value, and obtaining an image with clear details and cleanness;
the step 1.4 specifically comprises the following steps:
step 1.4.1 the loss function of the wavelet transformed coarse subband uses the mean square error measure MSE l ；
Wherein x (i, j) and y (i, j) represent the estimated image and the corresponding wavelet coefficient values of the net image, respectively, and c, w and h represent the channel, width and height of the input subband pair, respectively;
step 1.4.2, calculating the loss function of the wavelet transformation fine sub-band, introducing a weight factor delta and an adjustment factor beta into the mean square error measurement index (1), and calculating the loss function MSE of the fine sub-band h The following are provided:
wherein the weight factor δ is calculated by:
here, ave represents an average value of wavelet coefficients of each fine subband, each subband coefficient average value ave is calculated by formula (4), and the adjustment factor β is calculated by formula (5):
wherein σ represents the noise intensity;
when the noise level increases in said step 1.4, the amplitude of the noise in the sub-band increases and may be larger than the average value of the sub-band coefficients; to prevent these noise figures, which are greater than the average of the subband coefficients, from being enhanced, an adjustment factor β is used to intervene; if the variance sigma of the noise level is above 45, then the subband coefficient value is not less than 1.2 times the average value, and is considered as the image detail coefficient, and a delta=1.1 weight is given; thus suppressing coefficients less than 1.2 times the average value, which are considered to represent noise information; the ave of each fine subband is different, and is closely related to the noise coefficient and the characteristic coefficient of each subband;
step 2, training a network WCNN;
step 2.1, setting a data set comprising a training set, a verification set and a test set;
step 2.2, setting parameters of a training WCNN network;
and 2.3, setting a training platform of the network WCNN.
2. The method of claim 1, wherein the training set in step 2.1 is composed of 800 images of the dataset DIV2K, 200 images of the dataset BSD, and 4744 images of the dataset WED.
3. The method for denoising an image based on a wide convolutional neural network according to claim 2, wherein the validation set in step 2.1 is composed of 300 images of the data set RNI5 and the data set DIV 2K.
4. A method of denoising an image based on a wide convolutional neural network according to claim 3, wherein the test Set in step 2.1 consists of the image in data Set CSet8 and the image in Set 12.
5. The method according to claim 4, wherein the images in the training set in step 2.2 are 256×256, gaussian noise with a specific noise level, i.e., σ=5, 15, 25, 35, 45, 55, 65, and 75, are added to the clean images to generate 256×8000 image pairs, and the network model obtained by training the network WCNN using noise images with a superimposed low noise intensity, i.e., σ+.45, and using noise images with a superimposed high noise level, i.e., 45< σ+.75, is denoted as WCNN1, and the network model obtained by training the network WCNN2; when the noise intensity variance of the test noise image is not more than 45, the network WCNN1 is used for denoising; if the variance of the noise intensity of the test noise image is greater than 45, the WCNN2 network is used to denoise.
6. The image denoising method based on the wide convolution neural network according to claim 5, wherein the step 2.3 builds a WCNN network in a TensorFlow framework, updates the WCNN network by using an Adam optimizer, the activation function is ReLU, and the learning rate of all the subnets is initially set to 9×10 -4 The learning rate was reduced by one third after every 16 cycles, training the WCNN network with NVIDIA RTX 2080 Ti.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110071024.6A CN112801897B (en) | 2021-01-19 | 2021-01-19 | Image denoising method based on wide convolution neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110071024.6A CN112801897B (en) | 2021-01-19 | 2021-01-19 | Image denoising method based on wide convolution neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112801897A CN112801897A (en) | 2021-05-14 |
| CN112801897B true CN112801897B (en) | 2024-02-09 |
Family
ID=75810485
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110071024.6A Active CN112801897B (en) | 2021-01-19 | 2021-01-19 | Image denoising method based on wide convolution neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112801897B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116385280B (en) * | 2023-01-09 | 2024-01-23 | 爱芯元智半导体(上海)有限公司 | Image noise reduction system and method and noise reduction neural network training method |
| CN116703772B (en) * | 2023-06-15 | 2024-03-15 | 山东财经大学 | Image denoising method, system and terminal based on adaptive interpolation algorithm |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10032256B1 (en) * | 2016-11-18 | 2018-07-24 | The Florida State University Research Foundation, Inc. | System and method for image processing using automatically estimated tuning parameters |
| CN110276726A (en) * | 2019-05-13 | 2019-09-24 | 南昌大学 | A kind of image deblurring method based on the guidance of multichannel network prior information |
| CN110599409A (en) * | 2019-08-01 | 2019-12-20 | 西安理工大学 | Convolutional neural network image denoising method based on multi-scale convolutional groups and parallel |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8280185B2 (en) * | 2008-06-27 | 2012-10-02 | Microsoft Corporation | Image denoising techniques |
-
2021
- 2021-01-19 CN CN202110071024.6A patent/CN112801897B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10032256B1 (en) * | 2016-11-18 | 2018-07-24 | The Florida State University Research Foundation, Inc. | System and method for image processing using automatically estimated tuning parameters |
| CN110276726A (en) * | 2019-05-13 | 2019-09-24 | 南昌大学 | A kind of image deblurring method based on the guidance of multichannel network prior information |
| CN110599409A (en) * | 2019-08-01 | 2019-12-20 | 西安理工大学 | Convolutional neural network image denoising method based on multi-scale convolutional groups and parallel |
Non-Patent Citations (2)
| Title |
|---|
| 段立娟 ; 武春丽 ; 恩擎 ; 乔元华 ; 张韵东 ; 陈军成 ; .基于小波域的深度残差网络图像超分辨率算法.软件学报.2019,(第04期),全文. * |
| 陈清江 ; 石小涵 ; 柴昱洲 ; .基于小波变换与卷积神经网络的图像去噪算法.应用光学.2020,(第02期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112801897A (en) | 2021-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112801897B (en) | Image denoising method based on wide convolution neural network | |
| CN103093433B (en) | Natural image denoising method based on regionalism and dictionary learning | |
| CN103077508B (en) | Transform domain non local and minimum mean square error-based SAR (Synthetic Aperture Radar) image denoising method | |
| CN101477681B (en) | Wavelet image denoising process based on adaptive sliding window adjacent region threshold | |
| CN102663702B (en) | Natural image denoising method based on regional division | |
| CN101477680A (en) | Wavelet image denoising process based on sliding window adjacent region data selection | |
| CN101950414A (en) | Non-local mean de-noising method for natural image | |
| Tang et al. | Noise estimation of natural images via statistical analysis and noise injection | |
| CN108564597A (en) | A kind of video foreground target extraction method of fusion gauss hybrid models and H-S optical flow methods | |
| CN104182939B (en) | Medical image detail enhancement method | |
| CN113808042B (en) | SAR image denoising method based on wavelet transformation and generation countermeasure network | |
| CN109447903B (en) | Method for establishing semi-reference super-resolution reconstruction image quality evaluation model | |
| CN110706186B (en) | Underwater information anti-interference method based on compressed sensing and Renyi entropy | |
| CN101957984B (en) | Image de-noising method based on parametric estimation of non-local shrinkage factor | |
| Rao et al. | Selective neighbouring wavelet coefficients approach for image denoising | |
| CN111311508B (en) | Noise reduction method for pavement crack image with noise | |
| CN104240208A (en) | Uncooled infrared focal plane detector image detail enhancement method | |
| CN104754183B (en) | A kind of real-time monitor video adaptive filter method and its system | |
| Ponomarenko et al. | Sharpness metric for no-reference image visual quality assessment | |
| CN106373104A (en) | Adaptive enhancement method of rock boring image | |
| CN102314675A (en) | Wavelet high-frequency-based Bayesian denoising method | |
| Sun et al. | Color image denoising based on guided filter and adaptive wavelet threshold | |
| CN101930605A (en) | Synthetic Aperture Radar (SAR) image target extraction method and system based on two-dimensional mixing transform | |
| Kumar et al. | Analysis of Wavelet Denoising of a Colour ImageWith Different Types of Noises | |
| Tayade et al. | Medical image denoising and enhancement using DTCWT and Wiener filter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |