CN111738956B - Image denoising system based on characteristic modulation - Google Patents

Image denoising system based on characteristic modulation Download PDF

Info

Publication number
CN111738956B
CN111738956B CN202010589835.0A CN202010589835A CN111738956B CN 111738956 B CN111738956 B CN 111738956B CN 202010589835 A CN202010589835 A CN 202010589835A CN 111738956 B CN111738956 B CN 111738956B
Authority
CN
China
Prior art keywords
module
modulation
noise
image
sampling
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010589835.0A
Other languages
Chinese (zh)
Other versions
CN111738956A (en
Inventor
闫子飞
杜佳芝
乔鑫
张宏志
左旺孟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology
Original Assignee
Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology filed Critical Harbin Institute of Technology
Priority to CN202010589835.0A priority Critical patent/CN111738956B/en
Publication of CN111738956A publication Critical patent/CN111738956A/en
Application granted granted Critical
Publication of CN111738956B publication Critical patent/CN111738956B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

An image denoising system based on feature modulation relates to the field of image processing. The problem of noise image denoising effect poor because of neglecting the influence of the noise level map on the subsequent layers in the noise image denoising method by simply connecting the noise level map and the noise image in the prior art is solved. The method comprises the steps of conducting multi-layer guidance on denoising of an original noise image by using a noise level graph, firstly conducting feature extraction on the original noise image and the noise level graph to obtain an extracted feature graph, then conducting multi-scale multi-level modulation on the corresponding feature graph to obtain a residual image feature graph, and enabling the original noise image to be denoised by using the residual image feature graph. The method is mainly used for denoising the noise image.

Description

Image denoising system based on characteristic modulation
Technical Field
The present invention relates to the field of image processing.
Background
Image denoising is a classic topic in underlying vision, and aims to recover a potential clean image from a noisy picture. With the development of deep learning algorithms, many deep denoising networks based on Convolutional Neural Networks (CNNs) have been proposed in recent years. Compared with the traditional model-based method, the deep learning-based denoising method has more excellent denoising performance. However, most deep denoising networks still lack flexibility in processing Additive White Gaussian Noise (AWGN) with different noise levels, even spatial variations. In particular, these methods are typically non-blind, i.e., require learning a specific model for each noise level. The noise images in the real world often correspond to different noise levels, so a large number of denoising models need to be trained and stored in advance, and the practicability of the denoising models in the image denoising task is obviously reduced.
In addition, there are also many scholars who propose several methods for solving the flexibility problem of non-blind image denoising. For example: a fast and flexible convolution-based denoising network (FFDNet) is proposed. For a given noise image with a specific noise level, FFDNet only needs to join (join) the noise image with its corresponding noise level map as input, and can use a single denoising network to process additive white gaussian noise with different noise levels, which is called as input joining method. This additional input noise level map can help achieve a balance between noise removal and image detail preservation, enabling FFDNet to be better applied in some real denoising tasks.
However, the above approach explores how to incorporate noise prior information into image denoising, but simply ties the noise level map and the noise image to the input layer. On one hand, the noise information does not contain the information of the image, and the interference irrelevant to the image can be introduced by simultaneously processing the noise information and the image information by using convolution operation; on the other hand, such processing mechanisms are insufficient in mining the complex interaction relationship between noise level and image information, and have certain limitations in balancing noise removal and image detail preservation. The following describes the two disadvantages in a specific example, specifically:
let y be the noisy image with additive white Gaussian noise, σ2Is the noise variance and x is its corresponding clean image. The task of non-blind image denoising refers to that the noise variance sigma and the y of a given noise image are calculated2In the case of (2), the clean image x is restored from y. Where σ is a scalar and y is an image of size H × W × C (grayscale image C1, color image C3). Therefore, we stretch σ into a size of H W and each element is the σ noise level map M. After stretching, the noise level map M not only provides the conditional information for modulating the convolution activation, but also provides a way to handle spatially varying noise by simply setting M (i, j) to the local noise level at the (i, j) location. Thus, the flexible non-blind image denoising model can be written as:
Figure BDA0002555936900000021
where Θ represents a network parameter. In flexible denoising networks such as FFDNet, a noise level map is usually introduced with the junction of y and M as an input. However, the input cascading method has a certain limitation in capturing the complex interaction between y and M, which results in a problem of insufficient image denoising effect. Therefore, the above problems need to be solved.
Disclosure of Invention
The invention aims to solve the problem that in the prior art, the noise image denoising effect is poor due to the fact that the influence of a noise level image on a subsequent layer is ignored in a noise image denoising mode after the noise level image and the noise image are simply connected. The invention provides an image denoising system based on characteristic modulation.
An image denoising system based on feature modulation comprises two feature extraction modules, a first residual shift modulation module, a processing module, an image restoration module and a subtracter;
the characteristic extraction module is used for extracting the characteristics of the original noise image to obtain a noise image characteristic extraction image and sending the obtained noise image characteristic extraction image to the first residual error shift modulation module;
the other characteristic extraction module is used for carrying out characteristic extraction on a noise level graph corresponding to the original noise image to obtain a noise level characteristic extraction graph and sending the obtained noise level characteristic extraction graph to the first residual error shift modulation module;
the first residual error shift modulation module is used for carrying out primary modulation on the received noise image characteristic extraction image and the noise level characteristic extraction image to obtain a noise image characteristic image and a noise level characteristic image after the primary modulation, and sending the noise image characteristic image and the noise level characteristic image to the processing module;
the processing module is used for carrying out space scale adjustment and multi-level modulation on the received noise image characteristic map and the noise level characteristic map after the initial modulation to obtain a residual image characteristic map and sending the residual image characteristic map to the image restoration module;
the image restoration module is used for restoring the received residual image characteristic diagram to obtain a residual image and sending the residual image to the subtracter;
the subtracter is further used for receiving the original noise image, and superposing residual features of all feature points in the received residual image on corresponding feature points in the original noise image respectively, so that the original noise image is denoised, and a denoised image is obtained.
Preferably, the processing module comprises a first-level scale scaling modulation module, a second-level scale scaling modulation module, a third-level scale restoration modulation module and a fourth-level scale restoration modulation module;
the primary scale scaling modulation module is used for carrying out primary down-sampling on the noise image characteristic map and the noise level characteristic map after the primary modulation to obtain a noise image characteristic map and a noise level characteristic map after the primary down-sampling, carrying out primary modulation on the noise image characteristic map and the noise level characteristic map after the primary down-sampling to obtain a noise image characteristic map and a noise level characteristic map after the primary modulation, and sending the obtained noise image characteristic map and the noise level characteristic map after the primary modulation to the secondary scale scaling modulation module and the tertiary scale restoration modulation module;
the two-level scale scaling modulation module is used for carrying out second-time down-sampling on the noise image characteristic map and the noise level characteristic map after the first-time modulation, carrying out second-level modulation on the noise image characteristic map and the noise level characteristic map after the second-time down-sampling to obtain a noise image characteristic map and a noise level characteristic map after the second-level modulation, and sending the obtained noise image characteristic map and the noise level characteristic map after the second-level modulation to the three-level scale restoration modulation module;
the three-level scale restoration modulation module is used for performing primary up-sampling on the noise image feature map subjected to secondary modulation and the noise level feature map, connecting the noise image feature map subjected to primary up-sampling with the noise image feature map subjected to primary modulation, performing feature fusion on the noise image feature map subjected to primary up-sampling and the noise image feature map subjected to primary modulation to obtain a first feature fusion map, performing third-level modulation on the first feature fusion map to obtain a noise image feature map subjected to third-level modulation, and sending the noise image feature map subjected to third-level modulation to the four-level scale restoration modulation module; meanwhile, the noise level characteristic diagram after the first up-sampling is connected with the noise level characteristic diagram after the first-stage modulation, so that the noise level characteristic diagram after the first up-sampling is subjected to characteristic fusion with the noise level characteristic diagram after the first-stage modulation to obtain a second characteristic fusion diagram, the second characteristic fusion diagram is subjected to third-stage modulation to obtain a noise level characteristic diagram after the third-stage modulation, and the noise level characteristic diagram after the third-stage modulation is sent to a four-stage scale restoration modulation module;
the four-level scale restoration modulation module is used for receiving the noise image characteristic map and the noise level characteristic map after the preliminary modulation; the image processing device is also used for performing second upsampling on the noise image feature map and the noise level feature map after the third-level modulation, and connecting the noise image feature map after the second upsampling with the noise image feature map after the preliminary modulation, so that the noise image feature map after the second upsampling and the noise image feature map after the preliminary modulation are subjected to feature fusion to obtain a third feature fusion map; meanwhile, the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the initial modulation are connected, so that the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the initial modulation are subjected to characteristic fusion, and a fourth characteristic fusion diagram is obtained;
and performing fourth-level modulation on the third feature fusion map by using the fourth feature fusion map to obtain a noise image feature map subjected to fourth-level modulation, and sending the noise image feature map subjected to fourth-level modulation to the image restoration module as a residual image feature map output by the processing module.
Preferably, the one-stage scaling modulation module comprises two down-sampling modules and a second residual shift modulation module;
the two down-sampling modules are respectively a first down-sampling module and a second down-sampling module, wherein the first down-sampling module is used for carrying out first down-sampling on the noise image characteristic diagram after the initial modulation and sending the noise image characteristic diagram after the first down-sampling to the second residual error shift modulation module; the second down-sampling module is used for carrying out first down-sampling on the noise level characteristic diagram after the initial modulation and sending the noise level characteristic diagram after the first down-sampling to the second residual error shift modulation module;
and the second residual shift modulation module is used for realizing the mutual modulation between the noise image characteristic diagram after the first down-sampling and the noise level characteristic diagram after the first down-sampling to obtain the noise image characteristic diagram and the noise level characteristic diagram after the first-level modulation.
Preferably, the two-level scale modulation module comprises two down-sampling modules and a third residual shift modulation module;
the two down-sampling modules are respectively a third down-sampling module and a fourth down-sampling module, wherein the third down-sampling module is used for carrying out second down-sampling on the noise image feature map after the first-level modulation and sending the noise image feature map after the second down-sampling to the third residual error shift modulation module; the fourth down-sampling module is used for performing second down-sampling on the noise level characteristic diagram after the first-stage modulation and sending the noise level characteristic diagram after the second down-sampling to the third residual error shift modulation module;
and the third residual shift modulation module is used for realizing the mutual modulation between the noise image characteristic diagram after the second down-sampling and the noise level characteristic diagram after the second down-sampling to obtain the noise image characteristic diagram and the noise level characteristic diagram after the second-level modulation.
Preferably, the three-level scale restoration modulation module comprises two upsampling modules, two linking modules and a fourth residual shift modulation module;
the two up-sampling modules are respectively a first up-sampling module and a second up-sampling module; the first up-sampling module is used for performing first up-sampling on the noise image characteristic map subjected to the second-level modulation to obtain a noise image characteristic map subjected to the first up-sampling; the second up-sampling module is used for carrying out first up-sampling on the noise level characteristic diagram after the second-stage modulation to obtain the noise level characteristic diagram after the first up-sampling;
the two connection modules are respectively a No. 1 connection module and a No. 2 connection module;
connecting the noise image feature map subjected to the first up-sampling with the noise image feature map subjected to the first-level modulation through a No. 1 connecting module, so that the noise image feature map subjected to the first up-sampling and the noise image feature map subjected to the first-level modulation are subjected to feature fusion to obtain a first feature fusion map;
connecting the noise level characteristic diagram after the first up-sampling with the noise level characteristic diagram after the first-stage modulation through a No. 2 connecting module, so that the noise level characteristic diagram after the first up-sampling and the noise level characteristic diagram after the first-stage modulation are subjected to characteristic fusion to obtain a second characteristic fusion diagram;
and the fourth residual shift modulation module is used for realizing the mutual modulation between the first characteristic fusion diagram and the second characteristic fusion diagram and obtaining a noise image characteristic diagram and a noise level characteristic diagram after the third-level modulation.
Preferably, the four-level scale restoration modulation module comprises two upsampling modules, two linking modules and a fifth residual shift modulation module;
the two up-sampling modules are respectively a third up-sampling module and a fourth up-sampling module; the third up-sampling module is used for performing second up-sampling on the noise image characteristic diagram after the third-level modulation to obtain a noise image characteristic diagram after the second up-sampling; the fourth up-sampling module is used for performing second up-sampling on the noise level characteristic diagram after the third-level modulation to obtain a noise level characteristic diagram after the second up-sampling;
the two connecting modules are respectively a No. 3 connecting module and a No. 4 connecting module;
connecting the noise image feature map subjected to the second upsampling with the noise image feature map subjected to the preliminary modulation through a No. 3 connecting module, so that the noise image feature map subjected to the first upsampling and the noise image feature map subjected to the preliminary modulation are subjected to feature fusion to obtain a third feature fusion map;
connecting the noise level characteristic diagram after the second up-sampling with the noise level characteristic diagram after the primary modulation through a No. 4 connecting module, so that the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the primary modulation are subjected to characteristic fusion to obtain a fourth characteristic fusion diagram;
and the fifth residual shift modulation module is used for performing fourth-level modulation on the third feature fusion map by using the fourth feature fusion map to obtain a fourth-level modulated noise image feature map.
Preferably, the first residual shift modulation module includes a first one-to-one depth convolution module, a first two-to-one depth convolution module, a first connection module, a first channel conversion convolution module, a first shift module, a first one-to-one adder, and a first two-to-one adder;
the input end of the first one-to-one depth convolution module is connected with the first input end of the first adder and the first input end of the first connection module at the same time and then serves as the first input end of the first residual error shift modulation module, and the first input end of the first residual error shift modulation module is used for receiving a noise image characteristic extraction image;
the output end of the first one-to-one depth convolution module is connected with the second input end of the first one-to-one adder, and the output end of the first one-to-one adder is used as the first output end of the first residual shift modulation module and is used for outputting the preliminarily modulated noise image feature map;
the second input end of the first residual error shift modulation module is used for receiving a noise level characteristic extraction diagram;
the output end of the first connection module is connected with the input end of the first channel conversion convolution module, the output end of the first channel conversion convolution module is connected with the input end of the first second-depth convolution module, the output end of the first second-depth convolution module is simultaneously connected with the input end of the first shift module and the second input end of the first second adder, and the output end of the first second adder serves as the second output end of the first residual shift modulation module and is used for outputting the primarily modulated noise level characteristic diagram;
and the output end of the first shifting module is connected with the third input end of the first adder.
Preferably, the second residual shift modulation module includes a second first depth convolution module, a second connection module, a second channel conversion convolution module, a second shift module, a second first adder and a second adder;
the input end of the second depth convolution module is connected with the first input end of the second adder and the first input end of the second connection module at the same time and then used as the first input end of the second residual error shift modulation module, and the noise image feature map of the second residual error shift modulation module after the first down sampling;
the output end of the second deep convolution module is connected with the second input end of the second adder, and the output end of the second adder is used as the first output end of the second residual shift modulation module and is used for outputting the first-stage modulated noise image feature map;
the second input end of the second residual error shift modulation module is used for receiving the noise level characteristic diagram after the first downsampling;
the output end of the second coupling module is connected with the input end of a second channel conversion convolution module, the output end of the second channel conversion convolution module is connected with the input end of a second deep convolution module, the output end of the second deep convolution module is simultaneously connected with the input end of a second shift module and the second input end of a second adder, and the output end of the second adder is used as the second output end of a second residual shift modulation module and used for outputting a noise level characteristic diagram after the first-stage modulation;
the output end of the second shifting module is connected with the third input end of the second adder.
Preferably, the third residual shift modulation module includes a third first depth convolution module, a third second depth convolution module, a third concatenation module, a third channel conversion convolution module, a third shift module, a third first adder and a third second adder;
the input end of the third depth convolution module is connected with the first input end of the third adder and the first input end of the third combination module at the same time and then is used as the first input end of the third residual error shift modulation module, and the noise image characteristic map of the third residual error shift modulation module after the second down sampling;
the output end of the third depth convolution module is connected with the second input end of the third adder, and the output end of the third adder is used as the first output end of the third residual shift modulation module and is used for outputting the second-stage modulated noise image feature map;
the second input end of the third residual error shift modulation module is used for receiving the noise level characteristic diagram after the second downsampling;
the output end of the third link module is connected with the input end of a third channel conversion convolution module, the output end of the third channel conversion convolution module is connected with the input end of a third second depth convolution module, the output end of the third second depth convolution module is simultaneously connected with the input end of a third shift module and the second input end of a third second adder, and the output end of the third second adder serves as the second output end of a third residual shift modulation module and is used for outputting a noise level characteristic diagram after second-stage modulation;
the output end of the third shifting module is connected with the third input end of the third adder.
Preferably, the fourth residual shift modulation module includes a fourth first depth convolution module, a fourth second depth convolution module, a fourth connection module, a fourth channel conversion convolution module, a fourth shift module, a fourth first adder and a fourth second adder;
the input end of the fourth deep convolution module is connected with the first input end of the fourth adder and the first input end of the fourth connection module at the same time, and then is used as the first input end of the fourth residual shift modulation module for receiving the first feature fusion map;
the output end of the fourth deep convolution module is connected with the second input end of the fourth adder, and the output end of the fourth adder is used as the first output end of the fourth residual shift modulation module and is used for outputting a third-level modulated noise image feature map;
a second input end of the fourth linking module is connected with a first input end of the fourth second adder at the same time and then serves as a second input end of the fourth residual shift modulation module, and the second input end of the fourth residual shift modulation module is used for receiving the second feature fusion map;
the output end of the fourth coupling module is connected with the input end of a fourth channel conversion convolution module, the output end of the fourth channel conversion convolution module is connected with the input end of a fourth second deep convolution module, the output end of the fourth second deep convolution module is simultaneously connected with the input end of a fourth shift module and the second input end of a fourth second adder, and the output end of the fourth second adder serves as the second output end of a fourth residual shift modulation module and is used for outputting a noise level characteristic diagram after third-level modulation;
and the output end of the fourth shifting module is connected with the third input end of the fourth adder.
Preferably, the fifth residual shift modulation module includes a fifth first depth convolution module, a fifth second depth convolution module, a fifth linkage module, a fifth channel conversion convolution module, a fifth shift module and a fifth adder;
the input end of the fifth deep convolution module is connected with the first input end of the fifth adder and the first input end of the fifth linkage module at the same time, and then is used as the first input end of the fifth residual shift modulation module for receiving the third feature fusion map;
the output end of the fifth depth convolution module is connected with the second input end of the fifth adder, and the output end of the fifth adder is used as the first output end of the fifth residual shift modulation module and is used for outputting the fourth-level modulated noise image feature map;
a second input end of the fifth linking module is used as a second input end of the fifth residual error shift modulation module and is used for receiving the fourth feature fusion map;
the output end of the fifth coupling module is connected with the input end of the fifth channel conversion convolution module, the output end of the fifth channel conversion convolution module is connected with the input end of the fifth second depth convolution module, and the output end of the fifth second depth convolution module is connected with the input end of the fifth shifting module;
and the output end of the fifth shifting module is connected with the third input end of the fifth adder.
Preferably, the image denoising system minimizes a mean square error loss function by using the denoising system
Figure BDA0002555936900000071
Training is carried out;
wherein the content of the first and second substances,
Figure BDA0002555936900000072
Θ represents a parameter of web learning;
n represents the number of training data pairs in the training data set;
i is an integer, and i is 1,2,3 … … N;
xirepresenting a clear image obtained by denoising an ith noise image in a training data set;
yirepresenting the ith noisy image in the training dataset;
Figure BDA0002555936900000081
representing residual images learned by the network;
y represents the original noise image of the network input;
m represents a noise level map corresponding to the original noise image.
The invention has the advantages that: the method comprises the steps of conducting multi-layer guidance on denoising of an original noise image by using a noise level graph, firstly conducting feature extraction on the original noise image and the noise level graph to obtain an extracted feature graph, then conducting multi-scale multi-level modulation on the corresponding feature graph to obtain a residual image feature graph, and enabling the original noise image to be denoised by using the residual image feature graph.
Drawings
FIG. 1 is a schematic diagram of an image denoising system based on feature modulation according to the present invention;
FIG. 2 is a schematic diagram of a first residual shift modulation module;
FIG. 3 is a schematic diagram of a second residual shift modulation module;
FIG. 4 is a schematic diagram of a third residual shift modulation module;
FIG. 5 is a schematic diagram of a fourth residual shift modulation module;
fig. 6 is a schematic diagram of a fifth residual shift modulation module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
Referring to fig. 1, the present embodiment is described, and the image denoising system based on feature modulation according to the present embodiment includes two feature extraction modules, a first residual shift modulation module, a processing module, an image restoration module, and a subtractor;
the characteristic extraction module is used for extracting the characteristics of the original noise image to obtain a noise image characteristic extraction image and sending the obtained noise image characteristic extraction image to the first residual error shift modulation module;
the other characteristic extraction module is used for carrying out characteristic extraction on a noise level graph corresponding to the original noise image to obtain a noise level characteristic extraction graph and sending the obtained noise level characteristic extraction graph to the first residual error shift modulation module;
the first residual error shift modulation module is used for carrying out primary modulation on the received noise image characteristic extraction image and the noise level characteristic extraction image to obtain a noise image characteristic image and a noise level characteristic image after the primary modulation, and sending the noise image characteristic image and the noise level characteristic image to the processing module;
the processing module is used for carrying out space scale adjustment and multi-level modulation on the received noise image characteristic map and the noise level characteristic map after the initial modulation to obtain a residual image characteristic map and sending the residual image characteristic map to the image restoration module;
the image restoration module is used for restoring the received residual image characteristic diagram to obtain a residual image and sending the residual image to the subtracter;
the subtracter is further used for receiving the original noise image, and superposing residual features of all feature points in the received residual image on corresponding feature points in the original noise image respectively, so that the original noise image is denoised, and a denoised image is obtained.
The invention carries out space scale adjustment and multi-level modulation on the received noise image characteristic extraction image after preliminary modulation and the noise level characteristic extraction image to obtain the residual image characteristic image, thus increasing the receptive field of the network, improving the calculation efficiency and simultaneously mining the multi-scale information of the image.
The method considers the relevance between the original noise image and the noise level image, uses the noise level image to carry out multilayer guidance on the denoising of the original noise image, firstly carries out feature extraction on the original noise image and the noise level image to obtain an extracted feature image, and then carries out multi-scale multi-level modulation on the corresponding feature image to obtain a residual image feature image, the denoising of the original noise image is realized through the residual image feature image, the complex relation between the original noise image and the noise level image is mined in the whole denoising process, and the multi-scale modulation is continuously carried out on the noise image feature extraction image, so that the denoising effect is better. Further, referring specifically to fig. 1, the processing module includes a first-level scaling modulation module 1, a second-level scaling modulation module 2, a third-level scale restoration modulation module 3, and a fourth-level scale restoration modulation module 4;
the primary scale scaling modulation module 1 is configured to perform first downsampling on the noise image feature map and the noise level feature map after the preliminary modulation to obtain a noise image feature map and a noise level feature map after the first downsampling, perform first-level modulation on the noise image feature map and the noise level feature map after the first downsampling to obtain a noise image feature map and a noise level feature map after the first-level modulation, and send the obtained noise image feature map and the obtained noise level feature map after the first-level modulation to the secondary scale scaling modulation module 2 and the tertiary scale restoration modulation module 3;
the two-level scale scaling modulation module 2 is configured to perform second-time down-sampling on the noise image feature map and the noise level feature map after the first-time modulation, perform second-level modulation on the noise image feature map and the noise level feature map after the second-time down-sampling, obtain a noise image feature map and a noise level feature map after the second-level modulation, and send the obtained noise image feature map and the obtained noise level feature map after the second-level modulation to the three-level scale restoration modulation module 3;
the three-level scale restoration modulation module 3 is configured to perform first upsampling on the noise image feature map and the noise level feature map after the second-level modulation, and join the noise image feature map after the first upsampling with the noise image feature map after the first-level modulation, so that feature fusion is performed on the noise image feature map after the first upsampling and the noise image feature map after the first-level modulation, to obtain a first feature fusion map, perform third-level modulation on the first feature fusion map, to obtain a noise image feature map after the third-level modulation, and send the noise image feature map after the third-level modulation to the four-level scale restoration modulation module 4; meanwhile, the noise level characteristic diagram after the first up-sampling is connected with the noise level characteristic diagram after the first-stage modulation, so that the noise level characteristic diagram after the first up-sampling is subjected to characteristic fusion with the noise level characteristic diagram after the first-stage modulation to obtain a second characteristic fusion diagram, the second characteristic fusion diagram is subjected to third-stage modulation to obtain a noise level characteristic diagram after the third-stage modulation, and the noise level characteristic diagram after the third-stage modulation is sent to a four-stage scale restoration modulation module 4;
the four-level scale restoration modulation module 4 is used for receiving the noise image characteristic map and the noise level characteristic map after the preliminary modulation; the image processing device is also used for performing second upsampling on the noise image feature map and the noise level feature map after the third-level modulation, and connecting the noise image feature map after the second upsampling with the noise image feature map after the preliminary modulation, so that the noise image feature map after the second upsampling and the noise image feature map after the preliminary modulation are subjected to feature fusion to obtain a third feature fusion map; meanwhile, the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the initial modulation are connected, so that the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the initial modulation are subjected to characteristic fusion, and a fourth characteristic fusion diagram is obtained;
and performing fourth-level modulation on the third feature fusion map by using the fourth feature fusion map to obtain a noise image feature map subjected to fourth-level modulation, and sending the noise image feature map subjected to fourth-level modulation to the image restoration module as a residual image feature map output by the processing module.
In the preferred embodiment, down-sampling is used to reduce the spatial size of the feature map, up-sampling is used to enlarge the spatial size of the feature map, and up-and down-sampling is used to adjust the spatial scale of the feature map. The up and down sampling operations are common in the art in the field of image processing.
In the preferred embodiment, a specific structure of the processing module is provided, and the processing module sequentially performs adjustment and modulation of the spatial scale of the feature map 4 times through the first-level scaling modulation module 1, the second-level scaling modulation module 2, the third-level scale restoration modulation module 3, and the fourth-level scale restoration modulation module 4, so that the data accuracy of the obtained residual image feature map is higher. The invention takes the mode of 4-level modulation as an example, and explains the adjustment and modulation of the spatial scale of the characteristic diagram, and so on, the adjustment and modulation of the spatial scale of redundant 4-level can be carried out on the characteristic diagram.
Further, referring specifically to fig. 1 and 3, the one-level scaling modulation module 1 includes two down-sampling modules and a second residual shift modulation module;
the two down-sampling modules are respectively a first down-sampling module and a second down-sampling module, wherein the first down-sampling module is used for carrying out first down-sampling on the noise image characteristic diagram after the initial modulation and sending the noise image characteristic diagram after the first down-sampling to the second residual error shift modulation module; the second down-sampling module is used for carrying out first down-sampling on the noise level characteristic diagram after the initial modulation and sending the noise level characteristic diagram after the first down-sampling to the second residual error shift modulation module;
and the second residual shift modulation module is used for realizing the mutual modulation between the noise image characteristic diagram after the first down-sampling and the noise level characteristic diagram after the first down-sampling to obtain the noise image characteristic diagram and the noise level characteristic diagram after the first-level modulation.
In the preferred embodiment, a specific structure of the one-level scaling modulation module 1 is provided, which is simple, easy to implement and has a superior processing effect.
Further, referring specifically to fig. 1 and 4, the two-level scaling modulation module 2 comprises two down-sampling modules and a third residual shift modulation module;
the two down-sampling modules are respectively a third down-sampling module and a fourth down-sampling module, wherein the third down-sampling module is used for carrying out second down-sampling on the noise image feature map after the first-level modulation and sending the noise image feature map after the second down-sampling to the third residual error shift modulation module; the fourth down-sampling module is used for performing second down-sampling on the noise level characteristic diagram after the first-stage modulation and sending the noise level characteristic diagram after the second down-sampling to the third residual error shift modulation module;
and the third residual shift modulation module is used for realizing the mutual modulation between the noise image characteristic diagram after the second down-sampling and the noise level characteristic diagram after the second down-sampling to obtain the noise image characteristic diagram and the noise level characteristic diagram after the second-level modulation.
In the preferred embodiment, a specific structure of the two-level scaling modulation module 2 is provided, which is simple, easy to implement and has a superior processing effect.
Further, referring specifically to fig. 1 and 5, the three-level scale-restoration modulation module 3 includes two upsampling modules, two linking modules, and a fourth residual shift modulation module;
the two up-sampling modules are respectively a first up-sampling module and a second up-sampling module; the first up-sampling module is used for performing first up-sampling on the noise image characteristic map subjected to the second-level modulation to obtain a noise image characteristic map subjected to the first up-sampling; the second up-sampling module is used for carrying out first up-sampling on the noise level characteristic diagram after the second-stage modulation to obtain the noise level characteristic diagram after the first up-sampling;
the two connection modules are respectively a No. 1 connection module and a No. 2 connection module;
connecting the noise image feature map subjected to the first up-sampling with the noise image feature map subjected to the first-level modulation through a No. 1 connecting module, so that the noise image feature map subjected to the first up-sampling and the noise image feature map subjected to the first-level modulation are subjected to feature fusion to obtain a first feature fusion map;
connecting the noise level characteristic diagram after the first up-sampling with the noise level characteristic diagram after the first-stage modulation through a No. 2 connecting module, so that the noise level characteristic diagram after the first up-sampling and the noise level characteristic diagram after the first-stage modulation are subjected to characteristic fusion to obtain a second characteristic fusion diagram;
and the fourth residual shift modulation module is used for realizing the mutual modulation between the first characteristic fusion diagram and the second characteristic fusion diagram and obtaining a noise image characteristic diagram and a noise level characteristic diagram after the third-level modulation.
In the preferred embodiment, a specific structure of the three-level scale restoration modulation module 3 is provided, which is simple, easy to implement and superior in processing effect.
Further, referring specifically to fig. 1 and 6, the four-level scale restoration modulation module 4 includes two upsampling modules, two linking modules, and a fifth residual shift modulation module;
the two up-sampling modules are respectively a third up-sampling module and a fourth up-sampling module; the third up-sampling module is used for performing second up-sampling on the noise image characteristic diagram after the third-level modulation to obtain a noise image characteristic diagram after the second up-sampling; the fourth up-sampling module is used for performing second up-sampling on the noise level characteristic diagram after the third-level modulation to obtain a noise level characteristic diagram after the second up-sampling;
the two connecting modules are respectively a No. 3 connecting module and a No. 4 connecting module;
connecting the noise image feature map subjected to the second upsampling with the noise image feature map subjected to the preliminary modulation through a No. 3 connecting module, so that the noise image feature map subjected to the first upsampling and the noise image feature map subjected to the preliminary modulation are subjected to feature fusion to obtain a third feature fusion map;
connecting the noise level characteristic diagram after the second up-sampling with the noise level characteristic diagram after the primary modulation through a No. 4 connecting module, so that the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the primary modulation are subjected to characteristic fusion to obtain a fourth characteristic fusion diagram;
and the fifth residual shift modulation module is used for performing fourth-level modulation on the third feature fusion map by using the fourth feature fusion map to obtain a fourth-level modulated noise image feature map.
In the preferred embodiment, a specific structure of the four-level scale restoration modulation module 4 is provided, which is simple, easy to implement and superior in processing effect.
Furthermore, the specific structures inside the first to fourth residual shift modulation modules are the same.
Further, referring specifically to fig. 1 and fig. 2, the first residual shift modulation module includes a first one-to-one depth convolution module, a first two-to-one depth convolution module, a first connection module, a first channel conversion convolution module, a first shift module, a first one-to-one adder, and a first two-to-one adder;
the input end of the first one-to-one depth convolution module is connected with the first input end of the first adder and the first input end of the first connection module at the same time and then serves as the first input end of the first residual error shift modulation module, and the first input end of the first residual error shift modulation module is used for receiving a noise image characteristic extraction image;
the output end of the first one-to-one depth convolution module is connected with the second input end of the first one-to-one adder, and the output end of the first one-to-one adder is used as the first output end of the first residual shift modulation module and is used for outputting the preliminarily modulated noise image feature map;
the second input end of the first residual error shift modulation module is used for receiving a noise level characteristic extraction diagram;
the output end of the first connection module is connected with the input end of the first channel conversion convolution module, the output end of the first channel conversion convolution module is connected with the input end of the first second-depth convolution module, the output end of the first second-depth convolution module is simultaneously connected with the input end of the first shift module and the second input end of the first second adder, and the output end of the first second adder serves as the second output end of the first residual shift modulation module and is used for outputting the primarily modulated noise level characteristic diagram;
and the output end of the first shifting module is connected with the third input end of the first adder.
In the preferred embodiment, since the first to fourth residual shift modulation modules have the same internal specific structure, the principle of the first residual shift modulation module is analyzed by taking the first residual shift modulation module as an example:
the first one-to-one deep convolution module is used for carrying out convolution operation on the received noise image characteristic extraction image and then sending the convolution operation to the first one-to-one adder;
the first connection module is used for carrying out feature fusion on the noise level feature extraction graph and the noise image feature extraction graph to ensure that the fused feature graph is the noise level feature graph with image information, and sending the noise level feature graph with the image information to the first two-depth convolution module for convolution operation to generate two paths of feature graphs, wherein the first path of feature graph is sent to the first shift module, and the second path of feature graph is sent to the first two-adder;
the first shifting module is used for generating a shifting modulation map according to the received first path characteristic map, and the shifting modulation map is used for modulating the map output by the first one-to-one depth convolution module;
the first adder is used for performing feature superposition on the received three graphs to serve as a noise image feature graph after preliminary modulation, and the noise image feature graph after preliminary modulation serves as the input of the post-scale scaling modulation module;
and the first and second adders are used for superposing the received graphs to serve as a noise level characteristic graph after preliminary modulation, and the noise level characteristic graph serves as the input of the post-stage scaling modulation module.
Further, referring to fig. 1 and fig. 3 specifically, the second residual shift modulation module includes a second first depth convolution module, a second depth convolution module, a second connection module, a second channel conversion convolution module, a second shift module, a second adder, and a second adder;
the input end of the second depth convolution module is connected with the first input end of the second adder and the first input end of the second connection module at the same time and then used as the first input end of the second residual error shift modulation module, and the noise image feature map of the second residual error shift modulation module after the first down sampling;
the output end of the second deep convolution module is connected with the second input end of the second adder, and the output end of the second adder is used as the first output end of the second residual shift modulation module and is used for outputting the first-stage modulated noise image feature map;
the second input end of the second residual error shift modulation module is used for receiving the noise level characteristic diagram after the first downsampling;
the output end of the second coupling module is connected with the input end of a second channel conversion convolution module, the output end of the second channel conversion convolution module is connected with the input end of a second deep convolution module, the output end of the second deep convolution module is simultaneously connected with the input end of a second shift module and the second input end of a second adder, and the output end of the second adder is used as the second output end of a second residual shift modulation module and used for outputting a noise level characteristic diagram after the first-stage modulation;
the output end of the second shifting module is connected with the third input end of the second adder.
Further, referring to fig. 1 and 4 specifically, the third residual shift modulation module includes a third first depth convolution module, a third second depth convolution module, a third concatenation module, a third channel conversion convolution module, a third shift module, a third first adder, and a third second adder;
the input end of the third depth convolution module is connected with the first input end of the third adder and the first input end of the third combination module at the same time and then is used as the first input end of the third residual error shift modulation module, and the noise image characteristic map of the third residual error shift modulation module after the second down sampling;
the output end of the third depth convolution module is connected with the second input end of the third adder, and the output end of the third adder is used as the first output end of the third residual shift modulation module and is used for outputting the second-stage modulated noise image feature map;
the second input end of the third residual error shift modulation module is used for receiving the noise level characteristic diagram after the second downsampling;
the output end of the third link module is connected with the input end of a third channel conversion convolution module, the output end of the third channel conversion convolution module is connected with the input end of a third second depth convolution module, the output end of the third second depth convolution module is simultaneously connected with the input end of a third shift module and the second input end of a third second adder, and the output end of the third second adder serves as the second output end of a third residual shift modulation module and is used for outputting a noise level characteristic diagram after second-stage modulation;
the output end of the third shifting module is connected with the third input end of the third adder.
Further, referring to fig. 1 and 5 specifically, the fourth residual shift modulation module includes a fourth first deep convolution module, a fourth second deep convolution module, a fourth combination module, a fourth channel conversion convolution module, a fourth shift module, a fourth adder, and a fourth second adder;
the input end of the fourth deep convolution module is connected with the first input end of the fourth adder and the first input end of the fourth connection module at the same time, and then is used as the first input end of the fourth residual shift modulation module for receiving the first feature fusion map;
the output end of the fourth deep convolution module is connected with the second input end of the fourth adder, and the output end of the fourth adder is used as the first output end of the fourth residual shift modulation module and is used for outputting a third-level modulated noise image feature map;
a second input end of the fourth linking module is connected with a first input end of the fourth second adder at the same time and then serves as a second input end of the fourth residual shift modulation module, and the second input end of the fourth residual shift modulation module is used for receiving the second feature fusion map;
the output end of the fourth coupling module is connected with the input end of a fourth channel conversion convolution module, the output end of the fourth channel conversion convolution module is connected with the input end of a fourth second deep convolution module, the output end of the fourth second deep convolution module is simultaneously connected with the input end of a fourth shift module and the second input end of a fourth second adder, and the output end of the fourth second adder serves as the second output end of a fourth residual shift modulation module and is used for outputting a noise level characteristic diagram after third-level modulation;
and the output end of the fourth shifting module is connected with the third input end of the fourth adder.
Further, referring to fig. 1 and fig. 6 specifically, the fifth residual shift modulation module includes a fifth first depth convolution module, a fifth second depth convolution module, a fifth combination module, a fifth channel conversion convolution module, a fifth shift module, and a fifth adder;
the input end of the fifth deep convolution module is connected with the first input end of the fifth adder and the first input end of the fifth linkage module at the same time, and then is used as the first input end of the fifth residual shift modulation module for receiving the third feature fusion map;
the output end of the fifth depth convolution module is connected with the second input end of the fifth adder, and the output end of the fifth adder is used as the first output end of the fifth residual shift modulation module and is used for outputting the fourth-level modulated noise image feature map;
a second input end of the fifth linking module is used as a second input end of the fifth residual error shift modulation module and is used for receiving the fourth feature fusion map;
the output end of the fifth coupling module is connected with the input end of the fifth channel conversion convolution module, the output end of the fifth channel conversion convolution module is connected with the input end of the fifth second depth convolution module, and the output end of the fifth second depth convolution module is connected with the input end of the fifth shifting module;
and the output end of the fifth shifting module is connected with the third input end of the fifth adder.
Furthermore, before the image denoising system based on the characteristic modulation is used, the image denoising system is trainedMinimizing mean square error loss function using denoising system
Figure BDA0002555936900000161
Training is carried out;
wherein the content of the first and second substances,
Figure BDA0002555936900000162
Θ represents a parameter of web learning;
n represents the number of training data pairs in the training data set;
i is an integer, and i is 1,2,3 … … N;
xirepresenting a clear image obtained by denoising an ith noise image in a training data set;
yirepresenting the ith noisy image in the training dataset;
Figure BDA0002555936900000163
representing residual images learned by the network;
y represents the original noise image of the network input;
m represents a noise level map corresponding to the original noise image.
Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that features described in different dependent claims and herein may be combined in ways different from those described in the original claims. It is also to be understood that features described in connection with individual embodiments may be used in other described embodiments.

Claims (10)

1. An image denoising system based on feature modulation is characterized by comprising two feature extraction modules, a first residual shift modulation module, a processing module, an image restoration module and a subtracter;
the characteristic extraction module is used for extracting the characteristics of the original noise image to obtain a noise image characteristic extraction image and sending the obtained noise image characteristic extraction image to the first residual error shift modulation module;
the other characteristic extraction module is used for carrying out characteristic extraction on a noise level graph corresponding to the original noise image to obtain a noise level characteristic extraction graph and sending the obtained noise level characteristic extraction graph to the first residual error shift modulation module;
the first residual error shift modulation module is used for carrying out primary modulation on the received noise image characteristic extraction image and the noise level characteristic extraction image to obtain a noise image characteristic image and a noise level characteristic image after the primary modulation, and sending the noise image characteristic image and the noise level characteristic image to the processing module;
the processing module is used for carrying out space scale adjustment and multi-level modulation on the received noise image characteristic map and the noise level characteristic map after the initial modulation to obtain a residual image characteristic map and sending the residual image characteristic map to the image restoration module;
the image restoration module is used for restoring the received residual image characteristic diagram to obtain a residual image and sending the residual image to the subtracter;
the subtracter is also used for receiving the original noise image, and respectively superposing residual features of all feature points in the received residual image on corresponding feature points in the original noise image to realize the denoising of the original noise image and obtain a denoised image;
the processing module comprises a first-level scale scaling modulation module (1), a second-level scale scaling modulation module (2), a third-level scale restoration modulation module (3) and a fourth-level scale restoration modulation module (4);
the primary scale scaling modulation module (1) is used for carrying out primary down-sampling on the noise image characteristic map and the noise level characteristic map after primary modulation to obtain the noise image characteristic map and the noise level characteristic map after the primary down-sampling, carrying out primary modulation on the noise image characteristic map and the noise level characteristic map after the primary down-sampling to obtain the noise image characteristic map and the noise level characteristic map after the primary modulation, and sending the obtained noise image characteristic map and the noise level characteristic map after the primary modulation to the secondary scale scaling modulation module (2) and the three-level scale restoration modulation module (3);
the two-level scale scaling modulation module (2) is used for carrying out second-time down-sampling on the noise image characteristic map and the noise level characteristic map after the first-time modulation, carrying out second-level modulation on the noise image characteristic map and the noise level characteristic map after the second-time down-sampling to obtain a noise image characteristic map and a noise level characteristic map after the second-level modulation, and sending the obtained noise image characteristic map and the noise level characteristic map after the second-level modulation to the three-level scale restoration modulation module (3);
the three-level scale restoration modulation module (3) is used for performing primary up-sampling on the noise image feature map subjected to the secondary modulation and the noise level feature map, connecting the noise image feature map subjected to the primary up-sampling with the noise image feature map subjected to the primary modulation, performing feature fusion on the noise image feature map subjected to the primary up-sampling and the noise image feature map subjected to the primary modulation to obtain a first feature fusion map, performing third-level modulation on the first feature fusion map to obtain a noise image feature map subjected to third-level modulation, and sending the noise image feature map subjected to the third-level modulation to the four-level scale restoration modulation module (4); meanwhile, the noise level characteristic diagram after the first up-sampling is connected with the noise level characteristic diagram after the first-stage modulation, so that the noise level characteristic diagram after the first up-sampling is subjected to characteristic fusion with the noise level characteristic diagram after the first-stage modulation to obtain a second characteristic fusion diagram, the second characteristic fusion diagram is subjected to third-stage modulation to obtain a noise level characteristic diagram after the third-stage modulation, and the noise level characteristic diagram after the third-stage modulation is sent to a four-stage scale restoration modulation module (4);
the four-level scale restoration modulation module (4) is used for receiving the noise image feature map and the noise level feature map after the preliminary modulation; the image processing device is also used for performing second upsampling on the noise image feature map and the noise level feature map after the third-level modulation, and connecting the noise image feature map after the second upsampling with the noise image feature map after the preliminary modulation, so that the noise image feature map after the second upsampling and the noise image feature map after the preliminary modulation are subjected to feature fusion to obtain a third feature fusion map; meanwhile, the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the initial modulation are connected, so that the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the initial modulation are subjected to characteristic fusion, and a fourth characteristic fusion diagram is obtained;
performing fourth-level modulation on the third feature fusion map by using the fourth feature fusion map to obtain a noise image feature map subjected to fourth-level modulation, and sending the noise image feature map subjected to fourth-level modulation to an image restoration module as a residual image feature map output by the processing module;
the first residual error shift modulation module comprises a first one-to-one depth convolution module, a first two-to-one depth convolution module, a first connection module, a first channel conversion convolution module, a first shift module, a first one-to-one adder and a first two-to-one adder;
the input end of the first one-to-one depth convolution module is connected with the first input end of the first adder and the first input end of the first connection module at the same time and then serves as the first input end of the first residual error shift modulation module, and the first input end of the first residual error shift modulation module is used for receiving a noise image characteristic extraction image;
the output end of the first one-to-one depth convolution module is connected with the second input end of the first one-to-one adder, and the output end of the first one-to-one adder is used as the first output end of the first residual shift modulation module and is used for outputting the preliminarily modulated noise image feature map;
the second input end of the first residual error shift modulation module is used for receiving a noise level characteristic extraction diagram;
the output end of the first connection module is connected with the input end of the first channel conversion convolution module, the output end of the first channel conversion convolution module is connected with the input end of the first second-depth convolution module, the output end of the first second-depth convolution module is simultaneously connected with the input end of the first shift module and the second input end of the first second adder, and the output end of the first second adder serves as the second output end of the first residual shift modulation module and is used for outputting the primarily modulated noise level characteristic diagram;
and the output end of the first shifting module is connected with the third input end of the first adder.
2. The system for denoising an image based on feature modulation according to claim 1, wherein the one-level scale modulation module (1) comprises two down-sampling modules and a second residual shift modulation module;
the two down-sampling modules are respectively a first down-sampling module and a second down-sampling module, wherein the first down-sampling module is used for carrying out first down-sampling on the noise image characteristic diagram after the initial modulation and sending the noise image characteristic diagram after the first down-sampling to the second residual error shift modulation module; the second down-sampling module is used for carrying out first down-sampling on the noise level characteristic diagram after the initial modulation and sending the noise level characteristic diagram after the first down-sampling to the second residual error shift modulation module;
and the second residual shift modulation module is used for realizing the mutual modulation between the noise image characteristic diagram after the first down-sampling and the noise level characteristic diagram after the first down-sampling to obtain the noise image characteristic diagram and the noise level characteristic diagram after the first-level modulation.
3. The system for denoising an image based on feature modulation according to claim 1, wherein the two-level scale modulation module (2) comprises two down-sampling modules and a third residual shift modulation module;
the two down-sampling modules are respectively a third down-sampling module and a fourth down-sampling module, wherein the third down-sampling module is used for carrying out second down-sampling on the noise image feature map after the first-level modulation and sending the noise image feature map after the second down-sampling to the third residual error shift modulation module; the fourth down-sampling module is used for performing second down-sampling on the noise level characteristic diagram after the first-stage modulation and sending the noise level characteristic diagram after the second down-sampling to the third residual error shift modulation module;
and the third residual shift modulation module is used for realizing the mutual modulation between the noise image characteristic diagram after the second down-sampling and the noise level characteristic diagram after the second down-sampling to obtain the noise image characteristic diagram and the noise level characteristic diagram after the second-level modulation.
4. The system for denoising an image based on feature modulation according to claim 1, wherein the three-level scale restoration modulation module (3) comprises two upsampling modules, two linking modules and a fourth residual shift modulation module;
the two up-sampling modules are respectively a first up-sampling module and a second up-sampling module; the first up-sampling module is used for performing first up-sampling on the noise image characteristic map subjected to the second-level modulation to obtain a noise image characteristic map subjected to the first up-sampling; the second up-sampling module is used for carrying out first up-sampling on the noise level characteristic diagram after the second-stage modulation to obtain the noise level characteristic diagram after the first up-sampling;
the two connection modules are respectively a No. 1 connection module and a No. 2 connection module;
connecting the noise image feature map subjected to the first up-sampling with the noise image feature map subjected to the first-level modulation through a No. 1 connecting module, so that the noise image feature map subjected to the first up-sampling and the noise image feature map subjected to the first-level modulation are subjected to feature fusion to obtain a first feature fusion map;
connecting the noise level characteristic diagram after the first up-sampling with the noise level characteristic diagram after the first-stage modulation through a No. 2 connecting module, so that the noise level characteristic diagram after the first up-sampling and the noise level characteristic diagram after the first-stage modulation are subjected to characteristic fusion to obtain a second characteristic fusion diagram;
and the fourth residual shift modulation module is used for realizing the mutual modulation between the first characteristic fusion diagram and the second characteristic fusion diagram and obtaining a noise image characteristic diagram and a noise level characteristic diagram after the third-level modulation.
5. The system for denoising an image based on feature modulation according to claim 1, wherein the four-level scale restoration modulation module (4) comprises two upsampling modules, two concatenation modules and a fifth residual shift modulation module;
the two up-sampling modules are respectively a third up-sampling module and a fourth up-sampling module; the third up-sampling module is used for performing second up-sampling on the noise image characteristic diagram after the third-level modulation to obtain a noise image characteristic diagram after the second up-sampling; the fourth up-sampling module is used for performing second up-sampling on the noise level characteristic diagram after the third-level modulation to obtain a noise level characteristic diagram after the second up-sampling;
the two connecting modules are respectively a No. 3 connecting module and a No. 4 connecting module;
connecting the noise image feature map subjected to the second upsampling with the noise image feature map subjected to the preliminary modulation through a No. 3 connecting module, so that the noise image feature map subjected to the first upsampling and the noise image feature map subjected to the preliminary modulation are subjected to feature fusion to obtain a third feature fusion map;
connecting the noise level characteristic diagram after the second up-sampling with the noise level characteristic diagram after the primary modulation through a No. 4 connecting module, so that the noise level characteristic diagram after the second up-sampling and the noise level characteristic diagram after the primary modulation are subjected to characteristic fusion to obtain a fourth characteristic fusion diagram;
and the fifth residual shift modulation module is used for performing fourth-level modulation on the third feature fusion map by using the fourth feature fusion map to obtain a fourth-level modulated noise image feature map.
6. The system of claim 2, wherein the second residual shift modulation module comprises a second first depth convolution module, a second depth convolution module, a second connection module, a second channel conversion convolution module, a second shift module, a second adder, and a second adder;
the input end of the second depth convolution module is connected with the first input end of the second adder and the first input end of the second connection module at the same time and then used as the first input end of the second residual error shift modulation module, and the noise image feature map of the second residual error shift modulation module after the first down sampling;
the output end of the second deep convolution module is connected with the second input end of the second adder, and the output end of the second adder is used as the first output end of the second residual shift modulation module and is used for outputting the first-stage modulated noise image feature map;
the second input end of the second residual error shift modulation module is used for receiving the noise level characteristic diagram after the first downsampling;
the output end of the second coupling module is connected with the input end of a second channel conversion convolution module, the output end of the second channel conversion convolution module is connected with the input end of a second deep convolution module, the output end of the second deep convolution module is simultaneously connected with the input end of a second shift module and the second input end of a second adder, and the output end of the second adder is used as the second output end of a second residual shift modulation module and used for outputting a noise level characteristic diagram after the first-stage modulation;
the output end of the second shifting module is connected with the third input end of the second adder.
7. The feature modulation based image denoising system of claim 3, wherein the third residual shift modulation module comprises a third first depth convolution module, a third second depth convolution module, a third concatenation module, a third channel conversion convolution module, a third shift module, a third first adder and a third second adder;
the input end of the third depth convolution module is connected with the first input end of the third adder and the first input end of the third combination module at the same time and then is used as the first input end of the third residual error shift modulation module, and the noise image characteristic map of the third residual error shift modulation module after the second down sampling;
the output end of the third depth convolution module is connected with the second input end of the third adder, and the output end of the third adder is used as the first output end of the third residual shift modulation module and is used for outputting the second-stage modulated noise image feature map;
the second input end of the third residual error shift modulation module is used for receiving the noise level characteristic diagram after the second downsampling;
the output end of the third link module is connected with the input end of a third channel conversion convolution module, the output end of the third channel conversion convolution module is connected with the input end of a third second depth convolution module, the output end of the third second depth convolution module is simultaneously connected with the input end of a third shift module and the second input end of a third second adder, and the output end of the third second adder serves as the second output end of a third residual shift modulation module and is used for outputting a noise level characteristic diagram after second-stage modulation;
the output end of the third shifting module is connected with the third input end of the third adder.
8. The feature modulation based image denoising system of claim 4, wherein the fourth residual shift modulation module comprises a fourth first depth convolution module, a fourth second depth convolution module, a fourth combination module, a fourth channel conversion convolution module, a fourth shift module, a fourth first adder and a fourth second adder;
the input end of the fourth deep convolution module is connected with the first input end of the fourth adder and the first input end of the fourth connection module at the same time, and then is used as the first input end of the fourth residual shift modulation module for receiving the first feature fusion map;
the output end of the fourth deep convolution module is connected with the second input end of the fourth adder, and the output end of the fourth adder is used as the first output end of the fourth residual shift modulation module and is used for outputting a third-level modulated noise image feature map;
a second input end of the fourth linking module is connected with a first input end of the fourth second adder at the same time and then serves as a second input end of the fourth residual shift modulation module, and the second input end of the fourth residual shift modulation module is used for receiving the second feature fusion map;
the output end of the fourth coupling module is connected with the input end of a fourth channel conversion convolution module, the output end of the fourth channel conversion convolution module is connected with the input end of a fourth second deep convolution module, the output end of the fourth second deep convolution module is simultaneously connected with the input end of a fourth shift module and the second input end of a fourth second adder, and the output end of the fourth second adder serves as the second output end of a fourth residual shift modulation module and is used for outputting a noise level characteristic diagram after third-level modulation;
and the output end of the fourth shifting module is connected with the third input end of the fourth adder.
9. The feature modulation based image denoising system of claim 5, wherein the fifth residual shift modulation module comprises a fifth first depth convolution module, a fifth second depth convolution module, a fifth combination module, a fifth channel conversion convolution module, a fifth shift module and a fifth adder;
the input end of the fifth deep convolution module is connected with the first input end of the fifth adder and the first input end of the fifth linkage module at the same time, and then is used as the first input end of the fifth residual shift modulation module for receiving the third feature fusion map;
the output end of the fifth depth convolution module is connected with the second input end of the fifth adder, and the output end of the fifth adder is used as the first output end of the fifth residual shift modulation module and is used for outputting the fourth-level modulated noise image feature map;
a second input end of the fifth linking module is used as a second input end of the fifth residual error shift modulation module and is used for receiving the fourth feature fusion map;
the output end of the fifth coupling module is connected with the input end of the fifth channel conversion convolution module, the output end of the fifth channel conversion convolution module is connected with the input end of the fifth second depth convolution module, and the output end of the fifth second depth convolution module is connected with the input end of the fifth shifting module;
and the output end of the fifth shifting module is connected with the third input end of the fifth adder.
10. The feature modulation based image denoising system of claim 1, wherein the image denoising system is a system for minimizing the mean square error loss function using a denoising system
Figure FDA0002975652140000061
Training is carried out;
wherein the content of the first and second substances,
Figure FDA0002975652140000062
Θ represents a parameter of web learning;
n represents the number of training data pairs in the training data set;
i is an integer, and i is 1,2,3 … … N;
xirepresenting a clear image obtained by denoising an ith noise image in a training data set;
yirepresenting the ith noisy image in the training dataset;
Figure FDA0002975652140000071
representing residual images learned by the network;
y represents the original noise image of the network input;
m represents a noise level map corresponding to the original noise image.
CN202010589835.0A 2020-06-24 2020-06-24 Image denoising system based on characteristic modulation Active CN111738956B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010589835.0A CN111738956B (en) 2020-06-24 2020-06-24 Image denoising system based on characteristic modulation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010589835.0A CN111738956B (en) 2020-06-24 2020-06-24 Image denoising system based on characteristic modulation

Publications (2)

Publication Number Publication Date
CN111738956A CN111738956A (en) 2020-10-02
CN111738956B true CN111738956B (en) 2021-06-01

Family

ID=72650970

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010589835.0A Active CN111738956B (en) 2020-06-24 2020-06-24 Image denoising system based on characteristic modulation

Country Status (1)

Country Link
CN (1) CN111738956B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932474A (en) * 2020-07-30 2020-11-13 深圳市格灵人工智能与机器人研究院有限公司 Image denoising method based on deep learning

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104240203A (en) * 2014-09-09 2014-12-24 浙江工业大学 Medical ultrasound image denoising method based on wavelet transform and quick bilateral filtering
CN109191399A (en) * 2018-08-29 2019-01-11 陕西师范大学 Magnetic Resonance Image Denoising based on improved multipath matching pursuit algorithm
CN111028171A (en) * 2019-12-06 2020-04-17 北京金山云网络技术有限公司 Method, device and server for determining noise level of image

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3166072A1 (en) * 2015-11-06 2017-05-10 Thomson Licensing Method for denoising an image and apparatus for denoising an image
CN109285129A (en) * 2018-09-06 2019-01-29 哈尔滨工业大学 Image real noise based on convolutional neural networks removes system
CN109658344B (en) * 2018-11-12 2022-10-25 哈尔滨工业大学（深圳） Image denoising method, device and equipment based on deep learning and storage medium
CN110248190B (en) * 2019-07-03 2020-10-27 西安交通大学 Multilayer residual coefficient image coding method based on compressed sensing
CN110738605B (en) * 2019-08-30 2023-04-28 山东大学 Image denoising method, system, equipment and medium based on transfer learning
KR20190119548A (en) * 2019-10-02 2019-10-22 엘지전자 주식회사 Method and apparatus for processing image noise
CN110782406B (en) * 2019-10-15 2022-10-11 深圳大学 Image denoising method and device based on information distillation network
CN111127331B (en) * 2019-10-22 2020-09-08 广东启迪图卫科技股份有限公司 Image denoising method based on pixel-level global noise estimation coding and decoding network
CN111080541B (en) * 2019-12-06 2020-10-30 广东启迪图卫科技股份有限公司 Color image denoising method based on bit layering and attention fusion mechanism
CN111192211B (en) * 2019-12-24 2022-07-01 浙江大学 Multi-noise type blind denoising method based on single deep neural network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104240203A (en) * 2014-09-09 2014-12-24 浙江工业大学 Medical ultrasound image denoising method based on wavelet transform and quick bilateral filtering
CN109191399A (en) * 2018-08-29 2019-01-11 陕西师范大学 Magnetic Resonance Image Denoising based on improved multipath matching pursuit algorithm
CN111028171A (en) * 2019-12-06 2020-04-17 北京金山云网络技术有限公司 Method, device and server for determining noise level of image

Also Published As

Publication number Publication date
CN111738956A (en) 2020-10-02

Similar Documents

Publication Publication Date Title
KR101803471B1 (en) Deep learning system and learning method using of convolutional neural network based image patterning
CN110427968B (en) Binocular stereo matching method based on detail enhancement
CN112396607B (en) Deformable convolution fusion enhanced street view image semantic segmentation method
CN111429347A (en) Image super-resolution reconstruction method and device and computer-readable storage medium
CN113033570B (en) Image semantic segmentation method for improving void convolution and multilevel characteristic information fusion
CN111476249B (en) Construction method of multi-scale large-receptive-field convolutional neural network
CN112801895B (en) Two-stage attention mechanism-based GAN network image restoration algorithm
US20230005114A1 (en) Image restoration method and apparatus
CN110853039B (en) Sketch image segmentation method, system and device for multi-data fusion and storage medium
CN112364838B (en) Method for improving handwriting OCR performance by utilizing synthesized online text image
CN113076957A (en) RGB-D image saliency target detection method based on cross-modal feature fusion
CN111738956B (en) Image denoising system based on characteristic modulation
CN116051396B (en) Image denoising method based on feature enhancement network and GRU network
CN111260577B (en) Face image restoration system based on multi-guide image and self-adaptive feature fusion
CN113222819A (en) Remote sensing image super-resolution reconstruction method based on deep convolutional neural network
CN116309913A (en) Method for generating image based on ASG-GAN text description of generation countermeasure network
CN109272450B (en) Image super-resolution method based on convolutional neural network
CN107729885B (en) Face enhancement method based on multiple residual error learning
CN111861870B (en) End-to-end parallel generator network construction method for image translation
CN111667401B (en) Multi-level gradient image style migration method and system
CN111105364B (en) Image restoration method based on rank one decomposition and neural network
CN117078539A (en) CNN-transducer-based local global interactive image restoration method
CN116704506A (en) Cross-environment-attention-based image segmentation method
CN114155560B (en) Light weight method of high-resolution human body posture estimation model based on space dimension reduction
CN116188273A (en) Uncertainty-oriented bimodal separable image super-resolution method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant