CN109754376B - Image denoising method and device - Google Patents
Image denoising method and device Download PDFInfo
- Publication number
- CN109754376B CN109754376B CN201811629202.7A CN201811629202A CN109754376B CN 109754376 B CN109754376 B CN 109754376B CN 201811629202 A CN201811629202 A CN 201811629202A CN 109754376 B CN109754376 B CN 109754376B
- Authority
- CN
- China
- Prior art keywords
- image
- image sample
- noise
- sample set
- loss function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The embodiment of the application provides an image denoising method and device, and the specific implementation principle is as follows: inputting a noise image sample set into a convolutional neural network for training, outputting a denoised image sample set obtained by training, respectively calculating a corresponding mean-square error loss function value and a maximum posterior loss function value according to the noise-free image sample set and the denoised image sample set, obtaining a total loss function value according to the mean-square error loss function value and the maximum posterior loss function value, finally iteratively training the convolutional neural network based on the total loss function value obtained by calculation until a training termination condition is met, outputting an image denoising model obtained by training, and denoising an image to be denoised according to the image denoising model. Therefore, the convolution neural network is iteratively trained by combining the mean square error loss function value and the maximum posterior loss function value, and the removal effect of real noise can be greatly improved.
Description
Technical Field
The application relates to the technical field of computers, in particular to an image denoising method and device.
Background
At present, Image Denoising (Image Denoising) has been a very important topic in the field of Image restoration, and has high research value in both academic value and landing application. However, the current image denoising methods are generally established on an additive white gaussian noise model, however, the inventor researches and finds that the true noise model in the actual image is not the same, so that the current image denoising algorithms cannot remove the noise on the true image well.
Disclosure of Invention
In order to overcome the above-mentioned deficiencies in the prior art, the present application provides an image denoising method and apparatus, so as to solve or improve the above-mentioned problems.
In order to achieve the above purpose, the embodiments of the present application employ the following technical solutions:
in a first aspect, an embodiment of the present application provides an image denoising method applied to an electronic device, where the method includes:
acquiring a training sample set, wherein the training sample set comprises a noise image sample set and a noise-free image sample set corresponding to the noise image sample set;
inputting the noise image sample set into a convolutional neural network for training, and outputting a denoised image sample set obtained by training;
respectively calculating corresponding mean-square error loss function values and maximum posterior loss function values according to the noiseless image sample set and the denoising image sample set, and obtaining total loss function values according to the mean-square error loss function values and the maximum posterior loss function values;
iteratively training the convolutional neural network based on the total loss function value obtained through calculation until a training termination condition is met, outputting an image denoising model obtained through training, and denoising an image to be denoised according to the image denoising model.
In one possible embodiment, the step of obtaining a training sample set includes:
acquiring a plurality of frames of noise image samples continuously shot by image acquisition equipment based on configured image acquisition parameters to construct a noise image sample set, wherein the image acquisition parameters comprise image exposure parameters and shooting frequency parameters;
sequencing the pixel value of each pixel point in each frame of noise image sample in the noise image sample set, and eliminating abnormal pixel points with the pixel values smaller than a first preset pixel value and the pixel values larger than a second preset pixel value according to a sequencing result to obtain the noise image sample with the abnormal pixel points eliminated;
calculating the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated;
and obtaining a noiseless image sample set corresponding to the noise image sample set according to the relation between a preset noise image sample and the corresponding noiseless image sample and the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated.
In one possible embodiment, the noisy image sample and the corresponding noiseless image sample have the following relationship:
xn＝clip(yn+n(yn))
n(yn)～N(0,σn 2(yn))
wherein x isnFor noisy image samples, ynFor the corresponding noise-free image samples,n(yn) Is ynOf the noise distribution function ofn 2(yn) Variance obeyed by noise distribution function, where σn 2(yn)＝β1 nyn+β2 n，β1And beta2The method includes the steps of correlating image sensor parameters and sensitivity values of the image acquisition device.
In a possible implementation manner, the step of inputting the noisy image sample set into a convolutional neural network for training, and outputting a denoised image sample set obtained by training includes:
inputting the noise image sample set into a convolutional neural network, and sequentially extracting image characteristic information of each noise image sample in the noise image sample set through each layer of convolutional layer in the convolutional neural network;
and sending the image characteristic information of each noise image sample extracted by each convolution layer to a deconvolution layer symmetrically arranged with the convolution layer through a connecting layer, and generating a denoised image sample set after deconvolution according to the image characteristic information of each noise image sample through the deconvolution layer.
In one possible embodiment, the step of calculating the corresponding mean-square-error loss function value and the maximum posterior loss function value according to the noiseless image sample set and the noiseless image sample set respectively includes:
calculating a mean square error loss function value according to the noiseless image sample set and the denoised image sample set, specifically:
respectively calculating a square value of a difference value of each pixel point between each frame of noiseless image sample in the noiseless image sample set and the denoised image sample corresponding to each frame of noiseless image sample in the denoised image sample set, and taking the mean value of the square values of the difference values of each pixel point obtained by calculation as a mean square error loss function value; and
calculating a maximum posterior loss function value according to the noiseless image sample set and the denoised image sample set, which specifically comprises the following steps:
calculating the posterior probability of the Poisson noise existing in each noiseless image sample in the noiseless image sample set according to each noiseless image sample in the noiseless image sample set and the Poisson noise image after the Poisson noise is added into each noiseless image sample;
calculating posterior probability of Gaussian noise existing in each Poisson noise image according to the Poisson noise image after the Poisson noise is added into each noiseless image sample and the Gaussian noise image after the Gaussian noise is added into each noiseless image sample;
calculating to obtain the maximum posterior probability of each noiseless image sample with Poisson Gaussian noise according to the posterior probability of each noiseless image sample with Poisson noise and the posterior probability of each Poisson noise image with Gaussian noise;
and calculating a negative logarithm function value taking the maximum posterior probability of Poisson Gaussian noise existing in each noiseless image sample as an independent variable, and taking the negative logarithm function value as the maximum posterior loss function value.
In one possible embodiment, the step of deriving the total loss function value from the mean square error loss function value and the maximum a posteriori loss function value comprises:
respectively calculating a first weight loss function value of the mean square error loss function values and a second weight loss function value of the maximum posterior loss function values according to a preset weight proportion;
and obtaining the total loss function value according to the first weight loss function value and the second weight loss function value.
In a second aspect, an embodiment of the present application further provides an image denoising apparatus, which is applied to an electronic device, and the apparatus includes:
the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a training sample set, and the training sample set comprises a noise image sample set and a noise-free image sample set corresponding to the noise image sample set;
the training module is used for inputting the noise image sample set into a convolutional neural network for training and outputting a denoised image sample set obtained by training;
the computing module is used for respectively computing corresponding mean-square error loss function values and maximum posterior loss function values according to the noiseless image sample set and the noiseless image sample set, and obtaining total loss function values according to the mean-square error loss function values and the maximum posterior loss function values;
and the denoising processing module is used for iteratively training the convolutional neural network based on the total loss function value obtained by calculation until a training termination condition is met, outputting an image denoising model obtained by training, and denoising an image to be denoised according to the image denoising model.
In a third aspect, an embodiment of the present application further provides a readable storage medium, on which a computer program is stored, where the computer program is executed to implement the image denoising method described above.
Compared with the prior art, the method has the following beneficial effects:
the embodiment of the application provides an image denoising method and device, and the specific implementation principle is as follows: inputting a noise image sample set into a convolutional neural network for training, outputting a denoised image sample set obtained by training, respectively calculating a corresponding mean-square error loss function value and a maximum posterior loss function value according to the noise-free image sample set and the denoised image sample set, obtaining a total loss function value according to the mean-square error loss function value and the maximum posterior loss function value, finally iteratively training the convolutional neural network based on the total loss function value obtained by calculation until a training termination condition is met, outputting an image denoising model obtained by training, and denoising an image to be denoised according to the image denoising model. Therefore, the convolution neural network is iteratively trained by combining the mean square error loss function value and the maximum posterior loss function value, and the removal effect of real noise can be greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and it will be apparent to those skilled in the art that other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a schematic flow chart of an image denoising method according to an embodiment of the present application;
fig. 2 is a schematic diagram of a network structure of a convolutional neural network provided in an embodiment of the present application;
fig. 3 is a schematic functional block diagram of an image denoising device according to an embodiment of the present disclosure;
fig. 4 is a schematic block diagram of a structure of an electronic device for implementing the image denoising method according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
As a result of the technical problems known in the background art, the inventors of the present application discovered, during research, that with Deep Learning (DL) of the heteromilitary project in the field of classification and identification, etc., the field of image restoration quickly uses DL as an important research tool. Because supervised learning is adopted, real image noise and a corresponding noiseless image (GT) are used as a training sample set at the same time, so that the DL denoising algorithm can be really used in the actual life.
However, the inventor finds that the current DL denoising algorithm has the following problems in the research process:
firstly, when a training sample is acquired, a noise image sample is easy to acquire, but a corresponding noise-free image is extremely difficult to obtain, if a traditional denoising algorithm is used for denoising the noise image sample to be used as a noise-free image, the noise-free image is obviously inaccurate, and finally, the upper limit of the denoising capability of the image obtained through deep learning is only the traditional denoising algorithm.
Secondly, image denoising and image classification recognition are different, the interpretability of the current deep learning algorithm in the image denoising is insufficient, the current deep learning algorithm is difficult to determine and is certain to be most suitable for the image denoising, and in the actual training process, the phenomena that the training is difficult to converge, the loss of image details is serious after the training and the like often occur.
Thirdly, since all current image denoising methods assume that real image noise is gaussian distributed, a mean square error Loss function value MSE Loss is generally adopted as a Loss function of image denoising, but real image noise is not gaussian distributed actually, and a deep learning algorithm trained in this way has a problem that noise on a real image cannot be well removed as described in the foregoing background art.
Based on the above technical problem, the present inventors propose the following technical solution to solve or improve the above problem. It should be noted that the above prior art solutions have shortcomings, which are the results of practical and careful study by the inventor, and therefore, the discovery process of the above problems and the solutions proposed by the following embodiments of the present application for the above problems should be the contribution of the inventor to the present application in the process of invention creation.
Please refer to fig. 1, which is a schematic flow chart of an image denoising method according to an embodiment of the present application, and it should be noted that the image denoising method according to the embodiment of the present application is not limited by the specific sequence shown in fig. 1 and the following. The method comprises the following specific processes:
step S210, a training sample set is obtained.
In this embodiment, the training sample set includes a noise image sample set and a noise-free image sample set corresponding to the noise image sample set. In order to obtain a relatively accurate noiseless image sample set, the inventor of the present application, through long-term research, proposes the following scheme:
firstly, acquiring a plurality of frames of noise image samples continuously shot by an image acquisition device based on configured image acquisition parameters to construct a noise image sample set, wherein the image acquisition parameters comprise image exposure parameters and shooting frequency parameters.
Next, since the variance of the distribution of the image noise of each noise image sample in the noise image sample set is uncertain, the pixel value of a part of the image noise of the noise image sample is actually out of 0 to 255, but the final pixel value is fixed to 0 to 255 due to the problem of the actual digital representation. If the average value of each pixel point of the noisy image samples is used for subsequently generating corresponding noiseless image samples, the noiseless image samples are actually shifted, so that the noiseless image samples are inaccurate. In view of this, the inventor proposes that the pixel value of each pixel point in each frame of noise image sample in the noise image sample set is sorted, and according to the sorting result, an abnormal pixel point with the pixel value smaller than a first preset pixel value and the pixel value larger than a second preset pixel value is removed, so as to obtain the noise image sample with the abnormal pixel point removed. And then, calculating the average value of all pixel points of the noise image sample after the abnormal pixel points of all frames are eliminated.
And finally, obtaining a noiseless image sample set corresponding to the noise image sample set according to the relation between a preset noise image sample and the corresponding noiseless image sample and the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated.
Wherein the noisy image samples and the corresponding non-noisy image samples have the following relationship:
xn＝clip(yn+n(yn))
n(yn)～N(0,σn 2(yn))
wherein x isnFor noisy image samples, ynFor the corresponding noise-free image samples,n(yn) Is ynOf the noise distribution function ofn 2(yn) Variance obeyed by noise distribution function, where σn 2(yn)＝β1 nyn+β2 n，β1And beta2The method includes the steps of correlating image sensor parameters and sensitivity values of the image acquisition device.
The noiseless image sample acquired by the scheme is high in accuracy, and the image denoising capability obtained by subsequent deep learning can be effectively improved.
And S220, inputting the noise image sample set into a convolutional neural network for training, and outputting a denoised image sample set obtained by training.
In the design of the convolutional neural network, the convergence of the algorithm needs to be considered, and how to reduce the loss of the detail texture of the image while ensuring the denoising effect. The inventor of the application aims at the requirements, adopts an encoding-decoding mode, removes a normalization layer of the traditional convolutional neural network, and adds a large number of Long-Term Link connection layers. In detail, referring to fig. 2 in a possible implementation manner, the convolutional neural network designed in this embodiment includes a plurality of convolutional layer convergence as an encoding Encoder portion of the convolutional neural network and a plurality of anti-convolutional layer deconstruction as a decoding Decoder portion of the convolutional neural network. The convolutional layer constraint mainly has the functions of extracting the characteristic information of a noise image sample from an input noise image sample through a large number of convolutional layers and removing image noise; the role of the Deconvolution layer Deconvolution is to recover image details in the feature information of the noisy image samples.
In order to accelerate the convergence of the convolutional neural network in the training process, a large number of Long-Term Link connection layers are added into the convolutional neural network, and the feature information extracted by the preceding convolutional layer Convolition is directly added into the following Deconvolution layer. Therefore, the problem of gradient disappearance of the convolutional neural network under the condition of adopting a Back Propagation method. Meanwhile, the Long-Term Link connection layer can be regarded as adding the medium-high frequency image feature information extracted from the preceding Convolution layer constraint back to the following Deconvolution layer constraint, and the medium-high frequency image feature information is the texture details of the noise image sample, so that the problem of texture detail loss can be greatly relieved by adding the Long-Term Link connection layer.
Based on the above analysis, in the present embodiment, the noise image sample set is input into a convolutional neural network, image feature information of each noise image sample in the noise image sample set is sequentially extracted by each layer of convolutional layer in the convolutional neural network, and the image feature information of each noise image sample extracted by each layer of convolutional layer is sent to a deconvolution layer symmetrically arranged with the layer of convolutional layer through a connection layer, and a denoised image sample set after deconvolution is generated by the deconvolution layer according to the image feature information of each noise image sample.
Step S230, respectively calculating a corresponding mean-square-error loss function value and a maximum posterior loss function value according to the noise-free image sample set and the noise-free image sample set, and obtaining a total loss function value according to the mean-square-error loss function value and the maximum posterior loss function value.
In a possible implementation, the mean square error loss function value is calculated according to the set of noiseless image samples and the set of denoised image samples, and may specifically be:
and respectively calculating the square value of the difference value of each pixel point between each frame of the noiseless image sample in the noiseless image sample set and the denoised image sample corresponding to each frame of the noiseless image sample in the denoised image sample set, and taking the mean value of the square values of the difference values of each pixel point obtained by calculation as the mean square error LOSS function value MSE LOSS.
In a possible implementation, the maximum posterior loss function value is calculated according to the noiseless image sample set and the denoised image sample set, and may specifically be:
first, a maximum posterior probability P (poi | GT) of poisson noise existing in each of the noiseless image samples in the noiseless image sample set is calculated according to each of the noiseless image samples GT in the noiseless image sample set and a poisson noise image poi after poisson noise is added to each of the noiseless image samples. The specific calculation formula of P (poi | GT) may include:
next, from the Poisson noise image poi after adding Poisson noise to each of the noiseless image samples, the Gaussian noise image I after adding Gaussian noise to each of the noiseless image samplesnoise(noise), the maximum posterior probability P (I) of Gaussian noise existing in each Poisson noise image poi is calculatednoiseI poi). The specific calculation formula may include:
Inoise＝poi+∈，∈～N(0，σ)
according to the posterior probability P (poi | GT) of each noiseless image sample with Poisson noise and the posterior probability P (I) of each Poisson noise image with Gaussian noisenoiseI poi), calculating the maximum posterior probability P (I) of Poisson Gaussian noise existing in each noiseless image sample GT in the noiseless image sample setnoise| GT). Concrete computing deviceThe formula may include:
finally, calculating a negative logarithm function value logP (I) taking the maximum posterior probability of Poisson Gaussian noise existing in each noiseless image sample as an independent variablenoiseGT) and apply the negative log function value logP (I)noiseGT) as the maximum a posteriori LOSS function value MAP LOSS, and continuously minimizing the maximum a posteriori LOSS function value MAP LOSS.
Then, a first weight LOSS function value of the mean square error LOSS function value MSE LOSS and a second weight LOSS function value of the maximum posterior LOSS function value MAP LOSS may be respectively calculated according to a preset weight proportion, and then the total LOSS function value may be obtained according to the first weight LOSS function value and the second weight LOSS function value. For example, a sum of the first weight loss function value and the second weight loss function value may be calculated to obtain the total loss function value.
And S240, iteratively training the convolutional neural network based on the total loss function value obtained through calculation until a training termination condition is met, outputting an image denoising model obtained through training, and denoising an image to be denoised according to the image denoising model.
In detail, based on the total loss function value obtained by calculation, continuously updating the network parameters of the convolutional neural network, and repeating the training steps to perform iterative training until a training termination condition is met (for example, the total loss function value does not decrease, the iterative training times reach a preset number, and the like), outputting an image denoising model obtained by training, wherein the image denoising model can have an image denoising capability, and denoising an image to be denoised according to the image denoising model.
Therefore, the convolutional neural network is iteratively trained by combining the mean square error loss function value and the maximum posterior loss function value, and the removal effect of real noise can be greatly improved.
Further, referring to fig. 3, an embodiment of the present application further provides an image denoising apparatus 200, where the image denoising apparatus 200 may include:
an obtaining module 210, configured to obtain a training sample set, where the training sample set includes a noise image sample set and a noise-free image sample set corresponding to the noise image sample set.
And the training module 220 is configured to input the noise image sample set into a convolutional neural network for training, and output a denoised image sample set obtained by training.
And a calculating module 230, configured to calculate a corresponding mean-square-error loss function value and a corresponding maximum posterior loss function value according to the noiseless image sample set and the noiseless image sample set, and obtain a total loss function value according to the mean-square-error loss function value and the maximum posterior loss function value.
And the denoising processing module 240 is configured to iteratively train the convolutional neural network based on the calculated total loss function value until a training termination condition is met, output an image denoising model obtained by training, and perform denoising processing on an image to be denoised according to the image denoising model.
In a possible implementation manner, the obtaining module 210 may specifically obtain the training sample set by:
acquiring a plurality of frames of noise image samples continuously shot by image acquisition equipment based on configured image acquisition parameters to construct a noise image sample set, wherein the image acquisition parameters comprise image exposure parameters and shooting frequency parameters;
sequencing the pixel value of each pixel point in each frame of noise image sample in the noise image sample set, and eliminating abnormal pixel points with the pixel values smaller than a first preset pixel value and the pixel values larger than a second preset pixel value according to a sequencing result to obtain the noise image sample with the abnormal pixel points eliminated;
calculating the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated;
obtaining a noiseless image sample set corresponding to the noise image sample set according to the relation between a preset noise image sample and the corresponding noiseless image sample and the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated;
the noisy image samples and the corresponding non-noisy image samples have the following relationship:
xn＝clip(yn+n(yn))
n(yn)～N(0,σn 2(yn))
wherein x isnFor noisy image samples, ynFor the corresponding noise-free image samples,n(yn) Is ynOf the noise distribution function ofn 2(yn) Variance obeyed by noise distribution function, where σn 2(yn)＝β1 nyn+β2 n，β1And beta2The method includes the steps of correlating image sensor parameters and sensitivity values of the image acquisition device.
In a possible implementation manner, the training module 230 may specifically output a trained denoised image sample set by:
inputting the noise image sample set into a convolutional neural network, and sequentially extracting image characteristic information of each noise image sample in the noise image sample set through each layer of convolutional layer in the convolutional neural network;
and sending the image characteristic information of each noise image sample extracted by each convolution layer to a deconvolution layer symmetrically arranged with the convolution layer through a connecting layer, and generating a denoised image sample set after deconvolution according to the image characteristic information of each noise image sample through the deconvolution layer.
In a possible implementation, the calculating module 240 may specifically calculate the corresponding mean square error loss function value and the maximum a posteriori loss function value by the following manners:
calculating a mean square error loss function value according to the noiseless image sample set and the denoised image sample set, specifically:
respectively calculating a square value of a difference value of each pixel point between each frame of noiseless image sample in the noiseless image sample set and the denoised image sample corresponding to each frame of noiseless image sample in the denoised image sample set, and taking the mean value of the square values of the difference values of each pixel point obtained by calculation as a mean square error loss function value; and
calculating a maximum posterior loss function value according to the noiseless image sample set and the denoised image sample set, which specifically comprises the following steps:
calculating the posterior probability of the Poisson noise existing in each noiseless image sample in the noiseless image sample set according to each noiseless image sample in the noiseless image sample set and the Poisson noise image after the Poisson noise is added into each noiseless image sample;
calculating posterior probability of Gaussian noise existing in each Poisson noise image according to the Poisson noise image after the Poisson noise is added into each noiseless image sample and the Gaussian noise image after the Gaussian noise is added into each noiseless image sample;
calculating to obtain the maximum posterior probability of each noiseless image sample with Poisson Gaussian noise according to the posterior probability of each noiseless image sample with Poisson noise and the posterior probability of each Poisson noise image with Gaussian noise;
and calculating a negative logarithm function value taking the maximum posterior probability of Poisson Gaussian noise existing in each noiseless image sample as an independent variable, and taking the negative logarithm function value as the maximum posterior loss function value.
It can be understood that, for the specific operation method of each functional module in this embodiment, reference may be made to the detailed description of the corresponding step in the foregoing method embodiment, and no repeated description is provided herein.
Further, please refer to fig. 4, which is a block diagram illustrating a structure of an electronic device 100 for the image denoising method according to the embodiment of the present application. In this embodiment, the electronic device 100 may be implemented by a bus 110 as a general bus architecture. Bus 110 may include any number of interconnecting buses and bridges depending on the specific application of electronic device 100 and the overall design constraints. Bus 110 connects various circuits together, including processor 120, storage medium 130, and bus interface 140. Alternatively, the electronic apparatus 100 may connect a network adapter 150 or the like via the bus 110 using the bus interface 140. The network adapter 150 may be used to implement signal processing functions of a physical layer in the electronic device 100 and implement transmission and reception of radio frequency signals through an antenna. The user interface 160 may connect external devices such as: a keyboard, a display, a mouse or a joystick, etc. The bus 110 may also connect various other circuits such as timing sources, peripherals, voltage regulators, or power management circuits, which are well known in the art, and therefore, will not be described in detail.
Alternatively, the electronic device 100 may be configured as a general purpose processing system, for example, commonly referred to as a chip, including: one or more microprocessors providing processing functions, and an external memory providing at least a portion of storage medium 130, all connected together with other support circuits through an external bus architecture.
Alternatively, the electronic device 100 may be implemented using: an ASIC (application specific integrated circuit) having a processor 120, a bus interface 140, a user interface 160; and at least a portion of the storage medium 130 integrated in a single chip, or the electronic device 100 may be implemented using: one or more FPGAs (field programmable gate arrays), PLDs (programmable logic devices), controllers, state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.
Among other things, processor 120 is responsible for managing bus 110 and general processing (including the execution of software stored on storage medium 130). Processor 120 may be implemented using one or more general-purpose processors and/or special-purpose processors. Examples of processor 120 include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Software should be construed broadly to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
The processor 120 may execute the above embodiments, and in particular, the storage medium 130 may store the image denoising apparatus 200 therein, and the processor 120 may be configured to execute the image denoising apparatus 200.
Further, an embodiment of the present application also provides a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions may execute the image denoising method in any of the above method embodiments.
In summary, the embodiment of the present application provides an image denoising method and apparatus, and the specific implementation principle is as follows: inputting a noise image sample set into a convolutional neural network for training, outputting a denoised image sample set obtained by training, respectively calculating a corresponding mean-square error loss function value and a maximum posterior loss function value according to the noise-free image sample set and the denoised image sample set, obtaining a total loss function value according to the mean-square error loss function value and the maximum posterior loss function value, finally iteratively training the convolutional neural network based on the total loss function value obtained by calculation until a training termination condition is met, outputting an image denoising model obtained by training, and denoising an image to be denoised according to the image denoising model. Therefore, the convolution neural network is iteratively trained by combining the mean square error loss function value and the maximum posterior loss function value, and the removal effect of real noise can be greatly improved.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus and method embodiments described above are illustrative only, as the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
Alternatively, all or part of the implementation may be in software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as an electronic device, server, data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (8)
1. An image denoising method applied to an electronic device, the method comprising:
acquiring a training sample set, wherein the training sample set comprises a noise image sample set and a noise-free image sample set corresponding to the noise image sample set;
inputting the noise image sample set into a convolutional neural network for training, and outputting a denoised image sample set obtained by training;
respectively calculating corresponding mean-square error loss function values and maximum posterior loss function values according to the noiseless image sample set and the denoising image sample set, and obtaining total loss function values according to the mean-square error loss function values and the maximum posterior loss function values; wherein the step of calculating corresponding mean-square-error loss function values and maximum a posteriori loss function values, respectively, from the noiseless image sample set and the denoised image sample set comprises:
calculating a mean square error loss function value according to the noiseless image sample set and the denoised image sample set, specifically:
respectively calculating a square value of a difference value of each pixel point between each frame of noiseless image sample in the noiseless image sample set and the denoised image sample corresponding to each frame of noiseless image sample in the denoised image sample set, and taking the mean value of the square values of the difference values of each pixel point obtained by calculation as a mean square error loss function value; and
calculating a maximum posterior loss function value according to the noiseless image sample set and the denoised image sample set, which specifically comprises the following steps:
calculating the posterior probability of the Poisson noise existing in each noiseless image sample in the noiseless image sample set according to each noiseless image sample in the noiseless image sample set and the Poisson noise image after the Poisson noise is added into each noiseless image sample;
calculating posterior probability of Gaussian noise existing in each Poisson noise image according to the Poisson noise image after the Poisson noise is added into each noiseless image sample and the Gaussian noise image after the Gaussian noise is added into each noiseless image sample;
calculating to obtain the maximum posterior probability of each noiseless image sample with Poisson Gaussian noise according to the posterior probability of each noiseless image sample with Poisson noise and the posterior probability of each Poisson noise image with Gaussian noise;
calculating a negative logarithm function value taking the maximum posterior probability of Poisson Gaussian noise existing in each noiseless image sample as an independent variable, and taking the negative logarithm function value as the maximum posterior loss function value;
iteratively training the convolutional neural network based on the total loss function value obtained through calculation until a training termination condition is met, outputting an image denoising model obtained through training, and denoising an image to be denoised according to the image denoising model.
2. The image denoising method of claim 1, wherein the step of obtaining a training sample set comprises:
acquiring a plurality of frames of noise image samples continuously shot by image acquisition equipment based on configured image acquisition parameters to construct a noise image sample set, wherein the image acquisition parameters comprise image exposure parameters and shooting frequency parameters;
sequencing the pixel value of each pixel point in each frame of noise image sample in the noise image sample set, and eliminating abnormal pixel points with the pixel values smaller than a first preset pixel value and the pixel values larger than a second preset pixel value according to a sequencing result to obtain the noise image sample with the abnormal pixel points eliminated;
calculating the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated;
and obtaining a noiseless image sample set corresponding to the noise image sample set according to the relation between a preset noise image sample and the corresponding noiseless image sample and the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated.
3. The method of image denoising of claim 2, wherein the noisy image samples and corresponding non-noisy image samples have the following relationship:
xn＝clip(yn+n(yn))
n(yn)～N(0,σn 2(yn))
wherein x isnFor noisy image samples, ynFor the corresponding noise-free image samples,n(yn) Is ynOf the noise distribution function ofn 2(yn) Function clip (y) for variance obeyed by the noise distribution functionn+n(yn) Y) represents the noisy image sample compared to the noiseless image samplenThe noisy image sample xnSatisfies a distribution functionn(yn) Wherein, in the step (A),
，β1and beta2The method includes the steps of correlating image sensor parameters and sensitivity values of the image acquisition device.
4. The image denoising method of claim 1, wherein the step of inputting the noise image sample set into a convolutional neural network for training and outputting a trained denoised image sample set comprises:
inputting the noise image sample set into a convolutional neural network, and sequentially extracting image characteristic information of each noise image sample in the noise image sample set through each layer of convolutional layer in the convolutional neural network;
and aiming at each layer of the convolutional layer, sending the image characteristic information of each noise image sample extracted by the convolutional layer to a deconvolution layer symmetrically arranged by the convolutional layer through a connecting layer, and generating a denoised image sample set after deconvolution through the deconvolution layer according to the image characteristic information of each noise image sample.
5. The method of image denoising of claim 1, wherein said step of deriving a total loss function value from the mean square error loss function value and a maximum a posteriori loss function value comprises:
respectively calculating a first weight loss function value of the mean square error loss function values and a second weight loss function value of the maximum posterior loss function values according to a preset weight proportion;
and obtaining the total loss function value according to the first weight loss function value and the second weight loss function value.
6. An image denoising apparatus applied to an electronic device, the apparatus comprising:
the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a training sample set, and the training sample set comprises a noise image sample set and a noise-free image sample set corresponding to the noise image sample set;
the training module is used for inputting the noise image sample set into a convolutional neural network for training and outputting a denoised image sample set obtained by training;
the computing module is used for respectively computing corresponding mean-square error loss function values and maximum posterior loss function values according to the noiseless image sample set and the noiseless image sample set, and obtaining total loss function values according to the mean-square error loss function values and the maximum posterior loss function values; the calculation module specifically calculates a corresponding mean square error loss function value and a maximum posterior loss function value by the following methods:
calculating a mean square error loss function value according to the noiseless image sample set and the denoised image sample set, specifically:
respectively calculating a square value of a difference value of each pixel point between each frame of noiseless image sample in the noiseless image sample set and the denoised image sample corresponding to each frame of noiseless image sample in the denoised image sample set, and taking the mean value of the square values of the difference values of each pixel point obtained by calculation as a mean square error loss function value; and
calculating a maximum posterior loss function value according to the noiseless image sample set and the denoised image sample set, which specifically comprises the following steps:
calculating the posterior probability of the Poisson noise existing in each noiseless image sample in the noiseless image sample set according to each noiseless image sample in the noiseless image sample set and the Poisson noise image after the Poisson noise is added into each noiseless image sample;
calculating posterior probability of Gaussian noise existing in each Poisson noise image according to the Poisson noise image after the Poisson noise is added into each noiseless image sample and the Gaussian noise image after the Gaussian noise is added into each noiseless image sample;
calculating to obtain the maximum posterior probability of each noiseless image sample with Poisson Gaussian noise according to the posterior probability of each noiseless image sample with Poisson noise and the posterior probability of each Poisson noise image with Gaussian noise;
calculating a negative logarithm function value taking the maximum posterior probability of Poisson Gaussian noise existing in each noiseless image sample as an independent variable, and taking the negative logarithm function value as the maximum posterior loss function value;
and the denoising processing module is used for iteratively training the convolutional neural network based on the total loss function value obtained by calculation until a training termination condition is met, outputting an image denoising model obtained by training, and denoising an image to be denoised according to the image denoising model.
7. The image denoising apparatus of claim 6, wherein the obtaining module obtains the training sample set by:
acquiring a plurality of frames of noise image samples continuously shot by image acquisition equipment based on configured image acquisition parameters to construct a noise image sample set, wherein the image acquisition parameters comprise image exposure parameters and shooting frequency parameters;
sequencing the pixel value of each pixel point in each frame of noise image sample in the noise image sample set, and eliminating abnormal pixel points with the pixel values smaller than a first preset pixel value and the pixel values larger than a second preset pixel value according to a sequencing result to obtain the noise image sample with the abnormal pixel points eliminated;
calculating the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated;
obtaining a noiseless image sample set corresponding to the noise image sample set according to the relation between a preset noise image sample and the corresponding noiseless image sample and the average value of each pixel point of the noise image sample after each frame of abnormal pixel point is eliminated;
the noisy image samples and the corresponding non-noisy image samples have the following relationship:
xn＝clip(yn+n(yn))
n(yn)～N(0,σn 2(yn))
wherein x isnFor noisy image samples, ynFor the corresponding noise-free image samples,n(yn) Is ynOf the noise distribution function ofn 2(yn) Function clip (y) for variance obeyed by the noise distribution functionn+n(yn) Y) represents the noisy image sample compared to the noiseless image samplenThe noisy image sample xnSatisfies a distribution functionn(yn) Wherein, in the step (A),
，β1and beta2The method includes the steps of correlating image sensor parameters and sensitivity values of the image acquisition device.
8. The image denoising apparatus of claim 6, wherein the training module outputs a trained denoised image sample set by:
inputting the noise image sample set into a convolutional neural network, and sequentially extracting image characteristic information of each noise image sample in the noise image sample set through each layer of convolutional layer in the convolutional neural network;
and aiming at each layer of the convolutional layer, sending the image characteristic information of each noise image sample extracted by the convolutional layer to a deconvolution layer symmetrically arranged with the convolutional layer through a connecting layer, and generating a denoised image sample set after deconvolution through the deconvolution layer according to the image characteristic information of each noise image sample.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811629202.7A CN109754376B (en) | 2018-12-28 | 2018-12-28 | Image denoising method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811629202.7A CN109754376B (en) | 2018-12-28 | 2018-12-28 | Image denoising method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109754376A CN109754376A (en) | 2019-05-14 |
| CN109754376B true CN109754376B (en) | 2020-11-10 |
Family
ID=66403223
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811629202.7A Active CN109754376B (en) | 2018-12-28 | 2018-12-28 | Image denoising method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109754376B (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163827B (en) * | 2019-05-28 | 2023-01-10 | 腾讯科技（深圳）有限公司 | Training method of image denoising model, image denoising method, device and medium |
| CN110290289B (en) * | 2019-06-13 | 2021-07-09 | Oppo广东移动通信有限公司 | Image noise reduction method and device, electronic equipment and storage medium |
| CN110276730B (en) * | 2019-06-13 | 2023-03-28 | Oppo广东移动通信有限公司 | Image processing method and device and electronic equipment |
| CN110490824A (en) * | 2019-08-23 | 2019-11-22 | 广东浪潮大数据研究有限公司 | The method and device of image denoising |
| US11540798B2 (en) | 2019-08-30 | 2023-01-03 | The Research Foundation For The State University Of New York | Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising |
| CN111260579B (en) * | 2020-01-17 | 2021-08-03 | 北京理工大学 | Low-light-level image denoising and enhancing method based on physical noise generation model |
| CN111369456B (en) * | 2020-02-28 | 2021-08-31 | 深圳市商汤科技有限公司 | Image denoising method and device, electronic device and storage medium |
| CN111709891B (en) * | 2020-06-12 | 2023-11-24 | 北京小米松果电子有限公司 | Training method of image denoising model, image denoising method, device and medium |
| CN111861912B (en) * | 2020-06-30 | 2023-04-25 | 湖南傲英创视信息科技有限公司 | Linear array circumferential scanning stereoscopic panoramic image denoising method, system and storage medium |
| CN111899199B (en) * | 2020-08-07 | 2024-03-19 | 深圳市捷顺科技实业股份有限公司 | Image processing method, device, equipment and storage medium |
| CN112950501B (en) * | 2021-02-26 | 2023-10-13 | 平安科技（深圳）有限公司 | Noise field-based image noise reduction method, device, equipment and storage medium |
| CN113160089B (en) * | 2021-05-10 | 2022-10-04 | 福州大学 | Image denoising method based on local density loss |
| CN115082357B (en) * | 2022-07-20 | 2022-11-25 | 深圳思谋信息科技有限公司 | Video denoising data set generation method and device, computer equipment and storage medium |
| CN116051427B (en) * | 2023-03-31 | 2023-07-04 | 季华实验室 | Point cloud denoising model acquisition method, point cloud fusion method and related equipment thereof |
| CN116681618A (en) * | 2023-06-13 | 2023-09-01 | 强联智创（北京）科技有限公司 | Image denoising method, electronic device and storage medium |
| CN117094909B (en) * | 2023-08-31 | 2024-04-02 | 青岛天仁微纳科技有限责任公司 | Nanometer stamping wafer image acquisition processing method |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106981055A (en) * | 2017-03-30 | 2017-07-25 | 西安交通大学 | A kind of ICCD image de-noising methods |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106204468B (en) * | 2016-06-27 | 2019-04-26 | 深圳市未来媒体技术研究院 | A kind of image de-noising method based on ReLU convolutional neural networks |
| CN106683063B (en) * | 2017-01-22 | 2019-07-05 | 厦门美图之家科技有限公司 | A kind of method and apparatus of image denoising |
| CN109087255B (en) * | 2018-07-18 | 2022-03-08 | 中国人民解放军陆军工程大学 | Lightweight depth image denoising method based on mixed loss |
-
2018
- 2018-12-28 CN CN201811629202.7A patent/CN109754376B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106981055A (en) * | 2017-03-30 | 2017-07-25 | 西安交通大学 | A kind of ICCD image de-noising methods |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109754376A (en) | 2019-05-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109754376B (en) | Image denoising method and device | |
| CN108073902B (en) | Video summarizing method and device based on deep learning and terminal equipment | |
| EP3477550A1 (en) | Vehicle license plate classification method and system based on deep learning, electronic apparatus, and storage medium | |
| CN109544598B (en) | Target tracking method and device and readable storage medium | |
| CN108197652B (en) | Method and apparatus for generating information | |
| CN109146825B (en) | Photography style conversion method, device and readable storage medium | |
| CN109086654B (en) | Handwriting model training method, text recognition method, device, equipment and medium | |
| CN111553215B (en) | Personnel association method and device, graph roll-up network training method and device | |
| CN112800915B (en) | Building change detection method, device, electronic equipment and storage medium | |
| CN112215248A (en) | Deep learning model training method and device, electronic equipment and storage medium | |
| CN113378545B (en) | Aspect level emotion analysis method and device, electronic equipment and storage medium | |
| CN112270200B (en) | Text information translation method and device, electronic equipment and storage medium | |
| CN110209658B (en) | Data cleaning method and device | |
| Lu et al. | Image-specific prior adaptation for denoising | |
| CN113627361B (en) | Training method and device for face recognition model and computer program product | |
| Barczy et al. | Innovational outliers in INAR (1) models | |
| JP7207530B2 (en) | Information processing device, creation method and creation program | |
| CN115170919B (en) | Image processing model training and image processing method, device, equipment and storage medium | |
| CN114758130B (en) | Image processing and model training method, device, equipment and storage medium | |
| Xing et al. | An efficient method to remove mixed Gaussian and random-valued impulse noise | |
| US20190026646A1 (en) | Method to leverage similarity and hierarchy of documents in nn training | |
| CN111414921A (en) | Sample image processing method and device, electronic equipment and computer storage medium | |
| CN109829517B (en) | Target detection duplication removing method and device | |
| CN117456562B (en) | Attitude estimation method and device | |
| CN110717503A (en) | Classification method and device, electronic equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |