CN110728626A - Image deblurring method and apparatus and training thereof - Google Patents
Image deblurring method and apparatus and training thereof Download PDFInfo
- Publication number
- CN110728626A CN110728626A CN201810777095.6A CN201810777095A CN110728626A CN 110728626 A CN110728626 A CN 110728626A CN 201810777095 A CN201810777095 A CN 201810777095A CN 110728626 A CN110728626 A CN 110728626A
- Authority
- CN
- China
- Prior art keywords
- network
- training
- image
- generating
- graph
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 158
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000012795 verification Methods 0.000 claims abstract description 40
- 230000006870 function Effects 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 14
- 230000004913 activation Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 9
- 230000001902 propagating effect Effects 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 5
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 238000009826 distribution Methods 0.000 description 6
- 238000012512 characterization method Methods 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000003042 antagnostic effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The application provides a training method, a device, a system and a storage medium for a generative confrontation network for image deblurring. The training method comprises the following steps: preparing a matched fuzzy training image and a clear true value image; generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of a generating type countermeasure network; comparing training errors between the deblurred verification graph and the clear truth value graph by using a discriminant network of a generative confrontation network, wherein the discriminant network and the generative network are cascaded with each other; and back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition. The application also provides an image deblurring method and device using the trained generative confrontation network.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to a training method, an apparatus, a system and a storage medium for a generative confrontation network for image deblurring, and an image deblurring method and an apparatus using the trained generative confrontation network.
Background
For various reasons, images captured by image capture devices such as smartphones often exhibit blurring. Such blur includes blur caused by lens focus error, blur caused by atmospheric turbulence, blur caused by object movement, and the like. These blurs degrade the quality of the image and affect the user's shooting experience.
The blurred image is equivalent to the convolution result of the blur kernel and the sharp image. Thus, if the blur kernel is known, a sharp image can be reconstructed from the blurred image. This is mathematically equivalent to a "gray box" problem where the model parameters are solved for by a known model.
Indeed, some existing image deblurring methods use the above-mentioned ideas. However, in practice, it is not always possible to successfully know the cause of the blur and to infer the blur kernel. In addition, some images may be blurred for a variety of reasons in combination. In practice, therefore, it is practically difficult to reconstruct a sharp image from a blurred image with known blur kernels. In other words, in practice it is often necessary to face the "black box" problem of not knowing neither the model nor the parameters of the model.
Disclosure of Invention
The application provides a training method of a generative confrontation network for image deblurring. The generative confrontation network comprises a generative network and a discriminative network which are cascaded with each other. The training method comprises the following steps: preparing a matched fuzzy training image and a clear true value image; generating a deblurred verification graph based on the fuzzy training graph by utilizing the generation network; comparing training errors between the deblurred verification map and the clear truth map by using the discrimination network; and back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
According to an embodiment of the application, back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network may include: in a first updating period, fixing the parameters of the generated network, and updating the parameters of the judgment network through the back propagation; and in a second updating period, fixing the parameters of the judging network and updating the parameters of the generating network through the back propagation.
According to the embodiment of the present application, in the first update period, the loss function with the training error as an argument may be maximized to perform the update. In addition, in the second update period, the loss function may be minimized to perform the update.
According to an embodiment of the present application, preparing the paired fuzzy training graph and the distinct truth value graph may include: respectively shooting clear sample images aiming at various scenes to serve as the clear true value images; and respectively carrying out artificial blurring processing on each sample pattern to obtain the blurred training image.
According to an embodiment of the application, the artificial blur process may include at least one of: defocus blur, gaussian blur and motion blur.
According to an embodiment of the application, generating a deblurred verification graph based on the fuzzy training graph by using the generation network may include: performing convolution operation on the fuzzy training graph through a feature extraction layer of the generation network to extract a primary feature graph; performing nonlinear transformation on the primary feature map through an activation layer of the generation network to obtain a secondary feature map; and performing deconvolution operation on the secondary feature map through a reconstruction layer of the generation network to obtain the deblurred verification map.
According to an embodiment of the application, the activation layer may comprise a parametrically modified linear element.
According to the embodiment of the present application, the convergence condition may be that the generative confrontation network reaches nash equilibrium.
The application also provides a training device of the generative confrontation network for deblurring the image. The generative confrontation network comprises a generative network and a discriminative network which are cascaded with each other. The training apparatus includes: a training set preparation device for preparing a fuzzy training image and a clear true value image which are matched; a generator for generating a deblurred verification graph based on the fuzzy training graph by using the generation network; a discriminator for comparing the training error between the deblurred verification map and the clear truth map by using the discrimination network; and a back propagator that back propagates the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
The application also provides an image deblurring system. The system comprises: a processor; and a memory coupled to the processor and storing machine readable instructions. The machine-readable instructions are executable by the processor to perform operations comprising: preparing a matched fuzzy training image and a clear true value image; generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of a generating type countermeasure network; comparing training errors between the deblurred verification graph and the clear truth value graph by using a discriminant network of a generative confrontation network, wherein the discriminant network and the generative network are cascaded with each other; and back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
The present application also provides a non-transitory machine-readable storage medium. The non-transitory machine-readable storage medium stores machine-readable instructions. The machine-readable instructions are executable by a processor to perform operations comprising: preparing a matched fuzzy training image and a clear true value image; generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of a generating type countermeasure network; comparing training errors between the deblurred verification graph and the clear truth value graph by using a discriminant network of a generative confrontation network, wherein the discriminant network and the generative network are cascaded with each other; and back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
The application also provides an image deblurring method. The image deblurring method comprises the following steps: inputting the initial image into a generation network of a generation type countermeasure network trained according to the method; and generating a deblurred image based on the initial image using the generation network.
The application also provides an image deblurring device. The image deblurring device is provided with a generative confrontation network which is trained according to the method.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a simplified schematic diagram illustrating a generative countermeasure network for image deblurring according to an embodiment of the present application;
FIG. 2 is a flow chart illustrating a method of training a generative confrontation network according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating a generation network according to an embodiment of the present application;
FIG. 4 is an exemplary block diagram illustrating a generation network according to an embodiment of the present application;
FIG. 5 is an exemplary block diagram illustrating a discrimination network according to an embodiment of the present application;
FIG. 6 is a schematic diagram illustrating a training apparatus of a generative confrontation network according to an embodiment of the present application; and
fig. 7 is a schematic diagram illustrating a training system of a generative confrontation network according to an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant technical concepts and are not limitative of the technical concepts. It should be further noted that, for convenience of description, only portions related to the technical idea of the present application are shown in the drawings. It should be understood that, unless otherwise specified, ordinal words such as "first", "second", etc., used herein are used only to distinguish one element from another, and do not denote importance or priority. For example, the first update period and the second update period merely indicate that they are different update periods.
In addition, the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
As described above, the present application aims to provide a technical solution that can deblur an image without knowing a blur kernel. Specifically, the present application applies GAN (generic adaptive Net, Generative countermeasure network) to the field of image deblurring to solve the "black box" problem described above.
GAN is a two-person zero-sum game derived from the game theory, which essentially consists of at least one generation network and one discrimination network. The generation network generates more and more vivid images through continuous learning; the discrimination network enhances the discrimination capability between the generated image and the true (Ground Truth) image by continuously learning. By generating the confrontation between the network and the discrimination network, finally, the image generated by the generated network is close to a clear truth map and the discrimination network is successfully deceived. Such GAN can be used to generate high quality images, and thus is suitable for application scenarios such as image super resolution, image noise reduction, and image deblurring. Using GAN to perform image deblurring can eliminate the need for high computational cost markov chain (MarkovChain) and its approximation algorithm, thereby reducing the training difficulty and time cost of the network.
Fig. 1 is a simplified schematic diagram illustrating a GAN 1000 for image deblurring according to an embodiment of the present application.
As shown in fig. 1, the GAN 1000 includes a generation network 1100 and a discrimination network 1200 cascaded with each other. The generation network 1100 may generate an output image based on a randomly generated array or input specific data (e.g., a fuzzy training graph, described below).
During training of GAN 1000, it is desirable for generating network 1100 to generate a predetermined pattern. These predetermined patterns are used as "standard answers" in the training process, i.e., the clear truth map described above. The role of the decision network 1200 is to discriminate whether the input to the decision network 1200 is a sharp true value map or to generate an output image that the network 1100 "forges".
In the training process, the output image generated by the generation network 1100 is input to the discrimination network 1200. The decision network 1200 is used to distinguish whether its input is a clear truth map. Training is complete if the output image generated by the generation network 1100 can be "false-true" compared to the clear truth map. Thereafter, image generation may be performed with the trained generation network 1100 of GAN 1000.
Fig. 2 is a flow chart illustrating a training method 2000 for GAN 1000 according to an embodiment of the present application.
In step S2010, a fuzzy training map and a clear truth map of the pair are prepared. These paired fuzzy training images and the clean truth map may constitute a training set. Each fuzzy training graph will serve as the original input to the generation network 1100, and the matching clean truth map will serve as the basis for the discriminant network 1200. In order for the trained GAN 1000 to have good generalization capability, the training set should be of sufficiently large scale and sufficiently rich diversity.
According to one embodiment of the application, clear sample images can be respectively taken for various scenes to serve as the clear truth value map. These sample images may include landscapes, indoor environments, pedestrians, animals, vehicles, text, and the like. The sample image should have the final desired resolution and resolution. In addition, in order to obtain fuzzy training images paired with these clear truth value images, artificial fuzzy processing can be performed on each sample pattern respectively. Various artificial fuzzy algorithms can be used to simulate the causes of blur that are common in real life. For example, the image may be blurred out of focus to simulate blur caused by lens focus errors, gaussian blurred to simulate blur caused by atmospheric turbulence, motion blurred to simulate blur caused by object movement, and so forth. In addition, the image may be subjected to hybrid blurring with a certain weight. For example, defocus blur and motion blur may be simultaneously performed on an image with a weight to simulate blur in an image captured if the photographer moves and the lens is not successfully focused. To ensure that the trained GAN 1000 has good generalization capability, the training set may include at least 500 sets of paired fuzzy training images and clean truth images.
In step S2020, the deblurring verification graph 1170 is generated based on the fuzzy training graph 1110 using the generation network 1100. The image generation operation of the generation network 1100 is described in detail below with reference to fig. 3.
Fig. 3 is a schematic diagram illustrating a generating network 1100 according to an embodiment of the present application.
The generation network 1100 takes the fuzzy training graph 1110 as an initial input value to perform an image generation process. The size of the fuzzy training graph 1110 may not be limited. For example, the fuzzy training graph 1110 may have an arbitrary resolution and aspect ratio. The fuzzy training graph 1110 may be an RGB image and have three color channels of red, green, and blue. The image of each color channel is represented by pixel values at respective pixel points. These pixel values may be in the value range of [0,255 ].
According to one embodiment of the present application, the fuzzy training graph 1110 may be pre-cropped to fit a particular aspect ratio. For example, the fuzzy training graph 1110 may be cropped to have a size of 32 pixels by 32 pixels to match the CIFAR-10 dataset. Alternatively, the fuzzy training graph 1110 may be cropped to have a size of 227 pixels by 227 pixels to match the ImageNet data set. Still alternatively, the fuzzy training graph 1110 may be cropped to have a size of 224 pixels by 224 pixels to match the VGG16 and ResNet data sets. The cropping of the image may be manual cropping, for example, by a large number of online personnel using Amazon Mechanical Turn (AMT) services to crop the image to fit a particular aspect ratio while preserving the subject. In addition, the cropping may also be automatically extracted by an ROI (Region of Interest) extraction layer. For example, the ROI extraction layer may automatically generate a bounding box that frames out the target object, and automatically resize and crop the image to fit a particular aspect ratio based on this bounding box. The network parameters of the ROI extraction layer can be trained and optimized in the training process.
As shown in fig. 3, the generation network 1100 may include a feature extraction layer 1120. The feature extraction layer 1120 performs a convolution operation on the input image to extract image features. Therefore, the feature extraction layer 1120 is also referred to as a convolutional layer hereinafter. One convolutional layer is shown as an example. However, as will be appreciated by those skilled in the art, to enhance the characterization capabilities of the features, multiple convolutional layers may be included in the generation network 1100. Each convolutional layer may include a plurality of convolutional kernels, which are composed of weights (Weight) and offsets (Bias). The number of convolution kernels is also referred to as the number of eigen-channels. Each convolution kernel is sensitive only to certain features of the input layer and these features can be extracted by the convolution operation. The fuzzy training graph 1110 may be convolved by a feature extraction layer 1120 of the generation network 1100 to extract a preliminary feature graph, according to the present application.
Generally, the size of the convolution kernel is smaller than the size of the input layer, and therefore, each convolution kernel perceives only a partial region of the input layer, which is called a perceptual domain (perceptual Field). Each convolution kernel is then slid across the entire input layer in a particular step size (Stride) until all of the information of the input layer is extracted. In the process, through weight sharing, the convolution kernel can share and apply the weight and the offset of the convolution kernel to feature extraction of the whole input layer so as to greatly reduce the calculation load. However, weight sharing is not applicable to any application scenario. For some images, the user's region of interest is concentrated in a certain region of the image (e.g., the center region), and the image characteristics of this region are significantly different from other regions. In this application scenario, feature extraction may be performed on a specific region of an image through a local connection layer, and the convolution kernel weight of the local connection layer may not be shared in feature extraction on other image regions.
The generation network 1100 may also include a BN (Batch Normalization) layer 1130. The essence of the neural network learning process is to learn the distribution of data, and once the distribution of training data is different from that of verification data, the generalization capability of the network is greatly reduced. In addition, once the distribution of each batch of training data is different, the network needs to learn to adapt to different distributions in each iteration, which greatly reduces the training speed of the network. It is therefore advantageous to properly normalize (also called normalize) the data in the network. The BN 1130 functions to normalize the output of the previous layer network while ensuring that the feature distribution learned by the previous layer network is hardly distorted.
The generating network 1100 may also include an activation layer 1140. As described above, the convolution kernel only linearly transforms the initial image. However, linear transformations are insufficient for semantic characterization capabilities of image features. In order to enhance the semantic characterization capability of image features, a nonlinear activation layer is often required to be added. Such a non-linear activation layer 1140 may perform a non-linear transformation on the primary feature map to obtain a secondary feature map with a stronger semantic characterization capability. Different activation functions may be configured for activation layer 1140 depending on actual needs. For example, a sigmod function may be employed to activate features. The sigmod function is a sensitive region of a neuron where the slope is large in the middle and a suppressed region of a neuron where the slopes are gentle on both sides. The output value thereof is limited to the range of [0,1 ]. Alternatively, the features may be activated using a tanh function. the tanh function is similar to the curve of the sigmod function, but the output value of the tanh function is limited to the range of [ -1,1], and the entire function output value is centered at 0. Still alternatively, the activation layer 1140 may include a ReLU (Rectified Linear Unit). Compared with the sigmod function and the tanh function, the ReLU function has no problem of gradient dispersion when the input value is a positive number. In addition, since the ReLU function has only a linear relationship, the gradient calculation is much faster and less computationally expensive than the sigmod function and the tanh function, regardless of the forward propagation or the backward propagation.
According to one embodiment of the application, the activatable layer 1140 may comprise a PReLU (Parametric RectisedLinear Unit). PReLU is a variant of ReLU. In the negative region, the PReLU still has a small slope, so that the problem of gradient diffusion in the negative region, which cannot be overcome by the ReLU, can be avoided.
The generating network 1100 may include a fully connected layer 1150. In the fully-connected layer 1150, each neuron is connected to all neurons in the upper layer. Fully connected layer 1150 may summarize and summarize the features extracted for the first few convolutional layers to obtain a feature map embodying global features.
The generation network 1100 may also include a reconstruction layer 1160 to reconstruct the deblurred verification graph 1170. The reconstruction layer 1160 may include specific network layers such as an deconvolution layer, an activation layer, and a full connectivity layer. The deconvolution layer may perform the inverse of the feature extraction layer 1120 to obtain a reconstructed feature map or image. Reconstruction layer 1160 may include an active layer that is similarly configured to active layer 1140 to provide a non-linear transformation for image reconstruction. The reconstruction layer 1160 may include a fully-connected layer that may be configured similarly to the fully-connected layer 1150 to restore global features of the image.
In summary, the generation network 1100 is actually configured as a data processing structure similar to an encoder-decoder. The feature extraction layer 1120 through the fully-connected layer 1150 embody the functionality of an encoder that encodes the fuzzy training graph 1110 to extract semantic features of an image. A network-specific level of reconstruction 1160, including deconvolution, activation, and full-link layers, embodies the functionality of a decoder that performs image reconstruction based on semantic features of the image extracted by the encoder to obtain a deblurred verification map 1170. For purposes of illustration and not limitation, fig. 4 presents a specific example structure of the generating network 1100 shown in fig. 1.
In step S2030, the decision network 1200 is used to compare the training error between the deblurred verification map 1170 and the clean truth map. The discrimination network 1200 is similar to a conventional binary model. The trained discrimination network 1200 should have the following properties: when the input data is a clear true value graph, the output value is 1; when the input data is the deblurring verification map 1170, the output value is 0. That is, discrimination network 1200 should have the ability to verify a counterfeit. For purposes of illustration and not limitation, FIG. 5 sets forth a specific example structure of the discrimination network 1200 shown in FIG. 1.
In step S2040, the training error is propagated back through the generating network 1100 and the discriminating network 1200 to iteratively update the parameters of the generating network 1100 and the parameters of the discriminating network 1200 until the training error satisfies the convergence condition.
Since the training targets of the generating network 1100 and the discriminating network 1200 are different, the parameters of the generating network 1100 and the parameters of the discriminating network 1200 are not updated at the same time in the training process. In practice, the parameters of the generating network 1100 and the parameters of the discriminating network 1200 are alternately updated. In the first update cycle, the parameters of the generating network 1100 are "frozen" and only the parameters of the discriminating network 1200 are updated. The first update period is also called a discriminant network training period. In the second update cycle, the parameters of the discrimination network 1200 are "frozen" and only the parameters of the generation network 1100 are updated. The second update period is also referred to as a generation network training period. The first update period and the second update period may alternate in a variety of ways to iteratively update the network parameters. For example, 3-5 second update cycles may be performed after repeating 3-5 first update cycles. Alternatively, the updates may be alternated in sequence from the first update period to the second update period to the first update period.
To ensure that generating network 1100 and discriminating network 1200 train against each other, generating network 1100 and discriminating network 1200 have diametrically opposed training targets. In the first updating period, maximizing a loss function with the training error as an independent variable to perform the updating; and in the second update period, minimizing the loss function to perform the update. Equation (1) below shows the antagonistic training of GAN 1000.
In formula (1), G denotes an image generation operation of the generation network 1100, and D denotes a discrimination operation of the discrimination network 1200. First term on right side of equation (1)The clear true value map in the set Pdata (x) representing the clear true value map passes through the discriminant networkAn entropy of 1200. Second term on right side of equation (1)
The deblurring verification graph 1170 generated by the generation network 1100 passes the entropy of the discrimination network 1200.
In the first update period (i.e., the discriminant network training period), it is expected that both terms on the right side in equation (1) take a maximum value of 1. That is, when the clear truth map passes through the discrimination network 1200, a probability value of 1 is given; and when the deblurred verification graph 1170 passes through the discrimination network 1200, a probability value of 0 is given. In the second update period (i.e., the generation network training period), the first term on the right in equation (1) is no longer functional. At this time, the second term on the right side in the formula (1) indicates that it is desirable to give a probability value of 1 when the deblurred verification graph 1170 generated by the generation network 1100 passes the discrimination network 1200.
The training process may be iteratively performed multiple times until a convergence condition is satisfied. According to an embodiment of the present application, the convergence condition may be that GAN 1000 reaches nash equilibrium. In this case, the generation network 1100 fully simulates the data distribution of the sharp true value graph, i.e., generates the same deblurring verification graph 1170 as the sharp true value graph; the discrimination network 1200 cannot discriminate the authenticity any more, and gives a probability value of 0.5.
In the training process, a certain strategy can be adopted to verify the generalization ability of the GAN. For example, the training set may be randomly divided into a first training set and a second training set. The first training set and the second training set may have the same number of pairs of training images. The training method 2000 described above is then performed using the training image pairs of the first training set to obtain a trained GAN. Finally, it is checked whether the trained GAN can be generalized to a training image pair of the second training set. For example, the blurred training images of the training image pairs of the second training set are input into GAN to check whether the difference between the reconstructed deblurred verification image and the sharp true value image is within a threshold range. If the difference between the deblurred verification graph and the clear true value graph is still larger than the threshold value, the trained GAN may have insufficient generalization capability due to overfitting and the like. At this point, the first training set and the second training set may be re-partitioned and the training method 2000 described above may be repeated.
According to the embodiment of the application, the countermeasure training of the GAN is utilized, and under the condition that a specific model of the fuzzy core is not known, the deblurring model can still be established at low cost and the model parameters are perfected.
It should be noted that GAN 1000 utilizes the training of the generation network 1100 and the discriminant network 1200 in the training process to obtain a strong image generation capability. However, after training is complete, the image generation function can be completed using only the GAN 1000 generation network 1100 without the need to re-adapt and train a new sample set. That is, when using GAN 1000 for image deblurring, an initial image with blur defects is input into the generation network 1100 of GAN 1000 trained as described above. Then, the generating network 1100 using the GAN 1000 generates a deblurred image based on the initial image. Since GAN 1000 is already pre-trained to completion for common image scenes, image deblurring processing can be performed quickly and efficiently using such GAN 1000. Based on this, the application also provides an image deblurring device which is provided with the GAN 1000 trained according to the method.
Fig. 6 is a schematic diagram illustrating a GAN training apparatus 6000 according to an embodiment of the present application. According to an embodiment of the present application, the GAN includes a generation network and a discrimination network cascaded with each other. The training device 6000 for the image deblurred GAN comprises: a training set preparation device 6100 for preparing a fuzzy training image and a clear true value image which are paired; a generator 6200, generating a deblurred verification map based on the fuzzy training map by using the generation network; a discriminator 6300 for comparing the training error between the deblurred verification map and the clear truth map by using the discrimination network; and a back propagator 6400 back propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
The application also provides a computer system, which can be a mobile terminal, a Personal Computer (PC), a tablet computer, a server and the like. Referring now to FIG. 7, there is shown a schematic block diagram of a computer system suitable for use in implementing the terminal device or server of the present application: as shown in fig. 7, the computer system includes one or more processors, communication sections, and the like, for example: one or more Central Processing Units (CPUs) 701, and/or one or more image processors (GPUs) 713, etc., which may perform various suitable actions and processes according to executable instructions stored in a Read Only Memory (ROM)702 or loaded from a storage section 708 into a Random Access Memory (RAM) 703. The communication portion 712 may include, but is not limited to, a network card, which may include, but is not limited to, an ib (infiniband) network card.
The processor may communicate with the read-only memory 702 and/or the random access memory 703 to execute the executable instructions, connect with the communication part 712 through the bus 704, and communicate with other target devices through the communication part 712, so as to complete the operations corresponding to any one of the methods provided by the embodiments of the present application, for example: preparing a matched fuzzy training image and a clear true value image; generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of the GAN; comparing training errors between the deblurred verification graph and the clear true value graph by using a discrimination network of a GAN, wherein the discrimination network and the generation network are cascaded with each other; and back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
In addition, in the RAM703, various programs and data necessary for the operation of the device can also be stored. The CPU 701, the ROM702, and the RAM703 are connected to each other via a bus 704. The ROM702 is an optional module in case of the RAM 703. The RAM703 stores or writes executable instructions into the ROM702 at runtime, and the executable instructions cause the processor 701 to perform operations corresponding to the above-described communication method. An input/output (I/O) interface 705 is also connected to bus 704. The communication unit 712 may be integrated, or may be provided with a plurality of sub-modules (e.g., a plurality of IB network cards) and connected to the bus link.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
It should be noted that the architecture shown in fig. 7 is only an optional implementation manner, and in a specific practical process, the number and types of the components in fig. 7 may be selected, deleted, added or replaced according to actual needs; in different functional component settings, separate settings or integrated settings may also be used, for example, the GPU and the CPU may be separately set or the GPU may be integrated on the CPU, the communication part may be separately set or integrated on the CPU or the GPU, and so on. These alternative embodiments are all within the scope of the present disclosure.
Further, according to an embodiment of the present application, the processes described above with reference to the flowcharts may be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium having stored thereon machine-readable instructions executable by a processor to perform instructions corresponding to the method steps provided herein, such as: preparing a matched fuzzy training image and a clear true value image; generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of the GAN; comparing training errors between the deblurred verification graph and the clear true value graph by using a discrimination network of a GAN, wherein the discrimination network and the generation network are cascaded with each other; and back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program, when executed by a Central Processing Unit (CPU)701, performs the above-described functions defined in the method of the present application.
The method and apparatus, device of the present application may be implemented in a number of ways. For example, the methods and apparatuses, devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present application may also be embodied as a program recorded in a recording medium, the program including machine-readable instructions for implementing a method according to the present application. Thus, the present application also covers a recording medium storing a program for executing the method according to the present application.
The description of the present application has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the application in the form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. The embodiment was chosen and described in order to best explain the principles of the application and the practical application, and to enable others of ordinary skill in the art to understand the application for various embodiments with various modifications as are suited to the particular use contemplated.
Claims (12)
1. A training method of a generative confrontation network for image deblurring, the generative confrontation network comprising a generative network and a discriminative network cascaded with each other, the training method comprising:
preparing a matched fuzzy training image and a clear true value image;
generating a deblurred verification graph based on the fuzzy training graph by utilizing the generation network;
comparing training errors between the deblurred verification map and the clear truth map by using the discrimination network; and
propagating the training error back through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
2. The method of claim 1, wherein back-propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network comprises:
in a first updating period, fixing the parameters of the generated network, and updating the parameters of the judgment network through the back propagation; and
and in a second updating period, fixing the parameters of the judging network, and updating the parameters of the generating network through the back propagation.
3. The method of claim 2, wherein:
in the first updating period, maximizing a loss function with the training error as an independent variable to perform the updating; and
minimizing the loss function for the second update period.
4. The method of claim 1, wherein preparing the paired fuzzy training map and clear truth map comprises:
respectively shooting clear sample images aiming at various scenes to serve as the clear true value images; and
and respectively carrying out artificial blurring processing on each sample pattern to obtain the blurred training image.
5. The method of claim 4, wherein the artificial blur process comprises at least one of: defocus blur, gaussian blur and motion blur.
6. The method of claim 1, wherein generating, with the generation network, a deblurred verification map based on the blurred training map comprises:
performing convolution operation on the fuzzy training graph through a feature extraction layer of the generation network to extract a primary feature graph;
performing nonlinear transformation on the primary feature map through an activation layer of the generation network to obtain a secondary feature map; and
and carrying out deconvolution operation on the secondary feature map through a reconstruction layer of the generation network to obtain the deblurred verification map.
7. The method of claim 1, wherein the convergence condition is that the generative countermeasure network reaches nash equilibrium.
8. A training apparatus of a generative confrontation network for image deblurring, the generative confrontation network comprising a generative network and a discriminative network cascaded with each other, the training apparatus comprising:
a training set preparation device for preparing a fuzzy training image and a clear true value image which are matched;
a generator for generating a deblurred verification graph based on the fuzzy training graph by using the generation network;
a discriminator for comparing the training error between the deblurred verification map and the clear truth map by using the discrimination network; and
a back propagator back propagating the training error through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
9. An image deblurring system, the system comprising:
a processor; and
a memory coupled to the processor and storing machine-readable instructions executable by the processor to:
preparing a matched fuzzy training image and a clear true value image;
generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of a generating type countermeasure network;
comparing training errors between the deblurred verification graph and the clear truth value graph by using a discriminant network of a generative confrontation network, wherein the discriminant network and the generative network are cascaded with each other; and
propagating the training error back through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
10. A non-transitory machine-readable storage medium storing machine-readable instructions executable by a processor to:
preparing a matched fuzzy training image and a clear true value image;
generating a deblurring verification graph based on the fuzzy training graph by utilizing a generating network of a generating type countermeasure network;
comparing training errors between the deblurred verification graph and the clear truth value graph by using a discriminant network of a generative confrontation network, wherein the discriminant network and the generative network are cascaded with each other; and
propagating the training error back through the generating network and the discriminating network to iteratively update parameters of the generating network and parameters of the discriminating network until the training error satisfies a convergence condition.
11. A method of deblurring an image, the method comprising:
inputting an initial image into a generating network of a generative confrontation network trained according to the method of claim 1; and
generating, with the generation network, a deblurred image based on the initial image.
12. An image deblurring apparatus, characterized in that it is equipped with a generative confrontation network trained in accordance with the method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810777095.6A CN110728626A (en) | 2018-07-16 | 2018-07-16 | Image deblurring method and apparatus and training thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810777095.6A CN110728626A (en) | 2018-07-16 | 2018-07-16 | Image deblurring method and apparatus and training thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110728626A true CN110728626A (en) | 2020-01-24 |
Family
ID=69217267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810777095.6A Pending CN110728626A (en) | 2018-07-16 | 2018-07-16 | Image deblurring method and apparatus and training thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110728626A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111428575A (en) * | 2020-03-02 | 2020-07-17 | 武汉大学 | Tracking method for fuzzy target based on twin network |
| CN111462019A (en) * | 2020-04-20 | 2020-07-28 | 武汉大学 | Image deblurring method and system based on deep neural network parameter estimation |
| US20200372618A1 (en) * | 2018-05-09 | 2020-11-26 | Tencent Technology (Shenzhen) Company Limited | Video deblurring method and apparatus, storage medium, and electronic apparatus |
| CN112488947A (en) * | 2020-12-04 | 2021-03-12 | 北京字跳网络技术有限公司 | Model training and image processing method, device, equipment and computer readable medium |
| CN112991212A (en) * | 2021-03-16 | 2021-06-18 | Oppo广东移动通信有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN113205464A (en) * | 2021-04-30 | 2021-08-03 | 作业帮教育科技（北京）有限公司 | Image deblurring model generation method, image deblurring method and electronic equipment |
| WO2022100490A1 (en) * | 2020-11-16 | 2022-05-19 | Huawei Technologies Co.,Ltd. | Methods and systems for deblurring blurry images |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079149A (en) * | 2006-09-08 | 2007-11-28 | 浙江师范大学 | Noise-possessing movement fuzzy image restoration method based on radial basis nerve network |
| CN102542535A (en) * | 2011-11-18 | 2012-07-04 | 中国科学院自动化研究所 | Method for deblurring iris image |
| CN103458772A (en) * | 2011-04-07 | 2013-12-18 | 香港中文大学 | Method and device for retinal image analysis |
| CN103577814A (en) * | 2013-11-25 | 2014-02-12 | 中国科学院自动化研究所 | Weighting comparison method for motion-blur iris recognition |
| CN103604838A (en) * | 2013-12-03 | 2014-02-26 | 杭州协正信息技术有限公司 | Three-dimensional electronic nose imitating structure and information processing mechanism of animal olfactory organ |
| CN104765916A (en) * | 2015-03-31 | 2015-07-08 | 西南交通大学 | Dynamics performance parameter optimizing method of high-speed train |
| CN104867111A (en) * | 2015-03-27 | 2015-08-26 | 北京理工大学 | Block-blur-kernel-set-based heterogeneous video blind deblurring method |
| CN105957024A (en) * | 2016-04-20 | 2016-09-21 | 西安电子科技大学 | Blind deblurring method based on image block prior and sparse norm |
| CN106780364A (en) * | 2016-11-24 | 2017-05-31 | 天津大学 | Facial image deblurring method based on discernable feature |
| CN107563385A (en) * | 2017-09-02 | 2018-01-09 | 西安电子科技大学 | License plate character recognition method based on depth convolution production confrontation network |
| CN107590774A (en) * | 2017-09-18 | 2018-01-16 | 北京邮电大学 | A kind of car plate clarification method and device based on generation confrontation network |
| CN107730458A (en) * | 2017-09-05 | 2018-02-23 | 北京飞搜科技有限公司 | A kind of fuzzy facial reconstruction method and system based on production confrontation network |
| CN107945204A (en) * | 2017-10-27 | 2018-04-20 | 西安电子科技大学 | A kind of Pixel-level portrait based on generation confrontation network scratches drawing method |
| JP2018077615A (en) * | 2016-11-08 | 2018-05-17 | 株式会社ＰＫＳＨＡ Ｔｅｃｈｎｏｌｏｇｙ | Advertising image generation device, advertising image generation method and program for advertising image generation device |
| CN108198179A (en) * | 2018-01-03 | 2018-06-22 | 华南理工大学 | A kind of CT medical image pulmonary nodule detection methods for generating confrontation network improvement |
| CN108229381A (en) * | 2017-12-29 | 2018-06-29 | 湖南视觉伟业智能科技有限公司 | Face image synthesis method, apparatus, storage medium and computer equipment |
| CN108226892A (en) * | 2018-03-27 | 2018-06-29 | 天津大学 | A kind of radar signal restoration methods under complicated noise based on deep learning |
-
2018
- 2018-07-16 CN CN201810777095.6A patent/CN110728626A/en active Pending
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079149A (en) * | 2006-09-08 | 2007-11-28 | 浙江师范大学 | Noise-possessing movement fuzzy image restoration method based on radial basis nerve network |
| CN103458772A (en) * | 2011-04-07 | 2013-12-18 | 香港中文大学 | Method and device for retinal image analysis |
| CN102542535A (en) * | 2011-11-18 | 2012-07-04 | 中国科学院自动化研究所 | Method for deblurring iris image |
| CN103577814A (en) * | 2013-11-25 | 2014-02-12 | 中国科学院自动化研究所 | Weighting comparison method for motion-blur iris recognition |
| CN103604838A (en) * | 2013-12-03 | 2014-02-26 | 杭州协正信息技术有限公司 | Three-dimensional electronic nose imitating structure and information processing mechanism of animal olfactory organ |
| CN104867111A (en) * | 2015-03-27 | 2015-08-26 | 北京理工大学 | Block-blur-kernel-set-based heterogeneous video blind deblurring method |
| CN104765916A (en) * | 2015-03-31 | 2015-07-08 | 西南交通大学 | Dynamics performance parameter optimizing method of high-speed train |
| CN105957024A (en) * | 2016-04-20 | 2016-09-21 | 西安电子科技大学 | Blind deblurring method based on image block prior and sparse norm |
| JP2018077615A (en) * | 2016-11-08 | 2018-05-17 | 株式会社ＰＫＳＨＡ Ｔｅｃｈｎｏｌｏｇｙ | Advertising image generation device, advertising image generation method and program for advertising image generation device |
| CN106780364A (en) * | 2016-11-24 | 2017-05-31 | 天津大学 | Facial image deblurring method based on discernable feature |
| CN107563385A (en) * | 2017-09-02 | 2018-01-09 | 西安电子科技大学 | License plate character recognition method based on depth convolution production confrontation network |
| CN107730458A (en) * | 2017-09-05 | 2018-02-23 | 北京飞搜科技有限公司 | A kind of fuzzy facial reconstruction method and system based on production confrontation network |
| CN107590774A (en) * | 2017-09-18 | 2018-01-16 | 北京邮电大学 | A kind of car plate clarification method and device based on generation confrontation network |
| CN107945204A (en) * | 2017-10-27 | 2018-04-20 | 西安电子科技大学 | A kind of Pixel-level portrait based on generation confrontation network scratches drawing method |
| CN108229381A (en) * | 2017-12-29 | 2018-06-29 | 湖南视觉伟业智能科技有限公司 | Face image synthesis method, apparatus, storage medium and computer equipment |
| CN108198179A (en) * | 2018-01-03 | 2018-06-22 | 华南理工大学 | A kind of CT medical image pulmonary nodule detection methods for generating confrontation network improvement |
| CN108226892A (en) * | 2018-03-27 | 2018-06-29 | 天津大学 | A kind of radar signal restoration methods under complicated noise based on deep learning |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200372618A1 (en) * | 2018-05-09 | 2020-11-26 | Tencent Technology (Shenzhen) Company Limited | Video deblurring method and apparatus, storage medium, and electronic apparatus |
| US11688043B2 (en) * | 2018-05-09 | 2023-06-27 | Tencent Technology (Shenzhen) Company Limited | Video deblurring method and apparatus, storage medium, and electronic apparatus |
| CN111428575A (en) * | 2020-03-02 | 2020-07-17 | 武汉大学 | Tracking method for fuzzy target based on twin network |
| CN111428575B (en) * | 2020-03-02 | 2022-03-04 | 武汉大学 | Tracking method for fuzzy target based on twin network |
| CN111462019A (en) * | 2020-04-20 | 2020-07-28 | 武汉大学 | Image deblurring method and system based on deep neural network parameter estimation |
| WO2022100490A1 (en) * | 2020-11-16 | 2022-05-19 | Huawei Technologies Co.,Ltd. | Methods and systems for deblurring blurry images |
| US11741579B2 (en) | 2020-11-16 | 2023-08-29 | Huawei Technologies Co., Ltd. | Methods and systems for deblurring blurry images |
| CN112488947A (en) * | 2020-12-04 | 2021-03-12 | 北京字跳网络技术有限公司 | Model training and image processing method, device, equipment and computer readable medium |
| CN112991212A (en) * | 2021-03-16 | 2021-06-18 | Oppo广东移动通信有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN113205464A (en) * | 2021-04-30 | 2021-08-03 | 作业帮教育科技（北京）有限公司 | Image deblurring model generation method, image deblurring method and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110728626A (en) | Image deblurring method and apparatus and training thereof | |
| US11521011B2 (en) | Method and apparatus for training neural network model for enhancing image detail | |
| CN109410127B (en) | Image denoising method based on deep learning and multi-scale image enhancement | |
| CN111127308B (en) | Mirror image feature rearrangement restoration method for single sample face recognition under partial shielding | |
| CN111488865B (en) | Image optimization method and device, computer storage medium and electronic equipment | |
| Diamond et al. | Dirty pixels: Towards end-to-end image processing and perception | |
| Kolesnikov et al. | PixelCNN models with auxiliary variables for natural image modeling | |
| CN111161178A (en) | Single low-light image enhancement method based on generation type countermeasure network | |
| US20220301114A1 (en) | Noise Reconstruction For Image Denoising | |
| KR102095443B1 (en) | Method and Apparatus for Enhancing Image using Structural Tensor Based on Deep Learning | |
| CN112541877B (en) | Defuzzification method, system, equipment and medium for generating countermeasure network based on condition | |
| Kim et al. | BigColor: Colorization using a generative color prior for natural images | |
| Kim et al. | Multiple level feature-based universal blind image quality assessment model | |
| Wang et al. | Training very deep CNNs for general non-blind deconvolution | |
| US11741579B2 (en) | Methods and systems for deblurring blurry images | |
| Guan et al. | Srdgan: learning the noise prior for super resolution with dual generative adversarial networks | |
| Li et al. | Hdrnet: Single-image-based hdr reconstruction using channel attention cnn | |
| CN111415304A (en) | Underwater vision enhancement method and device based on cascade deep network | |
| Jiang et al. | Fast and high quality image denoising via malleable convolution | |
| Zhang et al. | Deep motion blur removal using noisy/blurry image pairs | |
| CN110728627A (en) | Image noise reduction method, device, system and storage medium | |
| Nah et al. | Clean images are hard to reblur: Exploiting the ill-posed inverse task for dynamic scene deblurring | |
| Rasheed et al. | LSR: Lightening super-resolution deep network for low-light image enhancement | |
| CN116664446A (en) | Lightweight dim light image enhancement method based on residual error dense block | |
| CN111583124A (en) | Method, device, system and storage medium for deblurring images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |