CN109410146A - A kind of image deblurring algorithm based on Bi-Skip-Net - Google Patents

Abstract

The invention relates to the field of digital image processing, in particular to an image deblurring method based on Bi-Skip-Net, which is a Bi-Skip-Net network to achieve blurred image restoration, and aims to solve the problem of existing deep learning deblurring algorithms. The time complexity is high, the texture restoration is inaccurate, and the restored image has problems such as grid effect. A Bi-Skip-Net network disclosed in the present invention is used as a generative network of GAN (Generative Adversarial Network), aiming to solve the shortcomings of the existing deep learning deblurring algorithms. The complexity is improved by 0.1s, and the average performance of image restoration is improved by 1dB.

Description

An Image Deblurring Algorithm Based on Bi-Skip-Net

technical field

The invention relates to the field of digital image processing, in particular to an image deblurring method based on Bi-Skip-Net, which realizes blurred image restoration through Bi-Skip-Net network.

technical background

Deblurring techniques are a subject of extensive research in the field of image and video processing. The blur caused by camera shake seriously affects the imaging quality and visual perception of the image in a certain sense. As an important branch in the field of image preprocessing, the improvement of deblurring technology directly affects the performance of other computer vision algorithms, such as foreground segmentation, object detection, behavior analysis, etc.; at the same time, it also affects the image coding performance. Therefore, it is imperative to study a high-performance deblurring algorithm.

Documents 1-3 introduce deblurring techniques for image and video processing, and deep learning deblurring algorithms; Document 1: Kupyn O, Budzan V, Mykhailych M, et al. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks[J].arXiv preprint arXiv:1711.07064, 2017. Document 2: Nah S, Kim T H, Lee K M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//CVPR.2017, 1(2):3. Literature 3: Sun J, Cao W, Xu Z, et al. Learning a convolutional neural network for non-uniform motion blurremoval[C]//Proceedings of the IEEE Conference on Computer Vision and PatternRecognition. 2015:769-777.

Generally speaking, image deblurring algorithms can be divided into traditional algorithms based on probability models and deblurring algorithms based on deep learning. The traditional algorithm uses a convolution model to explain the cause of blur, and the process of camera shake can be mapped to the blur kernel trajectory PSF (Point Spread Function). Restoring a clear image when the blur kernel is unknown is an ill-posed problem, so it is usually necessary to estimate the blur kernel first, and then use the estimated blur kernel to perform a deconvolution operation to obtain the restored image. The deblurring algorithm based on deep learning uses the deep network structure to obtain the latent information of the image, and then realizes the restoration of the blurred image. The deep learning deblurring algorithm can realize the two operations of blur kernel estimation and non-blind deconvolution for image restoration, and can also use the generative confrontation mechanism to restore the image. This patent aims to address the shortcomings of deblurring algorithms:

1) High time complexity,

2) Inaccurate texture recovery,

3) There is a checkerboard effect in the restored image.

SUMMARY OF THE INVENTION

The present invention proposes a Bi-Skip-Net network as a generative network of GAN (Generative Adversarial Network), aiming to solve the shortcomings of existing deep learning deblurring algorithms. By comparing with the existing optimal algorithms, the present invention improves the time complexity by 0.1s, and improves the image restoration performance by 1dB on average.

The technical solution provided by the present invention is as follows: (Note: the technical solution needs to be explained in natural language, and cannot be described in the way of "as shown in the figure". The method and technical solution are preferably written in the way of: step 1: step 2...)

The invention adopts the generation confrontation network mechanism to realize the restoration of the blurred image, and designs a Bi-Skip-Net network as the generator therein. Specific steps are as follows:

1): Input a blurred image, and obtain shallow features through a convolutional layer with a convolution kernel size of 7x7 and a stride of 1;

2): Pass the shallow feature through 3 residual blocks to obtain the depth feature at the current scale;

3): The mode of downsampling and adding residuals to deep features is used to obtain shallow features at the next scale;

4): Repeat steps 2 and 3 to obtain shallow features and depth features at different scales according to the specified number of downsampling times n, and do not obtain depth features at the minimum scale;

5): Take the shallow feature of the smallest scale as the basic feature;

6): Pass the shallow features of the previous scale through a convolutional layer with a convolution kernel size of 1x1 and a stride of 1 to obtain shallow dimension reduction features; pass the corresponding depth features through a convolution kernel with a size of 3x3 and a stride of 1. Obtain deep dimensionality reduction features for the convolutional layer of 2, and perform upsampling in series with the basic features; concatenate the upsampled features with shallow dimensionality reduction features to obtain basic features at the current scale;

7): Repeat step 6 until the sampling operation ends;

8): Pass the obtained basic features through a convolutional layer with a convolution kernel size of 7x7 and a stride of 1 to obtain residual features;

9): Add the residual feature to the input image to obtain the restored image;

…

The Bi-Skip-Net plus residual mode is used as the generator.

In step 4), the number of downsampling times is 5 according to the regulations.

Among them, the fuzzy image is obtained by the generator to obtain the restored image, and the task of discrimination is to distinguish the restored image from the clear image as much as possible; and the task of the generator is to deceive the discriminator as much as possible to reduce the ability to distinguish between the two images.

The Bi-Skip-Net network is composed of three parts: contract path (D), Skip path (S) and expand path (U). The Contract layer performs downsampling to achieve feature compression, the Skip layer is used to connect deep features and shallow features, and the expand layer performs upsampling. Where D*, S*, U* are the features at the corresponding downsampling scale.

In the feature operation at the sampling scale, in the contract path, the current feature obtains deep features through 3 residual blocks (3xResBlock), and uses the residual mode of pooling and convolution to obtain the next feature. Scale features; in the Skip path, the shallow features are compressed by 1x1 convolution, and the deep features are compressed by 3x3 convolution; in the expand path, the feature connection is realized by concat, and the feature is realized by 3x3 deconvolution upsampling.

The present invention has the following technical effects: because the present invention adopts the Bi-Skip-Net network as the generation network of GAN (Generative Adversarial Network), compared with the prior art, the following technical effects are obtained:

1. The time complexity is low; compared with the traditional method, the traditional motion blurring method adopts two steps of blur kernel estimation and non-blind deconvolution, and these two steps require multiple iterations to achieve a better restoration effect. Because of this, it also takes a long time to process a single motion blurred image; and the model designed in the present invention can avoid the time loss caused by multiple iterations of optimization.

2. Accurate texture recovery; compared with the traditional method, in the traditional method, the inaccurate estimation of the fuzzy kernel will cause the erroneous recovery of image information during the restoration process, while the non-blind de-scrolling operation often causes ringing effects in the texture part; the design of the present invention The dual-span connection network extracts deep features and shallow features at each layer scale. Through feature connection, the network can recover more detailed information to a certain extent.

3. There is no grid effect in the restored image. Compared with the existing deep learning methods, most of the existing deep learning methods use a deconvolution layer in the upsampling process, and because each deconvolution has a certain sawtooth effect, This makes the final restored image also have some jaggedness, that is, the checkered effect mentioned in the present invention.

In order to better understand the concept and principle of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments. However, the description of the specific embodiments does not limit the protection scope of the present invention in any way.

Description of drawings

Fig. 1 is the generative adversarial network mechanism of the present invention;

Fig. 2 is the Bi-Skip-Net network structure diagram of the present invention;

3 is a feature operation under a sampling scale of the present invention;

Figure 4 Generator design: Bi-Skip-Net+residual;

Figures 5a-d are subjective comparisons between the present invention and other algorithms; wherein,

Figure 5a: Blurred image;

Figure 5b: The recovery effect of Nah et al.;

Figure 5c: The recovery effect of Kupyn et al.;

Figure 5d: Restoration effect of Bi-Skip-Net.

Detailed ways

FIG. 1 shows the generative adversarial network mechanism adopted in the present invention. Among them, the fuzzy image is obtained by the generator to obtain the restored image, and the task of discrimination is to distinguish the restored image from the clear image as much as possible; and the task of the generator is to deceive the discriminator as much as possible to reduce the ability to distinguish between the two images.

The concrete steps of the embodiment of the present invention are as follows:

(1) Design generator and discriminator. The principle is shown in Figure 4. The fuzzy image of the building is passed through the Bi-Skip-Net generator to obtain a clear building image; any other fuzzy image can be generated by this model. Clear picture.

(2) Use the following loss function to train the network,

in For the adversarial loss function, is the conditional loss function, and λ is the weight of the conditional loss function.

by maximizing to optimize the discriminator D;

Optimize the generator G by minimizing Equation 3;

in The design is as follows:

Among them, L and S respectively represent the output and true value of the model at different levels, α is 1 or 2, and the entire conditional loss function is regulated by the number of channels c, the width w and the height h.

(3) Use the trained network as the final restoration model.

As shown in FIG. 1 , the method of the embodiment of the present invention adopts a generative adversarial network mechanism to achieve blurred image restoration. Figure 2 is a diagram of the Bi-Skip-Net network structure. Taking the network structure shown in Figure 2, a Bi-Skip-Net network is designed as the generator.

The discriminator parameters in this Bi-Skip-Net network structure diagram are shown in Table 1. Discriminator parameter table.

Table 1. Discriminator parameter table

# Floor parameter dimension step size 1 conv 32x3x5x5 2 2 conv 64x32x5x5 1 3 conv 64x64x5x5 2 4 conv 128x64x5x5 1 5 conv 128x128x5x5 4 6 conv 256x128x5x5 1 7 conv 256x256x5x5 4 8 conv 512x256x5x5 1 9 conv 512x512x4x4 4 10 fc 512x1x1x1 -

As shown in FIG. 2, the Bi-Skip-Net network designed in the embodiment of the present invention consists of three parts: contract path (D), including D0, D1, D2 and D3; Skip path (S), including S0, S1, S2 and S3; and expand path (U), including U0, U1, U2, and U3. The Contract layer performs downsampling to achieve feature compression, the Skip layer is used to connect deep features and shallow features, and the expand layer performs upsampling. Among them, D*(D0, D1, D2 and D3), S*(S0, S1, S2 and S3), U*(U0, U1, U2 and U3) are the features at the corresponding downsampling scale.

Figure 3 is a feature operation at a sampling scale. As shown in Figure 3, in the contract path, that is, the compression path, the current feature obtains deep features through three residual blocks (3xResBlock), and uses pooling (pooling) and volume The residual mode of product and addition is used to obtain the features of the next scale; in the Skip path, that is, the cross-connection path, the shallow features are compressed by 1x1 convolution, and the deep features are compressed by 3x3 convolution; in the expand path, Feature connection is realized through concat, namely concatenate, and feature upsampling is realized through 3x3 deconvolution.

Figure 4 Generator design: Bi-Skip-Net+residual, as shown in Figure 4, and finally the Bi-Skip-Net plus residual mode is used as the generator.

The comparison results between the implementation of the present invention and other algorithms are shown in Table 2. The test comparison between the present invention and other algorithms on the GoPro data set.

Table 2. Test comparison between the present invention and other algorithms on the GoPro dataset

Figures 5a-d are subjective comparisons of the present invention with other algorithms. Figure 5a is a blurred image, Figure 5b is the restoration effect of Nah et al., Figure 5c is the restoration effect of Kupyn et al., and Figure 5d is the restoration effect of Bi-Skip-Net of the present invention. The text "HARDWARE" in the lower left corner of the picture cannot be recognized or vaguely recognized in the other three pictures, but the present invention can restore and recognize it clearly. It can be seen from the subjective comparison of people that the present invention has obvious repairing effect on blurred images.

It should be noted that the purpose of publishing the embodiments is to help further understanding of the present invention, but those skilled in the art can understand that various replacements and modifications are possible without departing from the spirit and scope of the present invention and the appended claims of. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection of the present invention shall be subject to the scope defined by the claims.

Claims (5)

1. An image deblurring method based on Bi-Skip-Net, comprising the following steps: 1) Input a blurred image, and obtain shallow features through a convolutional layer with a convolution kernel size of 7x7 and a stride of 1; 2) Pass the shallow feature through three residual blocks to obtain the depth feature at the current scale; 3) The mode of downsampling and adding residuals to the deep features obtains the shallow features at the next scale; 4) According to the specified number of downsampling times n, repeat steps 2 and 3 to obtain shallow features and depth features at different scales, and do not obtain depth features at the minimum scale; 5) Take the shallow feature of the smallest scale as the basic feature; 6) Pass the shallow features of the previous scale through a convolutional layer with a convolution kernel size of 1x1 and a stride of 1 to obtain shallow dimension reduction features; pass the corresponding depth features through a convolution kernel with a size of 3x3 and a stride of 1 The convolutional layer of 2 obtains deep dimensionality reduction features, and upsampling them in series with basic features; concatenates the upsampled features and shallow dimensionality reduction features to obtain basic features at the current scale; 7) Repeat step 6 until the sampling operation ends; 8) Pass the obtained basic features through a convolutional layer with a convolution kernel size of 7x7 and a stride of 1 to obtain residual features; 9) adding the residual feature to the input image to obtain the restored image; 10) The model of Bi-Skip-Net plus residual is used as the generator. 2. image deblurring method according to claim 1, is characterized in that: Step 4) The number of downsampling times is 5 as specified. 3. image deblurring method according to claim 1, is characterized in that: The Bi-Skip-Net network is composed of three parts: contract path (D), Skip path (S) and expand path (U); the Contract layer performs downsampling to achieve feature compression, and the Skip layer is used to connect deep features and shallow features. Layer features, the expand layer is up-sampling; where D*, S*, U* are the features under the corresponding downsampling scale. 4. image deblurring method according to claim 3, is characterized in that: For the feature operation at the sampling scale, in the contract path, the current feature obtains deep features through 3 residual blocks (3xResBlock), and uses the residual mode of pooling and convolution to obtain the next feature. Scale features; in the Skip path, the shallow features are compressed by 1x1 convolution, and the deep features are compressed by 3x3 convolution; in the expand path, the feature connection is realized by concat, and the feature is realized by 3x3 deconvolution upsampling. 5. image deblurring method according to claim 1, is characterized in that: The generator described in step 10) is designed as follows, ① The following loss function is used to train the network, in For the adversarial loss function, is the conditional loss function, and λ is the weight of the conditional loss function; by maximizing to optimize the discriminator D; Optimize the generator G by minimizing Equation 3; in The design is as follows: Among them, L and S respectively represent the output and true value of the model at different levels, α is 1 or 2, and the entire conditional loss function is regulated by the number of channels c, the width w and the height h; ② Use the trained network as the final restoration model.