CN112200732B

CN112200732B - Video deblurring method with clear feature fusion

Info

Publication number: CN112200732B
Application number: CN202010368483.6A
Authority: CN
Inventors: 项欣光; 魏颢; 潘金山
Original assignee: Nanjing University of Science and Technology
Current assignee: Nanjing University of Science and Technology
Priority date: 2020-04-30
Filing date: 2020-04-30
Publication date: 2022-10-21
Anticipated expiration: 2040-04-30
Also published as: CN112200732A

Abstract

The invention discloses a video deblurring method with fused clear features. Firstly, selecting a plurality of continuous fuzzy video frames, estimating optical flow between the continuous frames by using an optical flow estimation network, twisting images of adjacent frames by using the estimated optical flow, then taking a twisted result and an original fuzzy frame sequence as input of a deblurring network, then selecting a plurality of clear frames, passing through a clear feature extraction module, obtaining clear features, fusing the clear features into the deblurring network, and finally outputting relatively clear video frames by the deblurring network. The method is robust to scenes with clear frames, clear frames of any scene can be used for feature fusion, and reconstruction of video frames is facilitated, so that the method is convenient and effective.

Description

Video deblurring method with clear feature fusion

Technical Field

The invention relates to an end-to-end video deblurring network, in particular to a video deblurring algorithm based on clear feature fusion.

Background

In recent years, with the development of portable imaging devices, such as mobile phones and cameras, image/video deblurring techniques have received increased attention. The reasons for the formation of blur are many, including the motion of the object during imaging, the shake of the camera and the depth of field, which provide much resistance to the study of computer vision (object detection, object recognition). Therefore, it is very necessary to research the deblurring algorithm.

At present, the deblurring algorithm is mainly divided into two types, one type is a method based on a physical model, and the other type is a method based on learning. Early, according to a degradation model (B = K × S + N), where B denotes a blurred image, K denotes a blur kernel, S denotes a sharp image, and N denotes additive noise. In the case of the known blurred image B, it is very difficult to solve the blurred kernel K and the sharp image S, and a multi-solution situation may occur, so that the problem is ill-conditioned. In order to constrain the solution space, people design some natural image priors, including L0 gradient priors, dark channel priors, etc., to constrain the natural image, and solve under a maximum posterior framework. However, the method is difficult to optimize based on a physical model, time-consuming, without universality, and requires artificially designed prior information for constraint.

Another learning-based approach is the one that is currently more popular. People learn the internal distribution of natural images from data sets through a reasonable neural network, and finally achieve the aim of deblurring.

The invention provides an end-to-end video deblurring network, which is different from image deblurring, and the video deblurring needs to consider the relation between adjacent frames.

Disclosure of Invention

The invention aims to provide an end-to-end video deblurring algorithm based on clear feature fusion, which is used for inputting a plurality of continuous blurred video frames and recovering clear intermediate frames.

The technical solution for realizing the purpose of the invention is as follows: a video deblurring algorithm based on sharp feature fusion comprises the following steps:

step A: designing an optical flow estimation module, inputting three continuously blurred frames into the optical flow estimation module, and outputting a result obtained by performing image torsion (image warp) on two adjacent frames;

and B, step B: designing a clear feature fusion module, inputting any clear three frames into the module, and outputting clear feature graphs with different scales;

step C: designing a deblurring module, stacking the result of the step A and the original input and then sending the result into the module, and fusing the result of the step B into the whole deblurring process to finally obtain a clear intermediate frame.

Compared with the prior art, the invention has the following remarkable advantages: the invention considers the motion compensation and the image deblurring at the same time. The method is robust to scenes with clear frames, clear frames of any scene can be used for feature fusion, reconstruction of video frames is facilitated, and the method is convenient and effective.

Drawings

FIG. 1 is an overall flow diagram of the present invention.

Fig. 2 is a diagram of a network architecture designed by the present invention.

FIG. 3 is a comparison graph of the deblurring effect of the present invention. The (a) is the blurred video frame, and the (b) is the reconstruction result.

Detailed Description

The invention is further described in the following with reference to the drawings.

As shown in fig. 2, the whole network includes 3 modules, which are an optical flow estimation module, a sharp feature fusion module and a deblurring module.

According to the first panel of fig. 1, the data preparation steps are as follows:

step 1, downloading a GOPRO _ Su data set as a training sample, wherein the GOPRO _ Su data set comprises 71 video sets, and each video is provided with a plurality of paired fuzzy-clear video frames;

and 2, dividing 71 videos into two parts, wherein 61 videos serve as training samples, and 10 videos serve as testing samples.

According to the second block of fig. 1, the steps of the optical flow estimation module are as follows:

step 1, presetting training parameters, including a learning rate 1e-6 of an optical flow estimation module, a learning rate 1e-4 of a deblurring module, and maximum iteration epoch =500;

step 2, three continuous blurred images { I ] are selected ₁ ,I ₂ ,I ₃ Estimate the optical flow between two adjacent frames by FlowNeTS { f } _1→2 ,f _3→2 Obtaining two 2-channel optical flow graphs respectively, performing image torsion (image warp) operation on the optical flow and an adjacent frame, and twisting the adjacent frame to an intermediate frame to finally obtain two 3-channel RGB images { warp } _1→2 ,warp _3→2 }. The process can be described as follows:

f _1→2 ＝F([I ₁ ,I ₂ ]),f _3→2 ＝F([I ₃ ,I ₂ ]),

warp _1→2 ＝W([f _1→2 ,I ₁ ]),warp _3→2 ＝W([f _3→2 ,I ₃ ])；

where F () denotes an optical flow estimation network, [ ] denotes the operation of image concat, and W () denotes warp operation.

According to the third panel of fig. 1, the steps of the sharp feature fusion module are as follows:

selecting three clear images (S) ₁ ,S ₂ ,S ₃ The three clear images are passed through a feature extraction module, which is composed of two layers of convolution respectively, to finally obtain 32256 pieces of 256 × 256 feature maps { feature _ coarse } and 64 pieces of 64 × 64 feature maps { feature _ fine }. The process can be described as follows:

Feature_coarse,feature_fine＝E([S ₁ ,S ₂ ,S ₃ ])

where E () represents the feature extraction network and [ ] represents the operation of the image concat.

According to the fourth panel of fig. 1, the step of deblurring the module is as follows:

step 1, obtaining a warp result { warp ] in an optical flow estimation module _1→2 ,warp _3→2 And original blurred video frame { I } ₁ ,I ₂ ,I ₃ Sending the fuzzy data into a deblurring module;

and 2, the deblurring module is a coding and decoding structure, wherein each coding block is formed by adding three residual blocks to 1-layer convolution, and each decoding block is formed by adding one deconvolution to the three residual blocks. In the process of deblurring, embedding the clear feature map { feature _ coarse, feature _ fine } obtained by the clear feature fusion module into an encoder part and a decoder part in the deblurring module to finally obtain a feature I ₂ Corresponding sharp image R ₂ . Clear image R generated by calculation ₂ With the original I ₂ Reference sharp image G ₂ To update the entire network. The process can be described as follows:

input＝[warp _1→2 ,I ₁ ,I ₂ ,I ₃ ,warp _3→2 ],

En_out＝En(input,feature_coarse,feature_fine),

De_out＝De(En_out,feature_coarse,feature_fine)；

where En () represents the encoder portion and De () represents the decoder portion.

And 3, training according to the steps until the maximum iteration number is reached, and finishing training when the generated model is reached.

According to the fifth panel of fig. 1, the test procedure is as follows:

step 1, inputting 10 video test samples obtained by a data preparation part into a trained model;

and 2, selecting the clear frame, namely selecting any clear frame different from the fuzzy scene, and inputting the clear frame into the trained model to finally obtain a clear image. Fig. 3 is a diagram of the visual effect of the present invention.

Claims

1. A video deblurring method based on clear feature fusion specifically comprises the following steps

Step A: designing an optical flow estimation module, inputting three continuous fuzzy frames into the optical flow estimation module, and outputting a result obtained by image torsion image warp of two adjacent frames;

and B: designing a clear feature fusion module, inputting any clear three frames into the clear feature fusion module, and outputting clear feature graphs of different scales;

step C: designing a deblurring module, stacking the result of the step A and three continuous blurred frames, sending the result into the deblurring module, and fusing the result of the step B into the whole deblurring process to finally obtain a clear intermediate frame; the step C specifically comprises the following steps:

step C01. Obtaining a result { warp after warp obtained in step A02 _1→2 ,warp _3→2 And original blurred video frame { I } ₁ ,I ₂ ,I ₃ Sending the fuzzy data into a deblurring module; in the process of deblurring, embedding the clear feature map { feature _ coarse, feature _ fine } obtained in the step B01 into a deblurring module to finally obtain a feature map I ₂ Corresponding sharp image R ₂ (ii) a Clear image R generated by calculation ₂ With original I ₂ Reference sharp image G ₂ To update the entire network;

c02, training according to the step C01 until the maximum iteration number is reached, and finishing training when the generated model is reached;

and C03, after the model training is finished, inputting three continuous fuzzy frames, and selecting any clear frame to extract clear features to finally obtain a clear image.

2. The method of claim 1, wherein step a specifically comprises the steps of:

a01, presetting training parameters, including a learning rate 1e-6 of an optical flow estimation module, a learning rate 1e-4 of a deblurring module, and maximum iteration epoch =500;

step A02. Selecting three continuous blurred images { I ₁ ,I ₂ ,I ₃ Estimating the optical flow between two adjacent frames by FlowNet S _1→2 ,flow _3→2 Obtaining two 2-channel optical flow graphs respectively, performing image torsion (image warp) operation on the optical flow and an adjacent frame, and twisting the adjacent frame to an intermediate frame to finally obtain two 3-channel RGB images { warp _1→2 ,warp _3→2 }。

3. The method of claim 1, wherein step B specifically comprises the steps of:

step B01, selecting three clear images { S ₁ ,S ₂ ,S ₃ And the three clear images are different from the content of the blurred image in the step A02, and the three clear images are passed through a feature extraction module to obtain 32 256 × 256 feature maps { feature _ coarse } and 64 × 64 feature maps { feature _ fine } respectively.