CN108257105A - A kind of light stream estimation for video image and denoising combination learning depth network model - Google Patents

Abstract

The invention discloses a joint learning deep network model for optical flow estimation and denoising of video images, which belongs to the field of image processing. The model includes a preprocessing module, an optical flow estimation module, and a denoising module. Each module adopts the Encoder-Decoder network structure. Using the sample data set, the preprocessing module is first trained separately, and then the relevant parameters of the preprocessing module are fixed, and the preprocessing module is trained at the same time. Module and optical flow estimation module, and finally fix the relevant parameters of the preprocessing module and optical flow estimation module. The overall training includes a deep network model of three modules. The trained deep network model can directly perform optical flow estimation and de-noising on noisy video images. noise processing. The joint learning deep network model proposed by the present invention has fast optical flow estimation and denoising speed and high precision, and is convenient for quickly processing a large number of video images in practice.

Description

A joint learning deep network model for optical flow estimation and denoising of video images

technical field

The invention relates to the field of image processing, and specifically refers to a joint learning deep network model for optical flow estimation and denoising of video images.

Background technique

Video images are faced with noise interference during acquisition, compression, storage, transmission and other links. Noise will significantly reduce the visual quality of video images and cause difficulties for subsequent intelligent analysis such as target recognition and tracking. Therefore, it is necessary to remove the noise in the video image under the premise of retaining the video information, improve the signal-to-noise ratio and improve the visual effect.

Because video images have time-domain correlation, optical flow estimation and video denoising can be combined to obtain better denoising effects. However, the existing joint optical flow estimation and video denoising algorithms require a large number of iterative operations and cost A large amount of computing resources and time are inconvenient for practical application, and optical flow estimation is easily disturbed by video noise, which affects the denoising effect. Therefore, to propose a fast and effective joint optical flow estimation and video denoising algorithm is an urgent problem in the field of video image processing.

Contents of the invention

In order to overcome the above disadvantages, the present invention aims to provide a joint learning deep network model for optical flow estimation and denoising of video images, using the deep network model to jointly learn optical flow estimation and video denoising from a large number of training samples to achieve It solves the problems of low accuracy of optical flow estimation, poor denoising effect and long time consumption in the prior art.

In order to solve the problems of the technologies described above, the technical solution proposed by the present invention is:

A joint learning deep network model for optical flow estimation and denoising of video images, characterized in that: the joint deep learning network model includes three modules: a preprocessing module, an optical flow estimation module and a denoising module, first using sample data Set the deep network model for training; then use the preprocessing module to perform preliminary denoising processing on the input noise images im_n1 and im_n2, and obtain the preprocessed image pair im_p1 and im_p2; use the optical flow estimation module to image pair im_p1 and im_p2 Perform motion estimation to obtain the optical flow estimation result flow; transform the noise image im_n2 according to the optical flow estimation result flow to obtain the image im_n2', and then use the image im_n2' and noise image im_n1 as the input image of the denoising module to obtain the corresponding noise image im_n1 The final denoised image im_dn.

The input noise images im_n1 and im_n2 are two adjacent frames of images in a video containing noise.

The number of sample data sets is not less than 20,000, wherein each sample includes two adjacent frames of noise images n1 and n2 in the video, the noise images n1 and n2 correspond to standard definition images p1 and p2, and the image pair p1 and p2 correspond to The optical flow estimation result of f.

The specific training method of the deep network model is to use the corresponding data in the sample data set to first train the preprocessing module separately; then fix the relevant parameters of the preprocessing module, and train the preprocessing module and the optical flow estimation module at the same time; finally fix the preprocessing module and the optical flow estimation module. The relevant parameters of the flow estimation module, the overall training includes a deep network model of three modules.

The preprocessing module, optical flow estimation module and denoising module adopt Encoder-Decoder network structure.

The Encoder-Decoder network structure includes an encoding Encoder part and a decoding Decoder part, wherein the encoding Encoder part includes M layers of convolutional layers, and the decoding Decoder part includes M subnetworks, each subnetwork includes 1 deconvolution layer and N convolutional layers Layer, when each sub-network layer of the Decoder part is deconvoluted, the image feature of the convolutional layer corresponding to the Encoder part is called, and the output result of the previous layer is used as the input of the next layer.

The deep network model is trained using the Caffe deep learning framework.

Beneficial effects of the present invention:

1) The optical flow estimation and denoising joint learning deep network model designed by the present invention can simultaneously solve the problem of optical flow estimation and denoising of noisy video in practice, and it is different from the joint optical flow estimation and video based on iterative operation in the prior art. Compared with the denoising algorithm, the optical flow estimation accuracy is high, and the denoising effect based on the optical flow estimation aid is better, and once the joint learning deep network model training is completed, the optical flow estimation and denoising speed are very fast, which is convenient for rapid implementation in practice. Process large numbers of video images.

2) For joint learning of deep network models, the present invention adopts a method of training a single network module separately and then training the overall network model, which can effectively reduce the network parameters in the training process and avoid over-fitting of the network model.

3) The method of the present invention uses a deep learning model to automatically learn the image features of optical flow estimation images and denoising images, and performs end-to-end optical flow estimation and image denoising without the need to estimate motion boundaries for assistance, and the Encoder-Decoder used The deep network model can fully mine the multi-dimensional features in the input image, which can improve the effect of optical flow estimation and denoising.

Description of drawings

Fig. 1 is the structural representation of joint learning depth network model among the present invention;

Figure 2 is two adjacent frames of noise images n1 and n2 in the video in the sample data set;

Figure 3 is the standard definition images p1 and p2 corresponding to the noise images n1 and n2 in the sample data set;

Figure 4 is the optical flow estimation result f corresponding to the image pair p1 and p2 in the sample data set;

Figure 5 is a schematic diagram of the Encoder-Decoder network structure;

Fig. 6 is a schematic structural diagram of a sub-network in the Encoder-Decoder network;

Figure 7 is two adjacent frames of noise images im_n1 and im_n2 in the video to be processed;

Figure 8 is the optical flow estimation results corresponding to noise images im_n1 and im_n2;

Fig. 9 is the denoising result of the noisy image im_n1.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figure 1, a joint learning deep network model for optical flow estimation and denoising of video images provided by this embodiment includes three modules: a preprocessing module, an optical flow estimation module and a denoising module. A sample data set of 30,000 samples, each of which includes two adjacent frames of noise images n1 and n2 in the video, as shown in Figure 2, and the standard definition images p1 and p2 corresponding to noise images n1 and n2, as shown in Figure 3 , the optical flow estimation result f corresponding to the image pair p1 and p2, as shown in Figure 4.

The preprocessing module, the optical flow estimation module and the denoising module adopt the Encoder-Decoder network structure, as shown in Figure 5, the Encoder-Decoder network structure includes an encoding Encoder part and a decoding Decoder part, and the encoding Encoder part includes a 6-layer convolutional layer c1 -c6, the number of feature maps of these 6 convolutional layers are 64, 64, 128, 128, 256, 512 respectively, the decoding Decoder part includes 5 subnetworks subnet1-subnet5, the structure of the subnetwork is shown in Figure 6, each subnetwork The network includes 1 convolutional layer and 4 deconvolutional layers. The number of feature maps of the convolutional layer in each sub-network is 64, and the number of feature maps of the 4 deconvolutional layers are 512, 256, 128, 64 respectively. The decoding When each sub-network layer of the Decoder part performs deconvolution, the image feature of the convolutional layer corresponding to the Encoder part is called, and the output result of the previous layer is used as the input of the next layer.

The joint learning deep network model is trained using the sample data set, and the deep learning model is trained using the Caffe environment installed on the Ubuntu system, and the ADAGRAD optimization algorithm is used for training. First, train the preprocessing module separately; then fix the relevant parameters of the preprocessing module, and train the preprocessing module and the optical flow estimation module at the same time; finally fix the relevant parameters of the preprocessing module and the optical flow estimation module, and train the deep network model including three modules as a whole. When training the preprocessing module alone or training the preprocessing module and the optical flow estimation module at the same time, the initial learning rate is 0.01, and the training times are 600,000 times. When the training times are 300,000, 400,000, and 500,000 times, the learning rate is divided by 10, and the reduction learning rate. The overall training includes a deep network model of three modules, the initial learning rate is 0.02, and the number of training times is 500,000 times. When the number of training times is 200,000, 300,000, and 400,000, the learning rate is divided by 8 to reduce the learning rate.

After training the joint learning deep network model, the model is directly used to process noisy video images. Input two adjacent noise images im_n1 and im_n2 in the video into the model, as shown in Figure 7, the optical flow estimation results corresponding to the noise images im_n1 and im_n2 can be directly and quickly obtained, as shown in Figure 8, and the noise image im_n1 is removed Noise results, as shown in Figure 9.

The above disclosure is only a preferred embodiment of the present invention, which certainly cannot limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims (7)

1. A kind of optical flow estimation and denoising joint learning deep network model for video image, it is characterized in that: this joint deep learning network model comprises three modules: preprocessing module, optical flow estimation module and denoising module, at first utilize The sample data set is used to train the deep network model; then, the preprocessing module is used to perform preliminary denoising processing on the input noise images im_n1 and im_n2, and the preprocessed image pair im_p1 and im_p2 are obtained; the image pair im_p1 is calculated using the optical flow estimation module Perform motion estimation with im_p2 to obtain the optical flow estimation result flow; transform the noise image im_n2 according to the optical flow estimation result flow to obtain the image im_n2', and then use the image im_n2' and noise image im_n1 as the input image of the denoising module to obtain the noise image The final denoised image im_dn corresponding to im_n1. 2. The optical flow estimation and denoising joint learning deep network model for video images according to claim 1, characterized in that: the input noise images im_n1 and im_n2 are two adjacent frames of images in a video containing noise. 3. The optical flow estimation and denoising joint learning deep network model for video images according to claim 1, characterized in that: the number of sample data sets is not less than 20000, wherein each sample includes the relevant There are two adjacent frames of noise images n1 and n2, the standard definition images p1 and p2 corresponding to the noise images n1 and n2, and the optical flow estimation result f corresponding to the image pair p1 and p2. 4. the optical flow estimation and denoising joint learning deep network model for video image according to claim 1, it is characterized in that: the concrete training method of described deep network model is to utilize the corresponding data in sample data set, at first separate training Preprocessing module; then fix the relevant parameters of the preprocessing module, and train the preprocessing module and the optical flow estimation module at the same time; finally fix the relevant parameters of the preprocessing module and the optical flow estimation module, and overall train the deep network model containing three modules. 5. The optical flow estimation and denoising joint learning deep network model for video images according to claim 1 or 4, characterized in that: the preprocessing module, optical flow estimation module and denoising module adopt Encoder-Decoder network structure. 6. The optical flow estimation and denoising joint learning deep network model for video images according to claim 5, characterized in that: the Encoder-Decoder network structure includes an encoding Encoder part and a decoding Decoder part, wherein the encoding Encoder part includes M layers of convolutional layers, the decoding Decoder part includes M subnetworks, each subnetwork includes 1 deconvolution layer and N convolutional layers, when deconvolution is performed on each subnetwork layer of the Decoder part, the encoding Encoder part is called The image features of the convolutional layer, the output of the previous layer is used as the input of the next layer. 7. The optical flow estimation and denoising joint learning deep network model for video images according to claim 4, characterized in that: the Caffe deep learning framework is used to train the deep network model.