KR102197089B1 - Multi-task learning architecture for removing non-uniform blur of single image based on deep learning - Google Patents

Abstract

The present embodiments provide a blur improvement method and apparatus capable of removing non-uniform blur from a single image through a multi-learning network model, wherein a blur improvement network and a motion prediction network designed with an encoder-decoder structure share an encoder and simultaneously go through learning.

Description

Deep learning-based multi-learning structure for removing single image non-uniform blur {MULTI-TASK LEARNING ARCHITECTURE FOR REMOVING NON-UNIFORM BLUR OF SINGLE IMAGE BASED ON DEEP LEARNING}

The technical field to which the present invention pertains is to a method and apparatus for improving single image non-uniform blur using a deep learning-based multiple learning structure.

The content described in this section merely provides background information on the present embodiment and does not constitute the prior art.

The camera's sensor accumulates light to obtain information about the scene, and sufficient exposure time is required. As a result, movement of the camera/object during the exposure time causes motion blur in the picture. Motion blur caused by camera/object movement decreases the visibility of photos, which affects the performance of various object recognition fields.

Restoring an image deteriorated by motion blur can obtain a visually improved image, and is a very essential pre-processing process in various object detection fields.

Korean Registered Patent Publication No. 10-0860967 (2008.09.24.) Korean Registered Patent Publication No. 10-1152525 (2012.05.25.) Korean Registered Patent Publication No. 10-1350460 (2014.01.03.) Korean Registered Patent Publication No. 10-1844332 (2018.03.27.)

Embodiments of the present invention are mainly aimed at removing non-uniform blur from a single image through a multi-learning network model that shares an encoder and undergoes simultaneous learning in a blur improvement network and a motion prediction network designed with an encoder-decoder structure.

Still other objects, not specified, of the present invention may be additionally considered within the range that can be easily deduced from the following detailed description and effects thereof.

According to an aspect of the present embodiment, a method of improving blur by a computing device includes receiving a single blur image, and removing non-uniform blur from the input single blur image through a multi-learning network model, The blur improvement network and the motion prediction network of the multi-learning network model provide a blur improvement method, characterized in that they share encoders with each other.

The encoder may be a motion-based feature extraction module to which a convolutional layer is applied.

The encoder may extract a presenter through a convolutional layer and increase the number of channels of the presenter through a residual block having a skip structure.

The blur improvement network includes the encoder and a first decoder, and the first decoder may restore an image from features extracted from the encoder.

The first decoder increases the resolution through a deconvolutional layer, adds a first convolution layer to a rear end of the deconvolution layer, and calculates the number of presenter channels in the first convolution layer. It is reduced to be symmetric with the encoder, the first convolution layer estimates a blur-improved image, and the spatial structure of the presenter can be maintained through skip connection between the symmetrical encoder and the first decoder.

The motion prediction network includes the encoder and a second decoder, and the second decoder may estimate an optical flow from features extracted from the encoder.

The second decoder estimates the optical flow through a deconvolution layer, adds a second convolution layer to a rear end of the deconvolution layer, and determines the number of presenter channels in the second convolution layer with the encoder. It is reduced to be symmetrical, and the second convolution layer estimates a front optical flow and a rear optical flow, and maintains the spatial structure of the presenter through skip connection between the symmetrical encoder and the second decoder.

The motion prediction network may apply a pseudo correct answer to an optical flow in a learning process.

The pseudo correct answer to the optical flow may be calculated by an optical flow estimation model that combines spatial pyramid formulation and deep learning.

The optical flow estimation model may be estimated through a convolution layer after stacking image pairs to be compared, or characteristics may be compared through a correlation layer using inputs from each other after passing through each convolution layer for an image pair to be compared. .

The blur improvement network and the motion prediction network may be learned simultaneously.

As described above, according to the embodiments of the present invention, a blur improvement network and a motion prediction network designed with an encoder-decoder structure share an encoder and remove non-uniform blur from a single image through a multi-learning network model that has undergone simultaneous learning. There is an effect that can be done.

Even if it is an effect not explicitly mentioned herein, the effect described in the following specification expected by the technical features of the present invention and the provisional effect thereof are treated as described in the specification of the present invention.

1 and 2 are diagrams illustrating an existing blur improvement model.
3 is a block diagram illustrating an apparatus for improving blur according to an embodiment of the present invention.
4 and 5 are diagrams illustrating a multi-learning network model of an apparatus for improving blur according to an embodiment of the present invention.
6 is a diagram illustrating an input image, motion information, and a reconstructed image processed by a multi-learning network model of a blur improvement apparatus according to an embodiment of the present invention.
7 is a flowchart illustrating a blur improvement method according to another embodiment of the present invention.

Hereinafter, in describing the present invention, when it is determined that the subject matter of the present invention may be unnecessarily obscured as matters apparent to those skilled in the art with respect to known functions related to the present invention, a detailed description thereof will be omitted, and some embodiments of the present invention will be It will be described in detail through exemplary drawings.

The present invention can be applied to target detection, military infrastructure monitoring system, and the like.

1 is a blur improvement model based on an existing blur kernel, and FIG. 2 is a diagram illustrating a blur improvement model that does not use the existing blur kernel.

In the conventional method shown in FIG. 1, after estimating a blur kernel, a blur-removed image is obtained through a non-blind deblurrer. Finding the blur kernel for every pixel in the image is an ill-posed problem. In order to solve the ill-posed problem, various assumptions are set in the blur kernel, and various physical models are used to limit motion information of cameras and objects in the blur kernel estimation process. There is a problem that it is difficult to accurately estimate the kernel because it is manually used.

The conventional method shown in FIG. 2 learns a high-dimensional nonlinear mapping relationship of an enhanced image from a blurred image directly without kernel estimation through a deep learning technique. The learning network learns motion information through a high-dimensional nonlinear mapping relationship between the blurred image and the correct answer image. There is no direct learning process that enables the learned network features to contain motion information semantically.

The blur improvement apparatus according to the present embodiment not only learns motion information through mapping between input correct answers, but also explicitly learns motion information through a multi-learning structure for additionally learning a motion estimation network. When using the same network, it is possible to secure more improved blur removal performance without loss of runtime than the existing deep learning method. Unlike the existing kernel-based method, there is no need to estimate accurate blur information (that is, motion information) during the test process, and complex repetitive operations are not required.

3 is a block diagram illustrating an apparatus for improving blur according to an embodiment of the present invention.

The blur improvement device 110 includes at least one processor 120, a computer-readable storage medium 130, and a communication bus 170.

The processor 120 may be controlled to operate as the blur improvement device 110. For example, the processor 120 may execute one or more programs stored in the computer-readable storage medium 130. One or more programs may include one or more computer-executable instructions, and when the computer-executable instructions are executed by the processor 120, the blur improvement apparatus 110 is configured to perform operations according to an exemplary embodiment. I can.

Computer-readable storage medium 130 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 140 stored in the computer-readable storage medium 130 includes a set of instructions executable by the processor 120. In one embodiment, the computer-readable storage medium 130 includes memory (volatile memory such as random access memory, nonvolatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, Flash memory devices, other types of storage media that can be accessed by the blur enhancement device 110 and store desired information, or a suitable combination thereof.

The communication bus 170 interconnects the various other components of the blur enhancement device 110 including the processor 120 and a computer-readable storage medium 140.

The blur improvement device 110 may also include one or more input/output interfaces 150 and one or more communication interfaces 160 that provide interfaces for one or more input/output devices. The input/output interface 150 and the communication interface 160 are connected to the communication bus 170. The input/output device (not shown) may be connected to other components of the blur improvement device 110 through the input/output interface 150.

4 and 5 are diagrams illustrating a multi-learning network model of an apparatus for improving blur according to an embodiment of the present invention.

The multi-learning network model is a learning network structure in which a plurality of layers are connected, and has an encoder-decoder structure. In the learning network model, a number of layers are connected through a network and include hidden layers. The layer may include parameters, and the parameters of the layer include a set of learnable filters. The parameters include weights and/or biases between nodes.

The multi-learning network model includes a blur improvement network and a motion prediction network. Both networks are designed with an encoder-decoder structure through stride convolution to refer to a wide area, and share some network parameters with each other.

The blur enhancement network first extracts a 64 channel presenter from a single input image through three convolution layers. Thereafter, the spatial size is halved through two residual blocks including stride convolutions, and the number of presenter channels is doubled.

The residual block has the advantage of learning better presenters because it can solve the problem that the gradient disappears during network learning. The residual block consists of three convolutional layers, and the first layer has a stride of 2. Also, the result of the residual block is obtained by summing the result of the last convolutional layer and the result of the first layer. The presenter reduced to 1/4 size (the channel is 256) then passes through three residual blocks, where the stride of all convolutional layers is 1.

Finally, in order to estimate an improved image having the same resolution as the input image, two deconvolution layers are used to increase the resolution by two times. After each deconvolution layer, one convolution layer was additionally designed, and the number of channels of the presenter was reduced in half to be symmetric with the encoder. The last convolutional layer directly estimates the three-channel enhancement image. Meanwhile, the spatial structure of the presenter at the front end is maintained through skip connection between the symmetrical encoder and the decoder presenter.

ReLU (Rectified Linear Unit) layer is used as a nonlinear function to increase the nonlinearity of the network. In addition, a batch normalization layer can be used to solve the problem of the internal covariate shift of the convolution presenter.

Another network of multiple learning structures, a motion prediction network (optical flow estimation network), shares the encoder structure of the blur improvement network. Additional networks for optical flow estimation are the deconvolution layer and the convolution layer of the decoder part. This can be designed similar to the decoder part of the blur enhancement network.

The final convolution of the motion prediction network immediately estimates the optical flow of the four channels. Each two-channel flow represents a front optical flow and a rear optical flow, respectively.

The multi-learning network model may convert an output of the first decoder of the blur improvement network into an input of an encoder, which is a motion-based feature extraction module, through a first feedback path during a learning process, and provide feedback. The conversion module converts the data value according to the format of the output and the input, and converts the data value through the first conversion module on the first feedback path.

The multi-learning network model may convert an output of the second decoder of the motion prediction network into an input of an encoder, which is a motion-based feature extraction module, through a second feedback path in the learning process, and feed back. The conversion module converts the data value according to the format of the output and the input, and converts the data value through the second conversion module on the second feedback path.

The multi-learning network model may convert an output of the first decoder of the blur improvement network into an input of the second decoder of the motion prediction network through a third feedback path in the learning process and feed back. The conversion module converts the data value according to the format of the output and the input, and converts the data value through the third conversion module on the third feedback path.

The multi-learning network model may convert the output of the second decoder of the motion prediction network into the input of the first decoder of the blur improvement network through the fourth feedback path in the learning process and provide feedback. The conversion module converts the data value according to the format of the output and the input, and converts the data value through the fourth conversion module on the fourth feedback path.

The multi-learning network model is trained to derive parameters that minimize the loss function.

The total loss function is designed as the sum of the cost functions of L _deblur and L _flow for learning the blur improvement network and motion prediction network. L _deblur can be designed as the sum of Perceptual Loss and Adversarial Loss as shown in Equation 1 according to the latest CNN-based super-resolution image restoration technology. λ can be set to 0.01, and other numerical values can be applied according to the required design requirements.

와 정답

를 수학식 2와 같이 설계된다.L _VGG is calculated through the presenter of the VGG network already learned in the field of image classification using the ImageNet database. L _VGG is an improved video

And correct answer

Is designed as in Equation 2.

Φ represents the presenter estimation network of the VGG network. The cost function using the presenter having such semantic characteristics can obtain a visually improved improved image. L _adv is a cost function that makes no distinction between the improved image and the correct answer image. For such adversarial learning, a discriminator network is required to classify whether an input image is actually a clear image or an improved image. In consideration of the Generative Adversarial Network (GAN) model, the discriminator network can be designed as shown in Table 1.

Here, Conv, LReLU, BN, and FC mean convolution, leaky ReLU, batch normalization, and fully connected layer, respectively. The discriminator in Table 1 takes as an input the result image or the correct answer image of the blur improvement network, and is learned by the cost function of Equation 3.

D and G represent discriminator and blur improvement networks, respectively. In order to learn that L _adv is _{indistinguishable} from the improved image and the correct answer image in L _deblur , the cost function of Equation 4 must be minimized.

In order to learn a motion prediction network (optical flow estimation network) from a single blur image, we first need to know the correct answer for the flow. However, in general, there is no such answer.

및

)을 도출한다. 옵티컬 플로우 추정 모델은 비교할 이미지 쌍을 스택한 후 컨볼루션 레이어를 거쳐 추정하거나 비교할 이미지 쌍을 각각의 컨볼루션 레이어를 거친 후 서로의 입력을 이용하는 상관 레이어(Correlation Layer)를 통해 특징을 비교한다.The blur improvement apparatus according to the present embodiment uses an optical flow estimation model using two images S[0] and S[M-1], and a pseudo correct answer for the flow (

And

). The optical flow estimation model stacks image pairs to be compared and then estimates them through a convolution layer, or compares features through a correlation layer using inputs from each other after estimating the image pairs to be compared through each convolution layer.

An example of an input image for learning a multi-learning network structure and an image of a correct answer for blur improvement and flow estimation is illustrated in FIG. 6.

L _flow is expressed as in Equation 5.

F _forward and F _backward represent a forward optical flow and a rear optical flow estimated by a motion prediction network (optical flow estimation network).

7 is a flowchart illustrating a blur improvement method according to another embodiment of the present invention. The blur improvement method may be performed by a blur improvement apparatus or a computing device.

In step S210, the processor receives a single blur image.

In step S220, the processor removes the non-uniform blur from the input single blur image through the multiple learning network model.

The blur improvement network and the motion prediction network of the multi-learning network model share encoders with each other. The blur improvement network and motion prediction network are learned simultaneously.

The encoder is a motion-based feature extraction module to which a convolutional layer is applied. The encoder extracts a presenter through a convolutional layer, and increases the number of channels of the presenter through a residual block having a skip structure.

The blur enhancement network includes an encoder and a first decoder, and the first decoder recovers an image from features extracted from the encoder. The first decoder increases the resolution through a deconvolutional layer, adds a first convolution layer to the rear end of the deconvolution layer, and determines the number of channels of the presenter in the first convolution layer symmetrically with the encoder. As much as possible, and the first convolution layer estimates the blur-enhanced image. The spatial structure of the presenter is maintained through skip connection between the symmetrical encoder and the first decoder.

The motion prediction network includes an encoder and a second decoder, and the second decoder estimates an optical flow from features extracted from the encoder. The second decoder estimates the optical flow through the deconvolution layer, adds a second convolution layer to the rear end of the deconvolution layer, and reduces the number of presenter channels in the second convolution layer to be symmetric with the encoder. And, the second convolution layer estimates the front optical flow and the rear optical flow. The spatial structure of the presenter is maintained through skip connection between the symmetrical encoder and the second decoder.

The motion prediction network applies a pseudo correct answer to the optical flow in the learning process. The pseudo-correct answer for the optical flow is calculated by an optical flow estimation model that combines spatial pyramid formulation and deep learning. The optical flow estimation model stacks image pairs to be compared and then estimates them through a convolution layer, or compares features through a correlation layer using inputs from each other after estimating the image pairs to be compared through each convolution layer.

The blur improvement apparatus may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general purpose or specific purpose computer. The device may be implemented using a hardwired device, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or the like. In addition, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The blur improvement apparatus may be mounted in a form of software, hardware, or a combination thereof on a computing device or server provided with hardware elements. Computing devices or servers include all or part of a communication device such as a communication modem for performing communication with various devices or wired/wireless communication networks, a memory storing data for executing a program, and a microprocessor for calculating and commanding a program. It can mean various devices including.

In FIG. 7, it is described that each process is sequentially executed, but this is only illustrative, and those skilled in the art may change the order shown in FIG. 7 without departing from the essential characteristics of the embodiment of the present invention. Or, by executing one or more processes in parallel, or adding other processes, various modifications and variations may be applied.

The operations according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. Computer-readable medium refers to any medium that has participated in providing instructions to a processor for execution. The computer-readable medium may include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. Computer programs may be distributed over networked computer systems to store and execute computer-readable codes in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment may be easily inferred by programmers in the technical field to which the present embodiment belongs.

The present embodiments are for explaining the technical idea of the present embodiment, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

Claims (11)

In the blur improvement method by the blur improvement device,
Receiving a single blur image; And
And removing non-uniform blur from the input single blur image through a multi-learning network model,
The blur improvement network and the motion prediction network of the multi-learning network model share encoders with each other,
The blur improvement method, characterized in that the blur improvement network and the motion prediction network are learned at the same time. The method of claim 1,
Wherein the encoder is a motion-based feature extraction module to which a convolutional layer is applied. The method of claim 2,
Wherein the encoder extracts a presenter through the convolutional layer and increases the number of channels of the presenter through a residual block having a skip structure. The method of claim 1,
Wherein the blur improvement network includes the encoder and a first decoder, and the first decoder restores an image from features extracted from the encoder. The method of claim 4,
The first decoder increases the resolution through a deconvolutional layer, adds a first convolution layer to a rear end of the deconvolution layer, and calculates the number of presenter channels in the first convolution layer. Reduce to be symmetric with the encoder, and the first convolution layer estimates a blur-improved image,
And maintaining the spatial structure of the presenter through skip connection between the symmetrical encoder and the first decoder. The method of claim 1,
Wherein the motion prediction network includes the encoder and a second decoder, and the second decoder estimates an optical flow from features extracted from the encoder. The method of claim 6,
The second decoder estimates the optical flow through a deconvolution layer, adds a second convolution layer to a rear end of the deconvolution layer, and determines the number of presenter channels in the second convolution layer with the encoder. Decrease to be symmetric, and the second convolution layer estimates a front optical flow and a rear optical flow,
And maintaining the spatial structure of the presenter through skip connection between the symmetrical encoder and the second decoder. The method of claim 6,
The motion prediction network applies a pseudo correct answer to an optical flow in a learning process. The method of claim 8,
The method for improving blur, characterized in that the pseudo correct answer to the optical flow is calculated by an optical flow estimation model that combines spatial pyramid formulation and deep learning. The method of claim 9,
In the optical flow estimation model, the image pairs to be compared are stacked and then estimated through a convolution layer, or the image pairs to be compared are evaluated through respective convolution layers, and features are compared through a correlation layer using inputs from each other. Blur improvement method characterized by. delete

Images