KR102334730B1 - Generative Adversarial Network for Joint Light Field Super-resolution and Deblurring and its Operation Method - Google Patents

Abstract

An adversarial neural network model apparatus for simultaneously performing light field super-resolution and deblurring and an operation method thereof are suggested. An adversarial neural network model apparatus for simultaneously performing light field super-resolution and deblurring according to an embodiment includes: a super-resolution network for receiving a subview image of a low-resolution or blurred light field and outputting a super-resolution reconstructed image; and a deblurring network for receiving the super-resolution reconstructed image, passing the same through at least one or more convolution blocks, and estimating an image from which super-resolution and motion blur have been removed. It is possible to provide an end-to-end network including the super-resolution network and the deblurring network.

Description

Adversarial neural network model apparatus for simultaneous performance of light field super-resolution and blur removal and its operation method

The following embodiments relate to an adversarial neural network model apparatus for simultaneously performing lightfield super-resolution and blur removal, and an operating method thereof.

Recently, light field imaging has attracted attention due to the increasing popularity of various studies such as viewpoint synthesis (angular domain super-resolution (SR)), image refocusing, and depth estimation. However, studies on image enhancement such as spatial domain super-resolution reconstruction of lightfield and motion blur removal have not received similar attention. This is because a single light field image contains a large amount of data and the parallax constraint must be maintained. Therefore, there are few notable studies on lightfield blur removal and super-resolution restoration recently.

In order to properly perform lightfield image enhancement, it is important to describe in detail the domain of the lightfield from which parallax information is acquired. One lightfield image has a two-dimensional spatial region and pixels dispersed in each two-dimensional region. Each additional area different from the existing 2D image generates subview images related to parallax. Each subview image of the light field means a two-dimensional image located in a specific angular direction as described by R. Ng (Non-Patent Document 11), and most of the coordinates of the space and angular region of the light field are (x , y) and (u, v) are used.

In early studies, light fields were acquired by creating multiple scenes by rotating a small object in front of a single camera. Recently, the user-friendly light field camera Lytro that can acquire light field images is similar in size to the latest digital camera, so it has convenience in light field acquisition. Although the Lytro camera is no longer produced, it is still used to acquire multi-view images. In the case of a multi-viewpoint image, it can be acquired through a micro lens located behind the main lens. Although the angle between the microlenses is small, it is mainly used for image depth estimation because it provides a plausible parallax effect. However, since these techniques greatly depend on the quality of the acquired light field, a light field enhancement technique is required after acquiring the light field image.

The super-resolution technique restores a blurred area generated by interpolated pixels of low-resolution images or fills the pixels at a new high-resolution position. Kim et al. It is known from the technique (nonpatent literature 1). Unlike a two-dimensional image, the super-resolution of a light field image can be performed in a two-dimensional spatial domain and each two-dimensional domain. The two-dimensional angular region super-resolution is also known as the viewpoint synthesis process.

Similar to super-resolution, the main task of blur removal is to clearly restore the blurred image given as input. However, in the present embodiment, the focus is on the blur caused by the movement of the camera and the object. Previous research on blur removal attempted to restore a point spread function or a blur kernel that expresses motion. In an existing study (Non-Patent Document 5), a specific light field camera movement for blur removal was introduced. In the previous study, a 3-DOF transformation model was introduced to simulate light field camera movement.

J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1646-1654, June 2016. D. Krishnan, T. Tay, and R. Fergus, "Blind deconvolution using a normalized sparsity measure," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 233-240, June 2011. J. Pan, Z. Hu, Z. Su, and M. H. Yang, “Deblurring text images via L0-regularized intensity and gradient prior,” In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2908, June 2014. T. H. Kim, K. M. Lee, B. Scholkopf, and M. Hirsch, “Online video deblurring via dynamic temporal blending network,” In Proc. of IEEE International Conference on Computer Vision, pages 4058-4067, 2017. P. P. Srinivasan, R. Ng, and R. Ramamoorthi, "Light field blind motion deblurring," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 3958-3966, July 2017. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, June 2016. P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pages 1125-1134, July 2017. Y. W. Tai, P. Tan, and M. S. Brown, "Richardson-Lucy deblurring for scenes under a projective motion path," IEEE Trans. on Pattern Analysis and Machine Intelligence, 33(8):1603-1618, August 2011. Y. Bok, H. G. Jeon, and I. S. Kweon, “Geometric calibration of micro-lense-based light field cameras using line features,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 39(2):287-300, February 2017. M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, "Depth from combining defocus and correspondence using light-field cameras," In Proc. of the IEEE International Conference on Computer Vision, pages 673-680, December 2013. R. Ng, "Fourier slice photography," ACM Trans. on Graphics, 24(3):735-744, July 2005.

The embodiments describe an adversarial neural network model apparatus for simultaneous performance of lightfield super-resolution and blur removal and an operation method thereof, and more specifically, end-to-end solving spatial domain super-resolution and motion blur removal of lightfield images simultaneously -end Provides network technology.

The embodiments made it possible to apply to multiple resolutions using down/upscale parameters, and to solve the problem of current light field image enhancement, an adversarial neural network model device that simultaneously performs super-resolution and motion blur removal and an operating method thereof is to provide

An adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal according to an embodiment includes: a super-resolution network that receives a subview image of a low-resolution or blurred light field and outputs a super-resolution reconstructed image; and a blur removal network for estimating an image from which super resolution and motion blur have been removed by receiving the super-resolution reconstructed image and passing at least one or more convolution blocks as an input, end including the super-resolution network and the blur removal network Can provide -to-end networks.

In the super-resolution network, the resolution of the reconstructed super-resolution image is down-sampled to a d scale, and the subview image input to obtain additional information about the neighboring subview image may be connected to at least neighboring images.

A discriminator network that trains the end-to-end network by dividing it into multiple steps to allow backpropagation of losses in super-resolution restoration and blur removal, and training with low-resolution and 6-DOF motion-blurred lightfield datasets may further include.

It provides a generative adversarial network (GAN) that processes the whole and local regions of an image, wherein the discriminator network is the super-resolution and motion blur-removed image predicted from the generator network in the first step. The discriminator network may be trained using the entire region, and additional local regions may be learned using the same discriminator network in a second step.

The operating method of the adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal performed through a computer device according to another embodiment receives a subview image of a low-resolution or blurred light field from a super-resolution network and receives a super-resolution outputting a restored image; and estimating the super-resolution and motion blur-removed image by receiving the super-resolution reconstructed image from the blur removal network and passing it through at least one or more convolution blocks, including the super-resolution network and the blur removal network It can provide an end-to-end network that

According to the embodiments, it is possible to apply to multiple resolutions using down/upscale parameters, and to solve the problem of current light field image improvement, an adversarial neural network model device that simultaneously performs super-resolution and motion blur removal and its operation method can be provided.

1 is a diagram illustrating a framework result according to an embodiment.
2A is a diagram for explaining an adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal according to an embodiment.
2B is a flowchart illustrating a method of operating an adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal according to an embodiment.
3 is a diagram illustrating a structure of a super-resolution network according to an embodiment.
4 is a diagram illustrating a structure of a blur removal network according to an embodiment.
5 is a diagram for describing a discriminator network according to an embodiment.
6 is a diagram illustrating a qualitative result image of an algorithm according to an exemplary embodiment.
7 is a diagram illustrating a qualitative result image according to an exemplary embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided in order to more completely explain the present invention to those of ordinary skill in the art. The shapes and sizes of elements in the drawings may be exaggerated for clearer description.

1 is a diagram illustrating a framework result according to an embodiment.

Fig. 1 (a) shows the central subview image of the low-resolution and 6-DOF motion-blurred light field, and (b) shows the light field subview image with super-resolution restoration and motion blur removed through the proposed technique. Here, the framework refers to an adversarial neural network model device for simultaneous performance of lightfield super-resolution and blur removal.

As studies on parallax-based image processing increase, research to restore low-resolution and motion-blurred lightfield images has become essential. These techniques are known as lightfield image enhancement processes, but there are few existing studies that solve two or more problems at the same time.

An embodiment of the present invention proposes a framework for simultaneously performing lightfield spatial domain super-resolution restoration and motion blur removal. Specifically, we create a simple network that trains on a low-resolution and 6-DOF motion-blurred lightfield dataset. Also, to improve the performance, we proposed a regional domain optimization technique for generative adversarial neural networks. The proposed framework is evaluated through quantitative and qualitative measurement, and it shows excellent performance compared to the existing state-of-the-art methods. The light field enhancement technique in this embodiment is defined as a 4-fold improvement in spatial domain resolution and 6-DOF motion model-based blur removal.

In the example below, we propose an end-to-end network that simultaneously solves spatial domain super-resolution and motion blur removal of lightfield images. Network training involves several stages of optimization, which are backpropagated through reconstruction and adversarial loss. In the generative adversarial neural network stage, the global and local structures of the estimated results and inputs are combined and used for multiple loss calculations. The local structure aims to force the unblurred area to be the same as the ground truth, similar to the image synthesis technique. The contribution of this embodiment is summarized as follows.

* Formulate 6-DOF lightfield camera motion on reduced spatial scales to generate low-resolution and motion-blurred lightfield datasets.

* We propose a framework that simultaneously performs lightfield spatial domain super-resolution restoration and motion blur removal.

* Provides local masking and improvement strategies to improve performance during network training.

The configuration of this embodiment is as follows. First, we propose a network that simultaneously performs super-resolution and blur removal, and introduces SRNet, DebNet, and discriminator networks in detail. Next, we introduce the 6-DOF light field blur model and the optimization technique applied to the network, and introduce the low-resolution and motion-blurred light field dataset created to train the network. After that, the details of implementation will be described, and quantitative and qualitative comparative analysis with other existing techniques will be performed.

2A is a diagram for explaining an adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal according to an embodiment.

Referring to Fig. 2a, in this embodiment, we propose an end-to-end network that takes as an input stacked subview images of a light field and generates as an output a result of spatial domain super-resolution restoration and motion blur removal of a specific subview image. .

An adversarial neural network model apparatus (adversarial network) for simultaneously performing lightfield super-resolution and blur removal according to an embodiment includes Joint SR and Deblurring (JSRD). The joint network defined by JSRD is simply composed of a super-resolution restoration network (super-resolution network) (SRNet, 210)) and a blur removal network (DebNet, 220).

JSRD is a network that takes an input of spatial size (H / d, W / d) and outputs a result of size (H, W). Since the network operates directly on the 2D image, the 4D light field representation (x, y, u, v) is reduced to 3D (x, y, a) and processed one by one through a cyclic technique. The domain reduction by stacking subview images will be described below.

The end-to-end network is trained in several stages to allow backpropagation of losses in super-resolution restoration and blur removal. Here we provide a generative adversarial neural network that processes with global and local mask regions. An adversarial neural network model apparatus for simultaneously performing lightfield super-resolution and blur removal according to an embodiment will be described in more detail below.

The adversarial neural network model apparatus for simultaneously performing lightfield super-resolution and blur removal according to an embodiment may include a super-resolution network 210 and a blur removal network 220 . Accordingly, it is possible to provide an end-to-end network including a super-resolution network and a blur removal network. In addition, according to an embodiment, it may further include a discriminator network. Therefore, it is possible to provide a generative adversarial network (GAN) that processes the entire image and local regions.

The super-resolution network 210 may receive a subview image 201 of a low-resolution or blurred light field and output a super-resolution reconstructed image 202 . Here, in the super-resolution network 210, the resolution of the super-resolution reconstructed image 202 is downsampled to a d scale, and the subview image 201 input to obtain additional information on the neighboring subview image is at least the neighboring images. can be connected with

The blur removal network 220 may receive the super-resolution reconstructed image 202 and pass it through at least one or more convolution blocks to estimate the super-resolution and motion blur-removed image 203 .

Embodiments may provide a generative adversarial network (GAN) that processes the entire image and local regions. In this case, the super-resolution network 210 and the blur removal network 220 may be included in the generator network, and may further include a discriminator network as follows.

The discriminator network trains the end-to-end network by dividing it into multiple steps to allow backpropagation of losses in super-resolution restoration and blur removal, and trains with low-resolution and 6-DOF motion-blurred lightfield datasets. can

Here, the discriminator network trains the discriminator network using the entire region of the super-resolution and motion blur-removed image 203 predicted from the generator network in the first step, and uses the same discriminator network in the second step. Thus, additional local areas can be learned.

2B is a flowchart illustrating a method of operating an adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal according to an embodiment.

Referring to FIG. 2B , a method of operating an adversarial neural network model apparatus for simultaneously performing light field super-resolution and blur removal performed through a computer device according to an embodiment is a subview of a low-resolution or blurred light field in a super-resolution network Step (S110) of receiving an image and outputting a super-resolution reconstructed image, and estimating an image from which super-resolution and motion blur are removed by receiving a super-resolution reconstructed image from a blur removal network and passing it through at least one or more convolution blocks (S120) may be included. Here, an end-to-end network including a super-resolution network and a blur removal network may be provided.

In addition, training the end-to-end network by dividing it into multiple steps to allow backpropagation of the losses of super-resolution restoration and blur removal in the discriminator network, and low-resolution and 6-DOF motion-blurred lightfield datasets The training step (S130) may be further included.

In the first step, the discriminator network is trained using the entire region of the super-resolution and motion blur-removed image predicted from the generator network in the first step, and in the second step, an additional local area is used using the same discriminator network. can learn

3 is a diagram illustrating a structure of a super-resolution network according to an embodiment.

)을 입력으로 받아 초해상도 복원 영상(

)을 출력한다.

의 해상도는 d 스케일로 다운샘플링되며, 이웃 subview 영상에 대한 추가적인 정보를 획득하기 위해 입력은 적어도 이웃한 영상들(

,

,

)과 연결한다. As shown in Fig. 3, in the first step of the framework, a cyclic U-Net was utilized to solve the spatial domain super-resolution restoration. The network is a low-resolution and blurred lightfield subview image (

) as an input and a super-resolution restored image (

) is output.

The resolution of is downsampled to the d scale, and the input is at least the neighboring images (

,

,

) is connected with

)과 연결되어 U-Net 내에서 처리된다. U-Net은 6개의 인코더와 5개의 디코더가 통합되어 있다. 각 인코더 계층에서는 채널의 수가 2배로 증가하지만 공간의 크기는 절반으로 줄어든다. 이와 반대로, 디코더 계층에서는 공간의 크기가 2배로 증가하고 채널 수가 절반으로 감소한다. 각각의 디코더 계층은 동일한 크기의 인코더 계층의 출력을 받는다는 점에 유의한다. 인코더의 출력은 디코더의 입력과 연결되어 있다. U-Net 블록은 컨볼루션/디컨볼루션, IN (인스턴스 정규화) 및 ReLU 활성화로 구성되어 있다. 마지막으로, U-Net의 출력은 2개의 간단한 Conv-IN-ReLU 블록을 통과한다. 첫 번째 블록의 출력

은 다음 영상을 위해 recurrent connection을 통해 전파한다. 두 번째 블록의 출력은 잔차(residual) 초해상도 영상

이며, 입력(

)을 더하여 블러된 영상의 초해상도 복원 영상(

)을 생성한다.First, the features are passed through two convolution blocks normalized by instance normalization. The channels of the first and second convolutional layers consist of 64 and 32, respectively, and the size of the filter is 9x9 and 3x3, respectively. The features extracted from the convolutional layer are the previously hidden features (

) and processed within U-Net. U-Net has 6 encoders and 5 decoders integrated. At each encoder layer, the number of channels is doubled, but the size of the space is halved. Conversely, in the decoder layer, the size of the space is doubled and the number of channels is halved. Note that each decoder layer receives the output of the encoder layer of the same size. The output of the encoder is connected to the input of the decoder. The U-Net block consists of convolution/deconvolution, IN (instance normalization) and ReLU activation. Finally, the output of U-Net is passed through two simple Conv-IN-ReLU blocks. output of the first block

propagates over the recurrent connection for the next video. The output of the second block is the residual super-resolution image

, and input (

) to the super-resolution reconstructed image (

) is created.

4 is a diagram illustrating a structure of a blur removal network according to an embodiment.

)은 conv-IN-ReLU의 조합으로 구성된 2개의 컨볼루션 블록을 통과한다. 처음 2개의 컨볼루션층에서 추출한 특징은 이전 영상의 숨겨진 특징인

와 연결한다. 연결된 특징들은 배치 정규화가 아닌 He et al.(비특허문헌 6)의 인스턴스 정규화를 사용하는 8개의 ResNet 블록을 통과하며, ResNet 블록은 conv-IN-ReLU-conv-IN 조합으로 구성되어 있다. ResNet 블록 내부에서는 특징이 ReLU로 활성화되고 2개의 컨볼루션 층을 통과한다. ResNet 블록 이후의 첫 번째 컨볼루션 블록은 다음 영상으로 숨겨진 특징인 출력

을 제공한다. 마지막 컨볼루션 블록은 잔차(residual) 블러 제거 영상(

)을 생성하고, 입력(

)을 더하여 최종적인 초해상도 및 모션 블러가 제거된 영상(

)을 추정한다.The next step, the de-blur network, processes the output of the super-resolution network. As shown in Fig. 4, the input of the blur removal network (

) passes through two convolution blocks composed of a combination of conv-IN-ReLU. The features extracted from the first two convolutional layers are the hidden features of the previous image.

connect with The connected features pass through 8 ResNet blocks using instance normalization of He et al. (Non-Patent Document 6) rather than batch normalization, and the ResNet block is composed of a conv-IN-ReLU-conv-IN combination. Inside the ResNet block, features are activated with ReLU and passed through two convolutional layers. The first convolution block after the ResNet block is the output, which is a feature hidden by the next image.

provides The last convolution block is a residual blur-removed image (

), and input (

) to the final super-resolution and motion blur removed image (

) is estimated.

5 is a diagram for describing a discriminator network according to an embodiment.

In the past few years, generative adversarial networks (GANs) have been introduced to synthesize plausible results given input images from different domains. While generative adversarial neural networks (GANs) have been widely used, the semantic information included in one image is reduced and performance is degraded. To solve this problem, local information for a specific area of the image was used. This local information enhances the relationships between pixels, producing smoother results.

)의 전체 영역을 사용하여 구별자를 훈련한다. 두 번째 단계에서는 동일한 구별자를 이용하여 추가 특정 영역을 학습한다. 추가 특정 영역이란

와

의 절대 차이에 의한 영역 M을 나타낸다. 이러한 방식으로 블러가 제거되지 않은 지역은 구별자를 통해 지역 prior로 처리된다. 구별자 네트워크는 Conv-IN-Leaky ReLU로 구성된 6개 블록과 Conv-IN-Sigmoid로 구성된 1개 블록으로 Isola et al.(비특허문헌 7)에서 제안한 기법과 유사하게 구성한다. In this example, we designed a simple discriminator network that trains in several steps. The first step is the image predicted from the generator (

) is used to train the discriminator. In the second stage, additional specific domains are learned using the same discriminator. What are additional specific areas?

Wow

represents the region M by the absolute difference of . In this way, the region where the blur is not removed is treated as a regional prior through the discriminator. The discriminator network is composed of 6 blocks composed of Conv-IN-Leaky ReLU and 1 block composed of Conv-IN-Sigmoid, similar to the technique proposed by Isola et al. (Non-Patent Document 7).

Finally, as shown in Fig. 5, in each layer except for the last layer, the size of the spatial domain is downsampled by 2 times, and the channel is increased to 64 scale.

는 SR-net, Deb-net을 통해 생성된 블러 제거 및 초해상도된 결과물이고,

는 초해상도 및 선명한 영상으로 ground truth를 의미한다. 도 5의 (a)는 마스킹 과정을 나타내고, (b)는 네트워크를 전체 영역으로 학습하는 경우를 나타내며, (c)는 네트워크를 마스킹된 영역으로 학습하는 경우를 나타낸다. here,

is the result of blur removal and super-resolution created through SR-net and Deb-net,

means ground truth with super-resolution and clear images. Fig. 5 (a) shows the masking process, (b) shows the case of learning the network as an entire region, and (c) shows the case of learning the network as the masked region.

The following describes the network learning technique.

6-DOF Lightfield Blur Model

In this embodiment, the blurred subview images are generated by simulating with 6-DOF light field camera motion. In particular, it warps sharp subview images according to motion and integrates them over time. This approach is similar to the method of warping a clear two-dimensional image with other homography proposed by Tai et al. (Non-Patent Document 8), an existing study. The final blurred image is generated by averaging the warped images. In the following, the 3-DOF transformation model and the 3-DOF rotation model are described to represent the complete 6-DOF model.

First, the 3-DOF transformation model will be described.

The 3-DOF light field transformation model is derived from two-dimensional in-plane motion in the x- and y-axis and one-dimensional out-of-plane motion in the z-axis. Blurred pixels of the lightfield (B) are obtained from an integrated version (G) of the sharpened lightfield that has been warped by moving it along the axis. Also, it can be simply expressed as the following equation.

[Equation 1]

는 전체 시간 T에서 각각의 노출 시간 t의 in-plane 및 out-of-plane 카메라 변환을 나타낸다. x (및 y)는 다운샘플 (또는 업 샘플) 스케일을 나타내는 d로 나누어진다. 여기에서 라이트필드의 매개 변수는 단순화를 위해 4D (x, y, u, v)에서 2D (x, u)로 축소하였다.Here, (x, u) represents spatial and angular domain coordinates,

represents the in-plane and out-of-plane camera transformations for each exposure time t at the total time T. x (and y) is divided by d representing the downsample (or upsample) scale. Here, the parameters of the light field are reduced from 4D (x, y, u, v) to 2D (x, u) for simplicity.

Next, the 3-DOF rotation model will be described.

및 y축

)과 in-plane의 두 가지 모델로 구성된 회전 변환은 보다 정확한 이해를 위해 각각 분리하여 설명한다.Equation 1 is extended to add an additional 3-DOF rotation approximation to model 6-DOF motion. out-of-plane (x-axis

and y-axis

) and in-plane, the rotational transformation, which consists of two models, will be described separately for a more accurate understanding.

Out-of-plane Rotation: Model a fronto-parallel subview image (G(u)) warped by a 3Х3 homography matrix. This can be defined as the following equation.

[Equation 2]

)는 초점거리와 y축과 x축을 따른 회전각을 의미한다. At this time, (

) is the focal length and rotation angles along the y and x axes.

)에 대한 추가적인 매개변수가 된다. 따라서 이는 다음 식과 같이 표현될 수 있다.Out-of-plane is a transformation model (

) as an additional parameter. Therefore, it can be expressed as the following equation.

[Equation 3]

는 각 x축 및 y축을 따라 업데이트된 in-plane 모션을 나타낸다. here,

denotes the updated in-plane motion along each x-axis and y-axis.

If Equation 1 is modified using out-of-plane rotation, it can be expressed as the following equation.

[Equation 4]

의 거리만큼 분리되며, 이 매개변수들은 시차효과를 생성하는 라이트필드 카메라 기준선으로 활용된다. Bok et al.(비특허문헌 9)에서 제공하는 툴 박스를 사용하여 캘리브레이션을 진행하면

값은 0.9 픽셀이지만 정확한 위치를 제공하기 위해 1픽셀로 정한다. in-plane 회전을 모델링하면 다음 식과 같이 나타낼 수 있다.In-plane Rotation: Lightfield cameras rotate along a single axis when rotated in-plane. This axis intersects the central subview image. Each subview image is

is separated by a distance of , and these parameters are used as a reference line for the light field camera to create a parallax effect. When calibration is performed using the toolbox provided by Bok et al. (Non-Patent Document 9),

The value is 0.9 pixels, but we set it to 1 pixel to provide an accurate location. Modeling the in-plane rotation can be expressed as the following equation.

[Equation 5]

[Equation 6]

는 in-plane 각도를 나타낸다. here,

is the in-plane angle.

Finally, by integrating Equations 5 and 6 into Equation 4, the entire 6-DOF model lightfield camera motion can be expressed as the following Equation.

[Equation 7]

Low-resolution and motion-blurred lightfield datasets

In this step, we describe the details of the dataset for network learning of the proposed technique. Existing blur datasets mainly focus on removing blur from 2D or 3D images (videos) rather than light fields. Recently, large-scale light field datasets for super-resolution studies in each domain have been provided.

Therefore, in order to generate a large-scale light field dataset with low resolution and non-uniformly blurred light field, light field data were acquired from Lytro Illum for real images and 3D scenes were collected from UnrealCV for synthetic images. The 3D scene was included to give more changes to the color gamut because the Lytro image tends to have low intensity. In the implementation, d is applied to 4 to reduce the spatial coordinates by 4 times to provide a subview image with a spatial size of 80x128. The dataset was divided into 360 light fields for the training set and 40 light fields for the test set. The population ratio of data captured with lytro and 3D scenes is 50:50 in both sets. The 6-DOF model simulated the shutter time effect (T = 15) in a lightfield camera using Equation 7. In particular, in the 6-DOF model, the Bezier curve for the transformation model and spherical linear interpolation for the rotation model were parameterized at every time t(t∈T).

optimization

In the initial stage, the super-resolution network and the blur removal network are individually trained using L1 and perceptual losses, respectively. The perceptual loss is obtained by obtaining the L2 error from the predicted image and ground truth features through the pre-trained VGG-19 network. In the implementation phase, the final features are acquired from conv3.3 of the VGG-19 network. Training for the super-resolution and blur removal networks was performed for 30 epochs, respectively. This approach is part of video content optimization. Expressed as an expression, it can be expressed as the following expression.

[Equation 8]

는 특정 반복에서 손실 계산을 대체하는 이진 값이다. here,

is a binary value that replaces the loss calculation in a particular iteration.

The detailed super-resolution loss function can be expressed as the following equation.

[Equation 9]

는 VGG에서 추출한 특징이다.

는 노이즈를 줄이기 위해 포함된 총 분산 손실을 의미한다. Here, S' is the super-resolution version generated from the low-resolution and blurred subview image B. S is the ground truth of the super-resolution image, N is the number of pixels,

is a feature extracted from VGG.

is the total dispersion loss included to reduce noise.

In the same way, the loss function of the blur removal network can be defined as the following equation.

[Equation 10]

In the next step, only the perceptual loss of the blur removal network is utilized. During this phase, all variables are thoroughly backpropagated. 200 epochs are used to iterate over the generator and discriminator networks. The discriminator in the first 100 epochs takes the entire area of the predicted image as input. The objective function of this step is defined as an adversarial loss with a gradient penalty added as in the WGAN-GP approach. Therefore, the global area penalty can be expressed as the following equation.

[Equation 11]

와 C는 기울기와 critic 구별자 함수이다. where x represents the mixed distribution of predicted D and ground truth (G),

and C are the slope and critic discriminator functions.

The adversarial loss of the global area can be defined as the following equation.

[Equation 12]

Here, P _G and P _D are the distributions of the ground truth and predicted super-resolution deblurred images.

The next step enforces the similarity between the ground truth and the local domain. A binary mask M is created as the absolute difference between D and S, and the non-blurred area is set to 1 and the rest to 0. This mask is similar to the extracted light field map studied by Tao et al. (Non-Patent Document 10). The study was to differentiate the refocused lightfield image into a blurred area and a non-blurred area. The difference is that the map contains only values 0 and 1, so it is a simplified version. With this approach, unblurred regions are given priority to be processed in the generative adversarial network. In this procedure, Equation 11 can be modified as the following Equation.

[Equation 13]

Here, the input x and the final computation are processed using a mask M.

Masking the final calculation requires maintaining local gradient penalty consistency. Thus, another adversarial loss of using this area is added. Using the same discriminant C, the adversarial loss can be defined as the following equation.

[Equation 14]

The next 100 epochs iterate using the sum of the global and local adversarial losses. In general approach, the total objective function can be written as

[Equation 15]

는 상수 값 100이다.here,

is the constant value 100.

The experimental results are described below.

During training, each subview image (G) of the ground truth is randomly cropped into a patch with a size of 128 Х 128 pixels. d is set to 4, and B, which is a low resolution and blurred version, has a spatial size of 32Х32, and batch size is applied to 1, so that subview images are processed as random sequences. A total of 5 x 5 subview images are sampled from 5 sequential frames in a spiral order. All networks have a learning rate of 0.0001 and are trained using the ADAM optimizer for a total of 260 epochs. The first 60 epochs are used to train the SRNet and DeblurNet separately, the next 100 epochs are used to train the global region, and the last 100 epochs are used to combine the global and local regions of the discriminator. . The training took about 6 days using a GTX 1080 GPU.

Quantitative results

Experimental results were compared with publicly available state-of-the-art super-resolution and blur removal algorithms operating at 5 Х 5 angular resolution. The light field blur removal algorithm of Srinivasan et al. (Non-Patent Document 5) was included although spatial resolution was limited. Also, the recommended optimization-based algorithm, Krishnan et al. (Non-Patent Document 2) and Pan et al. (Non-patent document 3) technique was utilized. For the training-based case, Kim et al. The technique of (Non-Patent Document 4) is utilized. The 2D blur removal technique was applied to each subview image individually. In Table 1, the value of each item is measured by averaging all PSNR and SSIM in the 40 LF test set.

[Table 1]

The first and second best performing techniques are indicated by blue and red text, respectively. As shown in Table 1, the method including the refine strategy in the technique proposed in this example showed the highest score in terms of the overall resolution LF. This indicates that the proposed dataset and learning-based algorithm improve the light field image and perform better than the state-of-the-art algorithm. This was achieved due to an accurate geometry approximation when generating the dataset.

Qualitative Results

6 is a diagram illustrating a qualitative result image of an algorithm according to an exemplary embodiment.

In order to qualitatively observe the performance, as shown in Fig. 6, visual results were presented. Experimental results of L ^* and Refine techniques show sharply restored edges than other techniques. Although the experimental result of Refine method shows a higher metric score, in reality, ^{the result of L*} shows clearer visual results at both large and small edges. This is because reconstructed pixels with small edges are usually in un-deblurred regions. Therefore ^{, using L*} , this region is prioritized in super-resolution and blur-removing networks. Epipolar plane images (EPIs) are displayed below and next to each subview image of FIG. 6 to show light field coherence. Further results can be found on the qualitative results in the supplementary data.

To analyze the effect of optimization, an ablation study was performed by learning the network with the same hyperparameters while excluding the loss functions. To numerically compare the performance, the maximum signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics were used. The higher the PSNR and SSIM, the better the performance.

[Table 2]

As can be seen in Table 2, ^{SR + Deb, which depends only on L Con} , showed a weak result for learning the geometry of the 6-DOF model. In the case of Equations 11 and 12 (+ Glob) where the global region exists in the adversarial loss, the weighting performance is increased by 1.5 dB in the PSNR scale. Finally, in the case of the additional local loss Equations 13 and 14 (+ Loc), a result showing a higher PSNR is shown.

7 is a diagram illustrating a qualitative result image according to an exemplary embodiment.

This method is sensitive and in some cases, as shown in FIG. 7 , a jiggling noise was generated in the weak corner. Therefore, we use an improvement strategy that utilizes the weights trained on L ^{* using L Con optimization.} The improvement strategy is expressed as Refine in Table 2, and the result of the algorithm comparison experiment can be checked through Table 2.

As described above, the embodiments propose a specialized neural network-based light field enhancement framework to solve spatial super-resolution and motion blur removal of light field images. The network is fully-convolutional and designed in a recursive approach with the goal of operating at various resolutions. In addition, a low-resolution and parallax-consistent 6-DOF motion-blurred lightfield dataset was used for network training. Local adversarial and refinement strategies were introduced in the optimization procedure to improve performance. The proposed method showed superiority to other state-of-the-art methods in both quantitative and qualitative aspects.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (5)

In the adversarial neural network model apparatus for simultaneous performance of lightfield super-resolution and blur removal,
a super-resolution network that receives a subview image of a low-resolution or blurred light field and outputs a super-resolution reconstructed image; and
A blur removal network that receives the super-resolution reconstructed image and passes it through at least one or more convolution blocks to estimate the super-resolution and motion blur-removed image
including,
To provide an end-to-end network comprising the super-resolution network and the blur removal network,
The super-resolution network,
The subview image is passed while the super-resolution reconstructed image is output, and two convolution blocks each having filters of different sizes, a U-Net in which six encoders and five decoders are integrated, and the U- Contains convolution blocks through which the output of Net is passed,
The resolution of the reconstructed super-resolution image is downsampled to a d scale, and the subview image input to obtain additional information about the neighboring subview image is connected to at least neighboring images,
The blur removal network is
As the super-resolution reconstructed image is passed, the super-resolution and motion blur-removed image is estimated so that two convolution blocks, eight ResNet blocks, and output of the ResNet blocks each having filters of different sizes An adversarial neural network model device, comprising convolution blocks passed through. delete According to claim 1,
A discriminator network that trains the end-to-end network by dividing it into multiple steps to allow backpropagation of losses in super-resolution restoration and blur removal, and training with low-resolution and 6-DOF motion-blurred lightfield datasets
Further comprising, adversarial neural network model device. 4. The method of claim 3,
It provides a generative adversarial network (GAN) that processes into global and local regions of an image,
The discriminator network is
In the first step, the discriminator network is trained using the entire region of the super-resolution and motion blur-removed image predicted from the generator network, and in the second step, an additional local region is learned using the same discriminator network. to do
characterized in that, adversarial neural network model device. In the operating method of an adversarial neural network model device for simultaneous performance of light field super-resolution and blur removal performed through a computer device,
receiving a subview image of a low-resolution or blurred light field from a super-resolution network and outputting a super-resolution reconstructed image; and
estimating an image from which super-resolution and motion blur have been removed by receiving the super-resolution reconstructed image from a blur removal network and passing it through at least one or more convolution blocks
including,
To provide an end-to-end network comprising the super-resolution network and the blur removal network,
The super-resolution network,
The subview image is passed while the super-resolution reconstructed image is output, and two convolution blocks each having filters of different sizes, a U-Net in which six encoders and five decoders are integrated, and the U- Contains convolution blocks through which the output of Net is passed,
The resolution of the reconstructed super-resolution image is downsampled to a d scale, and the subview image input to obtain additional information about the neighboring subview image is connected to at least neighboring images,
The blur removal network is
As the super-resolution reconstructed image is passed, the super-resolution and motion blur-removed image is estimated so that two convolution blocks, eight ResNet blocks, and output of the ResNet blocks each having filters of different sizes A method of operation of an adversarial neural network model device, comprising the convolution blocks passed through.

Images