KR101996730B1 - Method and apparatus for reconstructing single image super-resolution based on artificial neural network - Google Patents

Abstract

A single image high resolution reconstruction method and system using artificial neural network is presented. A single image super-resolution method using an artificial neural network according to an embodiment includes a step of pre-processing an input image to provide a low resolution image and a self high-resolution image ; Reconstructing a residual image using predetermined learning data for the low-resolution image and the self-high-resolution image; And reconstructing the input image into a high-resolution image through an image reconstructed from the low-resolution image, the self-high-resolution image, and the residual image, wherein the low-resolution image and the self- Reconstructing the residual image may include restoring a high frequency component of the overall image structure of the low resolution image using a wide acceptance area and residual image learning based on the external exemplar image through the entire residual image restoration network; And reconstructing the high frequency component of the detailed local image structure repeatedly appearing in the self-high-resolution image based on the internal example image through the local residual image restoration network or distributed over the entire image, The image can be restored at a high resolution using the complementary relationship of the example images.

Description

[0001] METHOD AND APPARATUS FOR RECONSTRUCTING SINGLE IMAGE SUPER-RESOLUTION [0002] BASED ON ARTIFICIAL NEURAL NETWORK [

The following embodiments relate to a single image high resolution reconstruction method and system using an artificial neural network and more particularly to a method and system for reconstructing a single image based on a deep convolutional neural network that learns a mapping relationship between a low resolution image and a high resolution image And more particularly, to a method and system for reconstructing high-resolution images.

Recent development of memory integration technology and artificial intelligence research has emphasized the importance of image based computer vision technique research based on Big Data. However, these studies have been studied based on clear image acquisition, and involve the necessity of research on image quality deterioration due to mechanical limitations and environmental factors during image acquisition.

A high-resolution reconstruction technique of a single image has been studied to solve the limitations of resolution through an algorithm when the low-resolution image can not be obtained due to environmental or mechanical constraints. Although the recent high resolution reconstruction technology has proved its performance in the direction of new developments beyond the performance of existing technologies, the reconstruction technology based on external exemplar images using Convolutional Neural Network (CNN) And a high-resolution reconstruction result is distorted.

Korean Patent No. 10-1780057 discloses such a high resolution image restoration method and apparatus and describes a technique for restoring a high resolution image by sequentially applying a plurality of image processing layers to a low resolution image to restore a high resolution image .

 W. Shi, J. Caballero, F. Huszr, J. Totz, AP Aitken, R. Bishop, D. Rueckert, and Z. Wang, "An efficient sub-pixel convolutional neural network , "In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874-1883, July, 2016.  S. Schulter, C. Leistner, and H. Bischof, "Fast and accurate image upscaling with super-resolution forests," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791-3799, June, 2015.  C. Dong, C. Loy, K. He, and X. Tang, "Learning a deep convolutional network for image super-resolution," In Proc. of European Conference on Computer Vision, pp. 184-199, September, 2014.  J. Kim, J. K. Lee, and K. M. Lee, " Accurate image super-resolution using very deep convolutional networks, "In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646-1654, July, 2016.  D. Glasner, S. Bagon, and M. Irani, "Super-resolution from a single image," In Proc. IEEE International Conference on Computer Vision, pp. 349-356, September, 2009.  A. Singh and N. Ahuja, "Super-resolution using sub-band self-similarity," In Proc. of Asian Conference on Computer Vision, pp. 552-568, November, 2014.  J. Yang, Z. Lin, and S. Cohen, "Fast image super-resolution based on in-place example regression," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1059-1066, June, 2013.  J.-B. Huang, A. Singh, and N. Ahuja, "Single image super-resolution from transformed self-exemplars," In Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197-5206, June, 2015.  A. Vedaldi and K. Lenc, "MatConvnet-Convolutional Neural Networks for MATLAB," In Proc. of the ACM International Conference on Multimedia, pp. 689-692, October, 2015.  J.-B. Huang, S. B. Kang, N. Ahuja, and J. Kopf. "Image completion using planar structure guidance," ACM Trans. on Graphics, vol. 33, no. 4, pp. 129, July, 2014.  Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," In Proc. of the IEEE, vol. 86, no. 11, pp. 2278-2324,1998.  M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. Alberi-Morel, "Low-complexity single-image super-resolution based on nonnegative neighbor embedding," In Proc. of British Machine Vision Conference, pp. 1-10, September, 2012.  R. Zeyde, M. Elad, and M. Protter, "On single image scale-up using sparse representations," In Proc. of International Conference on Curves and Surfaces, pp. 711-730, June, 2010.  D. Martin, C. Fowlkes, D. Tal, and J. Malik, "A database of human segmented natural images and their application to evaluating segmentation methods and measuring ecological statistics," In Proc. IEEE International Conference on Computer Vision, pp. 416-423, July, 2001.  L. Sun and J. Hays, "Super-resolution from internet-scale scene matching," In Proc. IEEE International Conference on Computational Photography, pp. 1-12, April, 2012.  https://github.com/jbhuang0604/SelfExSR.

Embodiments describe a single image high resolution reconstruction method and system using an artificial neural network, and more specifically, a deep convolutional neural network based learning method that learns a mapping function between a low resolution image and a high resolution image And single image super-resolution (SISR) technology.

Embodiments use a mutually complementary relationship between external and internal examples-driven SISR (SISR) techniques based on external and internal example images to provide a single high resolution image using an artificial neural network having robust high resolution image restoration capability in various image structures And to provide a restoration method and system.

A single image super-resolution method using an artificial neural network according to an embodiment includes a step of pre-processing an input image to provide a low resolution image and a self high-resolution image ; Reconstructing a residual image using predetermined learning data for the low-resolution image and the self-high-resolution image; And reconstructing the input image into a high-resolution image through an image reconstructed from the low-resolution image, the self-high-resolution image, and the residual image, wherein the low-resolution image and the self- Reconstructing the residual image may include restoring a high frequency component of the overall image structure of the low resolution image using a wide acceptance area and residual image learning based on the external exemplar image through the entire residual image restoration network; And reconstructing the high frequency component of the detailed local image structure repeatedly appearing in the self-high-resolution image based on the internal example image through the local residual image restoration network or distributed over the entire image, The image can be restored at a high resolution using the complementary relationship of the example images.

The step of pre-processing the input image to provide a low-resolution image and a self-high-resolution image may include providing the low-resolution image interpolated through an interpolation technique as a guide image for maintaining an overall image structure, A method for reconstructing a self-reconstructed high-resolution image using a high-resolution reconstruction technique based on an exemplar image, the method comprising the steps of: reconstructing a deep-reconstructed image using a complementary relationship between the external exemplar image and an internal- The input image can be restored with high resolution based on a Convolutional Neural Network (CNN).

In a single image super-resolution system using an artificial neural network according to another embodiment, an input image is pre-processed to provide a low-resolution image and a self-high-resolution image Image supply; A residual image restoring unit for restoring the residual image using the preset learning data for the low resolution image and the self high resolution image; And a high-resolution reconstruction unit that reconstructs the input image into a high-resolution image through the low-resolution image, the self-high-resolution image, and the reconstructed residual image, wherein the residual image reconstructing unit reconstructs the wide- An entire residual image restoration network for restoring a high frequency component of the entire overall image structure of the low resolution image using image learning; And a local residual image restoration network for restoring the high frequency component of the detailed local image structure repeatedly appearing on the self-high-resolution image based on the internal example image or distributed over the entire image, wherein the internal exemplar image and the external exemplar image The image can be restored at a high resolution using the complementary relation.

Wherein the image providing unit provides the low-resolution image interpolated through the interpolation technique as a guide image for maintaining the overall image structure, and the high-resolution restoration technique using the internal example image-based high- And reconstructs the input image based on a deep convolutional neural network (CNN) using the complementary relationship between the external exemplar image and the internal exemplar image. .

Wherein the region residual image restoration network is composed of a plurality of pixel encoding layers composed of a convolutional layer and a ReLu (Rectified Linear Unit) function, and the output feature map includes the self-high resolution The pixel information of the image can be transmitted.

The entire remaining image restoration network may include a plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality of the plurality A residual block may be formed.

Wherein the entire residual image restoration network performs a process of learning a gradient value using a skip-connection and a batch normalization layer through the residual block, The input feature map passes through a plurality of convolutional layers, the number of output feature maps remains the same as the number of input feature maps, the predetermined size of coverage area includes N remaining blocks and finally the convolutional layer As the size increases, the entire residual image can be restored and the high-resolution image structure can be restored.

According to embodiments, a single convolutional neural network-based single image super-resolution (SISR) method and system that learns the mapping relationship between a low-resolution image and a high-resolution image can be provided.

According to the embodiments, by using the mutually complementary relation of the external and internal examples-driven SISR techniques based on the external and internal example images, a single image using artificial neural network having robust high resolution image restoration ability in various image structures It is possible to provide an image restoration method and system.

FIG. 1 shows an example of a reconstructed high-resolution image.
2A is a schematic diagram for explaining a deep CNN model according to an embodiment.
FIG. 2B is a schematic diagram illustrating an entire residual image restoration network of a deep CNN model according to an embodiment.
3 is a block diagram illustrating a single image high resolution reconstruction system using an artificial neural network according to an exemplary embodiment.
4A and 4B are flowcharts illustrating a single image high resolution restoration method using an artificial neural network according to an exemplary embodiment.
FIG. 5 is a diagram for explaining an analysis of a deep CNN model according to an embodiment.
FIG. 6 is a diagram illustrating a result of high resolution reconstruction of an image according to an exemplary embodiment.
FIG. 7 shows a 4x high resolution reconstruction result for the Urban100 data set according to an embodiment.
FIG. 8 shows an 8x high resolution reconstruction for a Sun-hays80 dataset according to one embodiment.
Figure 9 shows the high resolution reconstruction results for the Sun-hays80 dataset according to one embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the embodiments described may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more fully describe the present invention to those skilled in the art. The shape and size of elements in the drawings may be exaggerated for clarity.

According to the embodiments described below, a high-resolution reconstruction technique based on a deep convolutional neural network (CNN) can solve the high-resolution reconstruction problem by using the complementary relationship between the external and internal example images. Hereinafter, the advantages and disadvantages of the high-resolution reconstruction technology based on the external and internal example images are analyzed, respectively, and a new CNN-based high-resolution reconstruction technique using the same is provided. Here, deep convolutive neural network (CNN) is a kind of deep learning, which can mean an artificial neural network for understanding images in the field of image processing or extracting various information from images.

First, we analyze the strengths and weaknesses of the high resolution reconstruction technology based on the external image.

Over the past decade, CNN-based high-resolution restoration technologies (non-patent documents 1, 3 and 4) have attracted attention as new solutions to overcoming the limitations of existing technologies. The deeper CNN model - based restoration technique learned the more sophisticated nonlinear mapping relationship between low resolution and high resolution images and showed higher performance than simple CNN model based techniques. VDSR (Non-Patent Document 4) used a repetitive deep structure of a 3x3 convolutional layer and a ReLu function as an example of such a deep CNN model-based high-resolution reconstruction technique. In particular, instead of restoring the entire high-resolution image, the high-resolution image and the residual image of the interpolated low-resolution image are restored to focus on the high-frequency image structure of the high-resolution image. This enhances the acceptability of the CNN model, It showed the best performance. Therefore, in this embodiment, it is possible to analyze the high-resolution restoration technology based on the external exemplar image based on the restoration performance of the VDSR.

FIG. 1 shows an example of a reconstructed high-resolution image.

(B) shows a restoration result based on an internal example image (non-patent document 8), (c) shows an example of a restored An example image-based restoration result (non-patent document 4) is shown.

In the wide acceptance area through the deep CNN model, a wide area of low resolution area can be seen in order to restore one pixel of a high resolution image, so that a more sophisticated mapping relation can be learned. In addition, the deep CNN model, which is specialized for restoration of high-frequency image structure, can reconstruct a clear high-resolution image of the edge region, which is a high-frequency image structure (122).

However, the deep CNN model can reconstruct a high-resolution image based on a wide coverage area, but is still insufficient to replace the self-imaged image information of the whole area of the image. The reconstructed high-resolution image structure in the reconstruction of the high-resolution image has a limit of distortion (112).

Next, we analyze the advantages and disadvantages of the high-resolution reconstruction technology based on the internal example image.

According to the experiment of the study (Non-Patent Document 5), a small high resolution patch of a natural image appears repeatedly in various scale images. Most internal image-based high-resolution reconstruction techniques are devised based on such self-similarity characteristics, so that a more extended example of the authorship. Creating a high resolution patch space is an important part to improve the high-resolution image reconstruction performance. To this end, the SelfExSR (Non-Patent Document 8) recently proposed a method of expanding the patch space in consideration of perspective distortion based on a multi-plane (non-patent document 10) in the input image, Improvement. Therefore, in the present embodiment, it is possible to analyze the high-resolution restoration technology based on the internal example image based on the restoration performance of SelfExSR.

Based on the self similarity characteristic, a high resolution image patch can be searched to use the entire image as a search space rather than a limited high resolution image search space. This is because high-resolution images can be reconstructed based on rich high-resolution information for image structures repeatedly appearing or distributed throughout the image, thus exhibiting high performance for fine image reconstruction (121).

On the other hand, the high-resolution reconstruction technique based on the internal example image can restore a high-resolution image by incrementally increasing the resolution of the image based on the small high resolution patch that is searched. This small, high-resolution patch-based reconstruction method requires restoration by observing a small high-resolution patch that overlaps the overall high-resolution image structure larger than the patch, so that it is easy to reconstruct distorted high-resolution image structure and distorted high- .

This disadvantage causes step artifacts in the high-frequency image structure of the reconstructed high-resolution image (111).

As described above, although the high-resolution reconstruction technique based on the external exemplar image can clearly implement the overall high-frequency image structure based on the wide acceptance region and the residual image learning, the high-resolution reconstruction technique based on the internal exemplar image may cause the step- . On the contrary, the high-resolution reconstruction technique based on the internal example image has excellent performance for the detailed image structure repeatedly or distributed over the entire image, whereas the high-resolution reconstruction technique based on the external exemplar image is easy to reconstruct the distorted image structure.

2A is a schematic diagram for explaining a deep CNN model according to an embodiment.

Referring to FIG. 2A, a deep CNN model according to an exemplary embodiment may be formed of a partial network of two parallel deep CNN structures. The partial network of the deep CNN structure is a global residual network, and the other partial network is a local residual network.

First, the entire residual image restoration network can recover a high frequency component of a large image structure by utilizing the wide receiving area of the deep CNN model. In addition, the local residual image restoration network can recover high frequency components of a clear image structure that is encoded without interference of peripheral pixels by utilizing three 1x1 convolutional layers of high resolution information based on the internal example image. This will be described in more detail below.

)을 전반적인 영상구조를 유지하기 위한 가이드 영상으로, 내부 예시영상 기반의 고해상도 복원 기술을 이용하여 복원한 자가 고해상도(Self High-Resolution: Self HR) 영상(

)을 지역적 영상구조의 세밀한 부분을 복원하기 위해 이용할 수 있다.The input of the deep CNN model according to one embodiment is an interpolated low-resolution (Interpolated LR) image interpolated with a Bicubic interpolation technique

) Is a guide image for maintaining the overall image structure, and the self-high-resolution (Self High-Resolution) image (Self HR) image

) Can be used to reconstruct the fine detail of the local image structure.

를 이용하여 입력영상

에 대하여 잔여영상을 복원하는 깊은 CNN 모델(

)을 다음 식을 최적화하여 학습할 수 있다. Given learning data

The input image

Deep CNN model to reconstruct the residual image

) Can be learned by optimizing the following equation.

[Formula 1]

The local residual image restoration network aims to transmit local image information of self high resolution image. For example, the local residual image restoration network may be composed of three pixel encoding layers composed of a 1x1 convolution layer and a ReLu (Rectified Linear Unit) function based on the research (Non-Patent Document 7). Mathematically, the output feature map of the i- th pixel encoding layer can be expressed by the following equation.

[Formula 2]

Here, F _i denotes an output feature map of the i-th pixel encoding layer, and W _i and B _i can denote a convolution filter and a bias of the i- th pixel encoding layer. The output feature map can convey the pixel information of the localized high-resolution image that is encoded without interfering with surrounding pixel information.

The entire residual image restoration network aims at restoring distortion-free high-resolution image structure by complementing the distorted overall image structure by the step distortion of self-high resolution image. The entire residual image restoration network can be composed of a series connection of several residual blocks to have a wide acceptance area based on the analysis of the high resolution restoration technology based on the preceding external image.

FIG. 2B is a schematic diagram illustrating an entire residual image restoration network of a deep CNN model according to an embodiment.

Referring to FIG. 2B, the entire residual image restoration network may include a plurality of residual blocks connected in series to have a wide coverage area. Here, the residual block can be steadily learned by using a skip-connection and batch normalization (BN) layer 202. FIG. Through each residual block, the input feature map passes through two 3x3 convolutional layers 201 and 204, and the number of output feature maps can be maintained at 64 equal to the number of input feature maps. The 3x3 receiving area can be expanded to (4N + 5) x (4N + 5) through the N remaining blocks and the last 3x3 convolutional layer 222 to restore the high resolution image structure.

Hereinafter, a single image high resolution restoration system and method using an artificial neural network according to an embodiment will be described.

3 is a block diagram illustrating a single image high resolution reconstruction system using an artificial neural network according to an exemplary embodiment.

Referring to FIG. 3, a single image super-resolution system 300 using an artificial neural network according to an exemplary embodiment includes an image providing unit 310, a residual image restoring unit 320, And a high-resolution reconstruction unit 330. FIG. Here, the single image high resolution reconstruction system 300 using the artificial neural network according to an exemplary embodiment may be a deep CNN model or at least one part of a deep CNN model according to the embodiment illustrated in FIGS. 2A and 2B.

The image providing unit 310 may pre-process the input image to provide a low-resolution image and a self-high-resolution image.

The image providing unit 310 provides the guide image for maintaining the overall image structure of the low resolution image interpolated through the interpolation technique, and provides the high resolution image restored by the internal example image based high resolution reconstruction technique to the local image structure Can be provided to restore details.

The residual image restoration unit 320 may restore the residual image using the preset learning data for the low resolution image and the self high resolution image.

Here, the residual image restoration unit 320 may include an entire residual image restoration network 331 and a region residual image restoration network 332. [

The entire residual image restoration network 331 can restore the high frequency components of the overall image structure using the low resolution image based on the wide acceptance area and the residual image learning based on the external exemplar image.

The entire residual image restoration network 331 may include a plurality of serial-connected image restoration areas 331, 332, 333, and 333 having a wide reception area based on the external image to compensate for the distortion of the entire image structure distorted by the step distortion of the self- A residual block may be formed.

More specifically, the entire residual image restoration network 331 can perform a stable gradient value learning process using the skip connection and the group normalization layer through the remaining blocks. Through each residual block, the input feature map passes through a plurality of convolutional layers, and the number of output feature maps can remain the same as the number of input feature maps. In addition, the entire remaining image restoration network 331 reconstructs the entire residual image as the size of the entire remaining image restoration network 331 is expanded through the N remaining blocks and finally the convolutional layer, and restores the high resolution image structure . For example, the entire residual image restoration network 331 passes through two remaining 3x3 convolutional layers 201 and 204 through the respective remaining blocks, and the number of output feature maps becomes equal to the number of input feature maps It can be kept at 64 in the same manner. The 3x3 receiving area can be expanded to (4N + 5) x (4N + 5) through the N remaining blocks and the last 3x3 convolutional layer 222 to restore the high resolution image structure.

The local residual image restoration network 332 can restore a high-frequency component of a fine local image structure repeatedly appearing in the self-high-resolution image based on the internal example image or distributed over the entire image.

More specifically, the region residual image restoration network 332 may be composed of a plurality of pixel encoding layers made up of a convolutional layer and a ReLu function. At this time, the output feature map can transmit the pixel information of the self-encoded high-resolution image without interference of neighboring pixel information. For example, the local residual image restoration network 332 uses high-resolution information based on the internal example image using three 1x1 convolutional layers to generate a high-frequency component of a clear small image structure encoded without intervention of peripheral pixels Can be restored.

The high-resolution reconstruction unit 330 reconstructs a low-resolution image, a high-resolution image, and a reconstructed residual image into a high-resolution image.

According to embodiments, a high-resolution reconstruction of a deep convoluted neural network (CNN) -based image can be performed using the complementary relationship between the external exemplar image and the internal exemplar image.

4A and 4B are flowcharts illustrating a single image high resolution restoration method using an artificial neural network according to an exemplary embodiment.

4A and 4B, a single image high resolution reconstruction method using an artificial neural network according to an exemplary embodiment includes a step 410 of providing a low resolution image and a high resolution image by pre-processing an input image, a low resolution image and a high resolution image (420) reconstructing a residual image using preset learning data and reconstructing a high-resolution image into a high-resolution image through a reconstructed low-resolution image, a self-high-resolution image, and a residual image . It is possible to reconstruct the image with high resolution by using the complementary relationship between the internal example image and the external example image.

In operation 420, restoring the residual image using the preset learning data for the low-resolution image and the self-high-resolution image may include using a wide acceptance region and residual image learning based on the external exemplar image through the entire residual image restoration network (421) reconstructing a high-frequency component of the entire overall image structure of the low-resolution image, and reconstructing the high-resolution local image of the low-resolution image repeatedly or through the local residual image restoration network, And recovering the high-frequency component of the structure (step 422).

Here, the single image high resolution reconstruction method using the artificial neural network according to one embodiment can be explained more concretely through the single image high resolution reconstruction system 300 using the artificial neural network according to the embodiment shown in FIG. The single image high resolution reconstruction system 300 using the artificial neural network according to an exemplary embodiment may include an image providing unit 310, a residual image restoring unit 320, and a high resolution restoring unit 330. Here, the residual image restoration unit 320 may include an entire residual image restoration network 331 and a region residual image restoration network 332. [

In operation 410, the image providing unit 310 may process the input image to provide a low-resolution image and a self-high-resolution image.

More specifically, the image data providing unit 310 provides the low-resolution image interpolated through the interpolation technology as a guide image for maintaining the overall image structure, and the high-resolution image reconstructed using the internal example image-based high- Can be provided to reconstruct the details of the local image structure.

In operation 420, the residual image restoration unit 320 may restore the residual image using the preset training data for the low-resolution image and the self-high-resolution image.

More specifically, in step 421, a high-frequency component of the overall image structure is restored using a low-resolution image based on the wide acceptance area and the residual image learning based on the external exemplar image based on the entire remaining image restoration network 331 .

Also, in step 422, the high-frequency component of the detailed local image structure repeatedly appearing on the internal example-image-based self-high-resolution image repeatedly or distributed over the entire image can be restored through the local residual image restoration network 332.

In operation 430, the high-resolution reconstruction unit 330 reconstructs a low-resolution image, a self-high-resolution image, and a reconstructed residual image into a high-resolution image.

Accordingly, it is possible to reconstruct a high-resolution image based on a deep convolutive neural network (CNN) using the complementary relationship between the external exemplar image and the internal exemplar image.

FIG. 5 is a diagram for explaining an analysis of a deep CNN model according to an embodiment.

Figure 5 is a block diagram (A) shows a Bicubic interpolation image, (b) shows a residual residual image restoration network result, (c) shows a result of a multiple input VDSR model, (d) shows the result using the deep CNN model according to one embodiment. Here, the deep CNN model can represent a single image high resolution reconstruction system or method using an artificial neural network, and is simply an image high resolution reconstruction technique.

In order to analyze the contribution of individual partial networks for reconstructing a high-resolution image, a high-resolution reconstruction using only the region-residual image restoration network can be performed on the same input image. In the case of a repetitive high-resolution image structure region, the local residual image restoration network can remove the ambiguity of the local small image structure of (a) through the transmission of the encoded pixel information of the self-high resolution image (b). As a result, the entire residual image restoration network can perform high frequency component restoration of the image structure for a clearer image.

The image high-resolution reconstruction technique according to an embodiment through cooperation of the two partial networks can be achieved by using a reconstruction technique based on the result of the reconstruction technique based on an individual external or internal example image without image structure distortion and step- A clear high-resolution image can be restored.

Additionally, a VDSR model of a multiple input layer may be considered as an alternative to the deep CNN model according to one embodiment (c).

FIG. 6 is a diagram illustrating a result of high resolution reconstruction of an image according to an exemplary embodiment.

Referring to FIG. 6, (a) shows an input image, (b) shows a result of a multiple input VDSR model, and (c) shows a result using a deep CNN model according to an embodiment.

As a result of comparing the results using the multi-input VDSR model and the deep CNN model according to an embodiment, the VDSR model of the multi-input layer can show a performance improvement with respect to a small image structure. However, Has the disadvantage of learning the disadvantages of image-based high-resolution technology equally.

On the other hand, in the case of using the deep CNN model according to an embodiment, the performance is improved not only for the small image structure but also for the edge area of the high frequency image structure.

Below One example of a learning method of a deep CNN model according to an embodiment will be described in more detail.

For example, the training data of the deep CNN model according to one embodiment is composed of a small image patch of 291 natural images of the study (Non-Patent Document 2), and the number of the training images is increased by applying inversion and rotation transformation to the patch . Self-high resolution images for deep CNN model input are restored using SelfExSR, and parameters can be used as default values of the study (Non-Patent Document 8). In order to study deep CNN model with various resolution restoration conversion factors, it is possible to use restored learning data by Bicubic interpolation technique after reducing to 2, 3 and 4 times resolution. The learning uses a min-batch gradient descent based on back-propagation (non-patent document 11), where the population size is 64, the momentum is 0.9 and the L2 weight The L2-weighted decay parameter can be set to 0.0001. Also, gradient clipping (non-patent document 4) technique can be used for fast learning speed. The initial learning rate is set to 0.01 and can be reduced by 0.1 for every 10 learning epochs. The learning rate is fixed after 40 learning epochs and can be stopped after 60 learning epochs.

Hereinafter, performance comparison and analysis of various high resolution image restoration techniques and image high resolution restoration techniques according to an embodiment will be performed.

For example, the experimental environment is an Intel Core i5 CPU (3.2 GHz), 12 GB RAM and an NVIDIA Titan GPU, and various natural image data sets (non-patent documents 12, 13 and 14) and an urban image data set We use Sun-Hays80 (non-patent document 15) for quantitative evaluation of 8x high resolution reconstruction conversion factor.

In order to benchmark the VDSR, the residual block of the overall residual image restoration network is set to 9, so that it has the same receiving area of 41x41, and only a single luminance image can be restored instead of the RGB three channel image. The high-resolution restoration results for the analysis (non-patent documents 5, 6, 8) can use the benchmark results of Huang's web page (non-patent document 16). The high-resolution image restoration technique according to one embodiment can be implemented using the MatConvNet Deep Learning Toolbox (Non-Patent Document 9), and the learning time can take about 10 hours. Further, performance enhancement experiments of various high resolution image restoration techniques based on the internal example images can be performed using the image high resolution restoration technique according to the embodiment.

Qualitative comparison evaluation

FIG. 7 shows a 4x high resolution reconstruction result for the Urban100 data set according to an embodiment.

More specifically, FIG. 7A is a VDSR (non-patent document 4), FIG. 7B is a SelfExSR (non-patent document 8), FIG. 7C is an image high resolution restoration technique according to an embodiment, ) Represents the original image. This is effective for evaluating the performance of high resolution reconstruction techniques based on road external and internal example images having repetitive high resolution image structures.

As shown in FIG. 7A, in the case of VDSR, a restoration technique specialized for a high-frequency image structure shows a vivid restoration result in a large high-resolution image structure of an edge portion. However, sufficient high-resolution information can not be obtained for a local small-resolution image structure, and thus a distorted restoration result is displayed.

As shown in FIG. 7B, in the case of SelfExSR, sufficient high resolution information is acquired through a self-illustrated high resolution patch space for a small high resolution image structure to restore a clear high resolution image. On the other hand, a small-sized high-resolution patch-based reconstruction method causes a staircase distortion phenomenon for a large high-resolution image structure.

As shown in FIG. 7C, in the case of the image high-resolution restoration technique according to the embodiment, the merit of the simple restoration technique is supplemented, and the performance of the restoration technique is improved by utilizing the merits of each restoration technique. .

Furthermore, a high resolution reconstruction experiment can be performed on an 8x resolution reconstruction conversion factor. Since the image high resolution reconstruction technique according to the embodiment does not perform learning on the resolution reconstruction conversion factor of 4 times or more, the stepwise high resolution image restoration method of SelfExSR can be used. The image high resolution reconstruction technique according to the embodiment can improve the intermediate result by applying the intermediate result of 2, 4 and 8 times to the restored high resolution image of SelfExSR. In a similar approach, VDSR performs a total of three high-resolution reconstructions for a 2x resolution conversion factor to restore 8x high-resolution images.

FIG. 8 shows an 8x high resolution reconstruction for a Sun-hays80 dataset according to one embodiment.

More specifically, FIG. 8A shows a VDSR (non-patent document 4), FIG. 8B shows a SelfExSR (non-patent document 8), FIG. 8C shows an image high- , (d) show the original image.

8, it can be seen that the image reconstruction technique according to the embodiment of the present invention restores a clear fence shape compared to the existing techniques (Non-Patent Documents 4 and 8) in the case of restoring an 8 times higher resolution image. The high resolution reconstruction results for the Sun-hays80 dataset according to a further embodiment are shown in FIG.

Quantitative comparison evaluation

It is possible to perform quantitative comparison evaluation on high resolution reconstruction techniques based on various external exemplar images (non-patent documents 3 and 4) and internal example images (non-patent documents 5, 6 and 8).

Table 1 shows the quantitative comparison evaluation result (PSNR / SSIM) for Set 5 (non-patent document 12), Set 14 (non-patent document 13), BSD 100 (non-patent document 14) and Urban 100 (non-patent document 8) data sets.

[Table 1]

In Table 1, the highest quantitative figure is indicated in red, and the second highest quantitative figure is indicated in blue.

According to one embodiment, the image high resolution restoration technique has a finer quantitative value than the VDSR which shows the best performance among the existing technologies for the natural image data set. However, for the Urban 100 data set having many repetitive high-resolution image structures, the image high-resolution restoration technique according to one embodiment shows a PSNR increase width (0.3 dB) larger than the VDSR.

Furthermore, without re-learning of the deep CNN model according to an embodiment, it is possible to use a self-high-resolution image based on existing internal example image-based high-resolution restoration techniques (non-patent documents 5 and 6) An improvement experiment can be performed.

As a result, it can be seen that the performance of existing restoration techniques using the image high resolution restoration technique according to one embodiment is higher than the existing performance by a high quantitative value which is improved by 1 dB or more.

According to embodiments, a new high-resolution reconstruction technique using CNN based on an example-driven SISR using an exemplary image among various existing high-resolution reconstruction techniques can be provided. The techniques according to these embodiments utilize the complementary relationship between the external and internal example-based SISR (external and internal example-based SISR) to overcome the drawbacks of the existing external or internal example image-based high resolution reconstruction techniques And enhances the merits of each technique to restore an enhanced high-resolution image.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing apparatus may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (7)

In a single image super-resolution method using an artificial neural network,
Processing the input image to provide a low-resolution image and a self-high-resolution image;
Reconstructing a residual image using predetermined learning data for the low-resolution image and the self-high-resolution image; And
And reconstructing the input image into a high-resolution image through the reconstructed low-resolution image, the self-high-resolution image, and the residual image
Lt; / RTI >
And restoring the residual image using the preset learning data for the low-resolution image and the self-high-
A high frequency component of the entire overall image structure of the low resolution image is restored using a wide acceptance area and residual image learning through a plurality of residual block structures connected in series based on the external exemplar image through the entire residual image restoration network step; And
The local residual image restoration network is used to display the self-high-resolution image based on the internal example image repeatedly using three 1x1 convolutional layers so as to be encoded without interference of neighboring pixels, or a detailed local Restoring the high-frequency component of the image structure
Lt; / RTI >
Reconstructing a high-resolution image using the complementary relationship between the internal exemplar image and the external exemplar image,
Processing the input image to provide a low-resolution image and a self-high-resolution image,
The interpolation technique provides the low resolution image as a guide image for maintaining the overall image structure and restores the detailed high resolution image of the local image structure using the high resolution reconstruction technique based on the internal example image , ≪ / RTI >
The high-resolution reconstruction of the input image based on a deep convolutional neural network (CNN) using the complementary relationship between the external exemplar image and the internal exemplar image
A single image high resolution reconstruction method using an artificial neural network. delete In a single image super-resolution system using an artificial neural network,
An image providing unit for providing a low-resolution image and a self-high-resolution image by preprocessing the input image;
A residual image restoring unit for restoring the residual image using the preset learning data for the low resolution image and the self high resolution image; And
And a high-resolution reconstruction unit that reconstructs the input image into a high-resolution image through the reconstructed low-resolution image, the self-high-
Lt; / RTI >
The residual image reconstructing unit reconstructs the residual image,
An entire residual image restoration network for restoring a high frequency component of an entire overall image structure of the low resolution image using a wide acceptance area and residual image learning through a plurality of residual block structures connected in series based on an external exemplar image; And
The high-frequency component of the detailed local image structure that appears repeatedly or distributed all over the image is restored by utilizing three 1x1 convolutional layers so that the self-high resolution image based on the internal example image is encoded without interference of peripheral pixels. Local residual image restoration network
Lt; / RTI >
Reconstructing a high-resolution image using the complementary relationship between the internal exemplar image and the external exemplar image,
The image providing unit,
The interpolation technique provides the low resolution image as a guide image for maintaining the overall image structure and restores the detailed high resolution image of the local image structure using the high resolution reconstruction technique based on the internal example image , ≪ / RTI >
The high-resolution reconstruction of the input image based on a deep convolutional neural network (CNN) using the complementary relationship between the external exemplar image and the internal exemplar image
A Single Image High Resolution Restoration System Using Artificial Neural Network. delete The method of claim 3,
The area residual image restoration network includes:
And a plurality of pixel encoding layers each of which is composed of a convolutional layer and a ReLu (Rectified Linear Unit) function. The output feature map includes pixel information of the self-high-resolution image encoded without interfering with neighboring pixel information
A Single Image High Resolution Restoration System Using Artificial Neural Network. The method of claim 3,
Wherein the entire residual image restoration network comprises:
A plurality of the residual blocks connected in series so as to have a wide receiving area based on the external exemplar image are formed so as to recover the distortionless high-resolution image structure by complementing the overall overall image structure distorted by the step-distortion phenomenon of the self-high- Being
A Single Image High Resolution Restoration System Using Artificial Neural Network. The method according to claim 6,
Wherein the entire residual image restoration network comprises:
A gradient gradient value learning process is performed using a skip-connection and a batch normalization layer through the residual block, and an input feature map is classified into a plurality of consecutive blocks The number of output feature maps remains the same as the number of input feature maps, and the acceptance area of a predetermined size passes through the N remaining blocks and finally the convolutional layer, Restoring the residual image to restore the high-resolution image structure
A Single Image High Resolution Restoration System Using Artificial Neural Network.

Images