KR101829287B1 - Nonsubsampled Contourlet Transform Based Infrared Image Super-Resolution - Google Patents

Abstract

The present invention relates to a method and an apparatus for infrared image super-resolution transform based on nonsubsampled contourlet transform (NSCT), which transform a low resolution infrared (LR IR) image into high resolution infrared (HR IR) image through an NSCT method by using a sparse dictionary and a residual dictionary. According to the present invention, the method comprises: (A) a step of a raw sparse dictionary generation module receiving training LR and HR IR image sets to form a training high frequency map; (B) a step of the raw sparse dictionary generation module performing the NSCT of the training LR IR image to form a training LR coefficient block; (C) a step of the raw sparse dictionary generation module forming a training HR coefficient block from the training high frequency map; (D) a step of the raw sparse dictionary generation module using the training LR coefficient block to form a raw LR sparse dictionary and using the training HR coefficient block to form a raw HR sparse dictionary; (E) a step of a residual sparse dictionary generation module to form residual HR and LR sparse dictionaries; (F) a step of a test module receiving a test LR IR image to form a test LR coefficient block; and (G) a step of the test module calculating the formed HR coefficient block to form an estimated HR IR image.

Description

[0001] Nonsubsampled Contourlet Transform Based Infrared Image Super-Resolution [0002]

The present invention relates to an infrared image super-resolution method and apparatus for converting a low-resolution infrared image to a high resolution by using a non-sample-to-image contour transformation method using a sparse dictionary and a residual dictionary.

An infrared (IR) thermal imaging camera can generate an infrared image that is not affected by visible light by thermal radiation of the scene. Thus, IR images can be applied to security surveillance, biomedical applications, military applications, and so on.

However, due to hardware limitations, the resolution of infrared images is very low. This low-resolution (LR) IR image can lead to failure when performing further analysis.

The Super-Resolution (SR) method is a technique for reconstructing high-resolution (HR) images through a single LR image or multiple LR images.

This ultra-high resolution method is one of the best solutions to improve the resolution of LR IR images. There are many super-resolution methods such as interpolation [1,2,3], reconstruction-based SR [4,5,6,7,8], and learning-based SR methods [9,10,11,12,13,14,15] . Generally, an SR image using interpolation lacks high-frequency details, has very smooth edges and a lot of unwanted artifacts.

To reconstruct an HR image in a reconstruction-based SR method, an LR image sequence of the same scene is required. Reconstruction based SR methods are limited due to low magnification. In the learning-based SR method, an HR image is generated in the corresponding LR image using a training set that includes a pair of LR and HR images.

Freeman et al. Proposed a method of learning the relationship between the LR image patch and the corresponding HR image patch [16] by learning the Markov network. Zeyde et al. Proposed upscale image sparse representation modeling [17]. Although this approach has good performance, it is very complex and requires a lot of memory.

So far, little research has been done on the SR of IR images. Yu et al. Proposed an algorithm based on a non-local method and a manipulated kernel regression.

Zhao et al. Proposed an SR technique that combines nonsampled contour transform (NSCT) and parabolic error edge preservation interpolation [19].

Wang et al. [10] consider robust ultra-high-resolution performance in simulated infrared images [20].

The above method improves performance, but has several drawbacks. First, the trained dictionary is an unstructured dictionary. This can lead to worse results. In addition, differential filters are employed to extract image features. However, these features are too simple to express effectively. Finally, one HR-LR dictionary pair is adopted to infer the HR IR image. However, one HR-LR dictionary pair is not enough to get good results.

[1] Lei Zhang and Xiaolin Wu, "An Edge-Guided Image Interpolation Algorithm via Directional Filtering and Data Fusion", IEEE Transaction on Image Processing (2006), Vol.15, No.8, pp.2226-2238. [2] Xiangjun Zhang and Xiaolin Wu, "Image Interpolation by Adaptive 2-D Autogressive Modeling and Soft Decision Estimation", IEEE Transaction on Image Processing (2008), Vol.17, No.6, pp.887-896. [3] Kai Guo, Xiaokang Yang, Hongyuan Zha, Weiyao Lin and Songyu Yu, "Multiscale Semilocal Interpolation with Antialiasing", IEEE Transaction on Image Processing (2012), Vol.21, No.2, pp. 615-625. [4] J. D. van Ouwerkerk, "Image super-resolution survey ", Image and Vision Computing (2006), Vol.24, No.10, pp.1039-1052. [5] SP Kim, NK Bose and HMValenzuela, "Recursive Reconstruction of High-Resolution Image from Noisy Under Sampled Multiframes", IEEE Transactions on Acoustics, Speech and Signal Processing (1990), Vol. 38, No. 6, pp. 1013-1027. [6] P. Vandewalle, S. Susstrunk and M. Vetterli, "A Frequency Domain Approach to Registration of Aliased Images with Application to Super-Resolution", EURASIP Journal on Applied Signal Processing (2006), pp. 1-14. [7] H. Stark, E. Oskoui, "High-resolution image recovery from image-plane arrays using Convex Projections", Journal of the Optical Society of America (1989), Vol.6, No.11, -1726. [8] A. J. Patti, M. I. Sezan and A. M. Tekalp, "Superresolution Video Reconstruction with Arbitrary Sampling Lattices and Nonzero Aperture Time", IEEE Transactions on Image Processing (1997), Vol.6, No.8, pp. [9] S. Baker and T. Kanade, "Limits on Super-Resolution and How to Break Them", IEEE Transaction on Pattern Analysis and Machine Intelligence (2002), Vol.24, No.9, pp.1167-1183. [10] W. T. Freeman, T. R. Jones and E. C. Pasztor, "Example-Based Super-Resolution", IEEE Computer Graphics and Applications (2002), Vol. 56-65. [11] Jianchao Yang, John Wright, Yi Ma and Thomas Huang, "Image Super-Resolution as Sparse Representation of Raw Image Patches", IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, June 23-28, pp.1-8. [12] Jianchao Yang, John Wright, Thomas Huang and Yi Ma, "Image Super-Resolution via Sparse Representation", IEEE Transaction on Image Processing (2010), Vol.19, No.11, pp.2861-2873. [13] Weisheng Dong, Lei Zhang, Rastislav Lukac, and Guangming Shi "Sparse Representation Based Image Interpolation with Nonlocal Autoregressive Modeling", IEEE Transaction on Image Processing (2013), Vol.22, No.4, pp.1382-1394 [14] Weisheng Dong, Guangming Shi and Xin Li "Nonlocal Image Restoration with Bilateral Variance Estimation: A Low-Rank Approach", IEEE Transactions on Image Processing (2013), Vol.22, No.2, pp.700-711. [15] Gwanggil Jeon, "Directionally Categorized Training Process for Hybrid Super-Resolution Algorithm", International Journal of Multimedia and Ubiquitous Engineering (2014), Vol.9, No.3, pp.107-116. [16] W. T. Freeman and Ce Liu, Markov random fields for super-resolution and texture Synthesis, Advances in Markov Random Fields for Vision and Image Processing, Chapter 10. MIT Press, 2011. [17] R. Zeyde, M. Elad and M. Protter, On single image scale-up using sparse-representations, Curves and Surfaces, Lecture Notes in Computer Science Volume 6920, 2012, pp. 711-730. [18] Hui Yu, Fusheng Chen, Zhijie Zhang and Chensheng Wang, "Single infrared image super-resolution combining non-local means with kernel regression", Infrared Physics & Technology, Vol.61, November 2013, pp.50-59. [19] Zhao Gang, Zhang Kai, Shao Wei and Yan Jie, "A Study on NSCT based Super Resolution Reconstruction for Infrared Image", TENCON 2013-2013 IEEE Region 10 Conference (2013), pp.1-5 [20] Jing Wang, Jason F. Ralph and John Goulermas, "An Analysis of a Robust Super Resolution Algorithm for Infrared Imaging", Proceedings of 6th International Symposium on Image and Signal Processing and Anal ... , pp.158-163. [21] Haicang Liu, Shutao Li and Haitao Yin, "Infrared surveillance image super resolution via group sparse representation", Optics Communication (2013), Vol.289, pp. 45-52. [22] M. Aharon, M. Eladan and A. M. Bruckstein, "K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation", IEEE Transactions on Signal Processing (2006), Vol.54, No.11, pp .4311-4233. [23] M. N. Do and M. Vetterli, "The Contourlet Transform: An Efficient Directional Multiresolution Image Representation", IEEE Transactions on Image Processing (2005), Vol.14, No.12, pp. 2091-2106. [24] Arthur L. da Cunha, Jianping Zhou and Minh N. Do, "The Nonsubsampled Contourlet Transform: Theory, Design and Applications", IEEE Transaction on Image Processing (2006), Vol.15, No.10, pp.3089 -3101. [25] Qiang Zhang and Baolong Guo, "Multifocus image fusion using the nonsubsampled contourlet transform", Signal Processing (2009), Vol. 89, No. 7, pp.1334-1346. [26] Huafeng Li, Yi Chai and Zhaofei Li, "Multi-focus image fusion based on nonsubsampled contourlet transform and focus regions detection", Optik-International Journal of Light and Electron Optics (2013), Vol.124.No.1, pp.40-51. [27] Ying Li, Jie Hu and Yu Jia, "Automatic SAR image enhancement based on nonsubsampled contourlet transform and memetic algorithm", Neurocomputing (2014), Vol.134, 25, pp.70-78. [28] Shuyuan Yang, Min Wang, Yan Xiong Lu, Weidong Qi and Licheng Jiao, "Fusion of multiparametric SAR images based on SW-nonsubsampled contourlet and PCNN", Signal Processing (2009), Vol.89, No.12, pp. 2596-2608. [29] Rosa Rodriguez-Sanchez, J.A. Garcia and J. Fdez-Valdivia, "Image inpainting with nonsubsampled contourlet transform ", Pattern Recognition Letters (2013), Vol.34, No.13, pp.1508-1518. [30] R. Rubinstein, M. Zibulevsky and M. Elad, "Double Sparsity: Learning Sparse Dictionaries for Sparse Signal Approximation", IEEE Transactions on Signal Processing (2010), Vol.58, No.3, pp.1553-1564 . Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, (1990) March 19-19, Phoenix, AZ, Vol. 5, pp. 2443-2446. [32] Wei Wu and Chenglin Zheng, "Single image super-resolution using self-similarity and generalized nonlocal mean ", TENCON 2013-2013 IEEE Region 10 Conference, (October 2013) October 22-25, Xi'n, China, pp. 1-4. [33] Yingying Zhang, Wei Wu, Yong Dai, Xiaomin Yang, Binyu Yan and Wei Lu, "Remote Sensing Image Super Resolution Based on Sparse Dictionaries and Residual Dictionaries", 2013 IEEE International Conference on Dependable, Autonomic and Secure Computing, (2013) December 21-22: Chengdu, China, pp.318-323 IEEE Transactions on Multimedia (2010), "A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference," by Yong-Sik Lee, Sang-ki Lv, Yanpeng Lv, Guobing Wu, Peng Peng, Fei Chen and Xufa Wang ), Vol. 12, No. 7, pp. 682-691. [35] "flir.com (1999-2013) FLIR Systems, Inc", [Online] Available: http://www.cvs.flir.com/l/6132/2011-08-02/1ARLRpi_ad_id=35322332185&gclid=COa2uNTZxccCFUEljgodZpQGTA . [36] Adrian Burlacu and Corneliu Lazar, "Image Feature Detection using Phase Congruency and Its Application in Visual Servoing", 2008 IEEE 4th International Conference on Intelligent Computer Comunication and Processing, (2010) August 29-30: Cluj-Napoca, pp. 47-52. [37] Simona E. Grigoresu, Nocolai Petkov and Peter Kruizinga, "Comparison of Texture Features Based on Gabor Filters", IEEE Transaction on Image Processing (2002), Vol.11, No.10, pp. 1160-1167.

SUMMARY OF THE INVENTION The present invention has been conceived to solve the above problems, and it is an object of the present invention to solve the problem that a dictionary prepared by introducing a rare dictionary into an IR image SR is an unstructured dictionary.

Further, the present invention aims to solve the problem that the differential filter is too simple to effectively display an image by using non-part sample contour transformation.

The present invention also solves the problem that one HR-LR dictionary pair is not sufficient to obtain good results by employing two pairs of HR-LR sparse dictionaries composed of a pair of primitive sparse dictionaries and a pair of remaining sparse dictionaries.

According to an aspect of the present invention, there is provided a method for generating training high frequency images, comprising the steps of: (A) receiving a training LR IR image set from a primitive rare dictionary generation module; (B) the primitive sparse pregeneration module transforms a training LR IR image into a training low resolution coefficient block to form a training low resolution coefficient block; (C) the primitive sparse pregeneration module forms training high resolution coefficient blocks in a training high frequency map; (D) the primitive spurious dictionary generation module forms a primitive low resolution sparse dictionary using training low resolution coefficient blocks, and generates a primitive high resolution sparse dictionary using the training high resolution coefficient blocks; (E) the remaining sparse dictionary creation module comprises: forming a remaining high resolution sparse dictionary and a remaining low resolution sparse dictionary; (F) receiving a test LR IR image from a test module to form a test low resolution coefficient block; And (G) the test module comprises generating a test high resolution coefficient block formed to form an estimated test HR IR image.

The step (A) of one aspect of the present invention comprises the steps of: (A-1) receiving the training LR IR image set and receiving training HR IR image set; (A-2) the primitive sparse pregeneration module expands the training LR IR image to form an expanded training LR IR image; And (A-3) forming the training high-frequency map, wherein the primitive spurious dictionary generation module is a difference between an expanded training LR IR image and a training HR IR image.

Further, in the step (A-2) of the aspect of the present invention, the primitive sparse pregeneration module enlarges the training LR IR image using bikigabic interpolation to form an enlarged training LR IR image.

The step (B) of one aspect of the present invention is characterized in that (B-1) the primitive sparse pregeneration module transforms the enlarged training LR IR image into a training feature vector low frequency map and a plurality of training features Generating a vector high frequency map; (B-2) the primitive spurious dictionary generation module performs block division of training feature vector low-frequency maps and a plurality of training feature vector high-frequency maps to form training low-resolution overlapping vectors for each; And (B-3) the primitive sparse pregeneration module combining training low resolution overlap vectors to generate training low resolution feature vectors to form training low resolution coefficient blocks.

Further, in the step (B-2) of one aspect of the present invention, the detail information of the training feature vector low-frequency map is extracted using the Gaussian difference filter before the elementary sparse dictionary generation module performs block division.

In addition, in the step (E) of the present invention, the residual sparse dictionary generation module forms a magnified estimated training LR IR image with reference to a raw low-scarcity dictionary, and refers to a raw high- Forming an estimated training HR IR image; (E-2) said residual sparse pre-production module using said estimated training HR IR image to form a residual high-resolution sparse dictionary; (E-3) the residual sparse pre-production module is configured to form a training residual IR image using a training HR IR image and an estimated training HR IR image; And (E-4) the residual sparse pre-production module comprises forming a residual low-sparse sparse dictionary using the formed training residual IR image.

The step (F) of one aspect of the present invention includes the steps of: (F-1) receiving the test LR IR image by the test module to form an enlarged test LR IR image; (F-2) generating a test feature vector low-frequency map and a plurality of test feature vector high-frequency maps by performing a non-part sample contour transformation on the enlarged test LR IR image by the test module; (F-3) the test module performs block division on one test feature vector low-frequency map and a plurality of test feature vector high-frequency maps to form a test low-resolution overlapping vector field for each test feature vector low-frequency map; And (F-4) Combining the low resolution overlapping vectors to form a test low resolution feature vector to form a test low resolution coefficient block.

The step (G) of one aspect of the present invention comprises the steps of: (G-1) calculating a rare expression by referring to a raw low-scarce dictionary from a formed test low-resolution coefficient block; (G-2) the test module calculates a test high-resolution coefficient block by referring to a raw high-resolution scarce dictionary to form an estimated test raw HR IR image; (G-3) The test module forms an estimated test residual IR image in the estimated test raw HR IR image with reference to the remaining high-resolution sparse dictionary and the remaining low-resolution sparse dictionary; And (G-4) the test module comprises forming an estimated test HR IR image using the estimated test raw HR IR image and the estimated test residual IR image.

In another aspect of the present invention, there is provided a method of generating training high frequency images, comprising the steps of: (A) receiving a training LR IR image set from a primitive rare dictionary generation module; (B) the primitive sparse pregeneration module transforms a training LR IR image into a training low resolution coefficient block to form a training low resolution coefficient block; (C) the primitive sparse pregeneration module forms training high resolution coefficient blocks in a training high frequency map; (D) the primitive spurious dictionary generation module forms a primitive low resolution sparse dictionary using training low resolution coefficient blocks, and generates a primitive high resolution sparse dictionary using the training high resolution coefficient blocks; (E) the remaining sparse dictionary creation module comprises: forming a remaining high resolution sparse dictionary and a remaining low resolution sparse dictionary; (F) receiving a test LR IR image from a test module to form a test low resolution coefficient block; And (G) the test module generates a test high resolution coefficient block that is formed to form an estimated test HR IR image, and provides a computer readable recording medium on which an infrared image ultra high resolution method of non-part sample contour transformation is recorded .

In another aspect of the present invention, a training LR IR image set is input, a training HR IR image set is received, a training high-frequency map is formed, and a training LR IR image is transformed into a non-specimen contour line to form training low resolution coefficient blocks A primitive sparse pregeneration module that forms a training high resolution coefficient block in a training high frequency map, forms a primitive low resolution sparse dictionary using training low resolution coefficient blocks, and generates a primitive high resolution sparse dictionary using training high resolution coefficient blocks; A residual sparse dictionary generation module that forms a residual high-resolution sparse dictionary and a residual low-resolution sparse dictionary; And a test LR IR image to generate a test low resolution coefficient block and a test high resolution coefficient block to form a test HR IR image by referring to the raw high resolution sparse dictionary, the raw low resolution sparse dictionary, the remaining high resolution sparse dictionary, and the remaining low resolution sparse dictionary Test modules.

According to another aspect of the present invention, there is provided a primitive sparse dictionary generation module comprising: an interpolation part for receiving training LR IR images and enlarging the training LR IR images to generate enlarged training LR IR images; A training IR image, which is the difference between the training HR IR image and the expanded training LR IR image; a training high frequency map forming unit, which forms a high frequency map; A non-specimen contour transformation unit for performing non-specimen contour transformation on an enlarged training LR IR image to form a training feature vector low-frequency map and a plurality of training feature vector high-frequency maps; A block dividing unit dividing the plurality of maps into blocks; A low resolution coefficient block calculator for transforming the divided blocks into training low resolution overlap vectors to collect low resolution overlap vectors to form training low resolution coefficient blocks; A high resolution coefficient block calculator for training high resolution coefficient blocks from training IR image high frequency maps; A low-resolution sparse dictionary portion forming a raw low-resolution sparse dictionary in the training low-resolution coefficient block; And a high-resolution sparse dictionary that forms a raw high-resolution sparse dictionary from a raw low-resolution sparse dictionary and a training high-resolution coefficient block.

In another aspect of the present invention, the interpolator receives training LR IR images and generates enlarged training LR IR images by using a bicubic interpolation method.

Yet another aspect of the present invention further includes a Gaussian difference filter for extracting details of the training feature vector low-frequency map before the block segmentation section performs block segmentation.

In yet another aspect of the present invention, the remaining sparse dictionary generation module comprises a low resolution estimator for forming an estimated training LR image magnified using a raw high resolution sparse dictionary and a raw low resolution sparse dictionary; A high resolution estimator for forming an estimated training raw HR IR image; A residual image forming unit for obtaining a residual IR image that is a difference between the estimated training raw HR IR image and the raw HR IR image; The remaining low-resolution sparse dictionary portion forming the remaining low-resolution sparse dictionary using the training HR IR image and the training residual IR image calculated using the estimated training HR IR image; And a residual high-resolution dictionary portion that forms a residual high-resolution sparse dictionary using the estimated training HR IR image.

According to another aspect of the present invention, the test module includes an interpolator receiving a test LR IR image and forming an enlarged test LR IR image using a bicubic interpolation method; A non-specimen contour transformer for forming a test low resolution coefficient block of non-specimen contour transformation by performing non-specimen contour transformation on an enlarged test LR IR image; Testing a non-specimen contour transformation with reference to a primitive low-resolution sparse dictionary; obtaining a rare representation from the low-resolution coefficient block; A high resolution coefficient block calculator for forming an estimation test high resolution coefficient block by referring to a raw high resolution scarce dictionary; A test high frequency map forming unit using the estimated test high resolution coefficient block to form a high frequency map of the estimated test IR image; Integrating the expanded test IR image with a high frequency map of the test IR image to obtain a raw HR IR image; A residual image estimator for estimating a residual IR image using a residual high-resolution sparse dictionary and a residual low-resolution sparse dictionary; And a high resolution image forming unit for obtaining an estimated test HR IR image by integrating the estimated test raw HR IR image and the estimated residual IR image.

The present invention solves the problem of introducing a sparse dictionary into an IR image SR, which is a trained dictionary unstructured dictionary.

In addition, the present invention solves the problem that the differential filter is too simple to effectively display an image using non-part sample contour transformation.

The present invention also solves the problems of the prior art that one HR-LR dictionary pair is not sufficient to obtain good results by employing two pairs of HR-LR sparse dictionaries composed of a pair of primitive sparse dictionaries and a pair of remaining sparse dictionaries.

1 is a conceptual diagram for explaining a non-part sample contour transformation applied to the present invention.
FIG. 2 is a block diagram of an infrared image super-resolution method for non-part sample contour transformation according to a preferred embodiment of the present invention.
FIG. 3 is a view showing the training process of FIG. 2. FIG.
FIG. 4A is a diagram illustrating a process of forming the feature vector of FIG. 3, and FIG. 4B is a diagram illustrating a combination process.
FIG. 5 is a diagram showing the test procedure of FIG. 2. FIG.
6 is an exemplary view of a training IR image having a natural scene.
7 is an illustration of a training IR image having a face.
Fig. 8 is a part of a test IR image, which shows a building IR image, a building B IR image, a building C IR image, (d) a face IR image, (e) a face B IR image , (f) a face C IR image.
Fig. 9 shows the experimental results using images. Fig. 9 (a) shows the results obtained by Yang's method, and (e) The result obtained by the K-SVD method, (f) the results obtained by Zhang's method, and (g) the results obtained by the proposed method.
Fig. 10 shows the results of experiments using images. Fig. 10 (a) shows the results obtained by Yang's method, and (e) The result obtained by the K-SVD method, (f) the results obtained by Zhang's method, and (g) the results obtained by the proposed method.
Figure 11 shows the effect on SR quality at the NSCT level, (a) PSNR at different NSCT levels, and (b) SSIM at different NSCT levels.
12 shows the influence of the number of directions of the NSCT on the SR quality, (a) PSNR of different directions of NSCT, and (b) SSIM of different directions of NSCT.
Fig. 13 shows the effect on the SR quality of the rare dictionary size and shows (a) a PSNR of another sparse dictionary size and (b) another sparse dictionary size SSIM.
Fig. 14 shows the effect of the low-frequency feature extraction method on the SR quality and shows the SSIM under (a) PSNR by another method on the extracted feature, and (b) by other methods on the extracted feature.
FIG. 15 is a configuration diagram of an infrared image super-resolution device for non-part sample contour transformation according to a preferred embodiment of the present invention.
FIG. 16 is a detailed configuration diagram of the primitive spurious dictionary generation module and the residual spurious dictionary generation module of FIG. 15; FIG.
17 is a detailed configuration diagram of the test module of Fig.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.

First, the terminology used in the present application is used only to describe a specific embodiment, and is not intended to limit the present invention, and the singular expressions may include plural expressions unless the context clearly indicates otherwise. Also, in this application, the terms "comprise", "having", and the like are intended to specify that there are stated features, integers, steps, operations, elements, parts or combinations thereof, But do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

The present invention solves the problem of introducing a sparse dictionary into an IR image SR, which is a trained dictionary unstructured dictionary. Sparse dictionaries are based on the sparse model of dictionary atoms for the basic dictionaries.

The non-specimen contour transformation (NSCT) has full displacement constant, multi-scale, and multiple directions to enrich the IR image function, avoid frequency aliasing, and preserve edge information completely. Thus, the non-sample-to-image contour transformation is used to solve the problem that the derivative filter is too simple to effectively represent the image.

Finally, two HR-LR sparse dictionary pairs consisting of a primitive sparse dictionary pair and a remaining sparse dictionary pair are employed. This solves the problem that one HR-LR dictionary pair is not enough to get good results.

The raw estimated HR IR image is first estimated as a primitive sparse dictionary pair. Subsequently, a similar process is applied to the raw estimation result with the remaining sparse dictionary pairs. Better results can be obtained with two pairs of scarce dictionaries.

In the following, we examine the decomposition of the non-specimen contour transformation, give details of the proposed method, and compare the experimental results with other SR reconstruction methods.

Contour transform (CT) has recently been introduced and widely used for image recognition, image noise reduction, and image enhancement.

The contour transformation (CT) obtains multi-scale decomposition and direction decomposition using a Laplacian pyramid (LP) and a directional filter bank (DFB), respectively.

Contour transformations can spontaneously represent natural images and capture outline information in all directions and at all scales.

Compared to a wavelet transform, the contour transform is an actual 2D discrete transform. Because of the downsampling and upsampling of LP and DFB, contour transformation lacks shift-invariance.

To overcome this problem, the nonparametric sample contour transformation (NSCT) proposed by Cunha et al. Is a shift variant version of contour transformation (CT) [24].

The non-specimen contour transformation (NSCT) [25, 26, 27, 28, 29] avoids frequency aliasing of the contour transformation (CT) and removes the down-sampler and up-sampler in the decomposition and reconstruction of the image to achieve shift invariance .

1 is a conceptual diagram for explaining a non-part sample contour transformation applied to the present invention.

Referring to FIG. 1, the non-part sample contour transform applied to the present invention is composed of two filter banks of a nonsubsampled pyramid (NSP) and a nonsubsampl-e-directional filter bank (NSDFB).

The non-subsampling pyramid decomposes the input image into high-frequency and low-frequency subbands at each non-subsampling pyramid resolution level.

The non-portion sampling directional filter bank decomposes the high frequency subbands into several direction subbands.

After the n-level decomposition, the non-sampling pyramid forms one low-frequency image and n high-frequency images having the same size as the input image.

It is the non-part sampling directional filter bank (NSDFB) that allows stepwise directional decomposition in the high-frequency images of the non-sampling pyramids at each scale.

의 고주파 이미지를 가지게 되며, 여기에서 k_i는 i 스케일의 방향 개수를 나타낸다.In this process, you can obtain a ^2k direction subimage with the same size as the input image. From these results, we can see that one low-

, Where k _i represents the number of directions of the i-scale.

Such non-specimen contour transformation was adopted because of its excellent properties of shift invariant, multiscale, and multi-directional.

FIG. 2 is a block diagram of an infrared image super-resolution method for non-part sample contour transformation according to a preferred embodiment of the present invention.

Referring to FIG. 2, an infrared image super-resolution method of non-part sample contour transformation according to a preferred embodiment of the present invention includes two steps of training step (S100) and test step (S200).

The training step 100 receives the training LR IR image set from the primitive sparse dictionary generation module (S110), and receives the training HR IR image set (S120).

Then, the primitive sparse dictionary creation module expands the training LR IR image to form a training high frequency map, which is the difference between the expanded training LR IR image and the training HR IR image (S130).

In addition, the primitive sparse dictionary generation module generates a training feature vector low-frequency map and a plurality of training feature vector high-frequency maps by transforming the expanded training LR IR image to a non-sample pattern, A training low resolution lapped vector is formed for each of a map and a plurality of training feature vector high frequency maps, and a training low resolution feature vector is formed by combining these training low resolution feature vectors to form a training low resolution feature block having training low resolution feature vectors as a set.

In addition, the primitive spurious dictionary generation module calculates the training high-resolution coefficient block in the training high-frequency map (S140).

The primitive spurious dictionary generation module forms a primitive low resolution sparse dictionary using the training low resolution coefficient block and generates a primitive high resolution sparse dictionary using the training high resolution coefficient block (S150).

On the other hand, the residual sparse dictionary generation module forms an estimated training LR IR image with reference to the raw low-scarce sparse dictionary, and forms an estimated training HR IR image with reference to the raw high-resolution sparse dictionary.

The remaining sparse dictionary generation module then forms a residual high-resolution sparse dictionary using the estimated training HR IR image and generates a residual IR image of the training formed after the training residual IR image is formed using the training HR IR image and the estimated training HR IR image To form a remaining low-resolution sparse dictionary (S160).

Meanwhile, in the test step, the test module receives the test LR IR image (S210), forms a magnified test LR IR image, and then performs non-part sample contour transformation to obtain a test feature low frequency map and a plurality of test feature vector high frequency After the map is generated, block division is performed to form a test low resolution overlapping vector for each of a single test feature vector low frequency map and a plurality of test feature vector high frequency maps, and a test low resolution feature vector is generated by combining the test low resolution feature vectors. (S220). ≪ / RTI >

The test module calculates a rare expression from the formed test low resolution coefficient block by referring to the raw low-scarcity dictionary, and then generates a test high-resolution coefficient block by referring to the raw high-resolution scarce dictionary to form an estimated test raw HR IR image.

The test module then forms an estimated test residual IR image from the estimated test raw HR IR image with reference to the remaining high resolution scarce dictionary and the remaining low resolution scarce dictionaries.

The test module forms an estimated test HR IR image using the estimated test raw HR IR image and the estimated test residual IR image (S230).

 In this training phase, two pairs of HR - LR sparse dictionaries are obtained from the training image set. In the test phase, the test LR IR image generates HR IR images with the help of two HR-LR sparse dictionary pairs obtained in the training phase.

The completed dictionary was used for image processing. However, the atoms used in advance have some rare structures based on more fundamental dictionaries. The existing completed dictionary can be formulated as an optimization problem of (Equation 1).

        (1)

Where B is an image coefficient block, and the coefficient vector phi is a scarce representation of B. t is the sparse tolerance of φ _i .

However, the dictionary D generated in this way is not structured [30]. This can lead to worse results. In order to overcome this problem, a rare dictionary D is used in the present invention.

Sparse dictionary D has a rare representation of the basic dictionary. In particular, the dictionary consists of two parts: a basic dictionary η and a sparse matrix representation C.

The sparse dictionary D is expressed by the following equation (2).

(2)

D = eta x C

Here, η is a basic dictionary with good regularity. C is an atomic representation matrix with good flexibility. A complete discrete cosine transform (DCT) dictionary, and a wavelet dictionary can be used as basic dictionaries.

In the present invention, a complete DCT dictionary is selected as a basic dictionary. Therefore, the sparse dictionary can be calculated as shown in Equation (3).

(3)

Where B is an image coefficient block, and the coefficient vector phi is a scarce representation of B. t is the sparse tolerance of φ _i , and s is the sparse tolerance of C.

 The process of obtaining a sparse dictionary can be divided into two procedures.

1) a procedure for sparse-coding the image coefficient block B, and 2) a procedure for updating the dictionary atom. In the step of solving the rare expression, any signal sparsely decomposing method can be used. In the present invention, an Orthogonal Matching Pursuit (OMP) method is used for sparse coding and a sparse K-SVD algorithm is used for pre-training.

In the step of updating the dictionary atom, every time only one atom is updated, that is, another atom is fixed to solve the optimal solution for the target equation. In the present invention, a rare K-SVD algorithm is used to solve Equation (3).

The primitive HR-LR sparse dictionary pair learns how to estimate the raw HR IR image from a single LR IR image input.

There is some gap between the estimated raw HR IR image and its corresponding reference HR IR image.

For convenience, the difference between the estimated HR IR image and the reference HR IR image may be referred to as a residual IR image. For better results, residual IR images should be estimated using residual sparse dictionaries.

For better results, residual IR images should be estimated using residual sparse dictionaries. Thus, the residual HR-LR sparse dictionary pair is introduced in the present invention to estimate the residual IR image. The sparse dictionary learning process is illustrated in FIG.

The framework of the sparse dictionary learning process is shown in FIG. The steps of the pre-learning process are as follows.

1) The primitive sparse dictionary generation module receives training LR IR images I _l ^m (S110) and expands it to Bicubic interpolation to generate expanded training LR IR images I _{l '} ^m (where m is 1, 2, 3, ..., M) (S141). The training IR image high frequency map I _hf ^{m is} then obtained, which is the difference between the training HR IR image and the corresponding expanded training LR IR image [32] (S130).

2) Extrapolate each image I _{l '} ^m into one level pyramid by performing a non-specimen contour transformation on the expanded training LR IR image.

As a result, one training feature vector low frequency map C _{llf '} ^m and four training feature vector high frequency maps C _lhf1' ^m , C _{lhf2 '} ^m , C _lhf3' ^m and C _{lhf4 '} ^m are obtained (S142).

4A is a framework showing a process of forming a low-resolution feature vector.

Referring to Figure 4a Gaussian difference: and using (DOG Difference of Gaussian) filter to _extract, the details of the training feature vector ^m low frequency map C _{llf 'training} feature vectors map the low frequency C _llf to ^m (S143).

5 map ^{_{C llf 'm, C lhf1'}} m, C lhf2 'm, C lhf3' m, C lhf4 'm is divided into blocks (S144) bibu sample outline conversion (NSCT) training low resolution coefficient blocks B = {b _l ^mk } (S145).

To do this, we obtain a block that overlaps the size of n * n in the five maps C _{llf '} ^m , C _lhf1' ^m , C _{lhf2 '} ^m , C _lhf3' ^m and C _{lhf4 '} ^m .

Thus, all the blocks are low-resolution training gyeopchip vector ^{_{B llf 'mk, B lhf1'}} mk, B lhf2 'mk, B lhf3' mk, B lhf4 ' by ^{mk (k = 1,2,3, ...,} K) are converted , Collecting low resolution overlap vectors from other maps to form a training low resolution feature vector b _l ^mk (where K is the total number of blocks in the map). The set of k = 1,2,3, ..., K of training low resolution feature vectors b _l ^mk is called the training low resolution coefficient block B = {b _l ^mk }. Then, the training IR image high frequency map I _hf ^m constitutes a training high resolution coefficient block B = {b _h ^mk } where n * n pixels overlap.

This will be described with reference to FIG. 4B. After block division, when abcd is formed for one map and efgh is formed for another map, the combination abcdefgh is formed.

3) Through Equation (4), we can obtain the primitive low-resolution sparse dictionary C _l and the rare expression φ.

(4)

Here, η is the DCT basic dictionary, and φ is a rare expression of b _l ^k . t is the signal sparse threshold, and s is the element sparse threshold. A raw high-resolution sparse dictionary C _h can be obtained from the raw low-resolution sparse dictionary C _l and the training high-resolution coefficient block (step S 150) using ηC _h = b _h ^k φ ^T (φφ ^T ) ^-1 .

4) Finally, the residual sparse dictionary generation module can estimate the magnified estimated training LR image I _{l '} ^m using the primitive sparse dictionaries C _l and C _h and obtain the estimated training primitive HR IR image I _ho (S161).

Hereinafter, the estimation method will be described in detail.

The remaining sparse dictionary generation module obtains the residual IR image which is the difference between the estimated training raw HR IR image and the raw HR IR image (S162).

The remaining sparse dictionary generation module then uses the estimated training HR IR image to form the remaining high resolution sparse dictionaries.

In addition, the remaining sparse dictionary generation module forms the remaining low-resolution sparse dictionary using the training residual IR image calculated using the training HR IR image and the estimated training HR IR image (S163).

The remaining HR sparse dictionary C _rh And the residual LR sparse dictionary C _rl can be trained through the estimated raw HR IR image and the residual IR image. The method used to obtain the primitive sparse dictionaries C _l and C _h described above is the same for the remaining HR sparse dictionaries C _rh And the residual LR sparse dictionary C _rl .

On the other hand, the high-resolution feature vector b _h ^k = η C _h α _i . Resolution feature vector, low-resolution tracing vector, or the same rare expression, finding the rare expression? _I is expressed by Equation (5) below.

(5)

F is a feature extraction operator, and? Represents an error tolerance degree. The optimization problem in equation (5) is NP-hard.

We can solve this problem by minimizing the l ^l norm and obtain the rare expression α _i using orthogonal matching tracing (OMP) as shown in Equation (6).

(6)

FIG. 5 illustrates a process of forming a HR IR image from a test LR IR image according to an exemplary embodiment of the present invention. Referring to FIG.

Referring to FIG. 5, a process of forming a high resolution image from a test IR image according to a preferred embodiment of the present invention comprises the following steps: 1) a test module receives a test LR IR image I _Ti (S210) To form an enlarged test LR IR image I _{Ti '} (S221).

Then, after enlarging the test LR IR image by non-part sample contour transformation to form four test feature vector high-frequency maps and one test feature vector low-frequency map (S122), the test low resolution coefficient The block B = {b _l ^k } (where k = 1, 2, ..., K) is obtained (S222).

2) Subsequently, the test module refers to the raw low-resolution sparse dictionary and uses the OMP of Equation 6 to test the low resolution coefficient block B = {b _l ^k } (k = 1,2, ..., K} to obtain the rare expression {alpha _i }.

Then, the estimated high-resolution coefficient block b _h ^{k *} is formed by referring to the raw high-resolution scarce dictionary using b _h ^{k *} = η C _h α _i, and using the estimated test high-resolution coefficient block, the high-frequency map I _hf ^* is obtained (S231).

3) Then, the magnified test IR image I _{Ti '} is integrated with the high frequency map I _hf ^* of the estimated test IR image to obtain an estimated test raw HR IR image (S232).

4) Similarly, the residual residual IR image I _r can be estimated using the residual rare HR-LR dictionary pair C _rh , C _rl (S233).

5) Finally, the estimated test raw HR IR image I _ho and the estimated residual IR image I _r are integrated to obtain the estimated test HR IR image I _h (S234).

Now, the present invention is compared with other methods such as bicubic interpolation, nearest neighbor interpolation, Yang's method, KSVD method, and Zhang's method [33].

The IR images used in the experiment consist of two categories: face images and natural landscape photographs.

While photographing natural landscape photographs, the facial images were taken from the USTC-NVIE database [34]. FLIR Tau 320 [35] is used to capture natural landscape images.

For the face image of the IR, select 50 HR IR images with the same size 228 * 300.

The training image is randomly selected from 34 HR in the 50 HR IR image and is selected to be used as the input for the test in the remaining 16 HR IR images. For natural scene IR images, 75 HR IR images are captured.

The IR image consists of two parts: a training sample and a test sample. The training sample contains 62 IR images and the test sample contains 13 IR images. The natural scene IR image is the same size 240 * 348. The HR IR image is downsampled by three times to construct the LR IR image. Some training images are shown in Figure 6 and Figure 7.

In the experiment, a patch of size 9 * 9 is used for the NSCT coefficient map and the HR IR image. And this patch overlaps 3 pixels between adjacent blocks. In each experiment, the size of the rare dictionary is 1024. Then divide the image into four directions. The DoG filter is used to extract the detail information from the low frequency map into a feature vector map. Evaluate the results of various methods visually and qualitatively in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Measure (SSIM).

Fig. 8 is a part of a test IR image, which shows a building IR image, a building B IR image, a building C IR image, (d) a face IR image, (e) a face B IR image , (f) a face C IR image.

Fig. 9 shows the experimental results using images. Fig. 9 (a) shows the results obtained by Yang's method, and (e) The result obtained by the K-SVD method, (f) the results obtained by Zhang's method, and (g) the results obtained by the proposed method.

Fig. 10 shows the results of experiments using images. Fig. 10 (a) shows the results obtained by Yang's method, and (e) The result obtained by the K-SVD method, (f) the results obtained by Zhang's method, and (g) the results obtained by the proposed method.

9 (a) and 10 (a) are original IR images of the natural scene image and the face image, respectively. Figs. 9 (b-g) and 10 (b-g) are IR images reconstructed by the nearest interpolation method, Bicubic interpolation method, Yang method, KSVD method, Zhang method and proposed method respectively. The three columns to the right of Figures 9 (a-g) and 10 (a-g) are enlarged versions of the displayed areas of the image in the three columns of light. Details of the results of the proposed method are as follows.

9 (b) to 10 (c) and 10 (b) to 10 (c) show that there is no high frequency component, no image detail, and a blurred portion appears. The reconstruction results of Yang, KSVD, Zhang and the proposed method are relatively clear, but reconstructed images of Yang, KSVD, Zhang and proposed method can reconstruct the image details.

In addition, the reconstruction of the image edge texture by KSVD method is improved compared with Yang 's method. It can be seen that the algorithm proposed in Figures 9-10 is most effective and the recovered detail is most similar to the reference HR IR image.

Table 1 shows that the proposed method achieves the highest PSNR and SSIM to prove superiority over the other methods.

(Table 1)

Next, examine the effect of the parameters on the results. The main parameters of the proposed method consist of four parameters: the number of NSCTs, the number of directions of NSCTs, the sparse dictionary size, and the level of the low frequency feature extraction method.

Figure 11 shows the effect on SR quality at the NSCT level, (a) PSNR at different NSCT levels, and (b) SSIM at different NSCT levels.

Figure 11 thus shows the effect of the level number of the NSCT on the results. It can be seen in Fig. 11 that one level is the best result. As the level number increases, the result is getting worse. The reason for this phenomenon is that noise may be introduced when the number of levels increases.

12 shows the influence of the number of directions of the NSCT on the SR quality, (a) PSNR of different directions of NSCT, and (b) SSIM of different directions of NSCT.

Figure 12 shows the effect of direction numbers on the results. If there is only one direction in the first level, the PSNR and SSIM are the lowest. As the direction number increases, PSNR and SSIM gradually increase. PSNR and SSIM tend to be stable when the number of directions increases. As the number of directions increases, the performance of the proposed method will be improved.

Fig. 13 shows the effect on the SR quality of the rare dictionary size and shows (a) a PSNR of another sparse dictionary size and (b) another sparse dictionary size SSIM.

Figure 13 shows the effect of sparse dictionary size on the results. Educate the dictionaries of size 256, 512, 1024 and 1200 and apply them to the same input image.

In FIG. 13, it can be seen that PSNR and SSIM tend to increase slowly when the size of the rare dictionary increases. Therefore, we conclude that the larger the dictionary dictionary size, the better the SR results.

Fig. 14 shows the effect of the low-frequency feature extraction method on the SR quality and shows the SSIM under (a) PSNR by another method on the extracted feature, and (b) by other methods on the extracted feature.

Fig. 14 shows the effect on the result of the feature extraction method for the low-frequency map. The phase integration filter [36], the Gabor filter [37], the gradient filter [12], and the DoG filter [32] are used to extract features of the low frequency map, respectively. It can be seen from Fig. 14 that the DoG filter is the best.

FIG. 15 is a configuration diagram of an infrared image super-resolution device for non-part sample contour transformation according to a preferred embodiment of the present invention.

Referring to FIG. 15, an infrared image super-resolution apparatus for non-part sample contour transformation according to an exemplary embodiment of the present invention includes a primitive sparse dictionary generation module 100, a residual sparse dictionary generation module 200, and a test module 300 Consists of.

The primitive sparse dictionary generation module 100 receives a training LR IR image set and receives a training HR IR image set.

The primitive sparse dictionary generation module 100 then expands the training LR IR image to form a training high frequency map that is the difference between the expanded training LR IR image and the training HR IR image.

In addition, the primitive sparse dictionary generation module 100 generates a training feature vector low-frequency map and a plurality of training feature vector high-frequency maps by transforming the enlarged training LR IR image to non- A training low resolution feature vector is generated for each feature low frequency map and a plurality of training feature vector high frequency maps, and a training low resolution feature vector is formed by combining the training low resolution overlap vector to form a training low resolution feature block having training low resolution feature vectors as a set.

In addition, the primitive sparse dictionary generation module 100 produces training high resolution coefficient blocks in the training high frequency map.

The primitive sparse dictionary generation module 100 forms a primitive low resolution sparse dictionary using training low resolution coefficient blocks and generates a primitive high resolution sparse dictionary using training high resolution coefficient blocks.

Meanwhile, the residual sparse dictionary generation module 200 forms an estimated training LR IR image with reference to the raw low-scarcity dictionary, and forms an estimated training HR IR image with reference to the raw high-resolution sparse dictionary.

Then, the residual sparse dictionary generation module 200 forms a residual high-resolution sparse dictionary using the estimated training HR IR image, and generates a training residual IR image using the training HR IR image and the estimated training HR IR image, The remaining IR images are used to form the remaining low-resolution sparse dictionaries.

Meanwhile, the test module 300 receives the test LR IR image, forms an enlarged test LR IR image, and then performs non-part sample contour transformation to generate a test feature low frequency map and a plurality of test feature vector high frequency maps Resolution block is divided into blocks to form a test low resolution overlapping vector for each of a test feature vector low-frequency map and a plurality of test feature vector high-frequency maps, and a test low-resolution feature vector is generated by combining the test low- To form a test low resolution coefficient block.

The test module 300 calculates a rare expression by referring to a raw low-scarc dictionary from a formed test low-resolution coefficient block, and then calculates a test high-resolution coefficient block by referring to the raw high-resolution scarce dictionary to form an estimated test raw HR IR image.

The test module 300 then forms an estimated test residual IR image from the estimated test raw HR IR image with reference to the remaining high resolution sparse dictionary and the remaining low resolution sparse dictionaries.

The test module 300 forms an estimated test HR IR image using the estimated test raw HR IR image and the estimated test residual IR image).

 Thus, two HR - LR sparse dictionary pairs are obtained from the training image set. Test LR IR images generate HR IR images with the help of two HR-LR sparse dictionary pairs obtained during the training phase.

FIG. 16 is a detailed configuration diagram of the primitive spurious dictionary generation module and the residual spurious dictionary generation module of FIG. 15; FIG.

15, the primitive spurious dictionary generation module of FIG. 15 includes an interpolation section 101, a training high-frequency map forming section 102, a non-sample contour transformation section 103, a Gaussian difference filter 104, Resolution coefficient block calculating unit 106, a high-resolution coefficient block calculating unit 107, a low-resolution sparse dictionary unit 108, and a high-resolution sparse dictionary unit 109. The low-

The remaining sparse dictionary generation module of Fig. 15 includes a low resolution estimation section 201, a high resolution estimation section 202, a residual image formation section 203, a residual low resolution dictionary section 204 and a remaining high resolution dictionary section 205 .

In this configuration, the interpolator 101 receives the training LR IR images I _l ^m (S110) and enlarges it with a bicubic interpolation method to generate expanded training LR IR images I _{l '} ^m (here, , M is 1, 2, 3, ..., M).

The training high frequency map forming unit 102 obtains a training IR image high frequency map I _hf ^m, which is the difference between the training HR IR image and the corresponding expanded training LR IR image [32].

Next, the non-part sample contour transformation unit 103 performs non-part sample contour transformation on the enlarged training LR IR image to decompose each image I _{l '} ^m into one level pyramid.

As a result, one training feature vector low frequency map C _{llf '} ^m and four training feature vector high frequency maps C _lhf1' ^m , C _{lhf2 '} ^m , C _lhf3' ^m and C _{lhf4 '} ^m are obtained.

Then, the difference Gaussian filter 104 extracts _'the details of the training feature vector ^m low frequency map C _llf' training feature vector in the low-frequency map C _llf ^m.

The block dividing unit 105 _{divides the} five maps C _{llf '} ^m , C _lhf1' ^m , C _{lhf2 '} ^m , C _lhf3' ^m and C _{lhf4 '} ^m into blocks, and the low resolution coefficient block calculating unit 106 (NSCT) training low-resolution coefficient block B = {b _l ^mk }.

To do this, we obtain a block that overlaps the size of n * n in the five maps C _{llf '} ^m , C _lhf1' ^m , C _{lhf2 '} ^m , C _lhf3' ^m and C _{lhf4 '} ^m .

Thus, all the blocks are low-resolution training gyeopchip vector ^{_{B llf 'mk, B lhf1'}} mk, B lhf2 'mk, B lhf3' mk, B lhf4 ' by ^{mk (k = 1,2,3, ...,} K) are converted , Collecting low resolution overlap vectors from other maps to form a training low resolution feature vector b _l ^mk (where K is the total number of blocks in the map). The set of k = 1,2,3, ..., K of training low resolution feature vectors b _l ^mk is called the training low resolution coefficient block B = {b _l ^mk }. Then, the training IR image high frequency map I _hf ^m constitutes a training high resolution coefficient block B = {b _h ^mk } where n * n pixels overlap.

On the other hand, the low-resolution sparse dictionary part 108 can obtain the primitive low-scarce sparse dictionary C _l and the sparse expression φ through equation (4).

Then, the high-resolution sparse dictionary portion 109 obtains a raw high-resolution sparse dictionary C _h from the raw low-resolution sparse dictionary C _l and the training high-resolution coefficient block.

On the other hand, the low-resolution estimation unit 201 of the residual sparse dictionary generation module can estimate the estimated training LR image I _{1 '} ^m magnified using the primitive sparse dictionaries C _l and C _h , and the high- Training Get the raw HR IR image I _ho .

Then, the residual image forming unit 203 obtains the residual IR image, which is the difference between the estimated training original HR image and the original HR image.

The residual low-resolution sparse dictionary part 204 forms a residual low-resolution sparse dictionary using a training HR IR image and a training residual IR image computed using an estimated training HR IR image.

Next, the remaining high-resolution dictionary part 205 forms a residual high-resolution sparse dictionary using the estimated training HR IR image.

17 is a detailed configuration diagram of the test module of Fig.

15, the test module of FIG. 15 includes an interpolator 301, a non-sample contour transformer 302, a Gaussian difference filter 303, a block divider 304, a low-resolution coefficient block calculator 305, A high-resolution coefficient block 307, a test high-frequency map forming unit 308, a raw high-resolution image forming unit 309, a residual image estimating unit 310 and a high-resolution image forming unit 311 do.

The interpolator 301 receives the test LR IR image I _Ti and forms an enlarged test LR IR image I _{Ti '} using bi-cubic interpolation.

The non-part sample contour transformation unit 302 transforms the enlarged test LR IR image into non-part sample contour lines to form four test feature vector high-frequency maps and one test feature vector low-frequency map.

Then, the Gaussian difference filter 303 extracts the detailed information of the test feature vector low-frequency map as a test feature vector low-frequency map.

The block dividing unit 304 divides the five maps into blocks and the low resolution coefficient block calculating unit 305 calculates the test low resolution coefficient block B = {b _l ^k } of the non-part sample contour transformation of the test LR IR image k = 1, 2, ..., K).

The sparse expression calculator 306 refers to the raw low-resolution sparse dictionary 108 and uses the OMP of Equation 6 to determine the test low resolution coefficient block B = {b _l ^k } (k = 1, 2,. ..., K} to obtain a sparse representation {α _i }.

The high resolution coefficient block calculating unit 307 forms an estimated test high resolution coefficient block b _h ^{k *} by referring to the raw high resolution scarce dictionary using b _h ^{k *} = η C _h α _i , and the test high frequency map forming unit 308 ) _Obtains the high frequency map I _hf ^* of the estimated test IR image, using the estimated test high resolution coefficient block.

The raw high-resolution image forming unit 309 integrates the enlarged test IR image I _{Ti '} with the high-frequency map I _hf ^* of the estimated test IR image to obtain an estimated test raw HR IR image I _ho .

The residual image estimating unit 310 may estimate the estimated residual IR image I _r using the residual rare HR-LR dictionary pair C _rh , C _rl .

Then, the high-resolution image forming unit 311 combines the estimated test raw HR IR image I _ho and the estimated residual IR image I _r to obtain an estimated test HR IR image I _h .

In the present invention, a rare expression is applied to solve the SR problem of the IR image based on the NSCT and the rare dictionary. Compared to the regular dictionary, the sparse dictionary has regularity and is used adaptively. In addition, two pairs of sparse dictionaries have been proposed in comparison with existing SR methods.

The primitive sparse dictionary pair is learned to reconstruct the raw HR IR image, but the reference HR image and some details are completely lost. Therefore, the residual information is reconstructed by learning the remaining pairs of rare dictionaries. Existing extracted features that extract gradient maps can not fully apply image features. In the present invention, the feature is extracted using NSCT. NSCT is multiscale, multi-directional, and invariant, so it enriches image features and prevents aliasing. Experimental results show that the proposed method is superior to the other methods.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments of the present invention are not intended to limit the scope of the present invention but to limit the scope of the present invention. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

100: primitive rare dictionary generation module 101: interpreter
102: training high frequency map forming unit, 103: non-specimen contour conversion unit
104: Gaussian filter 105: block divider
106: low resolution coefficient block calculating unit 107: high resolution coefficient block calculating unit
108: Low-resolution sparse dictionary part 109: High-resolution sparse dictionary part
200: Residual sparse dictionary generation module 201: Low resolution estimation part
202: high resolution estimation unit 203: residual image forming unit
204: Residual low-resolution dictionary part 205: Residual high-resolution dictionary part
300: Test module 301:
302: non-specimen contour conversion unit 303: Gaussian filter
304: block division unit 305: low resolution coefficient block calculation unit
306: Sparse Expression Calculator 307: High Resolution Coefficient Block
308: test high frequency map forming unit 309: raw high resolution image forming unit
310: residual image estimating unit 311: high resolution image forming unit

Claims (15)

(A) receiving a training LR IR image set from a primitive sparse dictionary generation module, receiving training HR IR image sets, and forming training high-frequency maps;
(B) the primitive sparse pregeneration module transforms a training LR IR image into a training low resolution coefficient block to form a training low resolution coefficient block;
(C) the primitive sparse pregeneration module forms training high resolution coefficient blocks in a training high frequency map;
(D) the primitive spurious dictionary generation module forms a primitive low resolution sparse dictionary using training low resolution coefficient blocks, and generates a primitive high resolution sparse dictionary using the training high resolution coefficient blocks;
(E) the remaining sparse dictionary creation module comprises: forming a remaining high resolution sparse dictionary and a remaining low resolution sparse dictionary;
(F) receiving a test LR IR image from a test module to form a test low resolution coefficient block; And
(G) the test module comprises generating a test high resolution coefficient block formed to form an estimated test HR IR image,
The step (E)
(E-1) the residual sparse dictionary generation module forms an estimated training LR IR image with reference to the raw low-scarce sparse dictionary, and forms an estimated training HR IR image with reference to the raw high-resolution sparse dictionary;
(E-2) said residual sparse pre-production module using said estimated training HR IR image to form a residual high-resolution sparse dictionary;
(E-3) the residual sparse pre-production module is configured to form a training residual IR image using a training HR IR image and an estimated training HR IR image; And
(E-4) said residual sparse pregeneration module comprises forming a residual low-scarcity sparse dictionary using formed training residual IR images,
The step (G)
(G-1) The test module calculates a rare expression by referring to a raw low-scarcity dictionary from a formed test low-resolution coefficient block;
(G-2) the test module calculates a test high-resolution coefficient block by referring to a raw high-resolution scarce dictionary to form an estimated test raw HR IR image;
(G-3) The test module forms an estimated test residual IR image in the estimated test raw HR IR image with reference to the remaining high-resolution sparse dictionary and the remaining low-resolution sparse dictionary; And
(G-4) The test module comprises forming an estimated test HR IR image using an estimated test raw HR IR image and an estimated test residual IR image. The method according to claim 1,
The step (A)
(A-1) receiving the training LR IR image set and receiving a training HR IR image set;
(A-2) the primitive sparse pregeneration module expands the training LR IR image to form an expanded training LR IR image; And
(A-3) an infrared image super-resolution method of non-specimen contour transformation comprising the step of forming a training high-frequency map wherein the primitive sparse pregeneration module is a difference between an expanded training LR IR image and a training HR IR image. The method according to claim 2,
Characterized in that in step (A-2), the primitive sparse pregeneration module forms an expanded training LR IR image by magnifying the training LR IR image using bikigabic interpolation to produce an infrared image of ultra-high resolution Way. The method according to claim 1,
The step (B)
(B-1) generating a training feature vector low-frequency map and a plurality of training feature vector high-frequency maps by transforming the training LR IR image of the original sparse dictionary creation module into a non-part sample contour transformation;
(B-2) the primitive spurious dictionary generation module performs block division of training feature vector low-frequency maps and a plurality of training feature vector high-frequency maps to form training low-resolution overlapping vectors for each; And
And (B-3) combining the training low resolution overlap vectors to generate a training low resolution feature vector to form a training low resolution coefficient block, wherein the primitive sparse pregeneration module combines the training low resolution overlap vectors to form a training low resolution coefficient block. The method of claim 4,
A method of ultra-high resolution imaging of a non-specimen contour by transforming a training feature vector low-frequency map detail information using a Gaussian difference filter before the elementary sparse dictionary generation module performs block segmentation in step (B-2). delete The method according to claim 1,
The step (F)
(F-1) receiving the test LR IR image from the test module and forming an enlarged test LR IR image;
(F-2) generating a test feature vector low-frequency map and a plurality of test feature vector high-frequency maps by performing a non-part sample contour transformation on the expanded test LR IR image by the test module;
(F-3) the test module performs block division on one test feature vector low-frequency map and a plurality of test feature vector high-frequency maps to form a test low-resolution overlapping vector field for each test feature vector low-frequency map; And
(F-4) Combining the low resolution overlapping vectors to form a test low resolution feature block to form a test low resolution feature block. delete (A) receiving a training LR IR image set from a primitive sparse dictionary generation module, receiving training HR IR image sets, and forming training high-frequency maps;
(B) the primitive sparse pregeneration module transforms a training LR IR image into a training low resolution coefficient block to form a training low resolution coefficient block;
(C) the primitive sparse pregeneration module forms training high resolution coefficient blocks in a training high frequency map;
(D) the primitive spurious dictionary generation module forms a primitive low resolution sparse dictionary using training low resolution coefficient blocks, and generates a primitive high resolution sparse dictionary using the training high resolution coefficient blocks;
(E) the remaining sparse dictionary creation module comprises: forming a remaining high resolution sparse dictionary and a remaining low resolution sparse dictionary;
(F) receiving a test LR IR image from a test module to form a test low resolution coefficient block; And
(G) the test module comprises generating a test high resolution coefficient block formed to form an estimated test HR IR image,
The step (E)
(E-1) the residual sparse dictionary generation module forms an estimated training LR IR image with reference to the raw low-scarce sparse dictionary, and forms an estimated training HR IR image with reference to the raw high-resolution sparse dictionary;
(E-2) said residual sparse pre-production module using said estimated training HR IR image to form a residual high-resolution sparse dictionary;
(E-3) the residual sparse pre-production module is configured to form a training residual IR image using a training HR IR image and an estimated training HR IR image; And
(E-4) said residual sparse pregeneration module comprises forming a residual low-scarcity sparse dictionary using formed training residual IR images,
The step (G)
(G-1) The test module calculates a rare expression by referring to a raw low-scarcity dictionary from a formed test low-resolution coefficient block;
(G-2) the test module calculates a test high-resolution coefficient block by referring to a raw high-resolution scarce dictionary to form an estimated test raw HR IR image;
(G-3) The test module forms an estimated test residual IR image in the estimated test raw HR IR image with reference to the remaining high-resolution sparse dictionary and the remaining low-resolution sparse dictionary; And
(G-4) The test module is configured to generate an estimated test HR IR image using an estimated test raw HR IR image and an estimated test residual IR image. A computer recording the infrared image ultra high resolution method of non- Readable recording medium. Training receives a set of LR IR images, receives a set of training HR IR images, forms training high-frequency maps, forms training low-resolution coefficient blocks by transforming training LR IR images into non-parametric contour maps, and training high- A primitive sparse pregeneration module to form a primitive low resolution sparse dictionary using training low resolution coefficient blocks and a raw high resolution sparse dictionary using training high resolution coefficient blocks;
A residual sparse dictionary generation module that forms a residual high-resolution sparse dictionary and a residual low-resolution sparse dictionary; And
A test LR IR image is received and a test low resolution coefficient block and a test high resolution coefficient block are calculated to form an estimated test HR IR image by referring to the raw high resolution sparse dictionary, the raw low resolution sparse dictionary, the remaining high resolution sparse dictionary and the remaining low resolution sparse dictionary A test module,
The remaining sparse dictionary generation module
A low resolution estimator for forming an estimated training LR image magnified using a raw high resolution sparse dictionary and a raw low resolution sparse dictionary;
A high resolution estimator for forming an estimated training raw HR IR image;
A residual image forming unit for obtaining a residual IR image that is a difference between the estimated training raw HR IR image and the raw HR IR image;
The remaining low-resolution sparse dictionary portion forming the remaining low-resolution sparse dictionary using the training HR IR image and the training residual IR image calculated using the estimated training HR IR image; And
Resolution residual dictionary forming a residual high-resolution sparse dictionary using an estimated training HR IR image,
The test module
An interpolator receiving the test LR IR image and forming an enlarged test LR IR image using bi-cubic interpolation;
A non-specimen contour transformer for forming a test low resolution coefficient block of non-specimen contour transformation by performing non-specimen contour transformation on an enlarged test LR IR image;
Testing a non-specimen contour transformation with reference to a primitive low-resolution sparse dictionary; obtaining a rare representation from the low-resolution coefficient block;
A high resolution coefficient block calculator for forming an estimation test high resolution coefficient block by referring to a raw high resolution scarce dictionary;
A test high frequency map forming unit using the estimated test high resolution coefficient block to form a high frequency map of the estimated test IR image;
Integrating the expanded test IR image with a high frequency map of the test IR image to obtain a raw HR IR image;
A residual image estimator for estimating a residual IR image using a residual high-resolution sparse dictionary and a residual low-resolution sparse dictionary; And
An infrared imaging ultra-high resolution device of non-specimen contour transformation comprising a high resolution image forming section for obtaining an estimated test HR IR image by integrating a presumptive test raw HR IR image and an estimated residual IR image. 12. The method of claim 10,
The primitive rare dictionary generation module
Training LR IR images are enlarged and expanded to generate training LR IR images;
A training IR image, which is the difference between the training HR IR image and the expanded training LR IR image; a training high frequency map forming unit, which forms a high frequency map;
A non-specimen contour transformation unit for performing non-specimen contour transformation on an enlarged training LR IR image to form a training feature vector low-frequency map and a plurality of training feature vector high-frequency maps;
A block dividing unit dividing the plurality of maps into blocks;
A low resolution coefficient block calculator for transforming the divided blocks into training low resolution overlap vectors to collect low resolution overlap vectors to form training low resolution coefficient blocks;
A high resolution coefficient block calculator for training high resolution coefficient blocks from training IR image high frequency maps;
A low-resolution sparse dictionary portion forming a raw low-resolution sparse dictionary in the training low-resolution coefficient block; And
Raw low resolution sparse dictionaries and training Infrared imaging ultra resolution devices for non-specimen contour transforms containing high resolution sparse dictionary parts forming a raw high resolution sparse dictionary from high resolution coefficient blocks. 12. The method of claim 11,
The interpolator receives the training LR IR images and generates enlarged training LR IR images using the bicubic interpolation method. 12. The method of claim 11,
Further comprising a Gaussian difference filter for extracting details of the training feature vector low frequency map before the block segmentation unit performs block segmentation. delete delete

Images