KR20140081481A - Block based image Registration for Super Resolution Image Reconstruction Method and Apparatus - Google Patents

Abstract

According to a method to restore a superresolution image using block-based image matching of the present invention, the blocks are divided to partially overlap in the outer regions thereof, when an image is divided into blocks, in order to prevent the occurrence the distortion of image information between the blocks during a block-based superresolution image restoring process, and an image matching process is performed. Also, during the image matching process, the matching process is performed such that the positions are compared in a region greater than a region of a restored image, thus increasing the accuracy of image matching.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a block-based image registration method and apparatus,

The present invention relates to a super-resolution image matching method and apparatus, and more particularly, to a method and apparatus for matching images using a binary cubic interpolation method with reference to a blocked image and a plurality of comparison images.

'This research was supported by the Ministry of Knowledge Economy and IDEC Platform Center.'

'This work was supported by the Ministry of Knowledge Economy (MKE) and the IDEC Platform Center (IPC).'

Image processing techniques for processing low-resolution images and obtaining high-resolution and high-quality images have been continuously proposed. As a background of these techniques, the following prior art documents can be found.

[1] 1991 CVGIP: Graphical Models and Image Proc. 53, pp. 231 - 239, M. Irani and S. S. Peleg, "Improving Resolution by Image Registration ", "

[2] Published by Jai Press, Green Location Materials, Connecticut, 1984 Advances in Computer and Image Processing, Vol. s. Edited by T. S. Huang, pp. 317 - 339, Al. Why. R. Y. Tsai and T. s. T. S. Huang, "Multiframe Image Restoration and Registration"

[3] In 1994 NASA Technical Report FIA-94-12, p. P. Cheeseman, B. Kanesfky, and Al. R. Kruft, J. < RTI ID = 0.0 > J. Stutz and al. R. Hanson, "Super-Resolved Surface Reconstruction from Multiple Images"

[4] IEEE International Conference on Acoustics, Speech, and Signal Processing, held in San Francisco, CA in 1992, Volume III, pp. 169-172, AM Tekalp, MK Ozakan and MI Sezan, "High-Resolution Image Reconstruction for Lower-Resolution Image Sequences and Space-Vision Image Restoration"

[5] Published by Addison-Wesley, al. Seed. R. C. Gonzalez and Al. this. &Quot; Digital Image Processing ", by R. E. Wood,

[6] In 2001, published by Wiley, New York, W. K. W. K. Pratt, "Digital Image Processing" Second Edition

[7] U.S. Patent No. 6,330,344

[8] U.S. Patent No. 5,969,848

As the manufacturing technology of the image capturing apparatus is developed, the resolution of the captured image is increasing day by day. However, there are cases where only low resolution images are sometimes obtained due to environmental constraints, cost, and the like. Two or more input images are required to improve the image quality and resolution seen by the human eye. By using the video sequence, a blurring scene, a dim figure, or an unclear subject of poor image quality can be reconstructed into an enhanced output image, which can then be easily observed and recognized. Conventional studies on reconstructing resolution enhancement images using low resolution images have been largely based on [1] iterative methods, [2] frequency domain methods, and [3] Bayesian statistical methods ). Up to now, among the aforementioned methods, [1] an iterative algorithm developed by Irani in 1991 and mainly reconstructing a resolution-enhanced image by image registration is still one of the prior art related to reconstruction of a resolution- It is the most reliable. Iterative methods mainly consist of three steps: initial guess, imaging process, and reconstruction.

However, it is known that the above-mentioned iterative method consumes more computation time as the magnification factor determined in this iteration method increases, that is, as the size of the reconstructed image becomes larger. Typically, the runtime of image reconstruction by iteration is several hours, depending on the performance of the computer system

One of the prior arts proposed to solve this problem has been proposed in Korean Patent No. 10-0570630 entitled " A method of generating a resolution enhanced image using a plurality of low resolution images ". This method interpolates and reconstructs the image using the improved resolution enhancement algorithms based on the iterative method and appropriate techniques such as initial interpolation, automatic image selection, and robust image matching.

However, there is still a problem that the amount of computation increases as the amount of data on the screen increases, even in the case of the above-described prior art, and there is a problem that can cope with an increase in the number of low-resolution images used as input .

Therefore, even when the number of low-resolution images increases, there is a need for a super-resolution image restoration technique capable of effectively reducing the calculation amount

Korean Patent No. 10-0570630 (registered on April, 2006) Korean Registered Patent No. 10-1200490 (registered on November 6, 2012)

[1] S. C. Park, M. K. Park, and M. G. Kang, "Super-resolution image reconstruction: A technical overview," IEEE Signal Process. Mag., Vol. 20, pp. 21. 2003. [2] Moshe, B. E. Assaf, Z. and Shree, K. N. "Video Super-Resolution Using Controlled Subpixel Detector Shifts", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, pp.1-2, June 2005 [3] Hiroyuki, T. Peyman, M. Matin, P. and Micheal, E., "Super-Resolution Without Explicit Subpixel Motion Estimation" IEEE Transactions on Image Processing vol.18, pp.1, September 2009

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to perform matching with two or more reference images using a block unit region division of a reference image and to restore a high resolution image using the matching result.

More specifically, the present invention aims at performing image matching processes by dividing an image into blocks on a block-by-block basis without a separate key frame.

It is another object of the present invention to improve the accuracy of motion vector calculation in the matching process of super resolution image restoration in block unit processing and to obtain an improved high resolution image in restoration.

It is another object of the present invention to make it possible to process a low resolution image of various sizes only by adjusting the number of block operations by performing a comparison operation on a region larger than a block unit in calculating a motion vector for the matching process, .

According to an aspect of the present invention, there is provided a method for restoring a super-resolution image, the method comprising: dividing at least one input image among a plurality of input images into a plurality of unit blocks to generate a unit block image step; Performing image matching using the unit block image and generating a matched unit block image; And generating the super resolution reconstructed image by synthesizing the matched unit block images.

The step of generating the matched unit block image may include generating a matching reference image including each of the unit block images and including more pixels than each of the unit block images. And may generate the matched unit block image using the matching reference image.

In addition, the step of generating the matched unit block image may generate the matched unit block image matching the size of the unit block from the resultant image after the matching process performed using the matching reference image.

According to another aspect of the present invention, there is provided an apparatus for restoring a super-resolution image, comprising: a division unit for dividing at least one input image among a plurality of input images into a plurality of unit blocks to generate the unit block image; A matching unit for performing image matching using the unit block image; A matching unit block generator for generating the matched unit block image using the image matching result of the unit block image; And a super resolution synthesizer for synthesizing the matched unit block images to generate a super resolution reconstruction image.

According to the super resolution image restoration method using the image matching of the block unit of the present invention, when the block is divided to prevent image information distortion between the block and the block during the block super resolution image restoration process, And the image matching process is performed. Even in the image matching process, the matching process is performed by comparing the positions in the area larger than the area of the restored image, so that the accuracy of the image matching can be improved.

In addition, by performing the block unit processing, it is possible to improve the accuracy of the motion vector calculation in the matching process of the super resolution image restoration and to obtain the high resolution image improved upon restoration.

Also, in the motion vector calculation for the matching process, the block phenomenon can be eliminated by performing a comparison operation over a wider area than the block unit, and low resolution images of various sizes can be processed only by controlling the number of block operations.

In addition, the super resolution image restoration technology can exhibit improved performance when using the present invention in a smart TV and various applications, and various applications can be applied by utilizing it.

In addition, the generating of the super resolution reconstructed image may generate the super resolution reconstructed image by applying a bi-cubic interpolation method to the matched unit block image.

FIG. 1 is a flowchart illustrating a method of restoring a block-based image-matched super-resolution image according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating an image matching process using a block image of FIG. 1 according to an exemplary embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method of restoring a super-resolution image using unit block image matching and bipartite interpolation according to another embodiment of the present invention.
FIG. 4 is a block diagram illustrating a conceptual configuration of a block-based image-matched super-resolution image restoration apparatus according to an embodiment of the present invention.
5 is a view showing divided unit blocks and a matching reference image according to an embodiment of the present invention.
FIG. 6 is a conceptual view illustrating a matching and super resolution restoring process of a divided block according to an exemplary embodiment of the present invention. Referring to FIG.
7 is a diagram illustrating an exemplary subpixel image matching process according to an embodiment of the present invention.
8 is a view illustrating an embodiment in which unit block images are matched to represent super-resolution images.

Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings.

Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

However, the present invention is not limited or limited by the embodiments. Like reference symbols in the drawings denote like elements.

Hereinafter, a method and apparatus for restoring a super-resolution image using block unit image matching according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 8

1 is a diagram illustrating a process of reconstructing a block-based image-matched super-resolution image according to an embodiment of the present invention.

Referring to FIG. 1, an input image is divided into a plurality of unit blocks (S110).

According to an embodiment of the present invention, each of the input images may be divided into unit blocks to perform image matching and super resolution restoration. In another embodiment, one of the input images is a main reference image, The remaining input images become sub reference images used in the matching process, so that only the main reference image is divided into unit blocks, and the sub reference images are all used for multi-image matching (or similarity search).

Image matching is performed using the unit block image (S120), and a matched unit block image is generated (S130).

In step S140, a super-resolution image is reconstructed by combining the unit block images matched through the unit block images matched with the image matching,

2 is a view illustrating an image matching process using a block image according to an embodiment of the present invention.

The block of the block image has a size larger than the divided area (unit block) of the reference image so that some of the matching reference images overlap each other (S210). The reason is that the part outside the area of the image is supplemented with image information through mirror padding. Mirror padding is a method of complementing a value outside the image range by symmetry of the outermost portion of the image.

In step S220, a matched unit block image is generated from the resultant image of the image matching process using the matching reference image. In this way, the matching unit block images are matched and the matched unit block images are generated.

Referring again to FIGS. 1 and 2, according to an embodiment of the present invention, in step S120, an area having a size larger than a unit block may be generated as a matching reference image for image matching. For example, if the unit block obtained by dividing the entire image equally is 16 x 16 pixels, the matching reference image may include 20 x 20 pixels including a unit block, which is larger than a unit block.

In this case, pixel-by-pixel matching can be performed by searching for the similarity between images using the matching reference image of 20 x 20 pixels or calculating the correlation based on the feature point. 16x16 pixels are cut out from the resultant image of the image matching process using the matching reference image to generate a matched unit block image (S220).

3 is a diagram illustrating a process of restoring a super-resolution image using a unit block image matching and a bipartite interpolation method according to an embodiment of the present invention.

First, the input image is divided into a plurality of unit blocks (S310). At this time, each input image can be used for matching in equal conditions.

The blocks of the block image have a size larger than the divided area (unit block) of the reference image, and a matching reference image having a size such that some areas overlap with each other is generated at step S320.

An integer pixel unit motion vector between the block main reference image and the pixels located at the same position of the sub reference image is extracted (S330).

The main reference image is expanded 8 times by using the bi-cubic interpolation method and the pixels of the sub reference image are spaced apart by 8 pixels to obtain the subpixel motion vectors and the most similar positions are matched in units of 1/8 pixel (S340) . In step S350, a subpixel motion vector between the pixels having the highest similarity is obtained and combined with the integer pixel motion vector described above.

In the process of obtaining the above-described integer pixel unit motion vector or subpixel motion vector, a deviation between pixels located at the same position between the main reference image and the sub reference image is obtained for each unit block. The sum of the deviations of these pixels is called a Sum of Absolute Difference (SAD). Small SAD means that there is little difference in image information in the region. That is, the similarity of images is high. The position of the sub reference image is shifted in units of integer pixels or, for example, in units of 1/8 pixels, and the SAD of the optimal integer pixel unit or the SAD of the sub pixel unit is obtained. The image information of the area beyond the range of the main reference image is replaced with the mirror padding image information described above.

The computation time is long to perform the SAD operation for all the positions. Therefore, in order to reduce the computation time, the position of the sub reference image may be shifted within ± 2 pixels in the vertical and horizontal directions, and the SAD operation may be performed.

In operation S360, the super-resolution image is reconstructed by connecting the divided blocks through the above process and matching the integer pixel motion vector with the sub-pixel motion vector.

4 is a block diagram of a block-based image-matched super-resolution image restoration apparatus according to an embodiment of the present invention.

The block-based image-matched super-resolution image restoring apparatus 400 includes a dividing unit 410 for dividing a reference image region and forming a unit block image by partitioning each divided region, A matching unit 420 for generating a matching reference image including more pixels than each block image, and a matching reference image, and performs a matching process on the matching reference image after the matching process, And a super resolution combining unit 440 that includes the matching unit block generating unit 430 for generating the matched unit block image and restores the super resolution image using the matched unit block image.

5 is a view showing divided unit blocks and a matching reference image according to an embodiment of the present invention.

The divided unit block 510 has a block 511 of the same size by dividing the main reference image. The matching reference image 520 is a block 521 in which a block image has a size larger than a divided area (unit block) of the reference image, and a matching reference image is generated by overlapping some areas of the matching reference images

FIG. 6 is a view illustrating an example of a matching process of divided blocks according to an embodiment of the present invention. Referring to FIG.

6 shows an example of a result 630 of a resolution-restored image obtained by finding an optimal SAD through a matching process including a block 621 of a matching reference image 620. [

7 is a diagram illustrating an exemplary subpixel image matching process according to an embodiment of the present invention.

The main reference image pixel 710 and the sub reference image pixel 720 are compared with each other. The sub reference image 720 is shifted within ± 2 pixels, for example, And a pixel 730 interpolated in the main reference image using the cubic interpolation method is interpolated.

8 is a view illustrating an embodiment in which unit block images are matched to represent super-resolution images.

FIG. 4 is a diagram illustrating a result of reconstructing the final reconstructed super-resolution image by obtaining an optimal SAD for each block through a process of matching the blocked low-resolution images to match the reference image.

Claims (8)

Generating a unit block image by dividing at least one input image among a plurality of input images into a plurality of unit blocks;
Performing image matching using the unit block image and generating a matched unit block image; And
Generating a super resolution reconstructed image by synthesizing the matched unit block images; The method according to claim 1,
Wherein the step of generating the matched unit block image comprises:
Generating a matching reference image including each of the unit block images and including more pixels than each of the unit block images;
Lt; / RTI >
Wherein the matching unit block image is generated using the matching reference image. 3. The method of claim 2,
Wherein the step of generating the matched unit block image comprises:
And the matching unit block image is generated in accordance with the size of the unit block from the resultant image of the image matching after the image matching process, . The method according to claim 1,
The step of generating the super resolution reconstructed image
Wherein the super resolution reconstructed image is generated by applying a cubic interpolation method to the matched unit block image. A division unit for dividing at least one input image among a plurality of input images into a plurality of unit blocks to generate a unit block image;
A matching unit for performing image matching using the unit block image;
A matching unit block generating unit for generating a matched unit block image using the matching result of the unit block image; And
A super resolution synthesizer for synthesizing the matched unit block images to generate a super resolution reconstruction image;
Resolution image. 6. The method of claim 5,
The matching unit may include:
Wherein the matching unit generates a matching reference image including each of the unit block images and including more pixels than each of the unit block images and performs the matching using the matching reference image. The method according to claim 6,
Wherein the matching unit-
Wherein the matching unit block image generating unit generates the matched unit block image matching the size of the unit block from the resultant image of the image matching performed using the matching reference image. 6. The method of claim 5,
Wherein the super-
Wherein the super resolution reconstructed image is generated by applying a cubic interpolation method to the matched unit block image.

Images