KR101675117B1 - Method and apparatus for reconstructing high-resolution image by using multi-layer low-resolution image - Google Patents

Abstract

Resolution texture layer images into a plurality of low resolution texture layer images and a plurality of low resolution base layer images, synthesizing a plurality of low resolution texture layer images to generate a high resolution texture layer image, synthesizing a plurality of low resolution base layer images, A high-resolution image generation method for generating a hierarchical image, synthesizing a high-resolution texture hierarchy image and a high-resolution base hierarchy image, and outputting a high-resolution image is disclosed.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for generating a high-resolution image using a low-

The present invention relates to resolution amplification of an image.

As the display size increases, a technique for effectively enlarging the image size is required. As the image enlargement technique, a linear interpolation filter including a bilinear filter, a bicubic filter, or a Lanczos kernel based filter may be used.

In addition, super-resolution image processing techniques are being used to improve the quality of SD (Standard Definition) video and HD (High Definition) video. The super resolution image processing method is based on a recursive operation method in which the processed image is repeatedly processed by using the input image to obtain reliable image information.

The present invention relates to a method and apparatus for reconstructing a high-resolution image using low-resolution images separated by layers.

According to an embodiment of the present invention, there is provided a method of generating a high-resolution image using a multi-layer low-resolution image, the method comprising: separating a low-resolution image into a plurality of low- Generating a high resolution texture layer image by synthesizing the plurality of low resolution texture layer images and synthesizing the plurality of low resolution base layer images to generate a high resolution base layer image; And synthesizing the high resolution texture layer image and the high resolution base layer image to output a high resolution image

The step of outputting the high-resolution image according to an exemplary embodiment may further include reconstructing a high-frequency component of the high-resolution image.

The separating step may separate the plurality of degraded texture layer images and the plurality of base layer images by separating each of the plurality of low resolution images into a low resolution texture layer image and a base layer image . The separating step may include generating a low-resolution base layer image by applying a filter for flattening an image while preserving an edge component of the image with respect to the low-resolution image. And generating a difference image of the low resolution image and the low resolution base layer image as the low resolution texture layer image.

The high resolution texture layer image and the high resolution base layer image generation step may include generating non-recursive filtering of the plurality of low resolution texture layer images and the high resolution texture layer image, The high-resolution base layer image can be generated through non-recursive filtering on the images.

The high-resolution texture layer image and the high-resolution base layer image generating step may include: weighting a first current frame of each low-resolution texture layer image, temporally adjacent frames of the first current frame, and spatially adjacent frames, Generating a high resolution texture layer image through sum filtering and performing a weighted sum filtering on a second current frame of each low resolution base layer image, temporally adjacent frames of the second current frame, and spatial adjacent frames, A hierarchical image can be generated.

The high-resolution image output step may synthesize the high-resolution image by weighted average filtering on the high-resolution texture layer image and the high-resolution base layer image.

The high-frequency component restoring step according to an exemplary embodiment may restore high-frequency components of the high-resolution image through non-articulation of the high-resolution image.

According to an embodiment of the present invention, there is provided an apparatus for generating a high-resolution image using a multi-layer low-resolution image, the apparatus comprising: a low-resolution image hierarchy separator for separating a low-resolution image into a plurality of low-resolution texture layer images and a plurality of low- A high resolution texture layer image generation unit for synthesizing the plurality of low resolution texture layer images to generate a high resolution texture layer image; A high-resolution base layer image generation unit for synthesizing the plurality of low-resolution base layer images to generate a high-resolution base layer image; And a high-resolution image synthesizer for synthesizing the high-resolution texture layer image and the high-resolution base layer image to output a high-resolution image.

The high-resolution image synthesis unit may include a high-frequency component reconstruction unit for reconstructing a high-frequency component of the high-resolution image.

The apparatus for generating a high-resolution image using the multi-layer low-resolution image according to an exemplary embodiment may further include a motion estimator for estimating motion information between a current frame of the low-resolution image and successive adjacent frames, , Weighted sum filtering is performed on the first current frame of each low resolution texture layer image, temporally adjacent frames of the first current frame, and spatially adjacent frames using motion compensated neighboring frames based on the motion information And the high resolution base layer image generating unit generates the high resolution base layer image using the motion compensated neighboring frames based on the motion information to generate a second current frame of each low resolution base layer image, The temporally adjacent frames of frame < RTI ID = Through the weighted sum filter for ever adjacent frames it may produce the high resolution base layer picture.

The high-resolution image synthesis unit may synthesize the high-resolution image by weighted average filtering on the high-resolution texture layer image and the high-resolution base layer image.

The high-frequency component reconstructing unit may transform the high-resolution image into the high-resolution image by performing first filtering to recover a damaged high-frequency component of the high-resolution image and second filtering to prevent noise amplification, The high resolution image can be synthesized by weighted sum filtering of the filtering result and the second filtering result.

According to an exemplary embodiment, at least one of the high resolution texture layer image generator, the high resolution base layer image generator, and the high resolution image composer may perform non-recursive weighted sum filtering on the input image.

According to another embodiment of the present invention, there is provided an apparatus for generating a high-resolution image using a multi-layer low-resolution image, the apparatus comprising: a low-resolution image multi- Separator; A hierarchical high-resolution image generation unit for generating a high-resolution image of the corresponding layer by combining the plurality of low-resolution images of the corresponding layer for each layer; And a multi-layer high-resolution image synthesizer for synthesizing the high-resolution images generated for each layer and outputting high-resolution images.

The present invention includes a computer-readable recording medium on which a program for implementing a high-resolution image generation method using a multi-layer low-resolution image according to an embodiment of the present invention is recorded.

FIG. 1 is a block diagram of a high-resolution image generating apparatus using a multi-layer low-resolution image according to an embodiment of the present invention.
Figure 2 shows the synthesis of high resolution images using a plurality of enlarged low resolution images.
FIG. 3 is a block diagram of a high-resolution image generating apparatus using a low-resolution image according to the related art.
FIG. 4 is a schematic diagram of a high-resolution image reconstruction method by a high-resolution image generation apparatus according to the related art.
FIG. 5 shows an exemplary block diagram for implementing a high-resolution image generating apparatus according to an embodiment.
6 is a schematic diagram of a high-resolution image reconstruction method of a high-resolution image generation apparatus according to an embodiment.
7 illustrates an exemplary block diagram for implementing a low-resolution image hierarchy separator of a high-resolution image generator according to an embodiment.
FIG. 8 illustrates an exemplary block diagram for implementing a high-resolution texture layer image generating unit or a high-resolution base layer image generating unit of the high-resolution image generating apparatus according to an exemplary embodiment.
FIG. 9 is a block diagram illustrating an example of a high-frequency component reconstruction unit in the high-resolution image synthesis unit of the high-resolution image generation apparatus according to an embodiment.
FIG. 10 is a block diagram of a high-resolution image generation apparatus using a multi-layer low-resolution image according to another embodiment of the present invention.
11 is a flowchart illustrating a method of generating a high-resolution image using a multi-layer low-resolution image according to an embodiment of the present invention.

FIG. 1 is a block diagram of a high-resolution image generating apparatus using a multi-layer low-resolution image according to an embodiment of the present invention.

According to an embodiment, a high-resolution image generation apparatus 100 using a multi-layer low-resolution image includes a low-resolution image hierarchy separating unit 110, a high-resolution texture layer image generating unit 120, a high-resolution base layer image generating unit 130, And an image combining unit 140. [

The low-resolution image hierarchy separating unit 110 according to an exemplary embodiment divides a low-resolution image input to the high-resolution image generating apparatus 100 using a multi-layer low-resolution image into a plurality of low-resolution texture layer images and a plurality of low- Separate.

The low resolution image layer separator 110 separates each of the plurality of low resolution images into a low resolution texture layer image and a base layer image, and as a result, a plurality of low resolution texture layer images and a plurality of low resolution base layer images May be separated.

The low-resolution image hierarchy separator 110 according to an embodiment separates a low-resolution image into two layers, a texture layer and a base layer. In the simplest example, a texture layer and a base layer of an image can be distinguished into a high frequency component and a low frequency component of an image according to a frequency characteristic of an image.

Alternatively, the base layer of an image according to an exemplary embodiment may be composed of a low frequency component and a strong directional edge component of an image. In this case, the texture layer according to an exemplary embodiment may include a high frequency component indicating fine detail information, excluding a low frequency component and a strong directional edge component in a low resolution image.

That is, the texture layer image according to one embodiment may be composed of image components other than the image component of the base layer from the original image. Accordingly, the low-resolution image layer separator 110 according to an embodiment separates the low-resolution base layer image from the low-resolution image and separates the low-resolution texture layer image composed of the remaining image components except for the low- have.

In order to separate a base layer image component including a low frequency component and a strong directional edge component of an image, a low resolution image layer separator 110 according to an exemplary embodiment separates a low resolution image, A flattening filter (hereinafter referred to as an edge-preserving flattening filter) can be used. Accordingly, the low-resolution image hierarchy separator 110 according to an exemplary embodiment can generate a low-resolution base-layer image by applying an edge-preserving smoothing filter to the low-resolution image.

The low-resolution image input to the low-resolution image hierarchy separator 110 according to an exemplary embodiment may be an image obtained by enlarging the original image. When a plurality of low-resolution images are used, the plurality of low-resolution images may be temporally continuous, or may be a plurality of low-resolution images spatially adjacent to each other.

The low-resolution texture layer images and the low-resolution base layer images separated into two layers by the low-resolution image layer separator 110 according to one embodiment are divided into a high-resolution texture layer image generator 120 and a high- Resolution base layer image generating unit 130 according to an embodiment of the present invention.

The high-resolution texture layer image generating unit 120 generates a high-resolution texture layer image by combining a plurality of low-resolution texture layer images. In addition, the high-resolution base-layer image generating unit 130 generates a high-resolution base-layer image by combining a plurality of low-resolution base-layer images.

The high-resolution texture layer image generating unit 120 and the high-resolution base layer image generating unit 130 according to an exemplary embodiment may perform spatial-temporal weighted sum filtering A high-resolution image of the layer can be generated.

Specifically, the high-resolution texture layer image generating unit 120 generates a high-resolution texture layer image based on the current frame of each low-resolution texture layer image and the weighted sum of the frames temporally adjacent to the current frame and spatially adjacent frames Using filtering, a high resolution texture layer image can be generated. In addition, the high-resolution base layer image generating unit 130 may perform weighted sum filtering on the current frame of each low-resolution base layer image, temporally adjacent frames of the current frame, and spatially adjacent frames , A high-resolution base layer image can be generated.

The high-resolution texture layer image generating unit 120 and the high-resolution base layer image generating unit 130 according to an exemplary embodiment may perform motion estimation or motion-compensated temporal or spatial compensation for a current frame of a low- Spatial adjacency frames may be used to perform weighted sum filtering on the current frame.

The high resolution texture layer image generating unit 120 and the high resolution base layer image generating unit 130 according to an exemplary embodiment may be configured to generate weighted sum of the weighted sum of the adjacent frames Temporal proximity, luminance difference, proximity to a strong edge, and reliability of motion estimation.

As an example of spatial proximity, the current area of the current frame and the spatial distance of the reference area of the adjacent frame, the distance between the center pixel positions, and the like can be used. Also, as an example of temporal proximity, a time difference between a current frame and an adjacent frame, a frame index difference, and the like can be used.

The high-resolution texture layer image generating unit 120 and the high-resolution base layer image generating unit 130 according to an exemplary embodiment of the present invention generate a plurality of low- Spatial weighted sum filtering using a weight based on temporal proximity or similarity can be performed to generate a high-resolution image of the corresponding layer.

As an example of the similarity, various numerical values representing the difference between the pixel values or the luminance of the corresponding pixels may be used.

In addition, since the high-resolution texture layer image generating unit 120 and the high-resolution base layer image generating unit 130 according to the embodiment use filtering using a plurality of continuous low-resolution images of the corresponding layer, the non-recursive filtering A high-resolution image of the layer can be generated.

The high-resolution image synthesis unit 140 synthesizes a high-resolution texture layer image and a high-resolution base layer image to output a high-resolution image. The high-resolution image synthesis unit 140 according to an exemplary embodiment may synthesize high-resolution images by weighted average filtering on a high-resolution texture layer image and a high-resolution base layer image.

The high-resolution image synthesizer 140 according to an exemplary embodiment may perform weighted average filtering to consider the edge direction of the original image while minimizing the damage of the texture component of the original image. Accordingly, the high-resolution image synthesis unit 140 according to an embodiment applies a first weight for minimizing the damage of the texture component of the original image to the high-resolution texture layer image, 2 weights can be applied to high-resolution base layer images.

The high-resolution image synthesis unit 140 reconstructs the high-frequency component of the initial high-resolution image generated by the weighted average filtering on the high-resolution texture layer image and the high-resolution base layer image, Can be generated.

In order to recover the high-frequency components of the initial high-resolution image, the high-resolution image synthesizer 140 according to an exemplary embodiment performs first filtering for restoring damaged high-frequency components of the high-resolution image and second filtering for preventing noise amplification . The high-resolution image synthesis unit 140 according to an exemplary embodiment may synthesize a high-resolution image by performing weighted sum filtering of an initial high-resolution image, a first filtering result for an initial high-resolution image, and a second filtering result.

The high-resolution image synthesis unit 140 according to an embodiment can generate a high-resolution image in which an initial high-resolution image and a high-frequency component are reconstructed by non-recursive weighted sum filtering on a high-resolution texture layer image and a high-resolution base layer image.

Accordingly, the apparatus 100 for generating a high-resolution image using a multi-layer low-resolution image according to an embodiment generates a high-resolution image by separating a low-resolution image into two or more layers and a plurality of low- It depends on recursive computation.

In general, existing techniques for reconstructing high-resolution images from low-resolution images are based on recursive computation. When a recursive operation is performed, the amount of computation is large according to an iterative operation for determining an optimal result. On the other hand, the high resolution image generating apparatus 00 according to an embodiment uses a non-recursive operation method, Relieved.

In addition, in generating a high-resolution image of a corresponding layer by weighted sum filtering of a plurality of low-resolution images of the layer in each layer, the temporal proximity, spatial proximity, luminance difference, reliability of motion estimation And so on, it is possible to perform weighted sum filtering based on the surrounding information of the current pixel.

Accordingly, the apparatus 100 for generating a high-resolution image using a multi-layer low-resolution image according to an exemplary embodiment of the present invention prevents damage to a texture component having a small correlation between a current pixel and surrounding information, , It is possible to generate a visually natural high-resolution image.

Figure 2 illustrates the synthesis of a high resolution image using a plurality of expanded low resolution images of related art.

As an example of a process of synthesizing a high resolution image using a plurality of low resolution images, a case where one high resolution image is synthesized by using low resolution images in which the width and height of the low resolution image are doubled, respectively, will be described below. Of the images 210, 220, 230, 240, the black circle pixel is a sample pixel before interpolation, and the white three model pixels represent interpolated pixels or blank pixels to which pixel values are not assigned.

In an ideal case, in order to synthesize a high-resolution image in which the width and size of the low-resolution image are enlarged by two, respectively, four low-resolution images 210, 220, 230, and 240 are required. Four enlarged low resolution images 210, 220, 230 and 240 may be combined to produce a high resolution image 250. [

However, if there are insufficient number of effective pixels required to generate a high resolution image, inconsistencies occur, or effective pixels are distributed nonuniformly in each region, aliasing from a high resolution image generated by combining a plurality of low resolution images aliasing, jagging artifact, or flickering artifact may occur.

Therefore, in order to prevent the distortion of the high-resolution image generated by the synthesis of the low-resolution image, it is necessary to reconstruct the high-resolution image by filtering or sampling the current pixel using the surrounding information in the process of synthesizing the low-

Hereinafter, referring to FIGS. 3 and 4, description will be made with respect to related technologies for reconstructing a high-resolution image using a low-resolution image.

FIG. 3 is a block diagram of a high-resolution image generating apparatus using a low-resolution image according to the related art.

Resolution image generating apparatus 300 according to the related art acquires motion information on the low-resolution image input from the motion estimating unit 320 and the high-resolution image reconstructing unit 310 reconstructs the motion information of the motion estimating unit 320 And reconstructs the high resolution image of the inputted low resolution image using the motion compensated peripheral information. The high frequency component reconstructing unit 330 applies a deblocking technique to the output image of the high resolution image reconstructing unit 310 to restore the high frequency image of the high resolution image.

The high-resolution image reconstruction unit 310 according to the related art uses linear filtering or nonlinear filtering for a plurality of low-resolution images for reconstructing a high-resolution image. When linear filtering or nonlinear filtering is used, a pixel in a flat area or a pixel in an edge area in which the current pixel has high correlation with surrounding pixels can be restored to a reliable prediction value by referring to the surrounding information. However, the image information of the pixels of the texture component having a low correlation with neighboring pixels may be damaged by the surrounding information.

FIG. 4 is a schematic diagram of a high-resolution image reconstruction method by a high-resolution image generation apparatus according to the related art. The high-resolution image reconstruction unit 310 of the high-resolution image generation apparatus 300 according to the related art follows the high-resolution image reconstruction method of FIG.

The graphs 410 and 440 respectively represent the first high resolution image synthesized from the low resolution image and the second high resolution image reconstructed using the linear filtering or nonlinear filtering with the pixels p1, p2, p3, p4, p5, p6, p8. < / RTI >

In the graph 410 relating to the first high-resolution image synthesized from the low-resolution image, the luminance values 412, 414, 416, 420, 422, 424, and 426 of the pixels are luminance values obtained by synthesizing sample pixels of the low- However, even if a low-resolution image is synthesized, pixels in the empty space 418 in which the luminance value is not determined as in the case of the pixel p4 may occur due to insufficient information or uneven pixel distribution.

At least one of the luminance values 412, 414, 416, 420, 422, 424, and 426 of the surrounding pixels p1, p2, p3, p5, p6, p7, and p8 in the graph 440 relating to the reconstructed second high- By using interpolation, the reconstructed pixel value 448 for pixel p4 in the empty space 418 of the first high resolution image can be determined. The luminance values 412, 414, 416, 418, 420, 422, 424 and 426 of all the pixels, that is, the pixel p4 and the surrounding pixels pi, p2, p3, p5, p6, p7 and p8, 444, 446, 448, 450, 452, 454, and 456 since the image is reconstructed by interpolation filtering for the luminance values 442, 444, 446, 448,

In the case of restoring the current pixel by referring to surrounding pixel information in a flat area or an edge area having high correlation with neighboring pixels, the result of interpolation filtering using the luminance values of neighboring pixels may be highly reliable. However, In the region of texture component with low correlation, interpolation filtering referring to neighboring pixels may damage texture components.

For example, the luminance values 412, 414, 416, 420, 422, 424 and 426 of the pixels of the first high resolution image are distributed with respective details information about the slope 430, Detail information has a low correlation with each other. However, the luminance values 442, 444, 446, 448, 450, 452, 454, and 456 of the pixels of the second high resolution image are shifted by the interpolation filtering so that the detail information of each pixel having low correlation is lost, ). ≪ / RTI >

Therefore, according to the related art, the high-resolution image reconstruction operation using the linear filtering or the non-linear filtering of the high-resolution image reconstructing unit 310 may degrade the detail information of the high-resolution image and the image texture associated with the texture component.

Although the high-frequency component restoring unit 330 of the high-resolution image generating apparatus 300 according to the related art restores high-frequency components of the reconstructed high-resolution image, the high- It is difficult to reconstruct high frequency components that have been damaged or lost through reconstruction of the image.

FIG. 5 shows an exemplary block diagram for implementing a high-resolution image generating apparatus according to an embodiment.

The high-resolution image generating apparatus 500 using a multi-layer low-resolution image according to an exemplary embodiment is a specific embodiment of the high-resolution image generating apparatus 100 according to an embodiment.

The high resolution image generation apparatus 500 using the multi-layer low resolution image according to an exemplary embodiment includes a low resolution image hierarchy separation unit 510, a high resolution texture layer image generation unit 520, a high resolution base layer image generation unit 530, An image synthesis unit 540 and a motion estimation unit 550. [ The high-resolution image synthesis unit 540 includes a high-frequency component reconstruction unit 545.

The high-resolution image generation apparatus 500 according to an embodiment of the present invention includes a low-resolution image hierarchy separator 510 for separating each frame of a low-resolution image into a low-resolution image frame of a base layer and a texture Resolution image frame of the layer.

The high resolution texture layer image generating unit 520 synthesizes low resolution image frames of the texture layer to generate a high resolution image frame of the texture layer. The high resolution base layer image generating unit 530 synthesizes the low resolution image frames of the base layer, Resolution image frame of the high resolution image.

In the high-resolution image generating apparatus 500 according to the embodiment, the motion information of the neighboring frame with respect to the current frame of the low-resolution image expanded by the motion estimating unit 550 is obtained. The high-resolution texture layer image generating unit 520 and the high-resolution base layer image generating unit 530 respectively generate the high-resolution texture layer image and the high-resolution base layer image by using the motion information or the prediction information obtained by the motion estimating unit 550, A high-resolution image of the layer can be reconfigured in a temporally and temporally adaptive manner.

Further, by considering at least one of temporal proximity, spatial proximity, reliability of motion estimation, and luminance similarity between the current frame and the neighboring frames, it is possible to estimate the temporal or spatial proximity of the current frame of the low- A reconstructed high-resolution image of the corresponding layer can be generated.

The high resolution image synthesis unit 540 synthesizes the high resolution texture layer image and the high resolution base layer image generated by the high resolution texture layer image generation unit 520 and the high resolution base layer image generation unit 530 to generate an initial high resolution image . Initial high resolution images can be synthesized by weighted average filtering of high resolution texture layer images and high resolution base layer images.

In addition, the high-frequency component restoring unit 545 of the high-resolution image combining unit 540 may restore the high-frequency component additionally to the initial high-resolution image. The high-frequency component reconstructing unit 545 according to an exemplary embodiment may use filters having different filter characteristics to restore high-frequency components while minimizing the amplification of the initial high-resolution image noise. For example, a bilateral filter for minimizing amplification of image noise and a band-pass filter for restoring a high-frequency component may be used together.

Accordingly, the high-resolution image generation apparatus 500 using the multi-layer low-resolution image according to an exemplary embodiment can generate a high-resolution image without noise amplification while preserving the texture component of the image through a hierarchical process for the low- have.

6 is a schematic diagram of a high-resolution image reconstruction method of a high-resolution image generation apparatus according to an embodiment.

The high resolution texture layer image generating units 120 and 520 and the high resolution base layer image generating units 130 and 530 and the high resolution image synthesizing units 140 and 540 of the high resolution image generating apparatuses 100 and 500 according to the embodiment 6 high resolution image reconstruction method.

The graph 410 shows the luminance of the pixels of the first high-resolution image generated through the synthesis of the low-resolution images. The high resolution image generation apparatuses 100 and 500 according to an embodiment use a low resolution image as a low resolution texture layer image and a low resolution base layer image as a low resolution image. The high-resolution base-layer image generator 130 and 530 according to the embodiment combines a plurality of low-resolution base-layer images to generate an initial high-resolution base layer image corresponding to the graph 610, The generation units 120 and 520 synthesize a plurality of low resolution texture layer images to generate an initial high resolution texture layer image corresponding to the graph 630. [

The initial high-resolution base layer image of the graph 600 is composed of image components of the base layer excluding the texture component of the image and luminance values 602, 604, 606 of the pixels p1, p2, p3, p5, p6, , 610, 612, 614, and 616 may be listed according to a predetermined slope 430. Pixel p4 of the empty space 608 among the initial high-resolution base layer images may exist.

The initial high-resolution texture layer image of the graph 620 is composed of texture components of the image, and the luminance values 622, 624, 626, 630, 632, 634, 636 of the pixels p1, p2, p3, p5, ) Is characterized by a small degree of correlation between neighboring pixels. There may be pixels 628 of empty space in the initial high resolution texture layer image.

The high resolution base layer image generating units 130 and 530 according to the embodiment may reconstruct the high resolution base layer image by supplementing the pixel p4 of the empty space 608 of the initial high resolution base layer image. The high-resolution base layer image generation unit 130 and 530 according to an exemplary embodiment may perform reconstruction (reconstruction) on the pixel p4 of the empty space 618, considering temporal proximity, spatial proximity, luminance distance, Gt; 648 < / RTI >

In a similar manner, the pixels p1, p2, p3, p4, p5, p6, p7 and p8 of the high-resolution base layer image are reconstructed considering the relationship with surrounding pixels, respectively, to obtain the luminance values 642, 644, 646, 648, 650, 652, 654, 656). Accordingly, the luminance distribution of some pixels of the high-resolution base-layer image synthesized and reconstructed from the low-resolution base-layer images by the high-resolution base-layer image generators 130 and 530 according to an exemplary embodiment is according to the graph 640. [

The high resolution texture layer image generating units 120 and 520 according to the embodiment may reconstruct a high resolution texture layer image by supplementing the pixels of the blank space 628 of the initial high resolution texture layer image. The high resolution texture layer image generation unit 120 and 520 according to an exemplary embodiment of the present invention may be configured to generate the high resolution texture layer image in consideration of temporal proximity and similarity degree, spatial proximity and similarity with peripheral pixels, luminance distance, And assign a reconstructed luminance value 668 to the pixel.

In a similar manner, pixels p1, p2, p3, p4, p5, p6, p7 and p8 of a partial region of the high resolution texture hierarchy image are reconstructed based on the relationship with surrounding pixels, 666, 668, 670, 672, 674, 676). Accordingly, the luminance distribution of some pixels of the high-resolution texture layer image synthesized and reconstructed from the low-resolution texture layer images by the high-resolution texture layer image generation unit 120, 520 according to an embodiment is according to the graph 660. [

The high-resolution image synthesis unit 140 and 540 according to an exemplary embodiment may generate the high-resolution texture layer image generated from the high-resolution texture layer image generation units 120 and 520 and the high- And combines the high-resolution base layer images generated by the units 130 and 530. The luminance values of some pixels of the high resolution image synthesized by the high resolution image synthesizer 140 and 540 according to the embodiment are calculated as luminance values 682, 684, 686, 688, 690, 692, 694, 696). According to the graph 680, it is confirmed that a high-resolution image in which detail information is stored around the slope 430 is generated while all the pixels in the empty space are supplemented.

7 illustrates an exemplary block diagram for implementing a low-resolution image hierarchy separator of a high-resolution image generator according to an embodiment.

The low-resolution image hierarchy separating unit 700 corresponds to a specific embodiment for implementing the low-resolution image hierarchy separating units 110 and 510 of the high-resolution image generating apparatuses 100 and 500 according to the embodiment.

The low-resolution image input to the low-resolution image hierarchy separating unit 700 may be an image obtained by enlarging the original image. The original image can be enlarged by various interpolation techniques such as bilinear interpolation, bi-cubic interpolation, and Lanczos interpolation.

The low-resolution image hierarchy separating unit 700 may use a bilateral filter to separate the low-resolution image into a basic component and a texture component. The bilateral filter is an edge-preserving flattening filter that can extract a base-layer image composed of low-frequency components while preserving strong edge regions from the input image. The low-resolution image hierarchy separator 700 according to an exemplary embodiment may include a bilateral filtering unit 710 and a subtractor 720.

When the low resolution image is input to the low resolution image hierarchy separator 700, the bilateral filtering unit 710 generates and outputs a low resolution base layer image from the input low resolution image, and the subtractor 720 subtracts the low resolution image component from the low resolution image base It is possible to output a low resolution texture layer image minus a layer image component.

The low-resolution image hierarchy separating unit 700 is an embodiment of the low-resolution image hierarchy separating unit 110 and 510 according to the embodiment. The hierarchical separating of the low-resolution image is not limited to the bilateral filter, May be implemented by various techniques used to extract the texture component of the texture.

FIG. 8 illustrates an exemplary block diagram for implementing a high-resolution texture layer image generating unit or a high-resolution base layer image generating unit of the high-resolution image generating apparatus according to an exemplary embodiment.

For convenience of explanation, a high resolution texture layer image generating unit or a high resolution base layer image generating unit is referred to as a 'high-resolution image generating unit of a predetermined layer'. The high-resolution image generation unit 800 of the predetermined hierarchy may generate the high-resolution texture layer generation units 120 and 520 or the high-resolution base layer generation units 130 and 530 of the high- This is an implementation example.

The high-resolution image generating unit 800 of the predetermined hierarchy includes a temporal / spatial weighted sum filtering unit 810 and a weight determining unit 820. The weight for weighted sum filtering of the temporal / spatial weighted sum filtering unit 810 is obtained from the weight determining unit 820. The temporal / spatial weighted sum filtering unit 810 and the weight determining unit 820 can use the motion information with the surrounding information of the current pixel to reconstruct a high-resolution image based on the surrounding information of the current pixel.

The high resolution texture layer image generating unit or the high resolution base layer image generating unit may be constituted by the same components as the temporally weighted sum filtering 810 and the weight determining unit 820 of the high resolution image generating unit 800 of the predetermined hierarchy. However, since the weights for the input image and the temporal weighted weight are different for each layer, the resulting image can be distinguished as a high-resolution texture layer image and a high-resolution base layer image.

Specifically, when a low-resolution texture layer image is input to a high-resolution image generator 800 of a predetermined layer, the temporal / spatial weighted sum filtering unit 810 transforms the current frame of the low-resolution texture layer image and temporally and spatially adjacent frames Weighted sum filtering is performed. The weight determining unit 820 determines a weight of the low resolution texture layer image based on motion information and temporal proximity of temporally adjacent frames, spatial proximity of spatially adjacent frames, and reliability of prediction information, Spatiotemporal weights are determined for each successive frame of the image. The temporal / spatial weighted sum filtering unit 810 performs weighted sum filtering on the temporal and spatial neighboring frames and the current frame of the low-resolution texture hierarchy image using the temporal / spatial weights determined by the weight determining unit 820, Can be output.

In a similar manner, when a low-resolution base layer image is input to a high-resolution image generation unit 800 of a predetermined layer, the temporal / spatial weighted sum filtering unit 810 multiplies the current frame of the low-resolution base layer image and the temporally and spatially adjacent frames Sum filtering. The weight determining unit 820 determines a low-resolution basic layer image based on motion information and temporal proximity of temporally adjacent frames, spatial proximity of spatially adjacent frames, and reliability of prediction information with respect to the current frame of the low- The temporal and spatial weights are determined for each successive frame of the hierarchical image. The temporal / spatial weighted sum filtering unit 810 performs weighted sum filtering on the temporal and spatial neighboring frames and the current frame of the low-resolution base layer image using the temporal / spatial weights determined by the weight determining unit 820, Can be output.

The high-resolution image generation unit 800 of a predetermined hierarchy reconstructs a high-resolution image of a corresponding layer through weighted sum filtering using temporal weighting or spatial weighting considering proximity and similarity with temporal or spatial surrounding information, , But it can be distinguished from a weight determination method for weighted sum filtering on an image of a texture layer and a low-resolution image of a base layer.

For example, the weight determining unit 820 determines a weight for each frame by using a sum of absolute difference (SAD) between a current block and a neighboring block, a spatial distance from the center pixel, Distance, distance from the strong edge direction, and luminance distance to the center pixel.

Since the texture layers have a low spatial correlation with each other, the spatial characteristics of the image components need to be preserved. The spatial weight determining unit 820 may weaken the weight based on the spatial distance from the center pixel, the distance from the strong edge direction, and the luminance distance to the center pixel in order to determine the weight for reconstruction of the high resolution texture layer image .

In order to reconstruct the high-resolution base layer image, the edge must be preserved, and therefore planarization is required considering the edge direction. Accordingly, the weight determining unit 820 can enhance the weight based on the distance from the strong edge direction to determine the weight for reconstruction of the high-resolution base layer image.

According to this method, the high-resolution image generator 800 of a predetermined layer generates a high-resolution image of the corresponding layer through a weighted sum using a plurality of low-resolution images of the corresponding layer. Therefore, Can be reconstructed.

The high-resolution texture layer image and the high-resolution base layer image output from the high-resolution image generation unit 800 of a predetermined hierarchical level are output to the high-resolution image synthesis units 140 and 540 of the high- Can be synthesized into one high resolution image.

The high-resolution image synthesis unit 140 and 540 according to the embodiment may synthesize high-resolution images by weighted averages of the high-resolution texture layer image and the high-resolution base layer image. For each weighted average of the high resolution texture layer image and the high resolution base layer image, each weight may be determined according to which layer of the texture component of the texture layer and the image component of the base layer is to be further emphasized.

The high-frequency component reconstructing unit 545 of the high-resolution image combining unit 540 according to an embodiment can reconstruct the high-frequency component generated by the weighted average of the high-resolution texture hierarchy image and the high-resolution base layer image .

FIG. 9 is a block diagram illustrating an example of a high-frequency component reconstruction unit in the high-resolution image synthesis unit of the high-resolution image generation apparatus according to an embodiment.

The high frequency component reconstructing unit 900 corresponds to an embodiment of the high frequency component reconstructing unit 545 of the high resolution image synthesizing unit 540 of the high resolution image generating apparatus 500 according to an embodiment.

The high frequency component restoring unit 900 includes a bilateral filtering unit 910, a bandpass filtering unit 920, multipliers 930, 940 and 950 and an adder 960. The high-frequency component restoring unit 900 may be a band-pass filter that is distinguished according to frequency bands, as an example of a filter for restoring damaged high-frequency components of an initial high-resolution image. Also, the high-frequency component restoring unit 900 may use at least one bilateral filter that is distinguished according to band characteristics, as an example of a filter for preventing noise amplification.

The bilateral filtering unit 910 may include a plurality of bilateral filter groups having different band characteristics according to frequency bands. In order to recover a high frequency component by frequency band, the band pass filtering unit 920 may be configured of a plurality of band pass filter groups that are distinguished by frequency bands.

The high frequency component reconstructing unit 900 performs a weighted sum filtering of the initial high resolution image, the result image of the bi-lateral filtering unit 910, and the result image of the band pass filtering unit 920 to generate a high- Can be generated. The weight values a, b, and c for each of the initial high resolution image, the resultant image of the bi-lateral filtering unit 910, and the resultant image of the bandpass filtering unit 920 may be determined according to which image component is emphasized .

The multipliers 930, 940 and 950 multiply the weighted values a, b and c by the initial high resolution image, the resultant image by the bilateral filtering unit 910, and the resultant image by the bandpass filtering unit 920, The values are summed by an adder 960. In the operation of the adder 960, a subtraction operation is performed inversely to a result corresponding to the bandpass filtering unit 920, whereby a high frequency component excluding an image component having passed through the frequency band can be restored.

The high frequency component reconstructing unit 900 performs a weighted sum filtering of an initial high resolution image, a result image of the bi-lateral filtering unit 910, and a result image of the band pass filtering unit 920, Can be restored, and a high-resolution image in which a high-frequency component is reconstructed can be output while noise amplification is prevented.

The high-frequency component reconstructing unit 900 is an example of the high-frequency component reconstructing unit 545 according to the embodiment. The filter for amplifying the image noise is not limited to the bilateral filter, and the filter for restoring the high- Pass filter, and may be implemented by various techniques used for the same purpose in the same technical field.

FIG. 10 is a block diagram of a high-resolution image generation apparatus using a multi-layer low-resolution image according to another embodiment of the present invention.

The high resolution image generating apparatus 1000 using a multi-layer low resolution image according to another embodiment includes a low resolution image multi-layer separator 1010, a hierarchical high resolution image generator 1020, and a multi-layer high resolution image synthesizer 1030, .

The low-resolution image multi-layer separator 1010 according to another embodiment separates the low-resolution image input to the high-resolution image generation apparatus 100 into low-resolution images classified into layers. The low-resolution image multi-layer separator 1010 according to another embodiment can separate the low-resolution images into a plurality of low-resolution images for each layer from at least one low-resolution image.

The low-resolution image multi-layer separator 1010 according to another embodiment outputs the low-resolution images separated into layers according to the layer to the layer-by-layer high-resolution image generator 1020 according to an embodiment.

The layer-by-layer high-resolution image generation unit 1020 according to another embodiment may further include a layer-by-layer low-resolution image generation unit 1022 corresponding to each layer, , A second layer high resolution image generation unit 1024, and an Nth layer high resolution image generation unit 1026. The layer-by-layer high-resolution image generation unit 1020 according to another embodiment combines a plurality of low-resolution images of the layer in each layer to generate a high-resolution image of the layer.

More specifically, a first-layer high-resolution image is generated from the first layer high-resolution image generation unit 1022, a second-layer high-resolution image is generated from the second layer high-resolution image generation unit 1024, The high-resolution image of the N-th layer is generated from the image generating unit 1026.

In this manner, the layer-by-layer high-resolution image generation unit 1020 generates high-resolution images of the corresponding layer for each layer and outputs the generated high-resolution images to the multi-layer high-resolution image synthesis unit 140 .

The multi-layer high-resolution image synthesis unit 1030 according to another embodiment synthesizes the inputted high-resolution images for each layer to output a high-resolution image. The multi-layer high-resolution image synthesis unit 1030 can determine the weight for each layer in consideration of characteristics of each layer of the high-resolution images for each layer. The multi-layer high-resolution image synthesis unit 1030 can synthesize high-resolution images through weighted average filtering of the high-resolution images by hierarchy using the hierarchical weights. In order to restore a high frequency component from the synthesized high resolution image, the multi-layer high-resolution image synthesis unit 1030 can amplify a high-frequency component while minimizing noise of a high-resolution image.

11 is a flowchart illustrating a method of generating a high-resolution image using a multi-layer low-resolution image according to an embodiment of the present invention.

In step 1110, the low-resolution image is divided into a plurality of low-resolution texture layer images and a plurality of low-resolution base layer images.

In step 1120, a plurality of low resolution texture layer images are synthesized to generate a high resolution texture layer image, and a plurality of low resolution base layer images are synthesized to generate a high resolution base layer image.

A high resolution texture layer image can be generated by weighted sum filtering of the current frame, temporally adjacent frames, and spatial adjacent frames of each low resolution texture layer image, and the current frame of each low resolution base layer image, A high-resolution base layer image can be generated through weighted sum filtering on the neighboring frames and the spatial neighboring frames. The weight of the weighted sum filtering may be determined based on at least one of spatial proximity or similarity, temporal proximity or similarity for each current frame of each adjacent frame, luminance distance, proximity to a strong edge, and reliability of motion estimation.

In step 1130, a high resolution texture layer image and a high resolution base layer image are synthesized to output a high resolution image. High resolution images can be synthesized by weighted average filtering on high resolution texture layer images and high resolution base layer images.

A weight for minimizing the damage of the texture component of the original image is determined for the high resolution texture layer image and a weight for the flattening considering the edge direction of the original image can be determined for the high resolution base layer image.

In order to recover the damaged high-frequency component of the high-resolution image, filtering for restoration of high-frequency components and filtering for preventing noise amplification may be performed on the high-resolution image.

The method and apparatus for generating a high-resolution image according to an exemplary embodiment uses weighted sum filtering for high-resolution image synthesis for each layer and weighted sum filtering for restoring high-frequency components of a high-resolution image, And the high-resolution image can be reconstructed from the low-resolution image while minimizing the damage of the high-frequency component and preventing the noise amplification in consideration of the surrounding information in time and space.

The above-described embodiments of the present invention can be embodied in a general-purpose digital computer that can be embodied as a program that can be executed by a computer and operates the program using a computer-readable recording medium. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

The present invention has been described with reference to the preferred embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

Claims (25)

Dividing a plurality of low resolution images into a plurality of low resolution texture layer images and a plurality of low resolution base layer images;
A first low resolution texture layer image of the plurality of low resolution texture layer images, temporally adjacent low resolution texture layer images of the first low resolution texture layer image, and spatial adjacent low resolution texture layer images of the first low resolution texture layer image, Generating a high resolution texture layer image through a first weighted sum filtering;
A first low-resolution base layer image of the plurality of low-resolution base layer images, temporally adjacent low-resolution base layer images of the first low-resolution base layer image, and a plurality of spatial low-resolution base layer images of the first low- Generating a high resolution base layer image through a second weighted sum filtering; And
Synthesizing the high resolution texture layer image and the high resolution base layer image to output a high resolution image,
Wherein the plurality of low resolution base layer images include low frequency components and strong directional edge components of the plurality of low resolution images and the plurality of low resolution texture layer images include low frequency components and strong directional components from the plurality of low resolution images, And the edge components of the low-resolution image are excluded. The method as claimed in claim 1, wherein the high-
And reconstructing the high-frequency component of the high-resolution image. delete delete The method according to claim 1,
Generating a first low-resolution base layer image by applying a filter for flattening an image while preserving an edge component of the image for the first low-resolution image among the plurality of low-resolution images; And
And generating a difference image of the first low resolution image and the first low resolution base layer image as the first low resolution texture layer image using the multi-layer low resolution image. The method of claim 1, wherein the high resolution texture layer image generation step comprises:
Generating the high resolution texture layer image through non-recursive filtering on the plurality of low resolution texture layer images,
In the high-resolution base layer image generation step,
And generating the high-resolution base layer image through non-recursive filtering on the plurality of low-resolution base layer images. delete The method according to claim 1,
The temporal neighbor images and the spatial neighbor images of the first low resolution texture layer image are determined from the motion estimation or motion compensation images in contrast to the first low resolution texture layer image,
Wherein the temporally adjacent images and the spatial neighbor images of the first low-resolution base layer image are determined from the motion-compensated or motion-compensated images in contrast to the first low-resolution base layer image. Image generation method. The method according to claim 1,
Wherein the weight of the first low-weight texture layer image of each of the first low-resolution texture layer images and the temporally adjacent low-resolution texture layer images of the first low- Is determined based on at least one of spatial proximity to the image, temporal proximity, luminance difference, proximity to a strong edge, and reliability of motion estimation,
Wherein the weight of the second weighted sum filtering is calculated by multiplying the temporally adjacent low resolution texture layer images of each of the first low resolution base layer images and the first low resolution base layer of the spatial low resolution base layer images of the first low resolution base layer image, Wherein the determination is based on at least one of spatial proximity to the image, temporal proximity, luminance difference, proximity to a strong edge, and reliability of motion estimation. The method as claimed in claim 1, wherein the high-
Wherein the high resolution image is synthesized by weighted average filtering of the high resolution texture layer image and the high resolution base layer image. 11. The method of claim 10,
The weighted average filtering is performed by applying a first weight for minimizing the damage of the texture component of the original image to the high resolution texture layer image and a second weight for flattening considering the edge direction of the original image, Wherein the method is applied to an image. 3. The method of claim 2, wherein the high-
And high-frequency components of the high-resolution image are reconstructed through non-artificial filtering of the high-resolution image. 13. The method of claim 12, wherein the high-
Wherein the first filtering result for restoring the damaged high-frequency component of the high-resolution image and the second filtering for preventing the noise amplification are weighted sum of the first filtering result for the high-resolution image and the second filtering result for the high- And synthesizing the high-resolution image by filtering the high-resolution image. 14. The method of claim 13,
Wherein the first filtering includes at least one bandpass filter that is distinguished according to a band characteristic. 14. The method of claim 13,
Wherein the second filtering includes at least one bilateral filter that is distinguished according to a band characteristic. The method according to claim 1,
Wherein the plurality of low resolution images are images obtained by enlarging original images. 6. The method of claim 5,
Wherein the filter for flattening an image while preserving an edge component of the image includes a bilateral filter. A low resolution image layer separator for separating a plurality of low resolution images into a plurality of low resolution texture layer images and a plurality of low resolution base layer images;
A first low resolution texture layer image of the plurality of low resolution texture layer images, temporally adjacent low resolution texture layer images of the first low resolution texture layer image, and spatial adjacent low resolution texture layer images of the first low resolution texture layer image, A high resolution texture layer image generation unit for generating a high resolution texture layer image through first weighted sum filtering;
A first low-resolution base layer image of the plurality of low-resolution base layer images, temporally adjacent low-resolution base layer images of the first low-resolution base layer image, and a plurality of spatial low-resolution base layer images of the first low- A high-resolution base layer image generating unit for generating a high-resolution base layer image through a second weighted sum filtering; And
And a high-resolution image synthesizer for synthesizing the high-resolution texture layer image and the high-resolution base layer image to output a high-resolution image,
Wherein the plurality of low resolution base layer images include low frequency components and strong directional edge components of the plurality of low resolution images and the plurality of low resolution texture layer images include low frequency components and strong directional components from the plurality of low resolution images, And the edge components of the low-resolution image are excluded. 19. The apparatus of claim 18, wherein the high-
And a high-frequency component reconstructing unit for reconstructing the high-frequency component of the high-resolution image. 19. The apparatus of claim 18, wherein the high-resolution image generator using the multi-
And a motion estimator for estimating motion information between the first low-resolution image and the consecutive neighboring images among the plurality of low-resolution images. 19. The apparatus of claim 18, wherein the high-
Wherein the high resolution image is synthesized by weighted average filtering of the high resolution texture layer image and the high resolution base layer image. 20. The apparatus of claim 19, wherein the high-
Wherein the first filtering result for restoring the damaged high-frequency component of the high-resolution image and the second filtering for preventing the noise amplification are weighted sum of the first filtering result for the high-resolution image and the second filtering result for the high- And the high-resolution image is synthesized by filtering the high-resolution image. 19. The method of claim 18,
Wherein at least one of the high resolution texture layer image generator, the high resolution base layer image generator, and the high resolution image composer performs non-recursive weighted sum filtering on the input image, High resolution image generation device using. delete A computer-readable medium having recorded thereon a program for implementing a method for generating a high-resolution image using a multi-layer low-resolution image according to any one of claims 1 to 5, 5 to 6, and 8 to 17, Recording medium.

Images