KR20180097342A - Method and apparatus for sharpening super resolution image with minimum image artifacts - Google Patents

Abstract

An image sharpening method and apparatus capable of reducing calculation while improving a subjective image quality to an original image quality are disclosed. A method for reconstructing a super-resolution image according to an aspect of the present invention comprises a step of dividing an input image into a low-frequency image having an input resolution and a high-frequency image having an input resolution, a step of generating a low-frequency image of a high-rank resolution from the input image; a step of reconstructing a high-frequency image of a high-rank resolution using the low-frequency image of the input resolution, the high-frequency image of the input resolution, and the low-frequency image of the high-rank resolution; and a step of generating a reconstructed image of an high-rank resolution using the low-frequency image of the high-rank resolution and the high-frequency image of the high-rank resolution.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for sharpening an image with minimal image artifacts,

The present disclosure relates to a super resolution image restoration technique, and more particularly, to a method and apparatus for sharpening a super resolution image.

Recently, various TV manufacturers have launched UHD (Ultra High Definition) TV with 4 times higher resolution of HDTV, so interest in UHD service is increasing. However, UHD contents are still insufficient, and UHD contents are important for successful UHD broadcasting service. Accordingly, a super resolution image restoration algorithm capable of converting existing contents into high quality UHD contents is receiving attention as a core technology of a mobile terminal and a TV.

Reconstruction-based technology that predicts and minimizes errors by using multiple low-resolution images and camera modeling to reconstruct super-resolution images, and low-resolution, high-resolution patch pairs And an example-based reconstruction technique for reconstructing an image using the reconstructed image. In particular, the self-similarity-based super resolution technique of the example-based technique does not require a huge amount of external information, and is getting a lot of attention because it restores a high-resolution image efficiently using the information of the self-similar region of the input image.

The above-mentioned super-resolution image restoration methods can generate a sharp image compared to a conventional interpolation method such as bilinear or bicubic. However, when compared with the image obtained from the beginning with high resolution, the sharpness of the image is insufficient. In order to compensate for this problem, it is required to further sharpen the image using information obtained from super resolution image restoration. In addition, there is a need for an improvement measure that can minimize the occurrence of artifacts that deteriorate the subjective image quality.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image sharpening method and apparatus capable of improving the subjective image quality to the original image level.

It is another object of the present invention to provide an image sharpening method and apparatus capable of minimizing the occurrence of image artifacts.

It is another object of the present invention to provide a super-resolution image sharpening method and apparatus capable of minimizing the amount of calculation required for interpolation of an image.

The technical objects to be achieved by the present disclosure are not limited to the above-mentioned technical subjects, and other technical subjects which are not mentioned are to be clearly understood from the following description to those skilled in the art It will be possible.

According to an aspect of the present disclosure, a method for restoring a super-resolution image can be provided. The method includes: dividing an input image into a low-frequency image having an input resolution and a high-frequency image having an input resolution; Generating a low-resolution image of an upper resolution from the input image; Reconstructing a high-resolution high-frequency image using the low-frequency image having the input resolution, the high-frequency image having the input resolution, and the low-resolution image having the high resolution; And generating a reconstructed image having an upper resolution using the low-frequency image of the higher resolution and the high-frequency image of the higher resolution.

The step of dividing the input image into a low-frequency image of the input resolution and a high-frequency image of the input resolution may be performed by applying a low-pass filter to the input image, The pass filter may be a Gaussian filter or a Butterworth filter.

In the super resolution image restoration method according to the present disclosure, the step of generating the low resolution image of the higher resolution from the input image may include interpolating the input image based on the higher resolution, Linear interpolation, bicubic interpolation, or lanth-order interpolation.

In the method of restoring super resolution image according to the present disclosure, restoring the high resolution image using the low resolution image having the input resolution, the high resolution image having the input resolution, and the low resolution image having the high resolution, Dividing the low-frequency image of the image into one or more patches; Retrieving a similar patch similar to the patch from low frequency images of the input resolution; Identifying a high frequency patch corresponding to the retrieved similar patch from a high frequency image of the input resolution; And reconstructing the high-resolution high-frequency image using the specified high-frequency patch.

In the super resolution image restoration method according to the present disclosure, the high frequency patch corresponding to the retrieved similar patch may be a patch in the high frequency image of the input resolution having the same coordinates as the retrieved similar patch.

In the super resolution image restoration method according to the present disclosure, when two or more similar patches are searched, each of the two or more similar patches has similarity information, and the two or more similar patches corresponding to each of the two or more similar patches It is possible to determine the combined high frequency patch by applying the weight based on the similarity information.

The super resolution image reconstruction method according to the present disclosure may further include amplifying the high resolution image having the higher resolution, wherein the step of amplifying the high resolution image having the higher resolution comprises: Determining a type of at least one image area included in the restored image of the image; Generating a high frequency amplification coefficient map based on the discriminated type of the image region; And amplifying the high-resolution high-frequency image using the high-frequency amplification coefficient map, wherein the step of generating the reconstructed image having the higher resolution using the high-resolution low-frequency image and the high-resolution high- , The amplified upper-resolution high-frequency image may be used instead of the higher-resolution high-frequency image.

In the method for reconstructing a super-resolution image according to the present disclosure, the type of the image region includes a flat region, an edge region, or a texture region, and the step of generating the high- Flatness region or the amplification factor for the texture region.

The super resolution image restoration method according to the present disclosure may further include a background projection process and the background projection process may include generating a restored image of the input resolution from the restored image of the higher resolution, ; Generating a difference image between the reconstructed image of the input resolution and the input image; Generating a difference image of an upper resolution from the difference image; And generating a final reconstructed image having an upper resolution by applying the difference image having the higher resolution to the reconstructed image having the higher resolution.

The generating of the reconstructed image having the input resolution from the reconstructed image having the higher resolution may include downsampling the reconstructed image having the higher resolution to the input resolution to reconstruct the reconstructed image having the input resolution, Can be generated.

In the super resolution image restoration method according to the present disclosure, the step of generating the difference image of the higher resolution from the difference image may generate the difference image of the higher resolution by upsampling the difference image to an upper resolution.

In the super resolution image restoration method according to the present disclosure, the final reconstruction image of the higher resolution, which is the output of the background projection process, is input again as the reconstruction image of the higher resolution which is the input of the background projection process, More than two times can be performed.

In the super resolution image restoration method according to the present disclosure, the bicubic interpolation is performed by grouping a plurality of pixels in a reference area, which is a block of NxN (N is an integer equal to or greater than 2), into N groups including N pixels step; Performing a first order interpolation for each of the N groups; And deriving a resultant pixel value by performing a second-order interpolation using a result of the first-order interpolation for each of the N groups, wherein the grouping step comprises: The N pixels belonging to the same row can be grouped into the same group.

In the super resolution image restoration method according to the present disclosure, when the reference area for deriving the one result pixel value overlaps with another reference area for deriving another result pixel value, The result of the primary interpolation for the group may be re-used as a result of the primary interpolation to derive the other resulting pixel value.

In the method of restoring super resolution images according to the present disclosure, the step of performing the first-order interpolation is performed using a cubic spline function, and the tension coefficient determining the continuity of the cubic spline function is determined by a spatial complexity Can be adaptively adjusted.

In the method of restoring a super-resolution image according to the present disclosure, the tension coefficient can be largely adjusted as the spatial complexity of the input image is lower.

In the super resolution image restoration method according to the present disclosure, the spatial complexity of the input image may be determined based on whether a ratio of the flat area included in the input image exceeds a predetermined threshold value.

In the super-resolution image restoration method according to the present disclosure, an interpolation parameter used in the step of generating the low-resolution image of the higher resolution from the input image is adaptively determined based on characteristics of the difference image or the difference image of the higher resolution .

In the super resolution image restoration method according to the present disclosure, the characteristic of the difference image or the difference image of the higher resolution is a statistical characteristic of a line artifact included in the difference image or the difference image of the higher resolution, The interpolation parameter used in the step of generating the low-resolution image of the higher resolution may be a tension coefficient determining the continuity of the cubic spline function.

According to another aspect of the present disclosure, a super-resolution image restoration apparatus can be provided. The apparatus includes a frequency domain division unit that divides an input image into a low-frequency image having an input resolution and a high-frequency image having an input resolution; A resolution enhancement unit for generating a low-resolution image of an upper resolution from the input image; A high frequency signal reconstruction unit for reconstructing a high-resolution high-frequency image using the low-frequency image having the input resolution, the high-frequency image having the input resolution, and the low-resolution image having the high resolution; And a frequency domain fusion unit for generating a reconstructed image of an upper resolution using the high-resolution low-frequency image and the high-resolution high-frequency image.

The features briefly summarized above for this disclosure are only exemplary aspects of the detailed description of the disclosure which follow, and are not intended to limit the scope of the disclosure.

According to the present disclosure, an image sharpening method and apparatus capable of improving the subjective image quality to the original image level can be provided.

Further, according to the present disclosure, an image sharpening method and apparatus capable of minimizing image artifact generation can be provided.

In addition, according to the present disclosure, a super-resolution image sharpening method and apparatus capable of minimizing the amount of calculation required for interpolation of an image can be provided.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below will be.

)에 따른 큐빅 스플라인 함수의 형태를 예시적으로 나타내는 도면이다.
도 10은 장력 계수(

)에 따른 초해상도 영상 복원 결과를 예시적으로 나타내는 도면이다.
도 11은 영상 아티팩트를 최소화하는 초해상도 선명화 방법의 또 다른 일 실시 예의 구성을 나타내는 블록도이다.
도 12는 도 11의 초해상도 영상 복원부(1110)의 구조를 예시적으로 나타내는 블록도이다.
도 13은 도 11의 배경 투사부(1120)의 구조를 예시적으로 나타내는 블록도이다.FIG. 1 is a view for explaining an embodiment of an apparatus and method for restoring a super-resolution image according to the present disclosure.
FIG. 2 is a view for schematically explaining a hierarchical super-resolution image restoration method.
FIG. 3 is an exemplary diagram illustrating image artifacts caused by regions determined to be edges.
4 is a block diagram exemplarily showing a configuration of the resolution enhancing unit 120 of FIG.
5 is a block diagram exemplarily showing a configuration of the background projection unit 170 of FIG.
6 is a diagram exemplarily showing a super resolution image generated by applying bilinear, bi-cubic, and lanchotz interpolation methods to the image interpolators 410 and 520, respectively.
FIG. 7 is an exemplary diagram illustrating a reference area when carrying out bicubic interpolation. FIG.
8 is a diagram illustrating an example of interpolation using cubic splines.
Fig. 9 is a graph showing the relationship between the tensile coefficient

) Of a cubic spline function according to an embodiment of the present invention.
10 is a graph showing a relationship between the tensile coefficient

In FIG. 5). FIG.
11 is a block diagram showing a configuration of still another embodiment of a super resolution sharpening method for minimizing image artifacts.
12 is a block diagram exemplarily showing a structure of the super-resolution image restoration unit 1110 of FIG.
13 is a block diagram exemplarily showing a structure of the background projection unit 1120 of FIG.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily understood by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. Parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.

In the present disclosure, when an element is referred to as being "connected", "coupled", or "connected" to another element, it is understood that not only a direct connection relationship but also an indirect connection relationship May also be included. Also, when an element is referred to as " comprising " or " having " another element, it is meant to include not only excluding another element but also another element .

In the present disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements, etc. unless specifically stated otherwise. Thus, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly a second component in one embodiment may be referred to as a first component .

In the present disclosure, the components that are distinguished from each other are intended to clearly illustrate each feature and do not necessarily mean that components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of this disclosure.

In the present disclosure, the components described in the various embodiments are not necessarily essential components, and some may be optional components. Thus, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Also, embodiments that include other elements in addition to the elements described in the various embodiments are also included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

FIG. 1 is a view for explaining an embodiment of an apparatus and method for restoring a super-resolution image according to the present disclosure. Hereinafter, the image restoration apparatus and method may be used in the same manner as the image sharpening apparatus and method.

In the following embodiments, a description will be given mainly to an example of a super-resolution image restoration process based on self similarity as shown in the block diagram of FIG. However, the present disclosure is not limited thereto and can be applied to other super-resolution image restoration methods and apparatuses.

Referring to FIG. 1, a super-resolution image restoration apparatus includes a frequency domain partitioning unit 110, a resolution enhancing unit 120, a similar patch searching unit 130, a high frequency patch mapping unit 140, an image region determining unit 150, A frequency-domain fusion unit 160, and / or a background projection unit 170. [

The input image may be divided into a low frequency region image L ₀ and a high frequency region image H ₀ in the frequency domain division unit 110. The low frequency region image L ₀ can be obtained by applying a low pass filter to the input image. The high frequency region image H ₀ can be obtained by using the difference between the input image and the obtained low frequency region image L ₀ . Thus, frequency domain division of the input image can be achieved. For example, the low pass filter may include a Gaussian filter, a Butterworth filter, and the like.

The image restoration method using self similarity can restore the image using both the low frequency region image (L ₀ ) and the high frequency region image (H ₀ ). Therefore, frequency domain segmentation is required for the input image.

The resolution enhancement unit 120 may perform an upsampling on the input image to convert the input image into an upper resolution image. The upper resolution image may mean an image having a resolution larger than that of the input image.

When upsampling an input image, an interpolation method can be used. For example, the interpolation method may include bilinear interpolation, bicubic interpolation, Lanczos interpolation interpolation, and the like. Alternatively, other methods may be used. The image up-sampled by the resolution enhancement unit 120 may be used as a low-frequency region image L ₁ of an upper resolution image.

The similar patch search unit 130 may divide the interpolated upper resolution image L ₁ by the resolution enhancement unit 120 in units of patches. The similar patch search unit 130 may search similar patches in the low frequency region image L ₀ of the input image for each patch constituting the upper resolution image L ₁ . Here, a patch may mean a set of pixels having a specific size. For example, a patch may mean a piece of video having a 5x5 pixel size.

For similar patch search, we can use intensity based similarity measure, similarity measure reflecting visual perception characteristics, and feature based similarity measure comparing object features in image.

Through the similar patch search, one or more similar patches can be retrieved from the low frequency region image L ₀ of the input image.

The high frequency patch mapping unit 140 can determine a patch corresponding to the searched optimal similar patch from the high frequency region image H ₀ of the input image. The high-frequency patch mapping unit 140 can restore the high-resolution region image H ₁ of an upper resolution using the determined patches. Here, the correspondence may mean that they have the same coordinates.

When the similar-patch searching unit 130 searches for a similar patch, the high-frequency patch mapping unit 140 may use a high-frequency image patch corresponding to one similar patch searched for. When the similar patch search unit 130 searches for a plurality of similar patches, similarity information for each of a plurality of similar patches may be given. In this case, the high frequency patch mapping unit 140 may apply a weight based on the degree of similarity to a plurality of corresponding high frequency image patches. It is possible to reconstruct the high-frequency region image H ₁ of an upper resolution by combining, for example, using a weighted high-frequency image patch.

When the similar-patch searching unit 130 divides the upper resolution image L ₁ into patch units, the patch may be divided so that two or more patches are overlapped. If the patches are divided so as to overlap, there may exist overlapped regions between the patches of the reconstructed high-resolution region image of high resolution. In this case, the high-frequency patch mapping unit 140 can restore the high-resolution image H ₁ by performing the post-processing step and combining the overlapped patches. For example, in the post-processing step, a weighted average value processing based on the similarity can be performed for overlapping regions between patches.

The frequency domain fusion unit 160 uses the high resolution image signal L ₁ interpolated by the resolution enhancement unit 120 and the high resolution high frequency signal H ₁ reconstructed by the high frequency patch mapping unit 140, Signals can be summed to generate a reconstructed image.

The frequency domain fusion unit 160 can generate a reconstructed image using the high frequency signal H ₁ generated by the high frequency patch mapping unit 140 as it is. Alternatively, the frequency domain fusion unit 160 may generate a reconstructed image using the signal H ₁ 'obtained by amplifying the high-frequency signal H ₁ generated by the high-frequency patch mapping unit 140. When the amplified high-frequency signal H ₁ 'is used, a reconstructed image that is visually clear can be generated.

Generally, an image region having different characteristics such as flat, edge, and texture may exist in an image.

When amplifying the high-frequency signal H ₁ generated by the high-frequency patch mapping unit 140, applying the same amplification factor to all the image regions may cause visual artifacts that are visually disturbed. Such image artifacts can be more visibly visualized, especially in the flat image region.

Therefore, it is necessary to adaptively amplify the high frequency signal in consideration of characteristics (flag, edge, texture, etc.) of the image region. Thus, the sharpness of the image can be improved while minimizing the deterioration of image quality due to image artifacts.

The video region determination unit 150 may perform the determination on the video region in order to adaptively apply the high frequency signal amplification coefficient according to the characteristics of the video region. The discrimination of the image region is performed on the image generated by adding the interpolated upper resolution image L ₁ from the resolution enhancement unit 120 and the upper resolution high frequency signal H ₁ restored by the high frequency patch mapping unit 140 . The image region determining unit 150 may output a high frequency amplification coefficient map after identifying the image region. Here, the high frequency amplification coefficient map may mean a map in which an image region is classified according to its characteristics, and then an amplification coefficient is assigned to each image region. The classification of the image region and / or the allocation of the amplification factor may be performed for each pixel of the image.

The optimal amplification factor for each image region may be determined based on a subjective image quality assessment or may be determined by an algorithm based on perceptual quality modeling.

When the amplification factor is determined based on the subjective image quality evaluation, the amplification coefficient of the image area is gradually increased to allow the participants to view the image, and then the amplification factor when the participant detects the image quality deterioration in the relevant image area It is possible to proceed with the experiment by recording the value.

As a result, the amplification coefficients of 1.5, 2.0, and 1.5 were determined for the flat image area, the edge image area, and the texture image area, respectively, as a result of performing experiments using four UHD resolution videos of 20 experiment participants . That is, it can be confirmed that the high-frequency signal can be amplified most greatly without the perceptual deterioration in the edge region.

The above experimental value can be used as the high frequency amplification factor for each region. Alternatively, a subjective image quality evaluation may be separately performed for an application scenario to obtain a high-frequency amplification coefficient.

In this case, the perceived image quality characteristics may be different for each participant. Therefore, it is possible to obtain statistical reliability by conducting experiments with a sufficient number of experiment participants.

The image area discrimination can be performed using Equation (1) below.

[Equation 1]

란 영상 프레임 내 i번째 픽셀의 x축 방향 1차 그라디언트 값을 의미하고,

는 영상 프레임 내 i번째 픽셀의 y축 방향 1차 그라디언트 값을 의미한다. 상기 수학식 1에서

는 영상 프레임 내 i번째 픽셀의 x축 방향 2차 그라디언트 값을 의미하고,

는 영상 프레임 내 i번째 픽셀의 y축 방향 2차 그라디언트 값을 의미하며,

는 영상 프레임 내 i번째 픽셀의 x축 방향 1차 그라디언트 값에 y축 방향 1차 그라디언트를 적용한 것을 의미한다. In Equation (1)

Denotes a first-order gradient value of an i-th pixel in an image frame in the x-axis direction,

Denotes the y-axis direction primary gradient value of the i-th pixel in the image frame. In Equation (1)

Denotes a secondary gradient value in the x-axis direction of the i-th pixel in the image frame,

Means a secondary gradient value in the y-axis direction of the i-th pixel in the image frame,

Means that the first-order gradient in the y-axis is applied to the first-order gradient value in the x-axis direction of the i-th pixel in the image frame.

In Equation (1), N and E denote empirically determined thresholds for discriminating image regions, and IND denotes an impulse function for outputting a value of 1 when the inequality in the function is satisfied. do.

When an image region is discriminated on a pixel-by-pixel basis according to Equation (1) and area adaptive sharpening is performed according to a discrimination result, an image region of pixels constituting the object is delicately defined differently for the same object between adjacent frames , Temporal artifacts such as image flicker may occur.

In order to mitigate the occurrence of temporal artifacts, a morphology filter such as erosion and / or dilation may be applied to the high-frequency amplification coefficient map to improve the image domain matching between adjacent frames.

The generated high frequency amplification coefficient map may be applied to the high frequency signal H ₁ generated by the high frequency patch mapping unit 140. For example, by multiplying the high-frequency amplification coefficient map by the generated high-frequency signal H ₁ , the high-frequency signal H ₁ 'amplified region-adaptively can be generated.

The frequency domain fusion unit 160 may generate a reconstructed image using the region-adaptively amplified high frequency signal H1 'and the upper resolution image L ₁ generated by the resolution enhancement unit 120. For example, the reconstructed image can be generated by adding the amplified high frequency signal H1 'and the upper resolution image L ₁ .

The background projection unit 170 can ensure consistency between the reconstructed image generated by the frequency domain fusion unit 160 and the input image. For this, the background projecting unit 170 may downsample the reconstructed image to the resolution of the input image, and then grasp the residual of the reconstructed image and the input image. The background projecting unit 170 can upscale the difference to an upper resolution and apply it to the restored image again. The process performed by the background projection unit 170 may be referred to as a backward projection process. In some cases, the background projection can be repeated several times. The super resolution image on which the background projection has been performed can be output as an output image.

FIG. 2 is a view for schematically explaining a hierarchical super-resolution image restoration method.

The super-resolution image restoration method described with reference to FIG. 1 can be repeatedly applied hierarchically. Through the repeated application of the super resolution image restoration method, the resolution of the input image can be increased step by step. For example, by increasing the resolution of the input image by 1.25 times, the resolution of the input image can be doubled (1.25x1.25x1.25? 2) by performing iteratively (or recursively) three times. Compared with the case where the resolution is doubled, the similarity between the resolutions is higher when the resolution is increased by 1.25 times. Therefore, as compared with the case of performing the process of doubling the resolution twice, it is possible to restore the high-frequency signal more accurately and accurately when the process of raising the resolution by 1.25 times is repeated three times. Therefore, when the resolution of the input image is hierarchically increased, it is possible to generate a restored image having a higher subjective image quality.

For example, the image to be subjected to the super resolution image restoration can be the input image of the super resolution image restoration apparatus shown in FIG. The super resolution image reconstruction method described with reference to FIG. 1 is performed on the input image, so that the upper resolution image of the first stage can be reconstructed. The upper resolution image of the first stage may be input again to the super resolution image restoration apparatus shown in FIG. The super resolution image reconstruction method described with reference to FIG. 1 is performed again on the upper resolution image of the first stage, and the upper resolution image of the second stage can be reconstructed. The above process can be repeated hierarchically until the resolution of the final output image reaches the target resolution.

The super resolution image reconstruction method according to the embodiment of the present disclosure can amplify a high frequency signal regionally adaptively within a range in which deterioration of image quality is minimized. Therefore, the restoring force of the high-frequency signal is improved and the image can be displayed more clearly.

However, if the image region discriminator 150 discriminates the image region differently from the cognitive viewpoint, a cognitively unnatural image can be generated through the region-adaptive high-frequency signal amplification process.

For example, as a result of performing experiments using four types of UHD resolution moving images of the above-mentioned experiment participants, the amplification factor in the edge image region can be set to be larger than the flatness or texture image region. In this case, if the video region determining unit 150 determines that the video region other than the edge is an edge, an image artifact may occur through a super-resolution image restoration process.

FIG. 3 is an exemplary diagram illustrating image artifacts caused by regions determined to be edges.

Referring to FIG. 3, there are line artifacts that appear to be drawn vertically or horizontally in an image. This may be a phenomenon in which a large amplification factor is applied to a high-frequency signal of pixels determined to be edges in a super-resolution image reconstruction process. In addition, when the super-resolution image restoring process is repeatedly performed as shown in FIG. 2, the line artifacts are propagated and amplified at each step, which may significantly degrade the perceived image quality.

In order to understand the cause of the line artifact, it is necessary to grasp the configuration of the resolution enhancing unit 120 and the background projecting unit 170 shown in FIG.

4 is a block diagram exemplarily showing a configuration of the resolution enhancing unit 120 of FIG.

Referring to FIG. 4, the resolution enhancer 120 may include an image interpolator 410. The image interpolating unit 410 may change the resolution of the input image and output an image having a higher resolution.

5 is a block diagram exemplarily showing a configuration of the background projection unit 170 of FIG.

5, the background projection unit 170 includes an anti-aliasing application unit 510, an image interpolation unit 520, an image difference calculation unit 530, and / or a difference reflection unit 540 ).

The anti-aliasing application unit 510 may receive the reconstructed image having the higher resolution and perform the anti-aliasing filtering operation. The image interpolating unit 520 may downsample the image subjected to the anti-aliasing filtering operation to the resolution of the input image. The image difference calculator 530 may calculate the difference between the downsampled reconstructed image output from the image interpolator 520 and the input image. The image difference calculation unit 530 can transmit the calculated difference to the image interpolation unit 520 again. The image interpolating unit 520 can upsample the difference output from the image difference calculating unit 530 to an upper resolution. The image interpolating unit 520 can transmit the upsampled difference to the difference reflecting unit 540. [ The difference reflecting unit 540 can generate a super resolution image using the upsampled difference and the restored image. For example, the difference reflecting unit 540 can generate a super resolution image by adding the upsampled difference value and the reconstructed image. If repeated background projection is required, the above-described process can be repeated by re-inputting the generated super resolution image as a reconstructed image, as indicated by a dotted line in FIG.

The resolution enhancement unit 120 generates an upper resolution image serving as a base of the super resolution image. The higher resolution image generated by the resolution enhancing unit 120 may affect the accuracy of the high frequency signal reconstruction and the image region discrimination. Therefore, the operation of the resolution enhancing unit 120 may affect the image quality of the super-resolution image.

The resolution enhancing unit 120 and the background projecting unit 170 shown in FIG. 4 and FIG. 5 may include the image interpolating units 410 and 520 in common.

The image interpolators 410 and 520 may adjust the resolution of an image through upsampling or downsampling. The types of interpolation may include bilinear interpolation, bicubic interpolation, Lanczos interpolation interpolation, and the like.

6 is a diagram exemplarily showing a super resolution image generated by applying bilinear, bi-cubic, and lanchotz interpolation methods to the image interpolators 410 and 520, respectively.

Table 1 below summarizes the advantages and disadvantages of each of the interpolation methods.

By Linear Bicubic Ranches Advantages - Less line artifacts
- Low computational complexity - Fair computational complexity
- relatively clear - Less line artifacts
- relatively clear Disadvantages - Globally blurred - Has line artifacts - High computational complexity
- Additional memory required

Applying the bilinear interpolation method to the image interpolators 410 and 520 has the advantage that the amount of computation is small and the line artifacts are comparatively less. However, overall the image looks blurry. That is, since the sharpness of the image is greatly reduced, the bilinear interpolation method may not be suitable for high-resolution image restoration.

When the lanth-second interpolation is applied to the image interpolators 410 and 520, the line artifacts are less and the images are relatively clear. However, a sinc function-based operation is required to obtain a kernel-level kernel. In addition, since the size of the kernel to be applied may be changed depending on the image size, additional memory may be required. In addition, as in the method described with reference to FIG. 2, when a plurality of layers of super-resolution images are generated, the memory requirement amount and the computation amount are increased, so that the lanth-order interpolation method may not be suitable.

According to an embodiment of the present invention, an interpolation method that has a smaller calculation amount than the lanchoth interpolation method and the bicubic interpolation method, and which minimizes the line artifact in the image super resolution process can be applied.

FIG. 7 is an exemplary diagram illustrating a reference area when carrying out bicubic interpolation. FIG.

In the case of bi-cubic interpolation, 16 pixel values of the input image can be referred to in order to calculate one pixel value of the result image. More generally, one resulting pixel value can be computed using values of pixels in the NxN (N is an integer greater than or equal to 2) reference range.

In this case, instead of performing the bicubic interpolation operation referring to 16 pixel values at one time, it is advantageous in terms of the calculation amount to perform unroll by performing five operations.

For example, referring to FIG. 7, 16 pixels in a 4x4 block of lower resolution, such as a solid line box or a dotted line box, can be used to calculate one pixel value of higher resolution. The sixteen pixels in the solid line box can be grouped into four groups each containing four pixels, such as shaded circles or shaded circles.

The bicubic interpolation operation according to an embodiment of the present disclosure may perform one-dimensional interpolation on each of the four groups, and then perform one-dimensional interpolation on four resultant values. That is, by performing a total of five one-dimensional interpolation operations, it is possible to calculate the same result as a two-dimensional interpolation operation performed on the reference range of 4 × 4.

As described above, if the bicubic interpolation operation for the reference range of 4x4 is performed by five 1-dimensional interpolation operations, the one-dimensional interpolation operations in the solid line box used for calculating the first result pixel value in FIG. 7, There may be overlapping one-dimensional interpolation operations between the one-dimensional interpolation operations in the dotted box used to calculate the resulting pixel values. For example, in FIG. 7, the pixels included in the shaded circles are included both in the solid line box and the dotted box. Therefore, among the one-dimensional interpolation calculation values derived for the calculation of the first result pixel value, the one-dimensional interpolation calculation values for the three shaded circles can be used as they are for the calculation of the second result pixel value. That is, when calculating the second result pixel value, since the one-dimensional interpolation operation is performed on the hatched circle positioned at the lowermost end in the dotted box and the final one-dimensional interpolation operation is performed on the four one-dimensional interpolation operation values, The amount of computation can be greatly reduced.

A cubic spline may be used for the one-dimensional interpolation operation. Here, a cubic spline may mean that a third order polynomial is used to predict a value between given points. The cubic spline can be expressed by the following equations (2) and (3).

&Quot; (2) "

&Quot; (3) "

, where

, where

In Equation (2), u denotes the difference between the coordinate when the coordinate s is mapped to the reference image resolution and the nearest integer coordinate i, and can be expressed as Equation (3). In Equation (3), f denotes a scale factor obtained by dividing the resultant image resolution by the reference image resolution, and floor () means that the downward operation is used to express a number including a decimal point as an integer.

,

,

,

구하기 위해서는 아래 수학식 4, 5, 6 및 7의 조건들이 고려될 수 있다. The coefficient

,

,

,

The conditions of equations (4), (5), (6) and (7) below can be considered.

&Quot; (4) "

&Quot; (5) "

&Quot; (6) "

&Quot; (7) "

8 is a diagram illustrating an example of interpolation using cubic splines.

와

사이를 연결함에 있어, 연속성(continuity)을 보장하기 위해 상기 수학식 6과 수학식 7의 조건이 추가로 고려될 수 있다.Two points of the equations (4) and (5)

Wow

The conditions of Equations (6) and (7) can be further considered in order to ensure continuity.

에서의 기울기가, 그 이전 지점인

와 그 이후 지점인

의 차와 비례하도록 설정함으로써 보간 함수가 연속적으로 나타나도록 할 수 있다. 이때 비례 정도는 장력 계수

를 통해 설정할 수 있다. Referring to Equation (6) above,

The slope at the previous point

And thereafter

So that the interpolation function can be continuously displayed. In this case,

.

에서의 기울기가, 그 이전 지점인

와 그 이후 지점인

의 차와 비례하도록 설정함으로써 보간 함수가 연속적으로 나타나도록 할 수 있다. 이때 비례 정도는 마찬가지로 장력 계수

를 통해 설정할 수 있다. Referring to Equation (7) above,

The slope at the previous point

And thereafter

So that the interpolation function can be continuously displayed. In this case,

.

는 현재 지점을 기준으로 보간 곡선을 형성함에 있어 그 이전 및 이후 지점과의 정합성을 어느 정도로 고려하였는가를 나타내는 계수라 볼 수 있다. 따라서, 그 값이 클수록 전후 정합성을 더 고려하였음을 의미할 수 있다.

Can be regarded as a coefficient indicating how much the consistency with the points before and after the interpolation curve is formed based on the current point. Therefore, the larger the value, the more likely it is to consider posterior consistency.

가 클수록 현재 지점뿐만 아니라 다른 여러 지점과 부드럽게 연결된 형태로 곡선이 구성될 수 있도록 곡선이 느슨하게 생성될 수 있다. 반대로,

가 작을수록 전후 정합성이 덜 고려되기 때문에 곡선이 비연속적이고 팽팽하게 생성될 수 있다. In other words,

Larger curves can be loosely created so that curves can be constructed in a smoothly connected fashion with other points as well as the current point. Contrary,

, The curves may be discontinuous and taut due to less consideration of post-war consistency.

)에 따른 큐빅 스플라인 함수의 형태를 예시적으로 나타내는 도면이다.Fig. 9 is a graph showing the relationship between the tensile coefficient

) Of a cubic spline function according to an embodiment of the present invention.

)가 0에 가까울수록, 도 9에 도시된 진한 실선과 같이 주어진 점 사이를 짧은 거리로 팽팽하게 연결하는 형태의 큐빅 스플라인 함수가 형성될 수 있다. 반대로, 장력 계수(

)가 커질수록, 도 9에 도시된 점선 및 옅은 실선과 같이 주어진 점 사이를 느슨한 형태의 3차식으로 연결하게 된다.Tension Factor (

Is closer to 0, a cubic spline function can be formed in such a manner that a given distance between the given points is tightly connected with a short distance as shown by the thick solid line shown in Fig. Conversely,

), The connection between the given points is loosened in a cubic fashion as indicated by the dotted line and the thin solid line in Fig.

)가 작을수록 보간 함수는 불연속적(discontinuous)인 형태를 가질 수 있다. 불연속적인 형태를 갖는 함수를 적용하여 영상을 보간하면, 보간된 영상에 임펄스(impulse) 값들이 나타날 확률이 높아진다.As shown in Fig. 9,

), The interpolation function may have a discontinuous shape. By interpolating an image using a discontinuous function, the probability that impulse values appear in the interpolated image increases.

Here, the impulse value may mean a value that suddenly increases or decreases, and the value bounces, unlike the neighboring pixel values. That is, it may be a value having discontinuity in an image.

The impulse values may be misinterpreted as edge pixels through the following super resolution process and amplified to appear as line artifacts.

1) When impulse values are input from the resolution enhancement unit 120, the similar-patch search unit 130 searches for a similar patch for a patch having an upper resolution including an impulse value. Also, the high frequency patch mapping unit 140 selects a corresponding high frequency signal. Therefore, the high-frequency signal of the impulse pixel portion can be further reinforced to appear as a line artifact.

2) The image region discrimination unit 150 can discriminate the introduced impulse value as an edge region. In this case, the high-frequency signal of the edge pixel is multiplied by a large amplification factor in the region-adaptive high-frequency signal amplification process, so that the line artifact can be highlighted.

3) As shown in FIG. 2, when the resolution of an image is hierarchically enhanced, line artifacts may be transited, amplified, and highlighted for each resolution layer.

)를 상향 조정하여, 영상 보간을 위한 함수를 더 느슨한 형태로 구성함으로써, 보간 과정에서의 임펄스 값의 유입을 최소화할 수 있다. Therefore, the tension coefficients used in the image interpolators 410 and 520

) Is adjusted upward to make the function for image interpolation more loose, so that the inflow of the impulse value in the interpolation process can be minimized.

At this time, there is a disadvantage in that when the tension coefficient is raised unconditionally without considering the image characteristic, the image may appear to be slightly blurred in raising the tension coefficient.

Therefore, it is desirable to minimize the line artifact while maintaining the sharpness by adjusting the tension coefficient adaptively according to the characteristics such as the spatial complexity of the image. That is, characteristics can be analyzed for each image frame, and the degree of upward adjustment of the tension coefficient can be adaptively set according to the result.

If the image contains a large number of flat areas, line artifacts may be visible. Therefore, the subjective image quality can be greatly deteriorated. Therefore, the tension coefficient can be greatly increased upward for an image including a plurality of flat regions.

On the other hand, in the case of an image including a large number of edges and / or texture regions of a complex image, a texture masking effect can be issued. That is, even in the presence of some line artifacts, line artifacts may be visibly less visible because they are obscured by surrounding complex texture patterns. Therefore, for an image including a plurality of edge and / or texture areas, the line artifact can be reduced without significantly reducing the sharpness of the image by raising the tension coefficient to an appropriate level.

As described above, in order to adaptively adjust the tension coefficient, the characteristics of the image frame can be analyzed first. To analyze the characteristics of the video frame, Equation 1 can be used. Using Equation (1), the ratio of each area pixel in the image frame can be grasped after dividing the image frame by the flatness, the edge and / or the texture area.

When the ratio of the flat area pixels in one image frame exceeds a specific threshold value, the corresponding image frame can be regarded as a frame in which the flat area predominates. For a frame in which the flat area predominates, the degree of upward tension coefficient can be set high. On the other hand, when the ratio of the flat area pixels in one image frame does not exceed the specific threshold value, the image frame can be regarded as a frame including a plurality of edges and / or textures. Thus, the upward degree of the tension coefficient can be set to an appropriate level for a frame having a high spatial complexity.

Alternatively, different tension coefficients may be applied even within the same image frame. In this case, the characteristics can be grasped for each of the plurality of regions constituting the image frame, and the tension coefficient can be adaptively applied in consideration of the detected characteristics.

)를 상향 조정하는 경우, 영상 보간 과정에서의 임펄스 값의 유입이 최소화될 수 있다. 그럼으로써, 엣지로 오판별되는 영역이 줄어들게 되므로, 라인 아티팩트로 인한 영상 왜곡 현상이 최소화될 수 있다.As described above,

), The inflow of the impulse value during the image interpolation process can be minimized. Thereby, since the region determined to be an edge is reduced, image distortion caused by line artifacts can be minimized.

)를 상향 조정한 보간 방법을 이용함으로써, 기존의 바이큐빅 보간법을 이용하였을 때와 비교하였을 때, 라인 아티팩트를 현저하게 줄일 수 있다.For example, when an input image having a resolution of 1920x1080 is enhanced to an image having a resolution of 3840x2160 according to the scheme of FIG. 1,

), The line artifact can be remarkably reduced when compared with the case using the existing bicubic interpolation method.

)에 따른 초해상도 영상 복원 결과를 예시적으로 나타내는 도면이다.10 is a graph showing a relationship between the tensile coefficient

In FIG. 5). FIG.

Referring to FIG. 10, in the image interpolators 410 and 520, the line artifacts in the vertical and / or horizontal directions are significantly reduced by using the interpolation method according to one embodiment of the present invention, rather than the existing bicubic interpolation method Can be confirmed.

According to one embodiment of the present invention, in performing the region adaptive super resolution image restoration of FIG. 1, by improving the interpolation method of the image interpolators 410 and 520, it is possible to provide a clear super resolution image, A method of minimizing line artifacts that may occur can be provided.

)를 적응적으로 상향시켜 적용할 수 있다. 그럼으로써, 유사패치 검색부(130)의 정확도를 향상시킬 수 있다. 또한, 영상 영역 판별부(150)에서의 엣지 오판별율을 줄일 수 있다. 그럼으로써, 초해상도 영상 복원 과정에서 발생할 수 있는 라인 아티팩트를 최소화할 수 있다.The interpolation method of the image interpolators 410 and 520 according to the embodiment of the present disclosure can minimize the amount of calculation for the duplicated items through the unroll-based calculation. Therefore, it is possible to have a smaller calculation amount than the conventional lanth-first and bicubic interpolation methods. It should also be noted that the interpolation scheme according to an embodiment of the present disclosure is based on the cubic spline matrix having a tensile coefficient

) Can be adaptively upgraded. Thus, the accuracy of the similar patch search unit 130 can be improved. In addition, the edge-to-edge discrimination ratio in the video area discrimination unit 150 can be reduced. Thus, the line artifacts that may occur in the super-resolution image restoration process can be minimized.

11 is a block diagram showing a configuration of still another embodiment of a super resolution sharpening method for minimizing image artifacts.

Referring to FIG. 11, the super resolution image restoration apparatus may include a super resolution image restoration unit 1110 and a background projection unit 1120.

As described through the embodiment shown in FIG. 1, the line artifacts of the super-resolution image may be less noticeable by adjusting the tension coefficient of the cubic spline upward.

In the case of the embodiment shown in FIG. 1, it is possible to determine the degree of upward adjustment of an appropriate tension coefficient adaptively according to the characteristics of the image frame. In the case of the embodiment shown in FIG. 11, after the super-resolution image is reconstructed by the basic Bicubic interpolation method, the line artifact occurrence degree can be grasped and the tension coefficient can be adjusted accordingly.

For example, the super-resolution image restoration unit 1110 may restore the super-resolution image by performing bi-cubic interpolation by substituting 0.5 as the initial value of the tension coefficient used in the image interpolation. The background projecting unit 1120 can determine the presence or absence of line artifacts by comparing the restored super resolution image and the input image.

As a result of the judgment by the background projecting unit 1120, if line artifacts exceeding a specific threshold exist, the image can be fed back to the super-resolution image restoring unit 1110. [ For example, the background projection unit 1120 may feed back the tension coefficient up signal to the super resolution image restoration unit 1110. Or the background projecting unit 1120 may feed back a signal indicating that the line artifact exceeds the threshold value to the super-resolution image restoring unit 1110. [

The super resolution image restoration unit 1110 can adjust the tension coefficient upward based on the feedback from the background projection unit 1120. [ The tension coefficient may be adjusted by an amount based on the feedback from the background projection section 1120 or by a predetermined amount. The super resolution image restoration unit 1110 may perform the super resolution image restoration process on the input image again based on the adjusted tension coefficient and transmit the result to the background projection unit 1120 again.

The above process can be repeated until the background projection unit 1120 determines that the line artifact degree does not exceed a specific threshold value.

When the background projection unit 1120 determines that the degree of line artifact does not exceed a certain threshold value or is appropriate, the background projection unit 1120 may finally output the upper resolution image subjected to the background projection.

The upper resolution image output as a result of FIG. 11 can be input as an input image of the next resolution layer as shown in FIG. The above process can be repeated until the target resolution image is generated.

At this time, the background projection unit 1120 analyzes the line artifact level for each resolution layer, and the image interpolation tension coefficient to be used in the super resolution image restoration unit 1110 can be increased. Alternatively, the tension coefficients derived from the first (or previous) layer may continue to be used for the higher resolution layers to reduce the amount of computation.

In the latter case, for the upper resolution layers other than the first resolution layer, the background projecting unit 1120 performs background projection on the image generated by the super-resolution image restoring unit 1110 and outputs the result immediately .

12 is a block diagram exemplarily showing a structure of the super-resolution image restoration unit 1110 of FIG.

Referring to FIG. 12, the super-resolution image restoration unit 1110 is similar to the structure except for the background projection unit 170 in the super-resolution image restoration apparatus shown in FIG. 12 includes a frequency domain division unit 1210, a resolution enhancement unit 1220, a similar patch search unit 1230, a high frequency patch mapping unit 1240, (1250) and / or a frequency domain fusion portion (1260). The operation of each unit of the super-resolution image reconstruction unit 1110 shown in FIG. 12 may be the same as or similar to the operation of the corresponding configuration of the super-resolution image reconstruction apparatus described with reference to FIG.

13 is a block diagram exemplarily showing a structure of the background projection unit 1120 of FIG.

13 includes an anti-aliasing applying unit 1310, an image interpolating unit 1320, an image difference calculating unit 1330, a line artifact analyzing unit 1340, and / or a difference reflecting unit 1350). Compared with the background projecting unit 170 shown in FIG. 5, the background projecting unit 1120 shown in FIG. 13 may further include a line artifact analyzing unit 1340.

The image difference calculator 1330 can calculate the difference between the image obtained by downsampling the super-resolution reconstructed image and the input image. Therefore, information on artifacts that are not present in the input image but are introduced in the super resolution process may be included in the calculated difference value.

The line artifact analyzing unit 1340 may generate a gradient histogram of the difference image. The gradient histogram can be generated by using the following equations (8) and (9).

&Quot; (8) "

&Quot; (9) "

는 (x,y) 좌표의 차분 값의 그라디언트 방향 정보를 의미한다. 상기 그라디언트 방향 정보는 각도 형식으로 나타낼 수 있다. 상기 수학식 8에 있어서, p는 구성할 히스토그램 빈(bin)의 개수를 의미한다. 히스토그램 빈이란 그라디언트 각도 범위를 동일한 크기의 여러 구간으로 나눈 것을 의미할 수 있다. 상기 p 에 큰 값을 넣을수록 세밀한 간격의 히스토그램이 생성될 수 있다. In Equation (8) above,

Denotes the gradient direction information of the difference value of the (x, y) coordinates. The gradient direction information may be expressed in an angle format. In Equation (8), p denotes the number of histogram bins to be constructed. A histogram bean can mean that the gradient angle range is divided into several sections of equal size. The larger the value of p is, the more detailed histograms of intervals can be generated.

값을 토대로 현재 그라디언트가 어느 히스토그램 빈에 포함되는지 결정될 수 있다. 예컨대, 현재 그라디언트가 b번째 히스토그램 빈에 포함되면, 상기 수학식 9에 따라, 해당 빈에 그라디언트 크기(magnitude) 값인

를 축적시킬 수 있다.As a result of the above equation (8)

Based on the value, it can be determined which histogram bin the current gradient is contained in. For example, if the current gradient is included in the b-th histogram bin, a gradient magnitude value

. ≪ / RTI >

The line artifact degree of the difference image can be estimated by referring to the gradient magnitudes of the vertical and horizontal angles (0 degree, 90 degree, 180 degree, 270 degree, 360 degree) in the gradient histogram constructed by the above method.

When the size of the calculated line artifact is equal to or larger than the threshold value, feedback (or a tension coefficient enhancement signal) is sent to the super-resolution image restoration unit 1110 as shown in Figs. 11 and 12, . At this time, the background projection unit 1120 may not reflect the difference.

When the size of the calculated line artifact is equal to or smaller than the threshold, the image interpolating unit 1320 can improve the difference to a higher resolution. The difference reflecting unit 1350 can output a super-resolution image by reflecting the improved difference in the resolution on the restored image.

The generated super resolution image may be the input image of the next resolution layer or the final super resolution image.

Although the exemplary methods of this disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, the illustrative steps may additionally include other steps, include the remaining steps except for some steps, or may include additional steps other than some steps.

The various embodiments of the disclosure are not intended to be all-inclusive and are intended to be illustrative of the typical aspects of the disclosure, and the features described in the various embodiments may be applied independently or in a combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays A general processor, a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is to be accorded the broadest interpretation as understanding of the principles of the invention, as well as software or machine-executable instructions (e.g., operating system, applications, firmware, Instructions, and the like are stored and are non-transitory computer-readable medium executable on the device or computer.

Claims (20)

A method for reconstructing a super-resolution image,
Dividing an input image into a low-frequency image having an input resolution and a high-frequency image having an input resolution;
Generating a low-resolution image of an upper resolution from the input image;
Reconstructing a high-resolution high-frequency image using the low-frequency image having the input resolution, the high-frequency image having the input resolution, and the low-resolution image having the high resolution; And
And generating a reconstructed image having an upper resolution using the low-frequency image of the higher resolution and the high-frequency image of the higher resolution. The method according to claim 1,
Dividing the input image into a low-frequency image having the input resolution and a high-frequency image having the input resolution,
A low pass filter is applied to the input image,
Wherein the low pass filter is a Gaussian filter or a Butterworth filter. The method according to claim 1,
Wherein the generating the low resolution image of the higher resolution from the input image comprises:
And interpolating the input image based on the higher resolution,
Wherein the interpolation is a bilinear interpolation, a bicubic interpolation, or a ranchus interpolation. The method according to claim 1,
The reconstructing the high-resolution high-frequency image using the low-frequency image having the input resolution, the high-frequency image having the input resolution, and the low-
Dividing the high-resolution low-frequency image into one or more patches;
Retrieving a similar patch similar to the patch from low frequency images of the input resolution;
Identifying a high frequency patch corresponding to the retrieved similar patch from a high frequency image of the input resolution; And
And reconstructing the high-resolution image having the higher resolution using the specified high-frequency patch. 5. The method of claim 4,
Wherein the high frequency patch corresponding to the retrieved similar patch comprises:
Wherein a patch in the high-frequency image of the input resolution having the same coordinates as the retrieved similar patch is a patch. 5. The method of claim 4,
If the retrieved similar patches are two or more,
Each of the two or more similar patches having similarity information,
And determining a combined high frequency patch by applying a weight based on the similarity degree information to at least two of the at least two high frequency patches corresponding to each of the at least two similar patches. The method according to claim 1,
The super resolution image restoration method includes:
Further comprising amplifying the high-resolution image of the higher resolution,
Wherein the step of amplifying the high-
Determining a type of one or more image regions included in the reconstructed image having the higher resolution;
Generating a high frequency amplification coefficient map based on the discriminated type of the image region; And
And amplifying the high-resolution image of the higher resolution using the high-frequency amplification coefficient map,
Wherein the generating of the reconstructed image having the higher resolution using the low-frequency image having the higher resolution and the higher-
Wherein the amplified higher resolution high frequency image is used instead of the higher resolution high frequency image. 8. The method of claim 7,
Wherein the type of the image area includes a flat area, an edge area or a texture area,
Wherein the step of generating the high-
Wherein an amplification factor for the edge region is set to be larger than an amplification factor for the flat region or the texture region. The method according to claim 1,
The super resolution image restoration method includes:
Further comprising a background projection process,
The background projection process includes:
Generating a restored image of the input resolution from the restored image of the higher resolution;
Generating a difference image between the reconstructed image of the input resolution and the input image;
Generating a difference image of an upper resolution from the difference image; And
And applying the difference image of the higher resolution to the reconstructed image of the higher resolution to generate a final reconstructed image of an upper resolution. 10. The method of claim 9,
Wherein the step of generating a reconstructed image of the input resolution from the reconstructed image of the higher resolution comprises:
And generating a reconstructed image having the input resolution by downsampling the reconstructed image having the higher resolution with the input resolution. 10. The method of claim 9,
Wherein the step of generating the difference image of the higher resolution from the difference image comprises:
And generating the difference image of the higher resolution by upsampling the difference image to an upper resolution. 10. The method of claim 9,
And inputting the final reconstructed image of the higher resolution, which is an output of the background projection process, as the restored image of the higher resolution which is the input of the background projection process, and performing the background projection process more than twice. The method of claim 3,
In the bicubic interpolation,
Grouping a plurality of pixels in a reference area, which is a block of NxN (N is an integer equal to or greater than 2), into N groups including N pixels;
Performing a first order interpolation for each of the N groups; And
And deriving a resultant pixel value by performing a second-order interpolation using the first-order interpolation result for each of the N groups,
Wherein the grouping comprises:
And grouping the N pixels belonging to the same column or the same row among the plurality of pixels in the reference area into the same group. 14. The method of claim 13,
If the reference area for deriving the one resulting pixel value overlaps another reference area for deriving another result pixel value,
Wherein the first interpolation result for the group included in the overlapping portion is reused as a first-order interpolation result for deriving the different result pixel value. 14. The method of claim 13,
Wherein performing the linear interpolation comprises:
Is performed using a cubic spline function,
The tension coefficient, which determines the continuity of the cubic spline function,
Wherein the input image is adaptively adjusted according to the spatial complexity of the input image. 16. The method of claim 15,
And adjusting the tension coefficient as the spatial complexity of the input image is lower. 16. The method of claim 15,
The spatial complexity of the input image may be expressed as:
And determining whether a ratio of a flat area included in the input image exceeds a predetermined threshold value. 10. The method of claim 9,
Wherein the interpolation parameter used in the step of generating the low-resolution image of the higher resolution from the input image is adaptively adjusted based on the characteristics of the difference image or the difference image of the higher resolution. 19. The method of claim 18,
The characteristic of the difference image or the difference image of the higher resolution is,
Wherein the difference image is a statistical characteristic of a line artifact included in the difference image or the difference image of the higher resolution,
Wherein the interpolation parameter used in the step of generating the low-resolution image of the higher resolution from the input image,
A super resolution image restoration method which is a tension coefficient determining the continuity of a cubic spline function. A super-resolution image restoration apparatus comprising:
A frequency domain division unit which divides the input image into a low frequency image having an input resolution and a high frequency image having an input resolution;
A resolution enhancement unit for generating a low-resolution image of an upper resolution from the input image;
A high frequency signal reconstruction unit for reconstructing a high-resolution high-frequency image using the low-frequency image having the input resolution, the high-frequency image having the input resolution, and the low-resolution image having the high resolution; And
And a frequency domain fusion unit for generating a reconstructed image of an upper resolution using the high-resolution low-frequency image and the high-resolution high-frequency image.

Images