KR20150107360A - Method and apparatus for generating of super resolution image - Google Patents

Abstract

Disclosed is a method for generating an image using a super resolution technique, which comprises the following steps: decomposing an input image into a low frequency image and a high frequency image by using a filter; generating an enlarged image of the low frequency component by interpolating the input image; searching a similar patch which has the highest similarity from the low frequency image by using the enlarged image as a patch unit; copying and mapping a patch in the high frequency image corresponding to the searched similar patch into a high frequency image of a higher layer which has a higher resolution than the high frequency image; and merging the enlarged image and the high frequency image of the higher layer to generate a high resolution image which has a higher resolution than the input image.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for generating an image using a super resolution technique,

[0001] The present invention relates to image processing, and more particularly, to parallel processing of images using a super resolution technique.

Recently, the emergence of various devices and UHDTV (Ultra High Definition TV) has emphasized the necessity of high resolution images. The super resolution technique is one of several methods for improving the resolution of an image, and is a technique for constructing a high resolution image from single or multiple images.

In addition, as the resolution of the image increases due to the development of camera technology, the amount of information to be processed has increased sharply. However, the processing performance of the conventional CPU (Central Processing Unit) has been limited, and the GPGPU (General Purpose Computing on Graphics Processing Unit) has been attracting interest. GPGPU refers to a technology that utilizes a GPU (Graphics Processing Unit) to perform calculations of general applications executed by a CPU. GPUs generally have a lower clock frequency than CPUs, but they can accelerate high parallelism of applications using multiple cores.

GPGPU is a platform that can be utilized by Compute Unified Device Architecture (CUDA) developed by NVIDIA Group and OpenCL (Open Computing Language) developed and distributed by Kronos Group. Among them, OpenCL is an open general-purpose parallel framework and has been highly evaluated for its versatility.

The present invention provides a method of reducing the processing time of a resolution algorithm using OpenCL and provides a result of applying an optimization technique to a memory model.

The present invention provides a method and apparatus for generating a high resolution image using a super resolution technique.

The present invention provides a method for parallelizing a super resolution technique based on OpenCL.

The present invention provides a memory optimization method applicable to a super resolution technique.

According to an embodiment of the present invention, an image generation method using a super resolution technique is provided. The method includes the steps of decomposing an input image into a low frequency image and a high frequency image using a filter, interpolating the input image to generate an enlarged image of a low frequency component, A step of searching for a similar patch having the highest similarity from the high frequency image and copying and mapping the patch in the high frequency image corresponding to the retrieved similar patch to an upper frequency high frequency image having a higher resolution than the high frequency image, And generating a high-resolution image having a higher resolution than the input image by fusing the high-frequency image of the upper layer.

Parallelization and high-speed processing are possible by implementing the super resolution technique using OpenCL. By applying the memory optimization technique, it is possible to solve the synchronization problem and the bottleneck which are generated in the global memory. Also, as the super resolution technique is implemented in OpenCL, the speed improvement is about 90 times as compared with the CPU implementation.

1 is a conceptual diagram illustrating a method for generating a high-resolution image using a super resolution technique algorithm according to an embodiment of the present invention.
2 is a block diagram illustrating a system configuration according to an embodiment of the present invention for implementing the method of FIG.
FIG. 3 shows an example of a kernel design for implementation of a super resolution technique algorithm according to an embodiment of the present invention.
4 is a diagram illustrating an example of a system configuration for memory optimization that can be applied to a super resolution technique algorithm according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a method for solving a contention condition problem when using global memory according to an embodiment of the present invention. Referring to FIG.
FIG. 6 is a diagram illustrating a contention condition problem according to texture memory usage. FIG.
FIG. 7 is a diagram illustrating another example of a system configuration for memory optimization that can be applied to a super resolution technique algorithm according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of the present invention, if the detailed description of related known structures or functions is deemed to obscure the subject matter of the present specification, the description may be omitted.

When an element is referred to herein as being connected or connected to another element, it may mean directly connected or connected to the other element, It may mean something. In addition, the description that includes a specific configuration in this specification does not exclude a configuration other than the configuration, and means that additional configurations can be included in the scope of the present invention or the scope of the present invention.

The terms first, second, etc. may be used to describe various configurations, but the configurations are not limited by the term. The terms are used for the purpose of distinguishing one configuration from another. For example, without departing from the scope of the present invention, the first configuration may be referred to as the second configuration, and similarly, the second configuration may be named as the first configuration.

In addition, the constituent elements shown in the embodiments of the present invention are shown independently to represent different characteristic functions, which do not mean that each constituent element is composed of separate hardware or a single software constituent unit. That is, each constituent unit is included in each constituent unit for convenience of explanation, and at least two constituent units of each constituent unit may form one constituent unit or one constituent unit may be divided into a plurality of constituent units to perform a function. The integrated embodiments and the separate embodiments of each component are also included in the scope of the present invention unless they depart from the essence of the present invention.

In addition, some of the components are not essential components to perform essential functions in the present invention, but may be optional components only to improve performance. The present invention can be implemented only with components essential for realizing the essence of the present invention, except for the components used for the performance improvement, and can be implemented by only including the essential components except the optional components used for performance improvement Are also included in the scope of the present invention.

The super resolution technique is superior to the conventional interpolation technique in restoration of high resolution images, and is particularly suitable for restoring high frequency detail components of an image. Despite these advantages, the super resolution technique requires a large amount of computation due to the complexity of the algorithm. GPGPU (General Purpose Computing on Graphics Processing Unit) technology is widely used to reduce the computation time of high-complexity applications by performing non-graphic operations on GPUs with high parallelism. In the present invention, we propose an optimization technique to solve the bottleneck based on the memory model by reducing the computation time by parallelizing the existing super resolution method using OpenCL (Open Computing Language) as a platform for using GPGPU.

The OpenCL platform is an open standard parallel computing framework that allows you to write programs that run on processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and DSP (Digital Signal Processor). According to the OpenCL model, the host processor manages the context, and based on this, multiple devices can operate in parallel. One task is divided into a plurality of work groups, and each of the work groups is composed of a plurality of work items simultaneously operating.

OpenCL defines a hierarchical memory model. Global memory has the greatest capacity, but its access time is very slow with 400 to 800 cycles, and local memory and private memory have a much faster access time. All work items are accessible to global memory, and work items in the same work group share local memory.

1 is a conceptual diagram illustrating a method for generating a high-resolution image using a super resolution technique algorithm according to an embodiment of the present invention.

1, an image generation method using a super resolution technique according to an embodiment of the present invention includes an LF (Low Frequency) / HF (High Frequency) decomposition step, a linear interpolation step, a similar patch search step, a patch copy and mapping step , An LF / HF fusion step, and a background projection step.

In the LF / HF decomposition step, the input image 100 can be decomposed into a low-frequency (LF) image 110 and a high-frequency (HF) image 120 using a Gaussian filter.

In the linear interpolation step, an original image (input image) 100 can be generated using an existing interpolation technique, for example, an image 130 enlarged by 1.25 times.

In the similar patch search step, the most similar patch 115 in the lower low frequency image layer can be searched for every patch 135. At this time, similarity between patches can be determined based on MSE (Mean Squared Error) value.

When a patch 115 having an optimal similarity is retrieved, the corresponding high frequency component 125 can be used to estimate the upper layer high frequency image 140. In other words, in the patch copying and mapping step, the high frequency component patch 125 corresponding to the most similar patch 115 found in the lower low frequency image layer is copied and mapped to the upper layer high frequency image 140, Can be generated.

In the LF / HF fusion step, the low-frequency image 130 and the high-frequency image 140 are finally fused to form a high-resolution image 150 that is one layer above the input image 100.

In the background projection step, the high-resolution image 150 can be projected to the input image 100 in the background.

The above-described process is repeated for several layers to obtain a high-resolution image.

2 is a block diagram illustrating a system configuration according to an embodiment of the present invention for implementing the method of FIG.

Referring to FIG. 2, a system for a super resolution technique according to the present invention may include a host 210 using a CPU and at least one device 220 using a GPU.

The host 210 can perform file input / output, context initialization, workflow control, and clean up using a CPU.

The device 220 can perform the process of FIG. 1 using the GPU to generate a high-resolution image having a resolution higher than that of the input image when the image is input from the host 210. FIG.

In other words, the device 220 performs the LF / HF decomposition step (step 1), the linear interpolation step (step 2), the similar patch search step (step 3), the patch copy and mapping step (step 4) HF fusion step (step 5) and background projection step (step 6). The high-resolution image generated through steps 1 to 6 may be transmitted to the host 210 and output.

The GPGPU technique can be used to implement parallelization and optimization of the super resolution technique according to the embodiment of the present invention. To do this, you must specify the number of work items and work groups, taking into account the GPU specification and image size. In the embodiment of the present invention, work items of [32x32] are designated as one work group and used.

In order to implement the algorithm of the super resolution technique according to the embodiment of the present invention, a total of 7 kernels can be designed as follows based on the task and synchronization timing. Table 1 shows the name of each kernel, the time it took to execute the kernel, and the percentage of each kernel in total execution time.

Kernel name Time (μs) ratio(%) Image convolution 5,998,678 1.77% Search for Similar Patches 323, 730.725 96.05% Image scaler 3,329.460 0.99% Image addition 1,089.007 0.32% Image subtraction 718.804 0.22% Image division 1,299,361 0.39% Initialize image 883.618 0.26%

Referring to Table 1, it is shown that similar patch search takes the most time during the entire execution time. From this, it can be seen that a technique for optimizing the time taken to search for a similar patch search kernel is required.

FIG. 3 shows an example of a kernel design for implementation of a super resolution technique algorithm according to an embodiment of the present invention.

A device performing a super resolution technique algorithm according to the present invention can separate a kernel as shown in FIG. 3 based on a task unit, a parallelization size, and global synchronization.

For example, in the super resolution technique algorithm according to the present invention, the LF / HF decomposition step can be executed by a "_kernel conform" kernel for convolution and a "_kernel Sub" kernel for image subtraction. The linear interpolation step can be performed by the " _kernel Scaler " kernel for the image scaler. The similar patch search step can be executed by the " _kernel SSSR " kernel for super resolution restoration. The patch copy and mapping steps can be performed by a "_kernel Add" kernel for image addition and a "_kernel Div" kernel for image division. The LF / HF fusion step can be performed by the "_kernel Add" kernel for image addition. Background projection steps can be performed by a "_kernel Convolute" kernel for convolution, a "_kernel Scaler" kernel for image scalers, and a "_kernel Sub" kernel for image subtraction.

At this time, memory use space can be minimized by using kernel for basic arithmetic operators such as image division, addition, subtraction. The kernel for the image scaler (linear interpolation) can perform up-scaling and down-scaling. The kernel for convolution can perform 3x3 Gaussian filtering or 11x11 anti-alias filtering. The kernel for super resolution reconstruction refers to self similarity based high resolution reconstruction.

As described above, it can be seen that the kernel design for realizing the super resolution technique according to the present invention takes the longest execution time in retrieving similar patches. Therefore, a memory access optimization technique is needed to minimize the time it takes to execute a similar patch search kernel. To this end, the present invention provides a method of optimizing a super resolution technique by applying a memory model of OpenCL.

According to OpenCL's hierarchical memory model, global memory is off-chip memory, the largest of all memory models, but with the slowest access time, local and private memory on-chip memory, It has advantages or size of limited resources.

There are two ways to allocate memory. In other words, there are two ways to set it as a normal buffer and an image buffer. If you set the memory as an image buffer, the image buffer will be allocated to texture memory, which is usually located inside the GPU, and will have a faster access speed than the global memory.

4 is a diagram illustrating an example of a system configuration for memory optimization that can be applied to a super resolution technique algorithm according to an embodiment of the present invention.

Referring to FIG. 4, the host 410 may maintain the memory space of the GPU in an isolated state after the first declaration of the GPU until the end of the super resolution technique algorithm according to the embodiment of the present invention.

All data except input and output can be stored in the memory space of the GPU.

The device 420 can allocate and reuse six memory spaces for realizing the super resolution technique according to the embodiment of the present invention. For example, as shown in FIG. 4, the necessary memory spaces I0, L0, H0, L1, H1, W1 (step 1 to step 6) of the super resolution technique according to the above- . ≪ / RTI >

At this time, the device 420, that is, the memory (I0, L0, H0, L1, H1, W1) allocated to the GPU may be a global memory or a texture memory.

If the memory allocated to the GPU is global memory, the global memory may encounter a synchronization problem in global memory, that is, a race condition because all work items are accessible.

FIG. 5 is a diagram illustrating a method for solving a contention condition problem when using global memory according to an embodiment of the present invention. Referring to FIG.

The race condition problem can occur when two or more work items are simultaneously accessible, such as global memory.

For example, as shown in FIG. 5 (a), when searching for a similar patch among the steps of the super resolution technique according to the present invention, when 9 work items refer to an address value of one pixel, .

The contention condition problem refers to a situation in which the result can not be predicted when multiple work items simultaneously access a pixel of an image declared in the global memory. There are three main ways to solve this problem.

First, it is a way to resynchronize the kernel. This has the disadvantage that additional data must be additionally stored inevitably, and an additional kernel must be executed.

Secondly, there is a way to use local memory. Local memory has the advantages of fast access speed, but it has only limited resources and it has a disadvantage that it needs to synchronize the boundary part additionally.

Third, there is a serialization method by Atom instruction. This is advantageous in that it can be implemented easily by utilizing the data value exchange method declared as an internal function.

Since the third method has less overhead than the second method, in the present invention, when the global memory in the GPU is allocated, the serialization method using an atomic function can be used.

As shown in FIG. 5 (b), when nine work items access the global memory at the same time, in the embodiment of the present invention, nine work items are serialized by exchanging data values through Atom instructions, It is possible to solve the contention problem due to access.

As described above, when the memory in the GPU is set as the image buffer, the texture memory is generally allocated to the texture memory, so that the texture memory has faster access speed than the global memory. However, since there is a characteristic that the image buffer can only be read or written, the above-mentioned conflict condition problem may occur.

FIG. 6 is a diagram illustrating a contention condition problem according to texture memory usage. FIG.

In general, texture memory is set as an image buffer, so that when multiple work items access the texture memory, it is possible to have read or write access. However, as shown in FIG. 6, when a plurality of work items simultaneously request a read and a write in the texture memory a, a race condition problem may occur.

In this case, according to the present invention, the contention condition problem due to simultaneous read and write requests of the texture memory can be solved by reusing the conflict method problem solution method by using the global memory and the texture memory together.

For example, in the super resolution technique according to the present invention, input of the kernel for executing similar patch search can be set to an image buffer, and output can be set to a general buffer. In this case, the contention condition problem of the image buffer can be solved through serialization using an internal function that causes internal transfer from the general buffer to the image buffer to use the intermediate result value directly above the image pyramid. For example, the internal function that causes internal transfer from the general buffer to the image buffer can use the "clBuffertoImage" function supported by the OpenCL API (application program interface).

FIG. 7 is a diagram illustrating another example of a system configuration for memory optimization that can be applied to a super resolution technique algorithm according to an embodiment of the present invention.

Referring to FIG. 7, the host 710 can maintain the memory space of the GPU in an isolated state after the memory space of the GPU is initially declared until the end of the super resolution technique algorithm according to an embodiment of the present invention.

All data except input and output can be stored in the memory space of the GPU.

The device 720 may allocate and reuse eight memory spaces for realizing the super resolution technique according to the embodiment of the present invention. For example, as shown in FIG. 7, the necessary texture memory spaces I0, L0, H0, L1, H1, and H1 may be obtained by performing each step (step 1 to step 6) of the super resolution technique according to the above- W1 and global memory space W1_G, H1_G. In other words, GPU memory can be optimized by mixing texture memory space and global memory space.

Meanwhile, memory transfer between PC and GPU is slower than GPU internal memory transfer. Therefore, in order to optimize the memory applicable to the super resolution technique algorithm according to the embodiment of the present invention, memory transfer can be minimized by reusing the memory declared in the GPU. In addition, frequently used kernel coefficients and function arguments can be stored in a constant memory with fast access speed to improve access speed.

In the present invention, a super resolution technique according to the present invention is implemented and tested using a CPU AMD-A8 3870 @ 3.00 Ghz, an 8 GB RAM, and a GPU Geforce 780. And we implemented it using OpenCL 1.1 version.

Table 2 shows the result of implementing a full high definition (FHD) image of 1920x1080 size using a super resolution technique according to the present invention proposed as a UHD (Ultra High Definition) image of 3840x2160 size. Table 2 shows average execution time and total execution time for each layer.

Run Time (s) Layer 1 (s) Layer 2 (s) Layer 3 (s) CPU 25.209 5.055 7.893 12.261 GPU (global) 0.388 0.074 0.118 0.182 GPU (image) 0.273 0.040 0.065 0.128

As shown in Table 2, when implemented using OpenCL, the speed was 64 times faster than the CPU, and the acceleration ratio could be improved up to 92 times when using the image buffer.

In the above-described embodiments, the methods are described on the basis of a flowchart as a series of steps or blocks, but the present invention is not limited to the order of the steps, and some steps may occur in different orders or simultaneously . It will also be understood by those skilled in the art that the steps depicted in the flowchart illustrations are not exclusive and that other steps may be included or that one or more steps in the flowchart may be deleted without affecting the scope of the invention You will understand.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the claims, and all technical ideas within the scope of the claims should be construed as being included in the scope of the present invention.

Claims (1)

Decomposing an input image into a low-frequency image and a high-frequency image using a filter;
Interpolating the input image to generate an enlarged image of a low frequency component;
Searching for a similar patch having the highest degree of similarity from the low-frequency images using the enlarged image as a patch unit;
And copying and mapping the patch in the high-frequency image corresponding to the retrieved similar patch to a higher-layer high-frequency image having a higher resolution than the high-frequency image; And
And generating a high-resolution image having a higher resolution than the input image by fusing the enlarged image and the high-frequency image of the upper layer.

Images