KR102064581B1 - Apparatus and Method for Interpolating Image Autoregressive - Google Patents

Abstract

 Image auto-regressive interpolation device and method performs diagonal interpolation for dark spots and vertical interpolation for gray spots in the GPU parallel optimization section to achieve better spatial detail and edge clarity. Since diagonal or vertical interpolation is independent of each other in the regression model, parallel optimization by CUDA is possible.

Description

Apparatus and Method for Interpolating Image Autoregressive}

The present invention relates to a method for automatic image regression interpolation. In particular, in the GPU parallel optimization section, when interpolating lost pixels, diagonal interpolation is performed for dark spots and vertical interpolation is performed for gray dots. Relates to an excellent image auto-regressive interpolation device and method.

High video quality (HD) products are becoming increasingly popular due to the high video quality.

Ultra HD resolution, however, requires effective algorithms with low time complexity for real-time reconstruction.

Interpolation is an important method for image processing and is a fundamental requirement for Ultra HD products that use low resolution (LR) images to produce high resolution (HR) images.

This interpolation method is widely used for image magnification, deinterlacing and noise removal. Conventional bilinear interpolation, cubic convolution interpolation and cubic spline interpolation are widely used in real time applications because of their relatively low complexity.

However, conventional interpolation methods can lead to interpolation artifacts such as blurring, ringing, serrations and zipper rings. In order to solve this problem, in recent years, an algorithm that is superior to the conventional interpolation method has been proposed to satisfy the demand for better display resolution.

The proposed interpolation method is proposed to estimate the covariance of HR images from the covariance of LR images and to interpolate missing pixels based on the estimated covariance. One of the most promising algorithms introduced recently is automatic regression modeling image interpolation, as proposed by Zhang and Wu, which uses blocky 2-D autoregressive (PAR) models to recover HR image blocks in blocks. .

Automatic regression modeling image interpolation is also used for image compression and can yield promising results.

However, the core of all these autoregressive models was a time-cost approach, which made it difficult to meet real-time reconstruction requirements in production.

Korean Patent Laid-Open No. 10-2017-0012019 ("Invention name: Operation method in a computing environment supporting a plurality of CPUs and a plurality of GPUs")

In order to solve this problem, the present invention performs diagonal interpolation for dark dots and interpolation in two stages of vertical interpolation for gray dots in the GPU parallel optimization section to provide excellent spatial detail and edge sharpness. It is an object of the present invention to provide a device and method for automatic image regression interpolation.

Image automatic regression interpolation method according to a feature of the present invention for achieving the above object,

Obtaining a low resolution image through down sampling from a high resolution image of the input image data;

Generating eight adjacent neighboring pixel matrices and four connected neighboring pixel matrices centered on the reference pixel to be interpolated in the obtained low resolution image;

Constructing at least one first matrix having the generated eight connected neighboring pixel matrices and the generated first connected neighboring pixel matrix having a first direction coefficient indicating a specific direction;

Performing first image interpolation for the reference pixel to be interpolated based on the plurality of first matrices; And

And improving the sharpness of the output image by reflecting the reference pixel subjected to the first image interpolation to the input image data.

Automatic image regression interpolation method according to a feature of the present invention,

A first step of obtaining a low resolution image through down sampling from a high resolution image of the input image data;

A second step of generating eight connected first neighboring pixel matrices positioned in diagonal directions adjacent to the first reference pixel to be interpolated in the obtained low resolution image and four connected first neighboring pixel matrices;

A third step of constructing at least one first matrix having the eight connected first neighboring pixel matrices and the first direction coefficients representing the diagonal directions by using the generated four connected first neighboring pixel matrices;

Performing a first image interpolation on the first reference pixel to be interpolated based on the plurality of first matrices;

A fifth step of improving the sharpness of the output image by reflecting the first reference pixel on which the first image interpolation has been performed by the diagonal interpolation method to the input image data;

Generating an eight connected second neighboring pixel matrix located in a vertical direction adjacent to the second reference pixel to be interpolated in the obtained low resolution image, and a four connected second neighboring pixel matrix;

A seventh step of constructing at least one second matrix having the eight connected second neighboring pixel matrices and the second direction coefficients representing the vertical direction by using the generated four connected second neighboring pixel matrices;

An eighth step of performing a second image interpolation on the second reference pixel to be interpolated based on the plurality of second matrices; And

And a ninth step of improving the sharpness of the output image by reflecting the second reference pixel on which the second image interpolation has been performed by the vertical interpolation method to the input image data.

Image automatic regression interpolation device according to a feature of the present invention,

An image input unit which obtains a low resolution image through down sampling from a high resolution image of the input image data;

A neighboring pixel matrix generator for generating eight adjacent neighboring pixel matrices and four connected neighboring pixel matrices centered on the reference pixel to be interpolated in the obtained low resolution image;

A direction coefficient matrix generator configured to form one or more first matrices having the eight connected neighbor pixel matrices and the first direction coefficients indicating a specific direction using the generated four connected neighbor pixel matrices;

An image interpolation calculator configured to perform first image interpolation on the reference pixels to be interpolated based on the plurality of first matrices; And

And an image output unit configured to reflect the reference pixel subjected to the first image interpolation to the input image data to improve the sharpness of the output image.

According to the above-described configuration, the present invention performs an interpolation method in the case of dark points when interpolating lost pixels, and interpolates in two stages of vertical interpolation in the case of gray points, thereby providing excellent spatial detail and edge sharpness.

Since the present invention is independent of each other when performing diagonal or vertical interpolation in the automatic regression model, parallel optimization by CUDA is possible.

1 is a diagram illustrating an interpolation model according to an embodiment of the present invention.
2 is a diagram illustrating two interpolation steps based on an image autoregressive interpolation model according to an exemplary embodiment of the present invention.
3 illustrates a decision model of parameters of 8-connected neighboring pixels and 4-connected neighboring pixels according to an embodiment of the present invention.
4 is a diagram illustrating an automatic image regression interpolation method according to an embodiment of the present invention.
FIG. 5 is a diagram schematically illustrating a configuration of an automatic image regression interpolation apparatus according to an exemplary embodiment of the present invention.
6 is a diagram illustrating a correspondence relationship between data and a CUDA thread according to an exemplary embodiment of the present invention.
7 is a diagram illustrating a result of comparing speeds between block sizes of sample image data.
8 illustrates asynchronous data transmission using Tesla K80 of the present invention.
9 is a diagram illustrating a comparison result between a single GPU and multiple GPUs.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Using the development technology of high-performance parallel computing, GPU-based acceleration is used from the scientific computing side for low cost and efficient power usage.

CUDA is a general-purpose parallel computing platform and programming model that leverages the parallel computing engine of NVIDIA GPUs to solve many complex computational problems. The pixels to be interpolated are independent of each other when performing diagonal or vertical interpolation in the automatic regression model, which is suitable for parallel optimization by CUDA.

Automatic Regression Modeling One of the main aspects of the image interpolation method is the use of appropriate training windows.

To capture the finer details of the pixels to be interpolated, we can use the training window, an octagonal window, with fewer computations needed, with four pixels being estimated simultaneously in a larger sleep window and serial computing is important.

Given the registers, shared memory, and other resource limitations of the GPU, if each thread's load is too heavy, then concurrency is greatly reduced. To calculate the pixels to be interpolated requires a large number of matrix operations, including multiple matrix transformations, matrix multiplications, and matrix inversions.

Thus, the present invention modifies the training window more concisely and achieves high speed on the GPU while maintaining high PSNR and subjective visual quality.

 1 is a diagram illustrating an interpolation model according to an embodiment of the present invention.

(1) image automatic regression interpolation model

The spatial detail and edge sharpness of an autoregressive interpolation model based on interpolation images show better performance than that produced by conventional interpolation methods.

The present invention shows significantly better results in terms of PSNR and subjective visual quality.

I _h is the HR image to be estimated and I _l is the observed LR image, which represents a down-sampled version of the HR image by two factors.

The present invention models the image as shown in Equation 1 below.

Here, W is a local window including 4 x 4 unknown pixels, as shown in FIG. 2, and V _{i, j} is random perturbation independent of spatial position and image signal.

는 픽셀 X(i,j)의

번째 이웃에 대한 자기 회귀 계수,

는 이미지들의 픽셀들,

는 HR 이미지에서 위치 i의 픽셀의 이웃이다.

Is equal to pixel X (i, j)

Regression coefficient for the first neighbor,

Is the pixels of the images,

Is the neighborhood of the pixel at position i in the HR image.

The image autoregressive interpolation model can describe the pixels known by the unknown pixels and can describe the unknown pixels by the known pixels according to geometric duality.

은 작은 지역에서 일정하거나 거의 일정하게 유지된다. parameter

Is constant or almost constant in small areas.

Structures such as edges and textures can be learned by fitting samples from the local window. The image autoregressive interpolation model consists of two parts: diagonal interpolation and vertical interpolation. Diagonal interpolation is used to calculate HR pixels, which are known neighboring pixels.

Pixels estimated in diagonal interpolation are considered known in HR images in vertical interpolation. First, the LR image is obtained from the HR image through down sampling.

2 is a diagram illustrating two interpolation steps based on an image autoregressive interpolation model according to an embodiment of the present invention, and FIG. 3 is a parameter of 8-connected neighbor pixels and 4-connected neighbor pixels according to an embodiment of the present invention. Figure showing the decision model of these.

2, white points are LR image pixels and dark and gray points are interpolated in two steps with missing HR pixels.

Afterwards, dark points are interpolated by diagonal interpolation, and finally gray points are interpolated by vertical interpolation.

Vertical interpolation is performed in a manner similar to diagonal interpolation. Hereinafter, only diagonal interpolation will be described.

2 shows a training window of the image automatic regression interpolation method of the present invention.

는 8-연결된 이웃 방향 계수이고,

는 4-연결된 이웃 방향 계수이며, W는 훈련 윈도우이다.2 is a spatial configuration of soft-decision adaptive interpolation (SAI), white dots are conventional LR pixels, dark dots are HR pixels lost,

Is an 8-connected neighbor direction coefficient,

Is the four-connected neighbor direction coefficient and W is the training window.

The white point is the LR pixel denoted by x and the dark HR denoted by y is the lost HR pixel where interpolation is performed by vertical interpolation.

The interpolation method is formulated to follow the Square Problem, and is represented by Equation 2 below, and the parameter λ is a Lagrangian Factor.

파라미터는 종래 기술에 의해 최소 제공법으로부터 하기의 [수학식 3]과 [수학식 4]와 같이 추정될 수 있다. 이러한 방법은 이미지가 부분적으로 고정되어 있다는 기하학적 이중성 가정에 기반하므로 LR 픽셀 공분산과 HR 픽셀 공분산의 사이에 대응된다.As shown in FIG. 3,

The parameters can be estimated by the conventional techniques as shown in Equations 3 and 4 below from the minimum providing method. This method is based on the geometric duality assumption that the image is partially fixed and therefore corresponds between LR pixel covariance and HR pixel covariance.

(8-연결된 이웃)과

(4-연결된 이웃)의 파라미터들의 결정 모델은 이미 알려진 픽셀에 의해 추정할 수 있다.As shown in FIG. 3,

(8-connected neighbors)

The decision model of the parameters of (four-connected neighbor) can be estimated by known pixels.

는 로컬 원도우의 내부의 M² 픽셀을 포함하는 데이터 벡터이다.here,

Is a data vector containing M ² pixels inside the local window.

을 포함하는 i^th행 벡터의 M²×4 행렬이고, B는 xi의 네 개의 4-연결된 이웃인

을 포함하는 i^th행 벡터의 M²×4 행렬이다. t={1,2,3,4}이고 M=4이다.A is the four 8-connected neighbors of xi

Is an M ² × 4 matrix of row vectors i ^th , where B is the four 4-connected neighbors of xi

Is an M ² × 4 matrix of row vectors i ^th containing. t = {1,2,3,4} and M = 4.

Finally, the following [Equation 5] and [Equation 6] can be obtained.

로 대체하여 [수학식 7]과 같이 다시 나타낼 수 있다.[Equation 2] described above

It can be replaced by Equation (7).

는 도 2에 도시된 바와 같이, 현재 원도우에서 9개의 미지의 픽셀의 벡터이다. 픽셀

는 하나의 블록 내의 추정된 픽셀이다. 이것은 선형 최소 제곱 최적화 문제가 된다.here,

Is a vector of nine unknown pixels in the current window, as shown in FIG. pixel

Is an estimated pixel in one block. This is a linear least squares optimization problem.

의 최적 추정은 다음의 [수학식 8]과 같이 정의된다.

The optimal estimate of is defined as [Equation 8] below.

Here, the sizes of C and D are 14x9 and 14x14, respectively.

로 다시 작성하며,

로 정의한다.Matrix operations are time consuming and require less intermediate steps. Matrix D is sparse, so

Rewrite as

It is defined as

Matrix C is decomposed into identity matrix I and matrix H, and I ₉ is a 9 × 9 identity matrix.

Equation 8 is redefined as in Equation 10 below.

The parameter λ of Equation 12 below is a Lagrangian Factor in Equation 2 described above.

The amount of calculation required by Equation (10) described above is smaller than that required by Equation (8).

Sample images in the experiment are Lena, Peppers, Pentagon and Airport as shown in Table 1.

Table 2 compares PSNR and CPU execution time. The execution time of the algorithm is compared with that of the soft-decision adaptive interpolation (SAI). All simulations were performed on Intel Core i7-920, 2.66GHz CPUs.

Table 2 shows that the CPU execution time (single thread) of the image automatic regression interpolation method of the present invention is slightly reduced than the CPU execution time of SAI. This greatly reduces the amount of computation required because the matrix is smaller.

The magnitudes of C and D in SAI are determined by 21 × 12 and 21 × 21, which is larger than the matrices in Equations 10 and 11.

The image automatic regression interpolation method of the present invention obtains one pixel in the sleep window when preparing for GPU parallel computing. That is, more data is duplicated between adjacent training windows. However, in SAI, four pixels can estimate a larger sleep window at the same time and reduce data redundancy.

Therefore, the next section optimizes the autoregressive interpolation model based on the GPU. The present invention uses many CUDA optimization strategies to make the most of the GPU, each of which helps to effectively solve this time consuming problem. PSNR and time are provided for each optimization.

(2) GPU-based optimization for images (automatic regression interpolation model)

A. Experimental Equipment

The present invention performs GPU based optimization for image autoregressive interpolation model in NVIDIA Tesla K80. This GPU-based optimization includes two GPUs, a 2 x 2496 CUDA core and a total of 26 SMX units.

Compared to previous GPU architectures, the number of registers and shared memory on the Tesla K80 has been greatly improved, nearly double that of the original. GPUs have the advantage of low cost and efficient power usage. In the experiments of the present invention, serial C code was executed on a CPU (Intel Core ™ i7-920, 2.66 GHz) with O2 compilation optimization.

When running a CUDA program, the default total amount of shared memory in all blocks of the Tesla K80 is 49,152B, and the total number of registers available per block is 65,536. The device specifications of the present invention are described in Table 3 below.

B. CUDA based algorithm flow

CUDA C provides a simple path for users familiar with the C programming language, making it easy to write programs to run on the device. In CUDA programs, the GPU is responsible for the parallel aspect of the kernel.

Parallel technology is now used in all aspects of science and technology, and GPU-based image processing models are widespread. Parallel computing has shown tremendous potential in image processing and is important in terms of time-consuming methods. CUDA code consists of three steps: transferring data to the GPU's memory, executing the CUDA kernel, and transferring the GPU's results to the CPU's memory.

4 is a diagram illustrating an automatic image regression interpolation method according to an exemplary embodiment of the present invention, and FIG. 5 is a diagram schematically illustrating a configuration of an automatic image regression interpolation apparatus according to an exemplary embodiment of the present invention.

Some applications do not need to send results back to the CPU, but are displayed directly on the GPU. The diagonal interpolation process and the vertical interpolation process go through the same process.

1) Transfer the input image data to the GPU and mirror the edges.

2) Generate four connected adjacent pixel matrices and eight connected adjacent pixel matrices in the slide training window for the training direction coefficients.

3) Construct matrices H and G with the direction coefficients of the previous step.

4) A pixel to be interpolated is calculated by using Equation 10 described above.

5) The result of the GPU is transferred to the memory of the CPU.

In the following optimization algorithm, the subject image data to be tested are Lena, Peppers, Pentagon and Airport, which are the most classic image data in image processing.

As shown in Table 1, the HR image size of Lena is 512 × 512, and the size of the down sampled LR image in the HR image is 256 × 256.

The automatic image regression interpolation method of the present invention has two steps, and all processes are performed in parallel on the GPU. When the diagonal interpolation is completed, the image data is not transferred to the CPU but is prepared for vertical interpolation. Mirror replication of the edges is required before each interpolation.

Vertical interpolation and diagonal interpolation go through the same process as in FIG. 4.

Image regression interpolation apparatus 100 according to an embodiment of the present invention is the image input unit 110, the neighbor pixel matrix generator 120, the direction coefficient matrix generator 130, the image interpolation calculator 140 and the image output The unit 150 is included.

The image input unit 110 obtains a low resolution image through down sampling from the high resolution image of the input image data (S100).

The neighboring pixel matrix generation unit 120 includes eight adjacent neighboring pixel matrices (Equation 3, Equation 5) and four connected neighboring pixel matrices (Equation 4, Equation 4) centered on the reference pixel to be interpolated in the obtained low resolution image. Equation (6) is generated (S102, S104).

The direction coefficient matrix generator 130 determines whether the local variance estimated from the LR pixel exceeds the threshold (S108), and if so, interpolates based on the automatic regression analysis. The interpolation is based on bicubic (S110).

If the regional deviation is lower than the threshold, it is interpolated based on bicubic, which can reduce the computation without sacrificing efficiency. Simple methods such as bicubic interpolation are sufficient to interpolate smooth areas.

The direction coefficient matrix generator 130 constructs one or more first matrices having the eight connected neighboring pixel matrices generated and the first direction coefficients representing the specific directions using the generated four connected neighboring pixel matrices (math (11) (S112).

The image interpolation calculator 140 performs first image interpolation on the reference pixel to be interpolated based on the plurality of first matrices (Equation 10) (S114).

The image output unit 150 improves the sharpness of the output image by reflecting the reference pixel on which the first image interpolation is performed to the input image data (S116).

Image automatic regression interpolation method (diagonal interpolation) is a first step of obtaining a low resolution image through down-sampling from a high resolution image of the input image data, and the diagonal direction adjacent to the first reference pixel to be interpolated in the obtained low resolution image A second step of generating eight connected first neighboring pixel matrices (Equation 3, Equation 5) and four connected first neighboring pixel matrices (Equation 4, Equation 6), wherein A third step of constructing at least one first matrix having eight connected first neighboring pixel matrices and first generated direction coefficients representing the diagonal directions by using the generated four connected first neighboring pixel matrices (Equation 11, Equation 12), a fourth step of performing a first image interpolation on the first reference pixel to be interpolated based on the plurality of first matrices (Equation 10), and a first image by diagonal interpolation. Reflected in the input to the first reference pixel image data by performing a liver and a fifth step of improving the sharpness of the output image.

The image autoregressive interpolation method (vertical interpolation method) then consists of eight connected second neighboring pixel matrices (Equation 3, Equation 5) located in the vertical direction adjacent to the second reference pixel to be interpolated in the obtained low resolution image. A sixth step of generating four connected second neighboring pixel matrices (Equation 4, Equation 6), the generated eight connected second neighboring pixel matrices, and the generated four connected second neighboring pixel matrices. A seventh step of forming one or more second matrices having a second direction coefficient indicating a vertical direction by using Equation 11 and Equation 12, and the second reference to be interpolated based on the plurality of second matrices. Eighth step of performing a second image interpolation for the pixel (Equation 10), and by reflecting the second reference pixel subjected to the second image interpolation by the vertical interpolation method to the input image data to improve the sharpness of the output image Ninth stage It includes.

In the case of dark points during interpolation of lost pixels in the input image data, diagonal interpolation of the first step, the second step, the third step, the fourth step, and the fifth step is performed, and In this case, the vertical interpolation of the sixth, seventh, eighth, and ninth steps is performed.

FIG. 6 is a diagram illustrating a correspondence relationship between data and a CUDA thread according to an exemplary embodiment of the present invention, and FIG. 7 is a diagram illustrating a result of comparing speeds between block sizes of sample image data.

C. CUDA-based Optimization

1) Optimization using shared memory and registers

GPU memory plays an important role in the CUDA optimization process. The latency associated with reading and writing data from global memory is a bottleneck for some large CUDA dependent programs.

Compared with global memory, shared memory and registers have high memory bandwidth and low access delay. For faster access, the original LR image data downsampled from the HR image is first converted to shared memory.

The correspondence between data and CUDA threads is shown in FIG. 6.

As shown in Fig. 6, the present invention copies the data of each block into the shared memory. The image data of each block is first copied to _shared__ Image [13] [13].

According to this data 64 pixels would be predicted if the block size is 8 × 8. To avoid forming a boundary, data duplication occurs between blocks.

Adjusting shared memory to registers is a classic optimization technique for CUDA programs.

All threads in the same block can communicate with each other so that all threads can share LR data in one block. However, you must consider the limited space of the shared memory and the Bank Conflict.

The largest size of shared memory in all blocks is 49,152 bytes. Therefore, all read or write requests belonging to n address memory and n individual memory banks can be accessed simultaneously, and it is important to avoid as many bank conflicts as possible.

Register memory is private with respect to a particular thread. That is, it is generally faster than shared memory because there are no bank conflicts in registers. In Tesla K80, L1 cache and shared memory share 128KB of configurable on-chip cache on each SMX.

All SMX share an L2 cache with a maximum size of 1.5MB. In the present invention, the number of available registers per block is 65,536 and the size of each register file is 32 bits.

In the next optimization process, operations on small matrices are completed in registers as shown in FIG. The parallel technique is to use one thread to process one slide window. Because the L1 cache can be used to handle register leaks, use the following command:

The present invention means that the L1 cache is further set using the "cudaFuncCachePerferL1" instruction, and the L1 cache is set to 48 KB in each SMX. The correspondence between data and threads is shown in FIG. Adjusting registers and shared memory is a conventional optimization technique for CUDA programs.

The present invention achieved a significant speed improvement of 79.3 for Lena.

One block has 8 × 8 threads, and one block must handle 8 × 8 unknown pixels.

It is necessary to create a 6x6 slide window for each unknown pixel, and one block of image data has a 13x13 boundary overlap, as shown in FIG.

Considering the various matrix sizes including 4 × 4, 5 × 9, and 1 × 14, the image autoregressive interpolation method of the present invention is very difficult to fine thread division because the matrix size is not fixed based on the granular model. Will be considered in subsequent work.

Table 4 shows the time comparison for single-threaded C CPU code after using shared memory and registering an 8 × 8 block size as a register.

2) The most appropriate block size is known to be expensive and expensive for Tesla K80's shared memory and register resources. Therefore, the number of threads per block should be reasonable, and if the block size is too small, resources may not be fully utilized.

However, if the block size is too large, the degree of parallelism will drop. In general, CUDA programs perform well when the number of threads per block is a multiple of 16.

However, it is important to consider that the optimal block size is not always the same and is closely related to all GPU resources. In order to achieve the best performance, numerous experiments have been conducted regarding speedup using different block sizes. As can be seen in Figure 7, 128 block size achieved the best performance in the method of the present invention. 86.4 for Lena, 83.9 for Peppers, 87.6 for Pentagon and 87.2 for Airport.

FIG. 8 illustrates asynchronous data transmission using Tesla K80 according to the present invention, and FIG. 9 illustrates a comparison result between a single GPU and multiple GPUs.

3) Further optimization using asynchronous data transfer

In the present invention, since the LR image data needs to be copied from the host device to the GPU device and then the HR image data is stored in the host memory, a relatively large overhead occurs when the image data volume is large.

The present invention can effectively hide the host device transfer time in a program using asynchronous data transfer (ADT).

Tesla K80 has two memory copy engines: host-device engine (H2D engine) and device-host engine (D2H engine).

This design allows you to manage the ADT through the stream. Here, a stream is a series of instructions that are executed in sequence. The LR image data is divided into four parts, and thus four streams are generated, and one stream is in charge of one data. Data transfer and data computing can be performed simultaneously with two different engines. As shown in Fig. 8, when the data amount is large but the required calculation is not complicated, the optimization effect by ADT becomes remarkable. In the present invention, four streams are generated in the GPU.

However, ADT is not suitable for use in all programs. ADT is useless if the transmission time between the host and the device is not sufficient to hide the time management of the stream. In the method of the present invention, the data transfer time is less than the kernel execution time, and the new edge should be mirrored when using the stream. In this case the speed slightly improves 88.3 speed for Lena, as shown in Table 5.

4) Optimization using Multi-GPU

Tesla K80 has two GPUs. Multi-GPU is suitable for single node desktop systems with multiple computing devices. Thus, the present invention can fully utilize the hardware resources provided by multi-GPU technology.

Since one GPU depends on one stream, the image autoregressive interpolation method of the present invention generates two streams.

Each GPU processes half of the image data, and different devices execute each instruction in parallel. After the final optimization, the speed-up performance is greatly improved by up to 147.3 times for Lena images as shown in [Table 6]. After final optimization, there is little change in PSNR and only a 0.01dB reduction for Lena occurs. 9 shows a comparison of a single GPU (without ADT) and multiple GPUs. Multi-GPU has a distinct advantage over one GPU.

The embodiments of the present invention are not only implemented through the apparatus and / or method, but may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention, a recording medium on which the program is recorded, and the like. Such implementations can be easily implemented by those skilled in the art to which the present invention pertains based on the description of the above-described embodiments.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

100: image auto-regressive interpolation device
110: image input unit
120: neighbor pixel matrix generator
130: direction coefficient matrix generator
140: image interpolation calculator
150: image output unit

Claims (11)

Obtaining a low resolution image through down sampling from a high resolution image of the input image data;
Generating eight adjacent neighboring pixel matrices and four connected neighboring pixel matrices centered on the reference pixel to be interpolated in the obtained low resolution image;
Constructing at least one first matrix having the generated eight connected neighboring pixel matrices and the generated first connected neighboring pixel matrix having a first direction coefficient indicating a specific direction;
Performing first image interpolation for the reference pixel to be interpolated based on the plurality of first matrices; And
Improving the sharpness of the output image by reflecting the reference pixel subjected to the first image interpolation to the input image data;
Performing the image interpolation,
And performing image interpolation on the reference pixel by Equation 1 and Equation 2 below.
[Equation 1]

Where I ₉ is a 9 × 9 identity matrix.
[Equation 2]

here,

, Parameter λ is the Lagrangian Factor. A first step of obtaining a low resolution image through down sampling from a high resolution image of the input image data;
A second step of generating eight connected first neighboring pixel matrices positioned in diagonal directions adjacent to the first reference pixel to be interpolated in the obtained low resolution image and four connected first neighboring pixel matrices;
A third step of constructing at least one first matrix having the eight connected first neighboring pixel matrices and the first direction coefficients representing the diagonal directions by using the generated four connected first neighboring pixel matrices;
Performing a first image interpolation on the first reference pixel to be interpolated based on the plurality of first matrices;
A fifth step of improving the sharpness of the output image by reflecting the first reference pixel on which the first image interpolation has been performed by the diagonal interpolation method to the input image data;
Generating an eight connected second neighboring pixel matrix located in a vertical direction adjacent to the second reference pixel to be interpolated in the obtained low resolution image, and a four connected second neighboring pixel matrix;
A seventh step of constructing one or more second matrices having the eight connected second neighboring pixel matrices and the second direction coefficients representing the vertical directions by using the generated four connected second neighboring pixel matrices;
An eighth step of performing a second image interpolation on the second reference pixel to be interpolated based on the plurality of second matrices; And
And a ninth step of improving the sharpness of the output image by reflecting the second reference pixel on which the second image interpolation has been performed by the vertical interpolation method to the input image data,
Performing the image interpolation,
And performing image interpolation on the reference pixel by Equation 3 and Equation 4 below.
[Equation 3]

Where I ₉ is a 9 × 9 identity matrix.
[Equation 4]

here,

, Parameter λ is the Lagrangian Factor. The method according to claim 1 or 2,
Generating the eight connected neighbor pixel matrix,
The eight connected neighbor pixel matrix further comprises the step of generating by the following equation (5).
[Equation 5]

here,

Is a data vector containing M ² pixels inside the local window, and A is the four 8-connected neighbors of xi

And M ² × 4 matrix of the i ^th row vector, t = {1,2,3,4} and Im M = 4, including. The method according to claim 1 or 2,
Generating the four connected neighbor pixel matrix,
And generating the four connected neighbor pixel matrices by Equation 6 below.
[Equation 6]

here,

Is a data vector containing M ² pixels inside the local window, and B is the four 4-connected neighbor of xi

Is an M ² × 4 matrix of row vectors i ^th containing. t = {1,2,3,4} and M = 4. delete The method of claim 2,
The performing of the second image interpolation may include:
And a first reference pixel which has performed the first image interpolation by the diagonal interpolation method, is regarded as a known pixel, and used when performing the second image interpolation. The method of claim 2,
Performing dark interpolation of the pixels lost from the input image data, performing diagonal interpolation of the first step, the second step, the third step, the fourth step, and the fifth step;
In the case of gray dots, performing the vertical interpolation of the sixth step, the seventh step, the eighth step, and the ninth step. An image input unit which obtains a low resolution image through down sampling from a high resolution image of the input image data;
A neighboring pixel matrix generator for generating eight adjacent neighboring pixel matrices and four connected neighboring pixel matrices centered on the reference pixel to be interpolated in the obtained low resolution image;
A direction coefficient matrix generator configured to form one or more first matrices having the eight connected neighbor pixel matrices and the first direction coefficients indicating a specific direction using the generated four connected neighbor pixel matrices;
An image interpolation calculator configured to perform first image interpolation on the reference pixels to be interpolated based on the plurality of first matrices; And
It includes an image output unit for improving the sharpness of the output image by reflecting the reference pixel subjected to the first image interpolation to the input image data,
And image interpolation for the reference pixel using Equation 1 and Equation 2 below.
[Equation 1]

Where I ₉ is a 9 × 9 identity matrix.
[Equation 2]

here,

, Parameter λ is the Lagrangian Factor. The method of claim 8,
And the eight connected neighbor pixel matrices are generated by Equation 3 below.
[Equation 3]

here,

Is a data vector containing M ² pixels inside the local window, and A is the four 8-connected neighbors of xi

And M ² × 4 matrix of the i ^th row vector, t = {1,2,3,4} and Im M = 4, including. The method of claim 8,
The four connected neighbor pixel matrix is generated by the following equation (4).
[Equation 4]

here,

Is a data vector containing M ² pixels inside the local window, and B is the four 4-connected neighbor of xi

Is an M ² × 4 matrix of row vectors i ^th containing. t = {1,2,3,4} and M = 4. delete

Images