KR102556777B1 - Efficient image restoration method and device based k-NN search - Google Patents

Abstract

The present invention relates to an image restoration apparatus and method based on a k-NN search, and an image restoration apparatus for restoring a distorted image based on a k-NN search, comprising: an input unit for receiving a distorted image; A processor unit predicting a residual image restored for the distorted image based on the distorted image input from the input unit, and applying the residual image predicted through the processor unit to the distorted image to restore and output the distorted image An encoder unit including an output unit, wherein the processor unit includes a plurality of stages and downsamples the distorted image received from the input unit; An image comprising a bottleneck unit comprising one stage and processing the distorted image downsampled by the encoder unit, and a decoder unit comprising a plurality of stages and upsampling the distorted image processed by the bottleneck unit to an initial resolution. Restoration equipment can be provided.

Description

k-NN search based image restoration device and method {Efficient image restoration method and device based k-NN search}

The present invention relates to an image restoration apparatus and method based on a k-NN searcher, and more particularly, as a method of applying localism between two different patches, operation considering locality and low computation by using pairwise localism between similar patches. It relates to a k-NN search based image restoration apparatus and method having complexity and wide receptive field.

With the rapid development of electronic communication technology and the advancement of related imaging equipment, video services are provided in many places through wired and wireless communication, and high-quality video recording is possible through smartphones and black boxes in everyday life.

However, when an image is blurry or image distortion occurs due to image noise caused by snow or rain, difficulties in image identification inevitably follow. Accordingly, there is a need for a method of effectively correcting and restoring image distortion.

Accordingly, various technologies are trending to show very excellent performance in the field of image restoration.

Among them, in the case of image restoration using global attention, excellent performance was shown by dividing the input image into patches of a certain size that do not overlap each other and considering the relationship between all patches, but the square of the input image resolution Since it has proportionally high computational complexity, it is not only difficult to apply to a task outputting an image having a high resolution, but also has a problem in that locality is not considered.

In addition, local attention showed excellent performance in various tasks by dividing the input image into non-overlapping patches of a certain size and then restoring the image by considering only the relationship between pixels in the patch, but had a relatively narrow receptive field. Therefore, there is a problem that is not suitable for image restoration.

Accordingly, there is a demand for a technique for restoring a distorted image with a large accommodation area and a low amount of computation.

As a prior art, there is Korean Patent Registration No. 10-2013777 entitled "Video Distortion Restoration Method and Apparatus Applying the Same".

In order to solve the above problems, the present invention aims to provide a k-NN search-based image restoration apparatus and method having a wide receptive field, low computational complexity and operation considering locality using pairwise localism between similar patches. there is

In order to solve the above problem, an image restoration apparatus based on k-NN search according to an embodiment of the present invention An image restoration apparatus for restoring a distorted image based on k-NN search, comprising: an input unit for receiving a distorted image; A processor unit that predicts the restored residual image for the distorted image based on the distorted image received from the input unit, and an output that restores and outputs the distorted image by applying the residual image predicted through the processor unit to the distorted image and an encoder unit comprising a plurality of stages and downsampling the distorted image received from the input unit; It may include a bottleneck part that is composed of one stage and processes the distorted image downsampled by the encoder part, and a decoder part that is composed of a plurality of stages and upsamples the distorted image processed by the bottleneck part to an initial resolution. there is.

In addition, each of the stages includes a patch partition unit that divides the distorted image into patches of a certain size, and a similar patch for each of the patches divided by the patch partition unit, finds a similarity, and derives a sum of weights. It may include a k-NN Transformer Block (KTB) unit.

The k-NN Transformer Block (KTB) unit may include a Layer Normalization (LN) unit normalizing the patch; A k-NN Local Attention (KLA) unit for deriving a weighted sum through similarity between the normalized patches and a Feed-Forward Network (FFN) unit for emphasizing and learning locality between the patches may be included.

In addition, the k-NN Local Attention (KLA) unit may include a matching unit that finds similar similar patches through k-NN matching using Locality Sensitive Hashing (LSH) for each patch; A grouping unit that sorts and groups the similar patches found in the matching unit, divides the grouped similar patches into chunks, and performs pair-wise regionalism between the similar patches included in the chunks to obtain the similar patches It may include a similarity derivation unit for obtaining a weighted sum through the similarity between them.

In addition, the locality sensitive hashing (LSH) applies the patch to a map in which a plurality of buckets are formed and rotates the map through a random rotation matrix so that the patches belonging to the same bucket are identical. It is characterized by having a hash value.

In addition, the pairwise regionalism is characterized by obtaining a degree of similarity between pixels constituting the similar patches for two adjacent similar patches.

In addition, in the image restoration method for restoring a distorted image based on k-NN search, the input step of receiving a distorted image; A prediction step of predicting a reconstructed residual image for the distorted image based on the distorted image and an output step of restoring and outputting the distorted image by applying the residual image to the distorted image, wherein the predicting step, downsampling the distorted image in an encoder unit composed of a plurality of stages; Processing the distorted image downsampled in a bottleneck unit composed of one stage and upsampling the distorted image processed in a decoder unit composed of a plurality of stages to an initial resolution, the stage comprising: A patch partition step of dividing the distorted image into patches of a certain size and applying K-NN local attention (KLA) to the patch divided in the patch partition step for each patch It is possible to provide an image restoration method including a derivation step of finding a similar patch, obtaining a similarity, and deriving a weighted sum.

According to the present invention, the k-NN search-based image restoration apparatus and method has operation considering locality and low computational complexity and can restore fine details of an image, so that a high quality image can be obtained from a distorted image.

In addition, since it provides connectivity between patches, it has the advantage of having a wide reception area, which is an advantage of globalism.

In addition, the balance between the amount of computation and performance can be adjusted according to the number of k, which is the number of similar patches in k-NN search.

In addition, the image obtained through the k-NN search-based image restoration apparatus and method according to the present invention is used as an input for various image-based tasks (image recognition, object detection, semantic segmentation, etc.) to prevent performance degradation due to distortion. It has the advantage that it can be used in various fields.

1 is a block diagram of an image restoration apparatus based on k-NN search according to an embodiment of the present invention.
2 is a diagram schematically showing the overall structure of an image restoration apparatus and method based on k-NN search according to an embodiment of the present invention.
FIG. 3 is a diagram schematically showing the structure of each stage of FIG. 2;
Figure 4 is a view schematically showing the structure of the KTB unit of Figure 3;
5 is a view showing the structure of the KTB unit of FIG. 4 in detail;
6 is a flowchart of an image restoration method based on k-NN search according to an embodiment of the present invention.
7 to 9 are exemplary diagrams schematically showing an image to which an existing method and the present invention are applied for qualitative evaluation of a k-NN search based image restoration method according to an embodiment.

Hereinafter, the description of the present invention with reference to the drawings is not limited to specific embodiments, and various transformations may be applied and various embodiments may be applied. In addition, the content described below should be understood to include all conversions, equivalents, or substitutes included in the spirit and scope of the present invention.

In the following description, terms such as first and second are terms used to describe various components, and are not limited in meaning per se, and are used only for the purpose of distinguishing one component from another.

Like reference numbers used throughout this specification indicate like elements.

Singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "include", "include" or "have" described below are intended to designate that features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist. should be construed, and understood not to preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 9 attached.

1 is a block diagram of an image restoration apparatus based on k-NN search according to an embodiment of the present invention, and FIG. 2 schematically shows the overall structure of an apparatus and method for image restoration based on a k-NN search according to an embodiment of the present invention. FIG. 3 is a diagram schematically showing the structure of each stage of FIG. 2, FIG. 4 is a view schematically showing the structure of the KTB unit of FIG. 3, and FIG. 5 is a detailed view of the structure of the KTB unit of FIG. it is a drawing

Referring to FIGS. 1 to 4, a k-NN search-based image restoration device (hereinafter simply referred to as 'image restoration device') 1 according to an embodiment of the present invention includes an input unit 10, a processor unit 20 and an output unit. A portion 30 may be included.

The input unit 10 receives a distorted image. The distorted image input through the input unit 10 may be transmitted to the processor unit 20 .

The processor unit 20 may predict a reconstructed residual image for the distorted image based on the distorted image transmitted from the input unit 10 . It can perform convolution on the encoder, bottleneck and decoder.

Referring to FIG. 2 , the processor unit 20 may have a U-shaped hierarchical structure in order to consider patterns of various scales.

More specifically, the processor unit 20 may include an encoder unit 21, a bottleneck unit 22 and a decoder unit 23.

The encoder unit 21 is composed of a plurality of stages and can downsample the distorted image input from the input unit 10 .

The bottleneck unit 22 is composed of one stage and can process a distorted image downsampled by the encoder unit 21 .

The decoder unit 23 is composed of a plurality of stages and can upsample the distorted image processed in the bottleneck unit 22 to an initial resolution.

Referring to FIG. 3 , each stage may include a patch partition unit 201 and a k-NN Transformer Block (KTB) unit 202 .

In this case, the encoder unit 21 and the decoder unit 23 may further include an interpolator 203 for downsampling and upsampling, respectively.

First, the patch partition unit 201 may divide the distorted image into patches of a certain size that do not overlap each other.

The KTB unit 202 may derive a sum of weights by finding a similar patch for each patch divided by the patch partition unit 201 and obtaining a similarity.

More specifically, referring to FIG. 4, the KTB unit 202 includes a Layer Normalization (LN) unit 202a, a k-NN Local Attention (KLA) unit 202b, and a Feed-Forward Network (FFN) unit 202c. can include

The LN (Layer Normalization) unit 202a may normalize the patches divided in the patch partition unit 201 . At this time, the LN unit 202a may refer to a technology commonly used in the art, but is not limited thereto.

The KLA (k-NN Local Attention) unit 202b may derive a weighted sum through similarities between normalized patches, and may be performed through a matching unit, a chunk division unit, and a similarity derivation unit.

The matching unit may find similar patches similar to each patch through k-NN matching using locality sensitive hashing (LSH) for each patch.

More specifically, the matching unit must calculate a pair-wise distance between two patches for k-NN matching. In this case, since the pairwise distance includes the quadratic complexity of the input length, the matching unit may calculate the pairwise distance using region sensitive hashing having a linear calculation complexity and perform k-NN matching.

Here, locality sensitive hashing (LSH) rotates a map through a random rotation matrix with respect to a map in which a plurality of divided buckets are formed. Afterwards, region-sensitive hashing ensures that patches belonging to the same bucket have the same hash value in the stopped map after rotation.

을 구형으로 투영된 패치

에 곱하면 다음의 식과 같이 해시값

이 할당될 수 있다.For region sensitive hashing, assuming there are m buckets, a random rotation matrix

patch as a spherical projection of

When multiplied by , the hash value is as follows

can be assigned.

Here, [;] may indicate the concatenation of two elements, and N may indicate the number of patches.

In this case, the matching unit may perform region-sensitive hashing several times to reduce the probability that similar patches belong to different buckets or that different patches belong to the same bucket in a single execution.

Referring to FIG. 5 , the grouping unit may sort and group similar patches found in the matching unit, and divide the grouped similar patches into chunks.

More specifically, the grouping unit first sorts the patches according to their hash value so that only patches with the same hash value can efficiently contribute to the query patch, and then divides the sorted patches into chunks each containing k patches for batch processing. Local Attention can be performed only between patches belonging to the same chunk.

That is, the grouping unit divides the grouped similar patches into a plurality of chunks by including similar patches having the same hash value in one chunk.

Here, the k value may mean the number of similar patches obtained through region sensitive hashing. For example, if one patch is selected as an input patch and three similar patches are derived through region sensitive hashing, the k value may mean 3. In this case, since the similar patches may include an input patch most similar to the input patch, one of the three similar patches may be an input patch.

이 해시값의 오름차순으로 패치를 정렬하는 순열을 나타낼 수 있다.as in the following expression

A permutation of arranging patches in ascending order of this hash value may be indicated.

를

가

와 같은 정렬된 패치로 정의할 수 있다. 이때,

에 대한

번째 청크

는

개의 패치가 포함되어 있다. In addition, for the above simplification, as

cast

go

It can be defined as an ordered patch such as At this time,

for

th chunk

Is

Patches are included.

The similarity deriving unit may perform pairwise regionalism between similar patches included in a chunk to obtain a weighted sum through the similarity between similar patches.

At this time, the similarity deriving unit may obtain a similarity between pixels constituting each similar patch in obtaining a similarity between similar patches.

More specifically, when obtaining a similarity between two patches, the similarity deriving unit may obtain a similarity between pixels existing at the same location.

는 쿼리(Query), 키(Key) 및 밸류(Value) 값으로 입력되어 각각 학습가능한 투영 함수

및

가 수행될 수 있다. 이때, 하나의 청크 안에 있는 k개의 패치들은 k ² 회 만큼 쌍별지역주의가 수행될 수 있다. 또한, 쌍별지역주의는 다음의 식과 같이 p번째 패치가 쿼리로 출력되고, q번째 패치가 키와 밸류로 사용될 수 있다. Also, sorted and input patches

is a projection function that can be learned by entering it as a query, key, and value value

and

can be performed. At this time, k patches in one chunk may be subjected to pairwise regionalism k ² times. In addition, in the pairwise regionalism, the p -th patch is output as a query and the q -th patch can be used as a key and value as in the following equation.

에서 쿼리패치로서

로 나타내고,

과 같이 k개의 출력 패치가 있다.For example, referring to FIG. 5, the chunk

As a query patch in

represented by,

There are k output patches as

As such, the similarity derivation unit may derive a similarity by calculating a similarity matrix between two patches of a query and a key k times in a pair-wise manner in order to improve locality.

에 대한 출력패치

는 가중 합계로 다음과 같이 계산될 수 있다.When k output patches are computed for a query patch, the pairwise outputs can be aggregated into a query patch, and the input patch

output patch for

is a weighted sum and can be calculated as:

는 패치 간의 쌍별 상대적 유사도이고, N _p 는 쿼리패치

가 속한 청크의 패치 인덱스 집합을 의미할 수 있다.here,

is the pairwise relative similarity between patches, and N _p is the query patch

This may mean a patch index set of chunks to which is included.

는 평탄화된 패치인

를 나타낼 수 있다.Also, here

is the flattened patch

can represent

In practice, since the number of patches in a bucket is often not divisible by the chunk size, patches with the same hash value can be classified into nearby chunks.

)에 기여할 수 있도록 한다. To handle this, the current chunk where the previous chunk contains the query patch (e.g.

) to contribute to

Therefore, localism can be performed 2 k times for each query patch.

The FFN (Feed-Forward Network) unit 202c is performed to emphasize and learn locality between patches, and may mean a technique commonly used in the art, but is not limited thereto.

The interpolator 203 may adjust the resolution through downsampling in the case of the encoder unit 21 and upsampling in the case of the decoder unit 23 .

As such, the processor unit 20 may predict a residual image for a distorted image through the patch partition unit 201, the KTB unit 202, and the interpolation unit 203.

Finally, the output unit 30 may restore and output the distorted image by applying the residual image predicted through the processor unit 20 to the distorted image.

를 만족시키는 영상복원이 이루어지도록 한다.Through this, the output unit 30

image restoration that satisfies the

는 복원된 영상을 의미하고,

는 왜곡영상,

는 잔여영상을 의미할 수 있다.here,

denotes a restored image,

is a distorted image,

may mean a residual image.

At this time, the output unit 30 optimizes the image restoration device of the present invention.

과 엣지손실

(Edge loss)을 활용할 수 있다.Charbonnier loss for

and edge loss

(Edge loss) can be used.

는 경험적으로 모든 실험에 대해 10^-3으로 설정되고 △는 라플라시안 함수를 나타낸다.here

is empirically set to 10 ^-3 for all experiments and Δ represents the Laplacian function.

은 초 매개 변수 λ가 두 손실의 비율을 제어하는

와

로 정의될 수 있다.Also, the total loss

where the hyperparameter λ controls the ratio of the two losses

and

can be defined as

A k-NN search-based image restoration method using the k-NN search-based image restoration apparatus will be described in detail below.

6 is a flowchart of an image restoration method based on k-NN search according to an embodiment of the present invention. 7 to 9 are exemplary diagrams schematically showing images to which an existing method and the present invention are applied for qualitative evaluation of a k-NN search based image restoration method according to an embodiment.

Referring to FIG. 6 , the image restoration method based on k-NN search according to an embodiment of the present invention may include an input step (S10), a prediction step (S20), and an output step (S30).

In the input step (S10), a distorted image may be input.

In the prediction step (S20), a residual image restored with respect to the distorted image received in the input step (S10) may be predicted.

More specifically, the prediction step (S20) includes downsampling the distorted image in the encoder unit, processing the downsampled distorted image in the bottleneck unit, and upsampling the processed distorted image to the initial resolution in the decoder unit. can include

In this case, the encoder unit and the decoder unit may be configured with a plurality of stages, and the bottleneck unit may be configured with one stage.

Here, each stage may include a patch partition step and a derivation step.

In the patch partitioning step, the distorted image may be divided into patches of a certain size.

In the derivation step, the patch partitioned in the patch partition step is applied to k-nearest local attention (K-NN local attention, KLA) to find similar patches for each patch, and the similarity can be obtained to derive the sum of weights.

In this case, in the case of the encoder unit and the decoder unit, after the derivation step, an interpolation step of adjusting the resolution through downsampling in the case of the encoder unit and upsampling in the case of the decoder unit may be further included.

Finally, in the output step (S30), the distorted image may be restored and output by applying the residual image predicted through the prediction step (S20) to the distorted image.

Since this has been described in the above device, a detailed description thereof will be omitted.

The comparative experiment results of the image restoration method according to the present invention are as follows and implemented in PyTorch.

The entire network was trained 800,000 times for a batch of 16 images with a resolution of 128x128 through the AdamW optimizer commonly used in this field.

The learning rate was initially set to 1x10 ^-4 , and a linear preparation strategy and cosine annealing were adopted for stable learning.

Basically, the chunk size k and the patch size r are set to 4, and k is set to 1 because there are almost no patches in the bottleneck (eg, the number of patches is 4x4 when the HW is 256 x 256).

The number of KTBs in each stage b was set to 2 in all stages.

In the KLA step, the number of hashes for multi-round region sensitive hashing (LSH) is set to 4.

The performance of the proposed method was verified in various image restoration tasks such as image noise removal, deblurring, and dearing. For performance evaluation, PSNR and SSIM were measured in the RGB space for noise removal and deblurring, and the evaluation was performed in the Y channel of the YCbCr color space.

[Experimental Example 1] Image Denoising

In practice, the entire network was trained using the SIDD data set containing 320 high-resolution images with noise.

Table 1 shows the quantitative evaluation of the actual noise rejection for the SIDD and DND data sets.

BM3D, DnCNN, MLP, CBDNet, RIDNet, AINDNet, VDN, SADNet, DANet, CycleISP, MPRNet, MIRNet, NBNet, DAGL, and Uformer in Table 1 are conventional noise removal methods, and KiT is an image restoration method according to the present invention. .

Since the DND data set does not provide real-world labels, the results were obtained from official benchmarks. The image restoration method according to the present invention outperforms state-of-the-art methods in both SIDD data sets and can achieve competitive performance in DND data sets.

Since the DND dataset does not provide training data, it could be Tab's performance.

As shown in Table 1, it is achieved using a network trained on the SIDD data set, and the robustness of the proposed method can be demonstrated as confirmed by the present invention's KiT with high numbers.

Subsequently, the qualitative results are described with reference to FIG. 7 as follows.

7 shows denoising images of various denoising methods for the SIDD data set.

More specifically, referring to FIG. 7 , the existing method outputs a reconstructed image with details lost, whereas the image restoration method of the present invention has a detailed structure thanks to the ability to capture adjacency with a non-local connection. You can see that the image with degraded performance was successfully restored.

[Experimental Example 2] Image Deblurring

Table 2 can show the performance of image deblurring to remove smearing on the GoPro data set.

The GoPro data set provides a composite blur image from which each image is obtained by averaging successive sharp images.

For training, 2,103 GoPro dataset images were used, 1,111 Go-Pro images, 3,758 RealBlur images, and 2,025 HIDE dataset images were evaluated.

As shown in Table 2, the excellent performance of the PSNR and SSIM metrics verified that KiT, the image restoration method of the present invention, is also helpful in restoring blurry images.

Referring to FIG. 8, (a) shows MPRNet, (b) MIMO-Net, and (c) shows an image restored from a blurry image of a GoPro data set with KiT, an image restoration method according to the present invention. As shown in FIG. 8, it can be easily found that the image restoration method of the present invention captures fine detail information, whereas other methods fail to capture detailed information.

[Experimental Example 3] Image Deraining

A network for image processing was trained using 13,712 pairs of clean rain images sampled from multiple data sets.

The deraining performance of removing light streaks present in images was evaluated for five data sets of Test100, Rain100H, Rain100L, Test2800, and Test1200.

As shown in Table 3, MPRNet has high values in some parts, but it was confirmed that KiT, the image restoration method of the present invention, showed high values in more items, and through the average score, KiT was 0.21dB compared to MPRNet. It was confirmed that a performance gain of

Referring to FIG. 9, (a) is DerainNet, (b) is PreNet, (c) is RESCAN, (d) is MPRNet, and (e) is an image derivation result through KiT, an image restoration method according to the present invention. shows a qualitative comparison of

As shown in FIG. 9, it was confirmed that KiT, the present invention, was the most excellent.

In addition, as shown in the results of FIGS. 7 and 9 described above, the present invention has the advantage of being able to process repeated textures as KLA aggregates patches of similar patterns.

In particular, since there are many patches with similar patterns in the image where light streaks exist, it was determined that the present invention can show superior performance in terms of quantity and quality compared to the compared methods.

The embodiments of the present invention described above can be implemented by various substitutions, modifications, and changes without departing from the technical spirit of the present invention, and such implementation can be performed by experts in the technical field to which the present invention belongs from the description of the above-described embodiments. that can be easily implemented.

1: Image Restoration Device
10: input unit
20: processor unit
21: encoder unit
22: bottle neck part
23: decoder unit
200: stage
201: patch partition unit
202: KTB part
202a: LN part
202b: KLA part
202c: FFN part
203: interpolation unit
30: output unit

Claims (7)

In the image restoration apparatus for restoring a distorted image based on k-NN search,
an input unit for receiving a distorted image;
A processor unit predicting a residual image reconstructed for the distorted image based on the distorted image input from the input unit; and
An output unit for restoring and outputting the distorted image by applying the residual image predicted through the processor unit to the distorted image;
The processor unit,
an encoder unit comprising a plurality of stages and downsampling the distorted image received from the input unit;
A bottleneck unit configured as one stage and processing the distorted image downsampled by the encoder unit; and
A decoder comprising a plurality of stages and upsampling the distorted image processed in the bottleneck to an initial resolution;
The stage is
A patch partition unit dividing the distorted image into patches of a certain size; and
and a KTB (k-NN Transformer Block) unit for deriving a sum of weights by finding similar patches for each of the patches divided by the patch partition unit and obtaining a similarity.
delete According to claim 1,
The KTB (k-NN Transformer Block) unit,
a layer normalization (LN) unit normalizing the patch;
A KLA (k-NN Local Attention) unit for deriving a weighted sum through the similarity between the normalized patches; and
Image restoration apparatus including a feed-forward network (FFN) unit that is performed to emphasize and learn the locality between the patches.
According to claim 3,
The KLA (k-NN Local Attention) unit,
a matching unit for finding similar similar patches through k-NN matching using locality sensitive hashing (LSH) for each patch;
a grouping unit that sorts and groups the similar patches found in the matching unit, and divides the grouped similar patches into chunks; and
and a similarity derivation unit that performs pair-wise local attention between the similar patches included in the chunk to obtain a weighted sum through the similarity between the similar patches.
According to claim 4,
The locality sensitive hashing (LSH, Locality sensitive hashing),
An image restoration apparatus according to claim 1 , wherein the patches belonging to the same bucket have the same hash value by applying the patch and rotating the map through a random rotation matrix to a map in which a plurality of divided buckets are formed.
According to claim 4,
The pairwise regionalism,
An image restoration apparatus characterized in that for the two similar patches positioned adjacent to each other, a degree of similarity between pixels constituting the similar patches is obtained.
In the image restoration method for restoring a distorted image based on k-NN search,
an input step of receiving a distorted image;
A prediction step of predicting a residual image reconstructed for the distorted image based on the distorted image; and
An output step of restoring and outputting the distorted image by applying the residual image to the distorted image;
The prediction step is
downsampling the distorted image in an encoder unit composed of a plurality of stages;
Processing the distorted image down-sampled in a bottleneck unit composed of one stage; and
Upsampling the distorted image processed in a decoder unit composed of a plurality of stages to an initial resolution;
The stage is
A patch partition step of dividing the distorted image into patches of a certain size; and
An image including a derivation step of deriving a weight sum by finding a similar patch for each patch by applying K-NN local attention (KLA) to the patch divided in the patch partition step and obtaining a similarity. restoration method.

Images