KR102624160B1 - Ultra-high-resolution image restoration system - Google Patents

Abstract

The present invention provides a super-resolution image restoration system using an artificial neural network, comprising: an image receiver for receiving an image; an image reuse unit that analyzes a motion vector of the image received by the image receiver and reuses duplicated portions of the image; a restoration unit that inputs the image received by the image receiver, excluding the portion reused by the image reuse unit, into an artificial neural network and restores it to a super-resolution image; and an output unit that outputs the image restored by the restoration unit.

Description

Ultra high resolution image restoration system{ULTRA-HIGH-RESOLUTION IMAGE RESTORATION SYSTEM}

The present invention relates to an ultra-high-resolution image restoration system, and particularly to an ultra-high-resolution image restoration system for shortening the time for restoring a low-resolution image to a high-resolution image.

Technology for restoring low-resolution images to super-resolution images has been applied and used in various fields, and various super-resolution restoration algorithms have been proposed. Super resolution is a technology that creates high-resolution images from low-resolution images and is used in applications such as astronomy and vehicle license plate recognition.

Recently, super resolution using deep learning has shown good performance, and active research is being conducted. Super resolution can be divided into SISR (Single Image Super Resolution), which creates high resolution from a single image, and VSR (Video Super Resolution), which creates high resolution using multiple frames of an image. In the case of super resolution, a large amount of computation is used to create a natural image, making real-time inference difficult. Since the ultra-high-resolution restoration algorithm uses a neural network, it involves a large number of parameter-based calculations, so it takes a long time to calculate, and the algorithm's memory usage is also high.

Therefore, there is a need for an ultra-high resolution image restoration system that can reduce the amount of computation and computation time.

This project (result) is the result of a local innovation project based on local government-university cooperation conducted in 2021 with funding from the Ministry of Education and support from the National Research Foundation of Korea (2021RIS-004).

This results was supported by "Regional Innovation Strategy (RIS)" through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(MOE)(2021RIS-004)

The purpose of the present invention is to provide a super-resolution image restoration system that can shorten the restoration time by reducing the weight of the existing ultra-high-resolution restoration algorithm by compressing and accelerating it.

In order to achieve the above object, the present invention provides a super-resolution image restoration system using an artificial neural network, comprising: an image receiver for receiving an image; an image reuse unit that analyzes a motion vector of the image received by the image receiver and reuses duplicated portions of the image; a restoration unit that inputs the image received by the image receiver, excluding the portion reused by the image reuse unit, into an artificial neural network and restores it to a super-resolution image; and an output unit that outputs the image restored by the restoration unit.

Preferably, the restoration unit may transmit difference values and movement vectors between video frames to the image reuse unit.

Preferably, the restoration unit includes an image cleaning module that removes image deterioration; A two-way propagation module that extracts features before and after the image; and an upsampling module that generates a super-resolution image from features before and after the image extracted from the two-way propagation module.

Preferably, the two-way propagation module is composed of a plurality of convolutional layers for extracting features before and after the image, uses only a portion of the plurality of convolutional layers, and pre- and post-frames are reused in the image reuse unit. Since it exceeds , only some convolution operations can be performed.

Preferably, the two-way propagation module can use only half of the convolutional layer.

According to the present invention, there is an advantage that ultra-high resolution restoration time can be shortened by compressing unnecessary neural networks.

In addition, the present invention has the advantage of being applicable to portable devices because it reduces the amount of computation by lightweighting the neural network structure.

Figure 1 shows a configuration diagram of an ultra-high resolution image restoration system according to an embodiment of the present invention.
Figure 2 shows an existing super-resolution restoration algorithm according to an embodiment of the present invention.
Figure 3 shows a configuration diagram of a restoration unit according to an embodiment of the present invention.
Figure 4 shows an algorithm of a super-resolution image restoration system according to an embodiment of the present invention.
Figure 5 shows a graph comparing time performance according to an artificial neural network model according to an embodiment of the present invention.
Figure 6 shows the weight distribution of an artificial neural network according to an embodiment of the present invention.
Figure 7 shows the output of super-resolution image restoration according to an artificial neural network model according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the exemplary embodiments. The same reference numerals in each drawing indicate members that perform substantially the same function.

The purpose and effect of the present invention can be naturally understood or become clearer through the following description, and the purpose and effect of the present invention are not limited to the following description. Additionally, in describing the present invention, if it is determined that a detailed description of known techniques related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Figure 1 shows a configuration diagram of an ultra-high resolution image restoration system 10 according to an embodiment of the present invention. Referring to FIG. 1, the ultra-high resolution image restoration system 10 may include an image reception unit 100, an image reuse unit 300, a restoration unit 500, and an output unit 700.

The ultra-high resolution image restoration system 10 can restore low-resolution images to ultra-high resolution images. The ultra-high resolution image restoration system 10 can shorten restoration time by accelerating and compressing the ultra-high resolution image restoration algorithm. The ultra-high resolution image restoration system 10 can restore a low-resolution image to a super-resolution image using a neural network-based image restoration algorithm.

The ultra-high resolution image restoration system 10 can compress unnecessary neural networks to make the algorithm lighter. The ultra-high resolution image restoration system 10 can maintain the same quality as the existing algorithm even if the algorithm is lightweight. The ultra-high-resolution image restoration system 10 can reduce the amount of calculation through lightweighting the algorithm. The ultra-high resolution image restoration system 10 can be applied to portable devices such as smartphones by increasing the speed of ultra-high resolution image restoration and reducing the amount of computation.

Figure 2 shows an existing super-resolution restoration algorithm according to an embodiment of the present invention. Referring to Figure 2, the existing super-resolution restoration algorithm undergoes image cleaning to remove deterioration of the input image and then inputs it to the backbone network. Afterwards, features before and after the image are extracted through forward/backward bidirectional propagation. Afterwards, a super-resolution image can be created by connecting and fusing the image and features before and after the image.

The difference between the existing super-resolution restoration algorithm and the present invention is that it reuses overlapping image parts using motion vectors and uses only a portion of the artificial neural network network. Specific details about the features of the present invention are described below.

The video receiver 100 can receive video. The image receiver 100 can receive images from a photographing device or a storage device. The image receiver 100 is connected to a third device through a wired or wireless network and can receive images captured or stored by the third device. The image receiver 100 may receive an image to be restored and transmit it to the image reuse unit 300.

The video receiver 100 may preprocess the received video. The image receiver 100 may perform preprocessing by normalizing the pick values of the received image. The image receiver 100 can normalize the pixels of the image to improve the computation speed in the artificial neural network.

The video reuse unit 300 can analyze the motion vector of the video received by the video receiver 100 and reuse the duplicated portion of the video. The image reuse unit 300 can reuse the duplicated portions because duplicated portions occur in consecutive images due to the nature of the video, which is a set of consecutive images. The image reuse unit 300 can reconstruct the original image by regenerating the prediction area using the motion vector and combining the restored errors in the prediction area.

The motion vector refers to the difference vector between the position coordinates of the current block and a reference block that is very similar. In the present invention, the motion vector may refer to the difference vector between the position coordinates of the front and back images. Movement vectors can eliminate temporal redundancy. A movement vector can describe the horizontal and vertical displacement from the most likely position in the previous or next frame, which is a reference frame. Movement vectors can provide information about where each video screen unit moves, from the front or back of the screen. The motion vector can greatly reduce the amount of data by encoding only the difference between the current macro block and the macro block of the previous screen whose motion was compensated by the motion vector.

The image reuse unit 300 can reduce the amount of calculation of the artificial neural network by reusing overlapping images. Through this, the image reuse unit 300 can improve the restoration speed. The image reuse unit 300 may receive difference values and movement vectors between image frames from the restoration unit 500 or the output unit 700.

Figure 3 shows a configuration diagram of the restoration unit 500 according to an embodiment of the present invention. Referring to FIG. 3 , the restoration unit 500 may include an image cleaning module 510, a bidirectional propagation module 530, and an upsampling module 550.

The restoration unit 500 can input the image received by the image receiver 100, excluding the portion reused by the image reuse unit 300, into an artificial neural network and restore it to a super-resolution image. The restoration unit 500 performs restoration by excluding overlapping portions of images input to the system for ultra-high resolution restoration, thereby improving calculation speed and reducing calculation amount.

Figure 4 shows the algorithm of the ultra-high resolution image restoration system 10 according to an embodiment of the present invention. Referring to FIG. 4, the restoration unit 500 may transmit the difference value and motion vector between image frames to the image reuse unit 300. The video reuse unit 300, which has received the difference value and motion vector between video frames, can analyze the motion vector and reuse the overlapping video portion.

The image cleaning module 510 can remove image deterioration.

The two-way propagation module 530 can extract features before and after the image. The two-way propagation module 530 can perform two-way propagation focusing on long-term and global propagation. The two-way propagation module 530 can demonstrate excellent performance in terms of restoration quality and efficiency of the artificial neural network.

The two-way propagation module 530 consists of a plurality of convolutional layers for extracting features before and after the image, uses only a portion of the plurality of convolutional layers, and skips the before and after frames reused in the image reuse unit 300. Therefore, only part of the convolution operation can be performed. Preferably, the two-way propagation module 530 can use only half of the convolutional layers. Since the two-way propagation module 530 generates redundant operations in a plurality of convolution layers, it is possible to solve the problems of increasing the amount of operations and reducing the operation speed due to the duplicate operations by using only a portion of the plurality of dry volume layers. The two-way propagation module 530 can solve the problems of increasing the amount of calculation and reducing the calculation speed by performing only part of the convolution operation.

The upsampling module 550 can generate a super-resolution image from features before and after the image extracted from the two-way propagation module.

The output unit 700 may output the image restored by the restoration unit 500.

Below, the simulation results of the present invention are described.

In this simulation, the inference time was measured and the models were analyzed on the NVIDIA Jetson Board for artificial neural network models showing good performance indicators. Through this simulation, it can be confirmed that the ultra-high-resolution image restoration system 10 according to the present invention is the most suitable model.

The performance of the super resolution model can be evaluated by PSNR (Peak Signal-to-noise ratio), which evaluates loss information, and SSIM (Structural Similarity Index Map), which evaluates similarity to visual image quality.

Recently, the inference performance was measured on the Jetson Xavier NX board for the SISR models LIFF and EDSR and the VSR models basicVSR, bascivVSR++, and Realbasic VSR, which are artificial neural network models that show high performance in two figures. For each model, the time to enlarge a 320x240 image by 4 times to 1280x960 was measured and tested in the MMEditting framework written based on PyTorch.

Figure 5 shows a graph comparing time performance according to an artificial neural network model according to an embodiment of the present invention.

Figure 6 shows the weight distribution of an artificial neural network according to an embodiment of the present invention.

Figure 7 shows the output of super-resolution image restoration according to an artificial neural network model according to an embodiment of the present invention.

Referring to Figures 5 to 7, the EDSR model using a convolution layer shows the fastest speed at 0.67 seconds, but shows low image quality. The LIFF model is based on EDSR with additional structures and shows the slowest speed at 13.78 seconds per sheet. BasicVSR models are based on Spynet, which extracts optical flow. Realbasic VSR shows good image quality at 1.51 seconds and is relatively faster than other models. Therefore, the present invention selected the RealbasicVSR model as a suitable model, reused overlapping image parts using movement vectors to accelerate and compress the model, and used only a portion of the artificial neural network network.

RealbasicVSR can be divided into Image Cleaning block, Spynet, Forward block and Backward block, and Upsample Layer. The Image Cleaning block and Forward and Backword blocks consist of a convolution layer and account for 70% of the total inference time. Since the weights are distributed close to 0, pruning techniques can be applied. In this simulation, part of the convolution layer was simply removed from the three modules, and the results were shown in (d) of Figure 7. Compared to the existing super-resolution restoration algorithm (c) in Figure 6, it was confirmed that the PSNR showed good performance with 31.2 and SSIM of 0.90, and the inference time was 30% faster.

When performance was measured for super resolution using deep learning on an embedded board, the Real basicVSR model was fast and showed good performance. The structure of the model was analyzed for real-time inference, and the weights of the convolution layers were distributed close to 0, showing a 30% improvement in inference performance even when the convolution layer was simply removed.

Although the present invention has been described in detail through representative embodiments above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. will be. Therefore, the scope of rights of the present invention should not be limited to the described embodiments, but should be determined not only by the claims described later, but also by all changes or modified forms derived from the claims and the concept of equivalents.

10: Ultra-high resolution image restoration system
100: Video receiver
300: Video reuse unit
500: restoration unit
510: Image cleaning module
530: Two-way radio wave module
550: Upsampling module
700: output unit

Claims (5)

In a super-resolution image restoration system using an artificial neural network,
A video receiving unit that receives video;
an image reuse unit that analyzes motion vectors of the image received by the image receiver and reuses duplicated frames in the image without inputting them into an artificial neural network;
a restoration unit that inputs the image received by the image receiver, excluding frames reused by the image reuse unit, into an artificial neural network and restores it to a super-resolution image; and
An output unit that outputs the image restored by the restoration unit,
The video receiver,
Preprocessing is performed to normalize the pixel values of the received image,
The video reuse department,
Regenerate the prediction area using the movement vector,
The restoration unit,
A bidirectional propagation module that extracts features before and after the image, which are the result of convolution operations of the front and back frames, using forward/backward bidirectional propagation of the image; and
It includes an upsampling module that generates a super-resolution image from features before and after the image extracted from the two-way propagation module,
The two-way propagation module is,
It consists of a plurality of forward convolution layers and a plurality of backward convolution layers learned to extract features before and after the image,
The two-way propagation module is,
Extracting features before and after the image using only some of the forward convolutional layers and the plurality of backward convolutional layers based on the distribution of weights, respectively.
Ultra-high resolution video restoration system.
delete According to claim 1,
The restoration unit,
Characterized in that it further includes an image cleaning module that removes image deterioration,
Ultra-high resolution video restoration system.
delete According to claim 1,
The two-way propagation module is,
Characterized in that only half of the convolutional layers are used for each of the plurality of forward convolutional layers and the plurality of backward convolutional layers,
Ultra-high resolution video restoration system.