KR20220008135A - Method and apparatus for image super resolution - Google Patents

Abstract

A method of operating a computing device operated by at least one processor includes the following steps of: instructing a teacher model by using learning data including a low-resolution image and a high-resolution image corresponding to the low-resolution image, and then, initially instructing a student model with the learning data; applying a weighted value to characteristic values with high importance among characteristic values generated during the procedure of instructing the teacher model, and then, additionally instructing the student model by using the important characteristic values; and inputting a random image into the student model, and then, outputting an image made by increasing the resolution of the random image. The student model is a deep learning model which is equal to or smaller in size than the teacher model. Therefore, the present invention is capable of reducing computational complexity and memory consumption.

Description

Image super-resolution processing method and apparatus

The present invention relates to a technique for increasing the resolution of an image.

Recently, high-resolution displays such as Ultra-High Definition (UHD) have appeared in the market, and accordingly, consumer demand for high resolution has increased. Therefore, interest in the Super-Resolution (SR) algorithm that can convert low-resolution (LR) images such as the existing Full-High Definition (FHD) into high-resolution (HR) images is growing. have.

For example, a polynomial-based interpolation method such as Bicubic Interpolation or a local patch-based super-resolution method using Linear Mapping uses high-quality, high-resolution images with low complexity and small amount of computation. can create Therefore, there is no problem in applying these techniques to an actual display system.

However, in these existing super-resolution techniques, there is a deterioration and deterioration of the image quality of the detailed part in the image restoration process. In addition, since existing techniques are based on relatively simple linear mapping, it is difficult to implement a complex and nonlinear low-resolution-high-resolution model.

In order to solve the above problem, various studies are being conducted on super-resolution algorithms based on deep learning recently, and they show higher performance compared to existing algorithms.

In particular, the convolutional neural network (CNN)-based super-resolution network shows high performance compared to the existing algorithms, and attempts are being made to implement it in a display system. Convolutional neural networks use multi-layered networks to precisely analyze complex nonlinear relationships between low-resolution inputs and high-resolution outputs. Because it learns convolutional filter parameters, it shows better performance than existing algorithms based on simple linear mapping.

However, the deep learning-based super-resolution algorithm has high complexity as it involves a large number of layers and parameters, and there may be problems in which the amount of computation and memory consumption increase. For example, Very Deep Super Resolution (VDSR), a representative deep learning-based super-resolution network, has 20 layers and requires more than 600,000 filter parameters.

This causes a problem in the process of implementing the deep learning-based super-resolution algorithm on the actual display. Specifically, low-power devices such as embedded systems and mobile devices with limited hardware resources have limited power consumption and memory size, which may become an obstacle to deploying deep learning-based super-resolution algorithms. Therefore, there is a need for a method to lighten the deep learning-based super-resolution algorithm and at the same time minimize the performance decrease.

The task to be solved is to provide a method of learning a teacher network using a learning image, learning a lightweight student network using the information of the learned teacher network, and outputting a high-resolution image using the student network. will be.

As a method of operating a computing device operated by at least one processor according to an embodiment, a teacher model is trained using a low-resolution image and training data including a high-resolution image corresponding to the low-resolution image, and the learning data The step of initially learning the student model, assigning weights to high-importance feature values among the feature values generated in the learning process of the teacher model, and further learning the student model using the important feature values, and optionally and inputting an image of the student model into the student model, and outputting an image with an increased resolution of the arbitrary image. The student model is a deep learning model having a size equal to or smaller than that of the teacher model.

The teacher model, a residual block including a plurality of convolutional layers for extracting features of the learning data, an activation function transferring the results of the convolutional layers, and a multi-layer scaling the result of the activation function (Residual Block) It may include at least one or more.

In the step of additionally learning the student model, the loss function of the student model may be modified using output values of each residual block of the teacher model.

In the step of further learning the student model, among the output values of each residual block, the teacher model is determined as important information for generating the high-resolution image from the low-resolution image, and the output values of the weighted residual blocks are used as the important features. values can be determined.

A computing device according to one embodiment includes a memory and at least one processor executing instructions of a program loaded into the memory. The program includes the steps of initially learning a student model using learning data including a low-resolution image and a high-resolution image corresponding to the low-resolution image, from a teacher model whose learning is completed with the learning data, generated in the learning process of the teacher model extracting output values, retraining the student model using the output values, inputting an arbitrary low-resolution image to the student model, and outputting a high-resolution image of the arbitrary image. include The student model is a deep learning model having a size equal to or smaller than that of the teacher model.

In the re-learning, weight is given to important output values determined by the teacher model as important information for generating the high-resolution image from the low-resolution image among the output values, and using the weighted important output values, the student We can modify the loss function of the model.

The teacher model and the student model may include at least one block including a plurality of convolutional layers for extracting features of the learning data and an activation function for transferring the results of the convolutional layers. The re-learning may include extracting output values from each block included in the teacher model.

In the re-learning, a loss function used for the initial learning and a loss function due to a difference between the important output values and output values output during the initial learning process of the student model may be used.

According to the present invention, it is possible to increase the resolution of an image by using a student network with a small number of parameters, thereby reducing the amount of computation and memory consumption and distributing it to hardware with limited resources.

In addition, according to the present invention, since specific information of high importance in the learning stage of the teacher network is transmitted to the student network using the knowledge distillation technique, the student network can be trained without changing the structure of the network or increasing the number of parameters.

1 is a block diagram of an image super-resolution apparatus according to an embodiment.
2 is an explanatory diagram of a teacher network according to an embodiment.
3 is an explanatory diagram of a student network according to an embodiment.
4 is a flowchart of a method of operating an image super-resolution apparatus according to an embodiment.
5 and 6 are explanatory diagrams of a knowledge dissemination method according to an embodiment.
7 is an exemplary diagram of a method of operating an image super-resolution apparatus according to an embodiment.
8 is a hardware configuration diagram of a computing device according to an embodiment.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. have.

Hereinafter, a network lightening method and a knowledge distillation technique applied to the present invention will be briefly described.

First, a lightweight architecture refers to a network for more efficiently learning an artificial neural network with a minimum of parameters. Examples include ResNet, DensNet, MobileNet, etc. In particular, SqueezeNet shows similar performance with 50 times fewer parameters compared to AlexNet. Specifically, SqueezeNet uses a Fire module that replaces 3*3 convolutional filters with a mixture of 1*1 and 3*3 filters. The Fire module includes a squeeze layer and an extension layer. The squeeze layer consists of only a 1*1 filter, and the extension layer includes a 1*1 filter and a 3*3 filter. Hereinafter, since SqueezeNet is a known technology, a detailed description thereof will be omitted.

Knowledge Distillation is one of the methods of transferring knowledge from a large number of large teacher networks learned through an ensemble technique to a single small student network. In this case, the teacher network is a high-performance neural network, and the student network is shallow and has low performance. Studies have been conducted using only the output values of the teacher network and the student network, or using hidden features.

The present invention proposes a method that can efficiently train a deep learning-based super-resolution network by using a knowledge distillation technique, and a network that minimizes performance reduction due to weight reduction by using SqueezeNet, one of the deep learning network lightweighting techniques do. Hereinafter, an image super-resolution apparatus proposed by the present invention will be described.

1 is a block diagram of an image super-resolution apparatus according to an embodiment, FIG. 2 is an explanatory diagram of a teacher network according to an embodiment, and FIG. 3 is an explanatory diagram of a student network according to an embodiment.

Referring to FIG. 1 , the image super-resolution apparatus 10 receives a low-resolution image and restores it to a high-resolution image. In this case, after learning the teacher network 100 in advance, the lightweight student network 200 may be learned using the learned teacher network 100 , and image resolution may be increased using the student network 200 .

By using the knowledge distillation technique, weight parameters of the student network 200 may be learned based on information of the teacher network 100 that has already been learned.

At this time, the student network 200 is learned in a way that imitates the behavior of the teacher network 100 . Through this, the student network 200 can obtain a higher recognition rate than when learning from the beginning using a general backpropagation algorithm.

The teacher network 100 and the student network 200 may be implemented in one computing device or distributed in separate computing devices. When distributed in separate computing devices, the teacher network 100 and the student network 200 may communicate with each other through a communication interface. The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, or the like.

Each of the teacher network 100 and the student network 200 may be one AI model or may be implemented with a plurality of AI models. And the image super-resolution device 10 may also be a single artificial intelligence model, may be implemented as a plurality of artificial intelligence models. Accordingly, one or a plurality of artificial intelligence models corresponding to the above-described configurations may be implemented by one or a plurality of computing devices.

Referring to FIG. 2 , the teacher network 100 includes a convolutional layer, a residual block, and an upscaling module. The teacher network 100 may be implemented as EDSR (Enhanced Deep Super Resolution), and each element constituting the teacher network 100 will be described below.

The convolutional layer serves to extract features of the input low-resolution image.

Each residual block includes nine residual modules, and one residual module consists of two convolutional layers, one Rectified Linear Unit (ReLU), and multi-layers.

The commutation linear unit is one of the activation functions, and the multi-layer serves to multiply the end of each module by a specific constant value. The constant value may be, for example, 0.1.

The upscaling module serves to increase the resolution of the image, can be implemented as a sub-pixel convolutional neural network (SPCNN), and exhibits better performance than polynomial interpolation such as bicubic interpolation.

The 3 remaining blocks and 9 remaining modules may lead to a skip connection with each other. Skip connection is a method in which an input value of one convolutional layer is directly added to an output value of another convolutional layer so that each convolutional layer predicts only a residual value between an input and an output.

Referring to FIG. 3 , the student network 200 includes a convolutional layer, a squeeze block, and an upscaling module. The student network 200 may be implemented with SqueezeSR, and each element constituting the student network 200 will be described below.

Like the teacher network 100, the first convolutional layer extracts features of the input low-resolution image. Thereafter, three squeeze blocks are connected, and each squeeze block includes a squeeze layer and an extension layer. Meanwhile, each squeeze block may be connected by a skip connection.

The squeeze layer refers to a 1x1 convolution filter, and the extension layer refers to a connection between a 1x1 convolution filter and a 3x3 convolution filter.

The squeeze layer reduces the number of input channels and passes them to the next layer. That is, the size of the channel delivered to the extension layer can be reduced. This means that the number of parameters is reduced by 9 times compared to the general CNN model.

Meanwhile, unlike the known SqueezeNet type, the student network 200 of the present invention does not include a layer for batch normalization and a downsampling layer.

The upscaling module for increasing the resolution of the image may be implemented as SPCNN like the teacher network 100 .

Hereinafter, a process in which the student network 200 receives information generated in the learning process of the teacher network 100 and additionally learns will be described.

4 is a flowchart of a method of operating an image super-resolution apparatus according to an embodiment, and FIGS. 5 and 6 are explanatory diagrams of a knowledge propagation method according to an embodiment.

Referring to FIG. 4 , the image super-resolution apparatus 10 initially learns the teacher network 100 and the student network 200 using learning data including a low-resolution image and a high-resolution image corresponding thereto (S110). In this case, the image used for initial learning is a color image, and may be an image having three color channels of RGB.

The image super-resolution device 10 connects the learned teacher network 100 and the student network 200 with a TAT (Teacher Attention Transfer) module (S120).

Since the feature map of the student network 200 and the feature map of the teacher network 100 are similar, if the feature map information determined to be important in the learning process of the teacher network 100 is transmitted to the student network 200, the student network ( 200) can be learned more efficiently.

In general, in a convolutional neural network, a convolution operation is performed using a filter for each channel of three-dimensional input data having three color channels. In this case, the filter plays a role in finding the characteristics of the input image. Since the size of the filter is smaller than the size of the image, the filter reflects the characteristics of a part of the image and outputs a result called a feature map.

In this case, the output values of the specific filter may be considered as more important factors in the learning process for increasing the image resolution. Therefore, it is possible to reflect importance by assigning weights to the output values of these filters.

Meanwhile, as a method of assigning weights, an attention mechanism-based TAT module may be used.

The attention mechanism (Attention Mechanism) is a structure that allows the model to focus on an important part, and every time the decoder predicts the output value, the entire input value from the encoder is referenced again. At this time, instead of referring to the entire input value at the same rate, attention is focused on a specific part of the input value that is related to the value to be predicted. Meanwhile, in a deep learning model, attention can be implemented as an importance vector expressed as a weight. Since the content of the attention mechanism is already known technology, a detailed description thereof will be omitted.

That is, when the student network 200 receives information from the teacher network 100 , the TAT module focuses on the output value of the filter determined to be important of the teacher network 100 , so that information of high importance for each channel is provided. By passing the weights to the student network 200 , the student network 200 helps the student network 200 to focus on more important information during learning.

The image super-resolution apparatus 10 transmits information including a weight for information having high importance for each channel of the teacher network 100 to the student network 200 ( S130 ).

Specifically, the TAT module obtains a feature map t_n output from each residual block constituting the teacher network 100 , and obtains an output feature map s_n from each squeeze block constituting the student network 200 . In this case, n is the number of residual blocks or squeeze blocks, and is a natural number from 1 to 3 with reference to the example of FIG. 5 .

The TAT module calculates the loss function by using the output value of each block. In this case, the value input to the loss function, ie, the output value of the TAT module, can be obtained through Equation 1.

[Equation 1]

In Equation 1, k denotes a variable for improving performance, c denotes the channel size of the output feature map of the teacher network 100, and c' denotes the channel size of the output feature map of the student network 200 it means. T_n and S_n are values input to the final loss function.

When k is 1, T_i may mean an average feature map of t_i, and S_i may mean an average feature map of s_i.

Meanwhile, the output feature map t_n of the teacher network 100 may be a vector or a tensor having a size of C*H*W. Here, C is the channel size of the output feature map, H is the height of the output feature map, and W is the width of the output feature map.

In order to obtain the attention vector for each channel of the teacher network 100 , global average pooling for extracting the average of the output feature map may be used. Global average pooling is applied to each output feature map t_n, and a tensor of size C*1*1 can be obtained as a result through an activation function.

Thereafter, the tensor is sequentially multiplied by s_n, which is the output feature map of the student network 200 . Meanwhile, by calculating the average of the tensor in units of channels, the size of the final tensor can be made 1*H*W.

The image super-resolution apparatus 10 further uses the loss function calculated through the TAT module to perform fine-tuning of the student network 200 (S140). In this case, the loss function used for fine adjustment may be as in Equation (2).

[Equation 2]

는 학습에 사용되는 정답 이미지(Ground Truth, GT)를 의미한다. loss_0는 S110 단계에서 학생 네트워크(200)의 학습 결과에 따른 손실함수로서, 학생 네트워크(200)의 출력 이미지와 학습 데이터에 포함된 정답 이미지 간의 차이에 의해 계산된 손실함수를 의미한다. loss_1, loss_2, loss_3은 학생 네트워크(200)와 교사 네트워크(100)의 각 필터의 출력 특징맵 간 차이에 의해 계산된 손실함수를 의미한다. T_n은 교사 네트워크(100)를 구성하는 잔여 블록의 출력값에 대한 TAT 모듈의 출력값, S_n은 학생 네트워크(200)를 구성하는 스퀴즈 블록의 출력값에 대한 TAT 모듈의 출력값을 의미한다.In Equation 2, y means a high-resolution image output through the student network 200,

means the correct answer image (Ground Truth, GT) used for learning. loss_0 is a loss function according to the learning result of the student network 200 in step S110, and means a loss function calculated by the difference between the output image of the student network 200 and the correct answer image included in the learning data. loss_1, loss_2, and loss_3 mean loss functions calculated by the difference between the output feature maps of each filter of the student network 200 and the teacher network 100 . T_n denotes an output value of the TAT module with respect to an output value of the remaining blocks constituting the teacher network 100 , and S_n denotes an output value of the TAT module with respect to an output value of a squeeze block constituting the student network 200 .

That is, the final loss function of the image super-resolution device 10 is loss_0, which is a loss function for the classification performance of the student network 200 , and a loss function indicating the difference between the classification result of the teacher network 100 and the student network 200 . It is calculated as the sum of loss_1 to loss_3.

Meanwhile, in the present specification, since there are three residual blocks of the teacher network 100 and three squeeze blocks of the student network 200 , three loss function terms are added, and the number of added loss function terms is not necessarily limited thereto.

7 is an exemplary diagram of a method of operating an image super-resolution apparatus according to an embodiment.

Fig. 7 (a) is a correct answer image, and Fig. 7 (b) is a result of restoring the image by bicubic interpolation. Bicubic interpolation is an interpolation method for reconstructing a hole by using the product of pixel values of 16 adjacent pixels and weights according to distance.

Fig. 7(c) shows the restoration result of the learned teacher network 100, and Fig. 7(d) shows the restoration result of the student network 200 learned from the teacher network 100 without adding parameters. .

Since the number of parameters of the student network 200 is 100 times lower than that of the teacher network 100 and the algorithm processing speed is about 10 times faster, the student network 200 can perform real-time processing.

8 is a hardware configuration diagram of a computing device according to an embodiment.

Referring to FIG. 8 , the image super-resolution apparatus 10 executes a program including instructions described to execute the operation of the present invention in the computing device 300 operated by at least one processor do.

The hardware of the computing device 300 may include at least one processor 310 , a memory 320 , a storage 330 , and a communication interface 340 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computing device 300 may be loaded with various software including an operating system capable of driving a program.

The processor 310 is a device for controlling the operation of the computing device 300 and may be various types of processors 310 that process instructions included in a program, for example, a central processing unit (CPU), an MPU (Central Processing Unit) It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 320 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 310 . The memory 320 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 330 stores various data and programs required for executing the operation of the present invention. The communication interface 340 may be a wired/wireless communication module.

According to the present invention, it is possible to increase the resolution of an image by using a student network with a small number of parameters, thereby reducing the amount of computation and memory consumption and distributing it to hardware with limited resources.

In addition, according to the present invention, since specific information of high importance in the learning stage of the teacher network is transmitted to the student network using the knowledge distillation technique, the student network can be trained without changing the structure of the network or increasing the number of parameters.

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. is within the scope of the right.

Claims (8)

A method of operating a computing device operated by at least one processor, comprising:
Learning a teacher model using learning data including a low-resolution image and a high-resolution image corresponding to the low-resolution image, and initially learning a student model with the learning data;
weighting feature values with high importance among feature values generated in the learning process of the teacher model, and further learning the student model using the important feature values; and
inputting an arbitrary image to the student model, and outputting an image with an increased resolution of the arbitrary image,
The student model is a deep learning model of the same size or smaller than the teacher model, the operating method. In claim 1,
The teacher model is
At least one residual block including a plurality of convolutional layers for extracting features of the training data, an activation function that delivers the results of the convolutional layers, and a multi-layer that scales the results of the activation function. to do, the way it works. In claim 2,
The step of further learning the student model is,
An operating method of modifying a loss function of the student model using output values of each residual block of the teacher model. In claim 3,
The step of further learning the student model is,
Of the output values of each of the residual blocks, the teacher model is determined as important information for generating the high-resolution image from the low-resolution image and determines output values of the weighted residual blocks as the important feature values. A computing device comprising:
memory, and
at least one processor executing instructions of a program loaded into the memory;
the program is
Initial learning of a student model using learning data including a low-resolution image and a high-resolution image corresponding to the low-resolution image;
extracting output values generated in the learning process of the teacher model from the teacher model for which learning is completed using the learning data, and re-learning the student model using the output values; and
instructions described for executing the steps of inputting an arbitrary low-resolution image to the student model and outputting a high-resolution image of the arbitrary image;
The student model is a deep learning model of the same size or smaller than the teacher model, computing device. In claim 5,
The re-learning step is
Among the output values, the teacher model assigns weights to important output values determined as important information for generating the high-resolution image from the low-resolution image, and uses the weighted important output values to correct the loss function of the student model , computing devices. In claim 6,
The teacher model and the student model are
At least one block including a plurality of convolutional layers for extracting features of the training data and an activation function for transferring the results of the convolutional layers,
The re-learning step is
A computing device for extracting output values from each block included in the teacher model. In claim 6,
The re-learning step is
A computing device using a loss function used for the initial learning and a loss function due to a difference between the important output values and output values output during an initial learning process of the student model.

Images