KR102192392B1 - Residual Network system for correcting low resolution image - Google Patents

Abstract

The present invention relates to a residual network for image correction. More specifically, in the present invention, the cascading residual network is an Efficient Residual Block, a cascading residual block composed of a plurality of the lightweight residual blocks, and a plurality of cascading residuals. Consisting of recursive blocks, each of the plurality of concatenated residual recursive blocks is connected to a plurality of convolutional blocks above the concatenated residual network, and each of the plurality of lightweight residual blocks is at least one previous convolutional block. Specific information is calculated using the result value calculated by the solution layer.

Description

Residual Network system for correcting low resolution image

The present specification relates to an artificial neural network system for low resolution images. More specifically, the present invention relates to a residual network system for correcting a low-resolution image in real time using a convolution neural network.

Super-Resolution (SR) is a computer vision task that reconstructs a High-Resolution (HR) image from a Low-Resolution (LR) image. In particular, it is related to Single Image Super-Resolution (SISR), which performs SR using a single LR image.

Because of the many-to-one mapping, SISR is generally difficult to compute HR images from LR images. Despite these difficulties, SISR is a very active area because it has the potential to overcome resolution limitations and can be used in various applications such as video streaming or surveillance systems.

Recently, a method based on a convolutional neural network (CNN) provides excellent performance in SISR work. From SRCNN with three convolutional layers to MDSR with more than 160 layers, the depth and overall performance of the network increase rapidly over time.

However, there is a problem that even if the quality of the SR image is improved by the in-depth learning method, it is often not suitable for actual scenarios.

Accordingly, the present invention proposes a network for correcting low-resolution images using a CNN having a recursive structure in order to design a practical lightweight deep learning model to solve this problem.

The technical problems to be achieved in the present specification are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description. I will be able to.

The cascading residual network for image correction according to the present invention includes: an Efficient Residual Block; A cascading residual block composed of a plurality of the lightweight residual blocks; And a plurality of concatenated residual recursive blocks, each of the plurality of concatenated residual recursive blocks is connected to a plurality of convolutional blocks above the concatenated residual network, and each of the plurality of lightweight residual blocks is previously Specific information is calculated using the result value calculated by at least one convolutional layer of.

In addition, in the present invention, the plurality of lightweight residual blocks are configured as a local chain connection in order to use a result value calculated by the at least one convolutional layer.

Further, in the present invention, each of the blocks constituting the chain residual network is configured in a recursive structure.

이고, u 번째 연쇄 잔차 재귀 블록이

인 경우, u 번째 특정 정보 H^u는 아래의 수학식을 통해서 계산된다.In addition, in the present invention, the u-th lightweight residual block

And the u-th chain residual recursive block is

In the case of, the u-th specific information H ^u is calculated through the following equation.

In addition, in the present invention, each of the plurality of light weight residual blocks includes two first convolution blocks and one second convolution block.

Further, in the present invention, the first convolution block is a group convolution of a specific size, and the second convolution is a pointwise convolution.

In addition, in the present invention, the specific size is variably determined by input information obtained from a user.

In addition, in the present invention, the chain residual network further includes an up-sampling block for processing low-resolution images of three different scales into one model.

In addition, the present invention provides a residual network performed on a computer through a computer-readable recording medium in which a program is recorded, for the residual network for image correction.

According to an embodiment of the present invention, compared to a general residual network, there is an effect of reducing the computational amount and parameters of the network.

In addition, according to an embodiment of the present invention, in a chain residual network for correcting a low-resolution image in real time, the size of the group convolution is adjusted based on information obtained from the user, thereby making the optimal use of the relationship between the network performance and the amount of computation. There is an effect of being able to select the group size of.

The effects obtainable in the present specification are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to aid in understanding of the present invention, provide embodiments of the present invention, and describe technical features of the present invention together with the detailed description.
1 is a diagram illustrating a network structure of a residual network for correcting a low-resolution image proposed in the present specification.
2 is a diagram illustrating the structure of a continuous residual recursive block proposed in the present specification.
3 is a diagram illustrating a structure of a residual block proposed in the present specification.
4 is a diagram illustrating PSNR performance of a continuous residual network proposed in the present specification.

In the present invention, since various transformations can be applied and various embodiments can be provided, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to be limited to a specific embodiment of the present invention, it is to be understood to include all conversions, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

One way to create a lightweight model is to reduce the number of parameters. There are many ways to make such a model, but the simplest and most effective way is to use a network with a recursive structure.

Over time, the depth and overall performance of the network have improved significantly, but even if the quality of the SR image is improved by deep learning methods, it is often not suitable for real scenarios.

Therefore, it is important to design a practical lightweight deep learning model that can be applied to actual applications. One way to create such a model is to reduce the number of parameters, and a simple and effective method for reducing parameters is to use a recursive network.

For example, a deeply-recursive convolutional network (DRCN) uses a recursive network to reduce redundant parameters and improves performance by adding a residual structure.

Compared with the standard CNN, this model effectively reduces the number of parameters of the model and shows good performance. However, this model has two drawbacks as follows.

1) The input image is upsampled before input to the CNN model.

2) Using a recursive network, increase the depth or width of the network to compensate for the loss.

This disadvantage is that the details of the image are maintained when the model is reconstructed, but there is a problem that the number of tasks and inference time are increased.

So far, most methods aimed at creating lightweight models have focused primarily on reducing the number of parameters.

However, as described above, in the actual operating scenario, the number of tasks and the situation in which the SR system operates on the mobile device must be considered.

In addition, the execution speed of the system is also an important factor in terms of user experience. In particular, battery capacity, which is highly dependent on the amount of computation being performed, can be a major consideration. In this regard, reducing the number of operations in a deep learning architecture is a necessary step.

Another scenario is related to applying the SR method to a video streaming service. The demand for streaming media is rapidly increasing, and a large amount of storage is required to store a large amount of multimedia data.

Therefore, it is essential to compress data using a lossy compression technique before storage, and afterwards, the data can be restored to its original resolution by applying the SR technique.

However, since delay is the most important factor in streaming services, the decompression process (ie, super-resolution) must be performed in real time, and for this, it is essential to make the SR method light in terms of the number of operations.

The present invention considers these issues and proposes a cascading residual network system for real-time correction of low-resolution images that improves the problems of the existing model.

Hereinafter, the chain residual network system proposed in the present invention is referred to as CARN and CARN-M.

The present invention builds a CARN model to improve performance, and extends the built model to CARN-M to optimize speed and number of tasks.

CARN and CARN-M receive low-resolution images and calculate high-resolution images as output.

The intermediate configuration of CARN and CARN-M proposed in the present invention may be designed based on deep residual learning for image recognition.

Specifically, the CARN and CARN-M proposed in the present invention may be configured as a CNN that receives a low-resolution image and corrects the received low-resolution image into a high-resolution image.

The CARN and CARN-M proposed in the present invention are lightweight to achieve real-time performance of low-resolution image restoration: 1) improve system performance through a concatenated residual network structure, and 2) efficient system construction in terms of computational load and model size. The residual block 300 and a recursive structure are used.

Hereinafter, a neural network applicable to the present invention will be described.

Efficient Neural Network

For a small and efficient neural network, the following can be considered.

1) Pre-linked network compression

2) Small but efficient model

In order to create such a small and efficient neural network, a deep compression technique consisting of pruning, vector quantization, and Hoffman coding can be used to reduce the size of a previously processed network.

Hereinafter, convolution and parameters of convolution that can be used for deep learning of a neural network that can be applied to the present invention will be described.

Kernel Size: The kernel size determines the view of the convolution. Usually used as 3x3 pixel in 2D

Stride: The stride determines the step size of the kernel as it traverses the image. The default is 1, but a stride of 2 can be used to downsample the image, similar to normal Max Pooling.

Padding: Padding determines how the sample border is adjusted. Padded convolution keeps the same output dimension as the input, whereas unpadded convolution can cut off part of the border if the kernel is greater than 1.

Input & Output Channels: The convolution layer takes a specific number (I) of input channels and calculates it as a specific number (O) of output channels. The number of parameters required in this layer can be calculated as I*O*K. K is the number of kernels.

Dilated Convolutions

Extended convolution means convolution in which another parameter, dilation rate, is applied to the convolutional layer. The dilation rate refers to the interval between kernels.

A 3x3 kernel with a dilation rate of 2 can have the same view as a 5x5 kernel while using 9 parameters.

If you use a 5x5 kernel and delete both the second column and row (compared to using a 3x3 kernel), you can provide a wider view at the same computational cost.

Extended convolution can be used especially in the real-time segmentation field, and can be used when a wide field of view is required and there is no room to use multiple convolutions or large kernels.

Separable Convolutions

In separate convolution, the kernel work can be divided into several stages. Assume that the convolution is equal to Equation 1 below, and that k is calculated by Equation 2.

[Equation 1]

y = conv(x, k)

[Equation 2]

k=k1.dot(k2)

In Equations 1 and 2, x is an input image, y is an output image, and k is a kernel.

Instead of performing K and 2D convolution as shown in Equations 1 and 2, since the same result as 1D convolution with k1 and k2 is obtained, such a convolution is referred to as a separate convolution.

In a neural network, depthwise separable convolution may be used. Separated convolution by depth refers to a method of performing spatial convolution without separating channels and then performing convolution by depth.

For example, assume that there is a 3x3 convolutional layer in 16 input channels and 32 output channels.

For every 16 channels, 32 3x3 kernels pass and 512 (16*32) feature maps are created. Afterwards, 1 feature map is merged and added from all input channels. If you repeat 32 times, you can get 32 output channels.

Under the same assumption, for depthwise separable convolution, 16 channels are searched with one 3x3 kernel to generate 16 feature maps. Before merging, 16 feature maps are traversed with 32 1x1 convolutions. As a result, we get 4068(16*32*3*3) parameters above, while 656(16*3*3 + 16*32*1*1) parameters.

1 is a diagram illustrating a network structure of a residual network for correcting a low-resolution image proposed in the present specification.

Referring to FIG. 1, in a chain residual network for correcting a low-resolution image in real time, global and regional chain residual connections may be additionally configured based on the residual network.

1A shows the structure of a general residual network, and (b) shows the structure of a chain residual network proposed in the present invention.

Specifically, the residual networks of FIGS. 1A and 1B differ in whether global and regional chain residual connections exist.

As shown in (b) of FIG. 1, in the concatenated residual network composed of concatenated residual blocks 100, the output of the middle layer is cascading to the upper layer and converged to a single 1x1 convolution layer. .

That is, the global chain residual connection is a structure in which lower blocks are connected to all upper blocks, respectively, and the convolutional residual block uses the convolutional block inside the network for the local chain residual connection.

The residual block of the general residual network shown in FIG. 1A can be expressed as Equation 3 below.

[Equation 3]

는 i번째 잔차 블록의 파라미터를 의미한다.In Equation 3, f denotes a convolution layer, and τ denotes an activation function.

Denotes a parameter of the i-th residual block.

At this time, as the activity function, ReLU or various activity functions may be used.

H ⁱ means result feature information of the i-th block, and final feature information H ^u of the residual network can be calculated by Equation 4 below.

[Equation 4]

Unlike Equations 3 and 4, the chain residual block of FIG. 2B may be expressed as Equation 5 below.

[Equation 5]

H ⁱ means result feature information of the i-th block, and final feature information H ^u of the residual network can be calculated by Equation 6 below.

[Equation 6]

In Equation 6, H ⁰ denotes the output of the first convolution layer. Referring to Equation 6, the final feature information can be calculated using the result value of the previous block through global chain residual connection.

As shown in (a) and (b) of FIG. 1, the main difference between a general residual network and a concatenated residual network lies in the cascading mechanism as described above.

As shown in (b) of FIG. 1, the chain residual network has a structure of global chain connections indicated by arrows, and each chain residual block has an additional 1 X 1 convolution layer behind it.

If you have a structure in which both the global and regional levels are chained, you have the following advantages.

1) Multi-level expression can be learned by integrating functions of several layers in the model.

2) Multi-level cascading connection works as a multi-level shortcut connection, allowing information to be quickly transferred from lower layers to higher layers.

This advantage is that in case of error back propagation, which is a network learning method, learning information can move through various paths, and further, the network can learn the right to decide on the direction of information movement by itself.

In addition, the chain residual network in FIG. 1 may include an up-sampling block 200 for processing low-resolution images of a plurality of scales (eg, x2, x3, x4) into one model.

The upsampling method can be applied to obtain a result image by increasing the resolution of the feature map that has passed through the convolutional layer using the image enlargement interpolation (bicubic), and then passing the feature map with the increased resolution through the final convolutional layer.

In this case, the x2 and x3 scales increase the resolution at once, but the x4 scale may use a method of applying x2 resolution expansion twice.

A low-resolution image can be reconstructed based on such a chain residual network, and details and context of the image can be simultaneously restored.

2 is a diagram illustrating the structure of a continuous residual recursive block proposed in the present specification.

Referring to FIG. 2, in the concatenated residual network illustrated in FIG. 1, a lightweight residual block 300 and a recursive structure may be applied to secure real time.

Specifically, the cascading block shown in (b) of FIG. 1 is composed of an efficient residual block (for example, a lightweight residual block 300) as shown in (a) of FIG. Can be, and can be configured in a recursive structure as shown in Figure 2 (b).

In this case, the recursive structure refers to a structure in which all convolutions inside the concatenation residual block use the same parameters in order to reduce the parameters of the network.

The lightweight residual block 300 refers to a block in which the existing residual block is lightened in terms of parameters and computational amount by using group convolution.

3 is a diagram illustrating a structure of a residual block proposed in the present specification. 3(a) shows a general residual block, and FIG. 3(b) shows a lightweight residual block 300.

As shown in (a) of FIG. 3, a general residual block consists of a convolutional layer and a regression function.

However, the lightweight residual block 300 of FIG. 3B may be configured with a plurality of group convolutions 400 and a regression function.

That is, the lightweight residual block 300 may be composed of two group convolutions (first convolution block) and one 1×1 convolution (second convolution block) having a specific size (for example, 3×3).

At this time, one 1x1 convolution may be applied to the depth-specific separation convolution described above, and two specific size group convolutions may be applied to a convolution in which the depth-specific convolution is expanded into a group.

The group expansion of the convolution by depth makes the model adjustment efficient. In other words, since the group size and performance are in a trade-off relationship, the user must select an appropriate group size.

For example, the user may input the size of the group, and the network may determine the size of the group by obtaining information indicating the size of the group from the user.

Accordingly, since the user can control the degree of weight reduction by adjusting the group size of the group convolution according to the situation, it is possible to reduce the amount of computation compared to a general residual block.

Specifically, the amount of calculation for the general residual block shown in FIG. 3(a) can be expressed as Equation 7, and the amount of calculation for the lightweight residual block 300 shown in FIG. 3(b) is Equation 8 It can be expressed as

[Equation 7]

[Equation 8]

In Equations 7 and 8, K is the size of the convolution kernel, C is the number of input/output channels, and F is the size of feature information.

Based on Equations 7 and 8, the difference in the amount of calculation between the general residual block and the lightweight residual block is shown in Equation 9 below.

[Equation 9]

In Equation 9, G means the size of a group.

Referring to Equation 9, when the size of all group convolutions is set to 3 X 3 and the number of channels is set to 64, and the lightweight residual block 300 is used instead of the general residual block, the amount of computation is calculated from 1.8 depending on the size of the group. It can be reduced to 14 times.

In addition, it is possible to reduce the number of parameters through the shared concatenated blocks as described in (b) of FIG. 2.

In addition, even in such a method, since the up-sampling block 200 shown in FIG. 1 needs to secondarily increase the number of channels in relation to the up-sampling ratio, the 3 X 3 convolution layer is 1 X inside the up-sampling block 200. 1 can be replaced with a convolutional layer.

Table 1 below shows the effects of each model.

[Table 1]

In Table 1, Baseline indicates a network to which global chain residual and regional chain residual linking are not applied, and CARN-NL indicates a network to which regional chain residual linking is not applied.

CARN-NG represents a network to which global chain residual connection is not applied, and CARN represents a network to which global and regional chain residual connection proposed in the present invention is applied.

4 is a diagram illustrating PSNR performance of a continuous residual network proposed in the present specification.

FIG. 4A shows the effect of using the lightweight residual block 300 and the recursive block in terms of PSNR and parameters, and FIG. 4B shows the lightweight residual block 300 and the lightweight residual block 300 in terms of PSNR and operation. This shows the effect of using a recursive block.

In FIGS. 4A and 4B, GConv represents the size of a group of group convolution.

In (a) and (b) of FIG. 4, the blue line represents the network model to which the recursive structure is not applied, and the orange line represents the network model to which the recursive structure and concatenated residual blocks are applied.

Even though all efficient models perform better than CARN models, which exhibit 28.70 PSNR, the number of parameters and operations is significantly reduced.

For example, the G64 shows a five-fold reduction in both parameters and operation.

So far, the present invention has been looked at around its preferred embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Furthermore, although each drawing has been described separately for convenience of description, it is also possible to design a new embodiment by merging the embodiments described in each drawing.

In addition, the present invention described above can be implemented as a computer-readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices storing data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Includes. In addition, the computer may include the control unit 130 of the device. Therefore, the detailed description above should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: chain residual block (100) 200: up-sampling block (200)
300: lightweight residual block (300) 400: group convolution

Claims (9)

In a residual network system for image correction,
Efficient Residual Block;
A cascading residual block composed of a plurality of lightweight residual blocks; And
It includes a cascading residual network composed of a plurality of concatenated residual recursive blocks,
Each of the plurality of concatenated residual recursive blocks is connected to a plurality of upper convolutional blocks of the concatenated residual network,
Each of the plurality of lightweight residual blocks calculates specific information using a result value calculated by at least one previous convolutional layer.
The method of claim 1,
The residual network system, wherein the plurality of lightweight residual blocks are configured with a local chain connection to use a result value calculated by the at least one convolutional layer.
The method of claim 1,
Each of the blocks constituting the concatenated residual network has a recursive structure.
The method of claim 1,
the u-th lightweight residual block

And the u-th chain residual recursive block is

In the case of, the u-th specific information H ^u is a residual network system calculated through the following equation.

The method of claim 1,
Each of the plurality of lightweight residual blocks includes two first convolution blocks and one second convolution block.
The method of claim 5,
The first convolution block is a group convolution of a specific size, and the second convolution is a pointwise convolution.
The method of claim 6,
A residual network system in which the specific size is variably determined by input information obtained from a user.
The method of claim 1,
The concatenated residual network further includes an up-sampling block for processing low-resolution images of three different scales into one model.
A computer-readable recording medium recording a computer program in which the residual network system for image correction of claim 1 is implemented.

Images