KR20210128605A - Method and Apparatus for Image Transform - Google Patents

Abstract

Provided are a method and device for image transformation for generating a high-resolution image from a low-resolution image. This embodiment of the present invention uses a pre-trained inference model based on a pair of low-resolution and high-resolution images for training with identical features such as resolution, aspect ratio, and alignment. The device for image transformation comprises: an input unit for acquiring an image; and an image conversion unit that creates an image.

Description

Image transformation apparatus and method {Method and Apparatus for Image Transform}

The present invention relates to an image conversion apparatus and method for generating a high-resolution image from a low-resolution image. More specifically, it relates to an image conversion apparatus and method in which a deep neural network-based inference model trained in advance using a low-resolution image for training and a high-resolution image pair generates a high-resolution image from a low-resolution image.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

In recent years, a super-resolution (SR) technology based on a deep learning-based deep neural network has been actively developed, and the performance of the SR is greatly improved compared to other previous technologies. However, in the existing deep neural network-based SR research, the data used for neural network training is mostly undistorted images, which are different in quality from the actual broadcasting images. Therefore, there is a limitation in that it is difficult to obtain a high-quality image of a target level when the result of SR implementation according to the existing research is applied to broadcasting contents.

As a representative technology of super-resolution deep neural networks, there is an Enhanced Deep Super Resolution network (EDSR) (see Non-Patent Document 1). In order to implement the super-resolution technology, the EDSR includes 32 residual blocks (RBs), a convolution layer, and an up-sampler.

Compared to ResNet's RB (Residual Block), EDSR's RB removes the BN (Batch Normalization) layer and has one ReLU (Rectified Linear Unit) layer. When EDSR using this structural change was announced, EDSR achieved State of the Art (SOTA) performance in the field of super-resolution implementation. However, compared to the general use of 64 feature map channels when implementing SR, EDSR uses 256 channels, so it ^{contains 43 × 10 6} vast parameters, so the calculation speed is slow. have.

As another super-resolution deep neural network, there is a Residual Channel Attention Network (RCAN) model in which the depth of a neural network is significantly increased for super-resolution implementation (see Non-Patent Document 3). Since it was difficult to train a deep network with the previously studied neural network structure to implement super-resolution, the RCAN model included a structure for training the deep network.

The RCAN model includes: Residual In Residual (RIR) using a long skip connection; Residual Group (RG) contained within the RIR and using a short skip connection; and a Residual Channel Attention Block (RCAB) inside the RG, which uses a CA (Channel Attention) for re-adjusting the characteristics of a channel unit based on inter-channel interdependence. In the RCAN model, approximately 800 convolutional layers were used to implement the structure characterized by RIR, RG, RCAB and CA. Using the experimental results of comparison with the existing method, the RCAN model showed more improved SR implementation results. However, compared to EDSR, ^{even though the number of parameters is reduced to 16×10 6} , the RCAN model also has a disadvantage in that the computation speed is slow because about 800 convolutional layers are used.

Accordingly, while implementing SR capable of obtaining a high-quality image of a target level from a low-resolution image for broadcasting including a lot of distortion, an image conversion apparatus having a simple structure and improved operation speed is required.

Non-Patent Document 1: B. Lim, S. Son, H. Kim, S. Nah, K. Lee. Enhanced Deep Residual Networks for Single Image Super Resolution, In CVPR workshop, 2017. Non-Patent Document 2: K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR 2016. Non-Patent Document 3: Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, Y. Fu. Image Super-Resolution Using Very Deep Residual Channel Attention Networks, In ECCV, 2018.

The present disclosure provides an inference model based on a deep neural network trained in advance based on a low-resolution image for learning and a high-resolution image pair in which characteristics such as resolution, aspect ratio, and alignment are identically matched. A main object of the present invention is to provide an image conversion apparatus and method for generating a high resolution image from a low resolution image by using the .

According to an embodiment of the present invention, the input unit for obtaining a low resolution (low resolution) image from the video content; and an image conversion unit generating a high resolution image for the video content from the low resolution image using an inference model based on a deep neural network, wherein the inference model includes a first By including a convolution layer, a plurality of Long Residual Blocks (LRB), a second convolution layer, a first skip connection, and a third convolution layer, the low-resolution image is converted into the first convolutional layer. A convolutional layer, the plurality of LRBs, the second convolutional layer, and the third convolutional layer are sequentially processed to generate the high-resolution image.

According to another embodiment of the present invention, there is provided an image conversion method used by an image conversion apparatus, the method comprising: obtaining a low resolution image from video content; And implemented based on a deep neural network, a first convolution layer, a plurality of Long Residual Blocks (LRB), a second convolution layer, a first skip connection, and a third It includes a convolutional layer, and is trained in advance based on a pair of a low-resolution image for training and a high-resolution image for training based on the same video content, and uses the low-resolution image as the first convolutional layer, the plurality of LRBs, and the A process of generating a high-resolution image of the video content from the low-resolution image by using an inference model that processes the second convolutional layer and the third convolutional layer in order It provides an image conversion method, characterized in that.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable recording medium to execute each step included in the image conversion method.

As described above, according to this embodiment, using an inference model based on a deep neural network trained in advance based on a low-resolution image for learning and a high-resolution image pair, high resolution from a low-resolution image ) By providing an image conversion apparatus and method for generating an image, there is an effect that it becomes possible to implement SR that obtains a high-quality, high-resolution image from a low-resolution image for broadcasting containing a lot of distortion.

In addition, according to the present embodiment, a deep neural network-based inference model trained in advance based on a pair of low-resolution images and high-resolution images for training in which characteristics such as resolution, aspect ratio, and alignment are identically matched. By providing an image conversion apparatus and method used, there is an effect that it is possible to reduce the complexity of the inference model and improve the training efficiency.

1 is a block diagram of an image conversion apparatus according to an embodiment of the present invention.
2 is a block diagram of an inference model according to an embodiment of the present invention.
3 is a block diagram of an LRB included in an inference model according to an embodiment of the present invention.
4 is a block diagram of an SRB included in an inference model according to an embodiment of the present invention.
5 is a flowchart of an image conversion method according to an embodiment of the present invention.
6 is a block diagram of a learning model of an image conversion apparatus according to an embodiment of the present invention.
7 is a flowchart of an image pair generation process according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

Also, in describing the components of the present embodiments, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced.

This embodiment discloses an image conversion apparatus and method for generating a high-resolution image from a low-resolution image. In more detail, using an inference model based on a deep neural network trained in advance based on a pair of low-resolution images and high-resolution images for training in which properties such as resolution, aspect ratio, and alignment are identically matched, An image conversion apparatus and method for generating a high-resolution image from a low-resolution image are provided.

1 is a block diagram of an image conversion apparatus according to an embodiment of the present invention.

In an embodiment of the present invention, the image conversion apparatus 100 generates a high resolution image from a low resolution image by using a pre-trained inference model. The image conversion apparatus 100 includes all or a part of the input unit 102 , the image conversion unit 104 , and the output unit 106 . In addition, the image conversion unit 104 includes an inference model (inference model, 110). Here, the components included in the image conversion apparatus 100 according to the present embodiment are not necessarily limited thereto. For example, the image conversion apparatus 100 may additionally include a training unit (not shown) for training an inference model, or may be implemented in a form that interworks with an external training unit.

Here, image transform refers to a method of generating a high-quality, clear image by processing a degraded image with an appropriate filter. Accordingly, the image conversion apparatus 100 according to the present embodiment performs image conversion for inferring a high-quality image from a low-quality image by using the inference model as a filter.

1 is an exemplary configuration according to the present embodiment, and various implementations including other components or other connections between components are possible depending on the types of inputs and outputs, the type of the inference model, and the training method of the inference model do.

The input unit 102 according to the present embodiment acquires a low-resolution image. Here, the low-resolution image may be an SD (Standard Definition) image obtained from video content for broadcasting stored in a storage device (not shown), but is not limited thereto, and any form requiring resolution and quality improvement It can be a video of A low-resolution image obtained from video content for broadcasting may include distortion.

The image conversion unit 104 according to the present embodiment generates a high-resolution image of the same video content from a low-resolution image by using the inference model 110 based on a deep neural network.

In addition, the high-resolution image may be a full high-definition (FHD) image, but is not limited thereto, and may be any type of image with improved resolution and quality than the input image.

The output unit 106 according to the present embodiment outputs a high-resolution image. The output unit 106 may provide the high-resolution image to the user in visual and/or audible form, or store it in a storage device.

Hereinafter, the structure and operation of the inference model 110 will be described using FIGS. 2 to 4 .

2 is a block diagram of an inference model according to an embodiment of the present invention.

The inference model 110 according to the present embodiment automatically generates a high-resolution image from a low-resolution image. The inference model 110 includes all or part of a plurality of Long Residual Blocks (LRBs) 202, a plurality of convolution layers (206, 208, 210), and a first skip connection (204). do.

The inference model 110 according to the present embodiment is implemented as a deep neural network based on a plurality of convolutional layers. The inference model 110 may be trained in advance using a deep learning-based learning model. The structure of the learning model and the training process of the learning model will be described later.

As shown in FIG. 2 , the inference model 110 calculates a low-resolution image in the order of the first convolutional layer 206 , the plurality of LRBs, the second convolutional layer 208 , and the third convolutional layer 210 . It can be processed to produce high-resolution images.

The first convolutional layer 206 located on the input side of the inference model 110 extracts a plurality of feature maps from the low-resolution image. An appropriate number of characteristic map channels extracted by the first convolutional layer 206 may be set by compromising the operation speed of the inference model 110 and the quality of the high-resolution image. 2 to 4, c (c is a natural number) indicates the number of characteristic map channels, for example, c may be 64.

As shown in FIG. 2 , the inference model 110 according to the present embodiment includes eight LRBs, but is not limited thereto, and the LRBs by compromising the operation speed of the inference model 110 and the quality of high-resolution images. An appropriate number of may be set. Each of the LRBs included in the inference model 110 is implemented with the same structure and performs the same function. Accordingly, the structure and operation of one LRB 202 will be described below.

3 is a block diagram of an LRB included in an inference model according to an embodiment of the present invention.

The LRB 202 according to the present embodiment includes all or a part of two Short Residual Blocks (SRBs) 302 , a second skip connection 304 , and a fourth convolutional layer 306 . The LRB 202 processes the input in order of two SRBs and a fourth convolutional layer 306 .

Each of the SRBs included in the LRB 202 is implemented with the same structure and performs the same function. Accordingly, the structure and operation of one SRB 302 will be described below.

4 is a block diagram of an SRB included in an inference model according to an embodiment of the present invention.

The SRB 302 according to this embodiment includes all of the third skip connection 402 , the first convolution group 404 , the concatenation layer 406 , and the second convolution group 408 . or includes some. Each of the first convolutional group 404 and the second convolutional group 408 includes two convolutional layers and one Rectified Linear Unit (ReLU). Here, ReLU is an activation function that limits the range of the output.

The SRB 302 processes the input in the order of the first convolutional group 404 , the concatenated layer 406 , and the second convolutional group 408 .

The first convolutional group 404 generates a residual output for the SRB 302 input. The concatenated layer 406 concatenates the input of the SRB 302 and the residual output of the first convolutional group 404 and transmits it to the second convolutional group 408 . The second convolution group 408 generates a residual output from the concatenated layer 406 result. The third skip connection 402 adds the input of the SRB 302 and the residual output of the second convolution group 408 .

The fourth convolutional layer 306 included in the LRB 202 generates a residual output from the output of the second SRB (the result of adding the input of the SRB and the residual output of the second convolution group). Also, the second skip connection 304 adds the input of the LRB 202 and the residual output of the fourth convolutional layer 306 .

The second convolutional layer 208 included in the inference model 110 generates a residual output from an output inside the last LRB (a result of adding the input of the LRB and the residual output of the fourth convolutional layer). The first skip connection 204 may transfer the characteristic of the low-resolution image to the output side of the inference model 110 by adding the output of the first convolutional layer 206 and the residual output of the second convolutional layer 208 .

The inference model 110 uses the third convolutional layer 210 to obtain the output of the first convolutional layer 206 (characteristics of the low-resolution image) and the residual output of the second convolutional layer 208 from the addition result. High-resolution images can be created.

As shown in FIGS. 2 to 4 , the inference model 110 is extended to the SRB 302 , the LRB 202 , and the inference model 110 , and the number of layers increases to deepen the depth of the neural network. Nevertheless, skip connections 204 , 304 , and 402 are included for each component of the inference model 110 , so that training for the learning model described later can proceed effectively.

Residual Block (RB) of ResNet (refer to Non-Patent Document 2) includes two convolutional layers, two BN (Batch Normalization) layers, and two ReLU layers. In addition, as compared to ResNet, the RB of EDSR (see Non-Patent Document 1) includes only one ReLU layer with all BN layers removed. In contrast, the SRB 302 according to the present embodiment includes four convolutional layers and a concatenated layer.

The BN layer used in ResNet is effective in classifying image classes, but is not used in the SRB 302 according to the present embodiment because it is known that the effect is small in the implementation of super-resolution converting images in pixel units. In addition, the SRB 302 uses the concatenated layer 406 in order to focus on the transformation of the high-frequency region of the image compared to the existing methods.

5 is a flowchart of an image conversion method according to an embodiment of the present invention.

The image conversion apparatus 100 according to the present embodiment acquires a low resolution image (S500). Here, the low-resolution image may be an SD image obtained from video content for broadcasting, but is not limited thereto, and may be any type of image requiring resolution and quality improvement.

The image conversion apparatus 100 generates a high-resolution image from a low-resolution image by using a pre-trained inference model (S502).

The inference model 110 is implemented as a deep neural network, and includes all or part of a plurality of LRBs, a plurality of convolutional layers, and skip connections. The inference model 110 may be trained in advance based on a pair of a low-resolution image for training and a high-resolution image for training.

The high-resolution image generated by the image conversion apparatus 100 may be an FHD image, but is not limited thereto, and may be any type of image with improved resolution and quality than the input image.

The image conversion apparatus 100 outputs a high-resolution image (S504). The high-resolution image may be provided to a user in a visual and/or audible form or stored in a storage device.

As described above, the image conversion apparatus 100 according to the present embodiment may have a deep learning-based learning model, and may perform a training process for the inference model using the provided learning model. This learning model is based on images with different resolutions for the same object, and converts low-resolution images into high-resolution images. It may be a pre-trained model to be able to generate.

6 is a block diagram of a learning model of an image conversion apparatus according to an embodiment of the present invention.

The learning model according to the present embodiment generates an image pair for training from an SD image and an FHD image obtained from the same video content, and trains the inference model 110 using the generated image pair. The learning model includes all or a part of the input unit 602 , the image pair generator 604 , and the inference model 110 . As described above, the learning model may further include a training unit (not shown) for training the inference model 110 , or may be implemented in a form that is linked with an external training unit.

The input unit 602 acquires an SD image, which is a low-resolution image, and an FHD image, which is a high-resolution image. Here, the SD image and the FHD image are obtained from the same broadcast video content. The resolution of the SD video is 720×480, and 4:3 (horizontal: vertical, hereinafter the same) ratio is used, and the FHD video resolution is 1920×1080 and uses a 16:9 ratio, so the two videos have resolution and aspect ratio. (aspect ratio), and do not have the same characteristics in terms of alignment. Also, a low-resolution image obtained from video content for broadcasting may include distortion.

The image pair generator 604 performs pre-processing for generating a learning image pair having the same characteristics, that is, an SD image for learning and an FHD image for learning, by correcting the resolution, aspect ratio, and alignment between the two input images. . Here, the SD image for learning may be used as an input of the inference model 110 , and the FHD image for learning may be used as a target image for training the inference model 110 .

When the two images do not have the same characteristics, the difference between the two images may show a strong edge characteristic. As the characteristics between the two images match, the strong edge characteristics can be relaxed. In deep learning-based learning, since training for a neural network is performed using a target image, it is very important to have the same characteristics between training images in terms of reducing the size of the inference model 110 and improving training efficiency.

7 is a flowchart of an image pair generation process according to an embodiment of the present invention.

The image pair generator 604 matches the start frames of the SD video image and the FHD video image (S700). When the number of frames of the SD video image and the FHD video image for the same video content do not match, the image pair generator 604 may match the start frames of the two video images. On the other hand, when the number of frames between the two images is the same, the image pair generator 604 may perform subsequent steps from the first frame.

The image pair generator 604 creates a list of frames to be used for learning after matching frame numbers between the SD video image and the FHD video image (S702). In this case, all frames of the two video images are not used, and frames may be selected at intervals of N-steps (N is a natural number). Also, when a compressed image is used, an I-frame (intra frame) pair having a relatively small degradation in image quality is preferentially selected to generate a list.

The image pair generator 604 matches the aspect ratio between the SD image and the FHD image (S704). With respect to the image pair included in the list, the image pair generator 604 may match the aspect ratio of the SD image and the FHD image. For example, when the SD image has a 4:3 ratio, the image pair generator 604 cuts the horizontal resolution of the 16:9 FHD image to fit the SD image. Alternatively, if the SD image includes a black region because the SD image has a 16:9 ratio, the image pair generator 604 cuts the image excluding the black region to fit the FHD image content.

The image pair generator 604 matches the resolution between the SD image and the FHD image (S706). Since the resolution of the FHD image cannot be generated by multiplying the resolution of the SD image by an integer multiple, the image pair generator 604 may adjust the vertical and horizontal resolutions of the SD image differently to match the resolutions between the two images.

The image pair generator 604 generates an image pair aligned at the same position using an image alignment algorithm, that is, an SD image for learning and an FHD image for learning (S708). In order to align two images with the same aspect ratio and resolution, an image alignment algorithm based on luminance information, which is the brightness of an image, or edge line information within the image, may be used.

Since the structure and operation of the deep neural network-based inference model 110 have already been described in detail, further detailed description will be omitted.

The training unit according to this embodiment uses the SD image for learning among the image pairs generated by the image pair generator 604 as an input of the inference model 110, and uses the FHD image for learning as a target for training of the inference model 110 ( target) is used as an image.

Based on the distance metric between the inference image and the target image generated by the inference model 110 using the SD image for learning, the training unit updates the parameters of the inference model 110, conduct training. Here, as the distance metric, any one capable of expressing a metric difference between two comparison objects, such as cross entropy, L1 or L2 metric, may be used.

Hereinafter, an experimental example for presenting the performance of the image conversion apparatus 100 according to the present embodiment will be described.

As a learning image pair for the experiment, drama contents for broadcasting were used, and the resolutions for them are as shown in Table 1.

As a comparison target, the RCAN model showing the best performance among the prior art was used. The image conversion apparatus 100 according to the present embodiment includes an inference model 110 trained using the learning model shown in FIG. 6 . The inference model 110 consists of 75 convolutional layers and includes about ^{3×10 6 parameters.} Compared with the RCAN model including 800 or more convolutional layers and about 16×10 ⁶ parameters, the inference model 110 according to the present embodiment has reduced complexity.

For performance comparison, the average required time per page for converting an SD image obtained from the contents shown in Table 1 into an FHD image, and a Peak Signal to Noise Ratio (PSNR) of the converted FHD image were measured. First, in the case of the average required time, the image conversion apparatus 100 according to this embodiment used 0.51 seconds per sheet, and showed an excellent increase in operation speed compared to 1.79 seconds per sheet of the RCAN model. Also, with respect to the converted FHD image, the PSNR shown in this example was 33.31 dB, which showed little difference compared to 33.35 dB of the RCAN model, confirming the similarity of image quality.

In contrast to the RCAN model, the inference model 110 included in the image conversion apparatus 100 includes simple components based on LRB, SRB, and the like, and a plurality of skip connections for effective training. Due to such a simple and effective structure, the image conversion apparatus 100 according to the present embodiment can suppress deterioration in quality of the converted image while increasing the operation speed.

As described above, according to the present embodiment, a high-resolution image is generated from a low-resolution image by using an inference model trained in advance based on a low-resolution image for learning and a high-resolution image pair. By providing an image conversion apparatus and method that includes a large amount of distortion, there is an effect that it becomes possible to implement SR that obtains a high-quality image of a target level from a low-resolution image for broadcasting that contains a lot of distortion.

Also, according to the present embodiment, an image conversion apparatus using an inference model trained in advance based on a pair of low-resolution images and high-resolution images for training in which characteristics such as resolution, aspect ratio, and alignment are identically matched. And by providing the method, there is an effect that it is possible to reduce the complexity of the inference model and improve the training efficiency.

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

Various implementations of the systems and techniques described herein include digital electronic circuitry, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable recording medium".

The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. These computer-readable recording media are non-volatile or non-transitory, such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device. media, and may further include transitory media such as carrier waves (eg, transmission over the Internet) and data transmission media. In addition, computer-readable recording media are distributed in networked computer systems, and computer-readable codes may be stored and executed in a distributed manner.

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

100: video conversion device 102: input unit
104: image conversion unit 106: output unit
110: inference model
202: LRB 204: first skip connection
302: SRB 304: second skip connection
402: third skip connection

Claims (10)

an input unit for obtaining a low resolution image from video content; and
An image conversion unit that generates a high-resolution image of the video content from the low-resolution image using an inference model based on a deep neural network
including,
The inference model includes a first convolution layer, a plurality of Long Residual Blocks (LRB), a second convolutional layer, a first skip connection, and a third convolutional layer, the low resolution The image conversion apparatus according to claim 1, wherein the high-resolution image is generated by processing the image in order of the first convolutional layer, the plurality of LRBs, the second convolutional layer, and the third convolutional layer. According to claim 1,
The inference model is
An image conversion apparatus, characterized in that it is trained in advance using a low-resolution image for learning and a high-resolution image pair for learning based on the same video content. According to claim 1,
The first convolutional layer,
Image conversion apparatus, characterized in that for extracting a plurality of feature maps (feature map) from the low-resolution image. According to claim 1,
The second convolutional layer generates a residual output with respect to an output of a last LRB among the plurality of LRBs, and the first skip connection is performed by adding the output of the first convolutional layer and the residual output. and the third convolutional layer generates the high-resolution image from the addition result. According to claim 1,
Each of the plurality of LRBs,
Including two Short Residual Blocks (SRBs), a second skip connection, and a fourth convolutional layer, the input of the LRB is processed in the order of the two SRBs and the fourth convolutional layer, characterized in that video converter. 6. The method of claim 5,
The fourth convolutional layer generates a residual output with respect to an output of a second SRB among the two SRBs, and the second skip connection adds the input of the LRB and the residual output. 6. The method of claim 5,
Each of the two SRBs,
Including a first convolutional group including at least one convolutional layer, a concatenation layer, a second convolutional group including at least one convolutional layer, and a third skip connection, the input of the SRB is processed in the order of the first convolutional group, the concatenated layer, and the second convolutional group. 8. The method of claim 7,
The first convolution group generates a residual output with respect to the input of the SRB, and the concatenation layer concatenates the input of the SRB and the residual output of the first convolution group and transmits it to the second convolution group; The second convolution group generates a residual output for the output of the concatenated layer, and the third skip connection adds the input of the SRB and the residual output of the second convolution group. . In the image conversion method used by the image conversion apparatus,
obtaining a low resolution image from video content; and
Implemented based on a deep neural network, a first convolution layer, a plurality of Long Residual Blocks (LRB), a second convolution layer, a first skip connection, and a third convolution It includes a convolution layer, is trained in advance based on a low-resolution image for training and a high-resolution image pair for training based on the same video content, and uses the low-resolution image to convert the low-resolution image to the first convolutional layer, the plurality of LRBs, and the first A process of generating a high-resolution image of the video content from the low-resolution image by using an inference model that processes two convolutional layers and the third convolutional layer in order
Image conversion method comprising a. A computer program stored in a computer-readable recording medium to execute each step included in the image conversion method according to claim 9.

Images