KR102423047B1 - Pre-Processing Apparatus And Method For Super-Resolution Device Implemented In Hardware - Google Patents

Abstract

The present invention is a pre-processing device for a super-resolution device that has a plurality of filters of a predetermined size and is implemented as an artificial neural network including N convolutional layers sequentially connected to perform image upscaling, and is input to the super-resolution device. An integrated padding size calculator that calculates the size of the integrated padding area representing the total size of the image being reduced due to image shrink of the convolutional layer, half the size of the integrated padding area in all directions along the outskirts of the input image to be upscaled of a super-resolution device implemented in hardware, including an integrated padding unit that performs integrated padding by expanding with It is possible to provide a pre-processing apparatus and method for a super-resolution device that simplifies a data flow to enable easy design and high-speed operation, and to sufficiently reflect edge region information in an input image.

Description

Pre-Processing Apparatus And Method For Super-Resolution Device Implemented In Hardware

The present invention relates to a pre-processing apparatus and method for a super-resolution apparatus, and to a pre-processing apparatus and method for a super-resolution apparatus implemented in hardware.

With the development of artificial neural networks, artificial neural networks are being actively used in various fields. Currently, artificial neural networks are being applied in a wide variety of fields such as image processing, object classification, object detection, speech recognition, and natural language processing, and the field of application continues to expand.

Meanwhile, due to the recent development of image technology, the demand for a high-resolution image is increasing, and thus, an upscaling technology for converting a low-resolution image into an ultra-high-resolution image is also attracting attention. Various algorithms have been proposed as image upscaling techniques, but it is known that the super-resolution technique, which is a method of calculating using an artificial neural network, shows better performance than an algorithm based on simple arithmetic operation.

1 shows an example of a super-resolution device composed of an artificial neural network, and FIG. 2 is a diagram for explaining an image size change by a convolution operation of a filter.

1 shows a Super-Resolution Convolutional Neural Network (FSRCNN) which is mainly used in the super-resolution technique.

In the super-resolution technique, an artificial neural network is constructed based on a Convolution Neural Network (CNN) that performs a convolution operation. However, CNN has a problem that not only requires a lot of memory and computation to improve image quality, but also the upscaling speed is slow. Accordingly, at present, a Super-Resolution Convolutional Neural Network (SRCNN) having a simple structure composed of a small number of convolutional layers or an FSRCNN as shown in FIG. 1 has been proposed and mainly used. S RCNN or FSRCNN has an advantage that it can be quickly executed even at the level of a user terminal such as a TV or a smartphone because the structure is not complicated.

Referring to FIG. 1 , the FSRCNN sequentially extracts a feature map from an input image including a plurality of convolutional layers (Conv.Layer 1 to Conv.Layer 5) each having a plurality of filters of a predetermined size and sequentially connected. and remapping the feature map output from the final convolution layer (Conv.Layer 5) to obtain an output image upscaled to have twice the resolution compared to the input image.

Here, a plurality of filters of each convolutional layer (Conv.Layer 1 to Conv.Layer 5) perform a convolution operation with an input image or a feature map output from a previous layer.

At this time, if the size of the filter is larger than 1 × 1, as shown in FIG. 2 , image shrinking occurs in which the size of the output image is reduced compared to the input image according to the characteristics of the convolution operation. do. Since most artificial neural networks include multiple convolutional layers of a sequentially connected structure, the input image is continuously reduced in size as it passes through each convolutional layer. In FIG. 1, the first, third, and fifth convolutional layers (Conv.Layer 1, Conv.Layer 3, Conv.Layer 5) of the five convolutional layers (Conv.Layer 1 to Conv.Layer 5) are 5, respectively. Since the filter has a size of × 5, 3 × 3, and 3 × 3, image shrink is generated.

In the case of an artificial neural network that performs object classification or object detection, since it is configured to extract features of an object from an image, it is not a problem even if image shrink occurs. On the other hand, since the super-resolution technique is configured to output the output image by upscaling the resolution of the input input image by two, it should be possible to prevent image shrink in which the size of the image is reduced.

In addition, when a convolution operation is performed between an image and a filter having a size greater than 1 × 1, relatively few operations are performed on the edge region of the image compared to other regions due to the convolution operation characteristic. Therefore, there is a problem that the artificial neural network does not sufficiently reflect the edge information of the image.

As described above, to prevent image shrink caused by a convolution operation with a filter in a plurality of convolutional layers of an artificial neural network, and to prevent a computational difference between the edge region and the remaining regions, the image is reduced as much as the edge A padding technique was introduced to extend the size of the image by adding pixels filled with specified data to the outside.

In the padding technique, in general, a zero-padding technique of adding a pixel having a data value of 0 outside the edge of an image is used.

However, the zero padding technique can prevent image shrink, which reduces the image size as each added pixel has a data value of 0 that does not include image information, but still does not sufficiently reflect the information of the edge area. there is a problem.

Also, the padding technique may be divided into a pre-padding technique and a post-padding padding technique, as shown in FIG. 3 , according to a time point at which padding is applied.

3 is a diagram for explaining a padding technique for a conventional artificial neural network.

In FIG. 3, (a) shows a pre-processing padding technique, and (b) shows a post-processing padding technique.

Referring to (a), in the preprocessing padding technique, the size of the input image (or feature map) applied to each convolutional layer is pre-padded in consideration of the size to be reduced due to image shrink, and then the padded image and the filter are applied. By performing the convolution operation, the size of the output image (or feature map) calculated and output from each convolution layer is maintained at the same size as the input image (or feature map).

On the other hand, as shown in (b), in the preprocessing padding technique, the output image (or feature map) is obtained by performing convolution operation on the input image (or feature map) to which each convolution layer is applied as it is with the filter, and the obtained By padding the output image (or feature map) by the reduced size due to image shrink, the size of the final image is maintained at the same size as the input image (or feature map).

However, in the case of the post-processing padding technique shown in (b), as the number of convolutional layers on which padding is performed increases, the long-dependency phenomenon in which the image is blurred is intensified. Pre-processing padding that performs

When the artificial neural network is implemented in software, each layer performs its own operation after all operations of the previous layer are performed. On the other hand, when the artificial neural network is implemented in hardware, the input image (or feature map) to which each convolution layer is applied due to problems such as the number of operators or memory capacity. Instead of performing a convolution operation on all pixels of , it is performed locally in an acceptable size unit. Therefore, when each convolution layer performs a convolution operation on an input image (or feature map) using a filter, it should always be considered by classifying whether the region to be calculated currently corresponds to the region to be preprocessed according to the number of cases. do. This causes a problem of greatly increasing the hardware implementation complexity.

Korean Patent Publication No. 10-2019-0062305 (published on June 5, 2019)

An object of the present invention is to provide a pre-processing apparatus and method for a super-resolution device that is easy to design and can perform high-speed operation by simplifying the data flow of the super-resolution device implemented in hardware.

Another object of the present invention is to provide a pre-processing apparatus and method for a super-resolution apparatus capable of sufficiently reflecting edge region information in an input image.

A preprocessing device for a super-resolution device according to an embodiment of the present invention for achieving the above object is implemented as an artificial neural network including N convolutional layers sequentially connected and provided with a plurality of filters of a predetermined size, respectively. As a preprocessor for a super-resolution device that performs upscaling, an integrated padding size calculation unit that calculates the size of an integrated padding area indicating the overall size of an image input to the super-resolution device is reduced due to image shrink of a convolution layer ; an integrated padding unit for performing integrated padding by expanding to half the size of the integrated padding area in all directions along the periphery of the input image to be upscaled; and a padding area filling unit that fills a pixel value of each pixel of a padding area that is an area extended by the integrated padding in a predetermined manner.

The padding area filling unit may first fill a corresponding area of the padding area with pixel values of a predetermined number of rows and columns of each edge of the input image, and fill a non-corresponding area with a pixel value of 0.

The padding area filling unit may copy pixel values of pixels located in one row and one column located at each edge of the input image, and repeatedly fill the corresponding padding area with pixel values of the copied rows and columns.

The padding area filling unit copies rows and columns of sizes corresponding to the sizes of the padding areas at each edge of the input image, and in the padding areas corresponding to the pixel values of rows and columns of sizes corresponding to the sizes in the copied padding area. can be filled

The padding area filling unit copies rows and columns of sizes corresponding to the sizes of the padding areas at each edge of the input image, and in the pixel areas corresponding to the pixel values of rows and columns of sizes corresponding to the sizes in the copied padding area. You can fill it by inverting it so that it becomes a symmetrical pattern.

The integrated padding size calculation unit includes the number of convolutional layers (α) having filters having a size greater than 1 × 1, the length of filters included in each convolutional layer (K _i ), the input image length (L _in ), and the output image Equation based on length (L _out )

Accordingly, the size of the integrated padding area can be calculated.

The super-resolution device may be implemented as hardware including a plurality of operators and memories.

A pre-processing method for a super-resolution device according to another embodiment of the present invention for achieving the above object is implemented as an artificial neural network including N convolutional layers sequentially connected and provided with a plurality of filters of a predetermined size, respectively. A pre-processing method for a super-resolution device performing upscaling, comprising: calculating a size of an integrated padding area indicating an overall size of an image input to the super-resolution device reduced due to image shrink of a convolutional layer; performing integrated padding by expanding to half the size of the integrated padding area in all directions along the periphery of the input image that is the upscaling target; and filling a pixel value of each pixel of a padding area that is an area extended by the integrated padding in a predetermined manner.

Therefore, the pre-processing apparatus and method for a super-resolution device according to an embodiment of the present invention calculates in advance the size to be padded in the entire super-resolution device implemented as an artificial neural network, and performs integrated pre-processing padding in advance as much as the calculated size. Then, it is applied as an input to the super-resolution device so that individual padding is not considered for each convolutional layer of the super-resolution device, thereby simplifying a data storage space allocation method or data flow. Therefore, it is possible to simplify the hardware structure, thereby reducing the cost and improving the operation speed. In addition, by filling the integrated preprocessing padded area with the value of a pixel located in an edge area of a predetermined size in the input image, edge area characteristic information can be sufficiently reflected at a level similar to that of the rest area.

1 shows an example of a super-resolution device configured with an artificial neural network.
FIG. 2 is a diagram for explaining an image size change due to a convolution operation of a filter.
3 is a diagram for explaining a padding technique for a conventional artificial neural network.
4 shows a schematic structure of a pre-processing apparatus for a super-resolution apparatus according to an embodiment of the present invention.
FIG. 5 is a diagram for explaining a concept in which an input image is integrated padding by the integrated padding unit of FIG. 4 .
FIG. 6 is a diagram for explaining a method in which the padding area filling unit of FIG. 4 fills the padding area with pixel values.
FIG. 7 is a diagram for explaining an operation of a conventional pre-processing padding technique and an integrated pre-processing padding technique of the present embodiment by comparison.
8 is a graph comparing performance changes of a super-resolution device according to a padding method.
9 shows a pre-processing method for a super-resolution device according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in several different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it does not exclude other components unless otherwise stated, meaning that other components may be further included. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

4 shows a schematic structure of a preprocessing device for a super-resolution device according to an embodiment of the present invention, and FIG. 5 is a diagram for explaining a concept in which an input image is integrated padding by the integrated padding unit of FIG. 4, FIG. 6 is a diagram for explaining a method in which the padding area filling unit of FIG. 4 fills the padding area with pixel values.

Referring to FIG. 4 , the pre-processing apparatus 100 of this embodiment receives the input image IN, and pads the input image IN in consideration of the size reduced due to image shrink in the super-resolution apparatus 200, and , the padded padding image (PIN) is transmitted to the super-resolution device 200 implemented as an artificial neural network.

The preprocessing apparatus 100 includes an integrated padding size calculation unit 110 , an integrated padding unit 120 , and a padding area filling unit 130 .

The integrated padding size calculator 110 calculates a size reduced due to image shrink in the super-resolution device 200 . That is, the size of the padding area to be extended in the input image IN is calculated.

The integrated padding size calculator 110 calculates a size in which the image applied to the super-resolution device 200 is reduced due to the image shrink. The integrated padding size calculator 110 calculates the number of convolutional layers having a filter of size greater than 1 × 1 and each convolution in the super-resolution device 200 implemented as an artificial neural network having N convolutional layers. First, check the size of the filter included in the layer and the stride size (s).

Then, the image size (L _in ) input to each of the identified α convolutional layers is checked, and the output image size (L _out ) is checked. In an artificial neural network, the input and output of each convolutional layer are considered in the form of a two-dimensional or three-dimensional matrix, but when implemented in hardware, they are stored in the real memory in the form of a one-dimensional vector. The input image size (L _in ) and the output image size (L _out ) may also be referred to as an input length and an output length. Also, the size of the filter (K _n ) is also

And when the filter length of the n-th convolutional layer among α convolutional layers having a filter size greater than 1 × 1 is K _n , the output length (L _in ) corresponding to the input length of the n-th convolutional layer ( L _out ) may be calculated by Equation (1).

If Equation 1 is extended to correspond to each of the α convolutional layers, it can be expressed as Equation 2.

If it is assumed that all stride sizes s in the α convolutional layers are 1, the size ΔL reduced due to image shrink in the super-resolution device 200 may be calculated as in Equation 3.

The size ΔL calculated according to Equation 3 is a size for which the preprocessing apparatus 100 needs to collectively expand and integrate the size of the input image IN, so it is referred to herein as the integrated padding size.

When the integrated padding size calculation unit 110 calculates the integrated padding size ΔL, the integrated padding unit 120 temporarily pads the input image IN by the calculated integrated padding size ΔL. At this time, as shown in FIG. 5 , the integrated padding unit 120 pads the two-dimensional input image IN with a half size (ΔL/2) of the integrated padding size (ΔL) in the omnidirectional outer direction. This is because the input image in the convolutional layer is reduced uniformly in all directions.

The padding area filling unit 130 obtains a padding image PIN by filling each pixel of the padding area collectively expanded by the integrated padding unit with pixel values in a predetermined manner.

In this case, the padding area filling unit 130 may fill the pixel values of all pixels of the padding area with 0 according to the zero padding technique. However, when the zero padding technique is applied as described above, there is a limitation in that the super-resolution device 200 does not sufficiently reflect the information of the edge region of the input image IN.

Accordingly, in this embodiment, the padding area filling unit 130 proposes other techniques for filling the padding area so that the super-resolution device 200 can sufficiently reflect the information of the edge area of the input image IN.

In FIG. 6, (a) shows a zero padding technique for filling a padding area with a pixel value of 0, (b) to (d) are three different filling techniques proposed in this embodiment, and (b) is an edge copy ( Edge Copy) technique, (c) shows a border copy technique, (d) shows a border mirror technique.

In FIG. 6 , the red hatched area is the area where a pixel value of 0 is filled in the padding area, and the blue hatched area is the area including the pixel value to be filled in the padding area in the input image and the padding filled with the pixel value according to this pixel value. indicates the area.

As shown in (a) of FIG. 6 , in the zero padding technique, all pixels of the padding area have a pixel value of 0 regardless of the input image IN. this is filled

On the other hand, in the edge copy technique of (b), pixel values of pixels located on edge lines of each side of the input image IN are copied in units of edge lines and filled in such a way that they extend outward. That is, as shown in (b), the padding area is filled in the form of repeatedly copying and pasting pixel values of pixels located in rows and columns at both ends of the input image IN. However, since the padding area is formed to surround the input image IN, the number of pixels in the x-axis and y-axis directions is greater than that of the input image IN. Therefore, even if the edge line of each side of the input image IN is copied and filled in the outer direction, as shown in (b), the four corner direction areas of the input image are not filled in the pixel area as shown in (b). Accordingly, in the edge copy technique, a pixel value is filled with 0, similar to the zero padding technique, for pixels in the corner area not filled with the outermost edge line.

And, in the boundary copy technique of (c), similar to the edge copy technique, pixel values of pixels located in the number of rows or columns corresponding to the padding area in the direction of each side of the input image IN are copied and located in the outer direction of the corresponding side. Fill in the padding area. As shown in (c), when the padding area extends from each side of the input image IN to the size of 4 rows or columns in the outward direction, the boundary copy technique uses 4 rows located at the edges of each side of the input image IN. Alternatively, the pixel values of the column are copied and filled in the 4 rows or columns of the padding area in the copied order. That is, an area of a predetermined size outside the input image IN is copied and filled again in the padding area as it is. And, like the edge copy technique, the pixel value is filled with 0 for the pixel in the corner area that is not filled with the pixel value of the input image IN.

On the other hand, the boundary mirror technique of (d) copies the pixel values of pixels located in the number of rows or columns corresponding to the padding area in the direction of each side of the input image IN, similar to the boundary copy technique, and is located in the outer direction of the corresponding side. Fill in the padding area. However, the boundary copy technique has a pattern in which the same boundary region is repeated by filling the padding region in the same order that they are copied from the input image IN, whereas in the boundary mirror technique, a number of copied rows or columns is inverted to fill the padding region. fill in Accordingly, a plurality of rows or columns of the input image IN and a padding area corresponding thereto have pixel values of a symmetric pattern with respect to each side of the input image IN. Also, in the boundary mirror technique, the pixel value is filled with 0 for the pixel in the corner area that is not filled with the pixel value of the input image IN.

Unlike the zero padding technique, the edge copy technique, the boundary copy technique, and the boundary mirror technique shown in (b) to (d) fill the padding area with the pixel value of the actual edge area of the input image IN, so that the input image It allows the information of the edge region of (IN) to be sufficiently reflected in the convolution operation of the super-resolution device 200 .

The super-resolution device 200 is implemented as an artificial neural network that sequentially extracts a feature map from an input image, including a plurality of convolutional layers sequentially connected and provided with a plurality of filters of a predetermined size, respectively, and is higher than the input image (IN). Outputs an output image OUT having a large predetermined resolution. The super-resolution device 200 may be implemented as FSRCNN as shown in FIG. 1 , but may also be implemented as SCRCNN or other types of artificial neural networks.

That is, in the present embodiment, the super-resolution device 200 receives the padding image PIN preprocessed in the preprocessing device 100 without directly applying the input image IN, unlike the existing ones, and padding according to a pre-learned method. An output image OUT having a higher resolution than the input image IN is obtained from the image PIN and outputted.

In this case, the super-resolution apparatus 200 performs a convolution operation without considering the sequential size reduction due to image shrink occurring according to the size of the filter in the plurality of convolutional layers. That is, pre-processing padding or post-processing padding in units of convolutional layers is not performed.

Instead, the pre-processing unit 100 calculates the overall size reduced by the image shrink caused by the plurality of convolutional layers of the super-resolution unit 200, and as shown in FIG. 5, as shown in FIG. Integrated padding is performed on the input image (IN) in a batch. That is, the pre-processing device 100 acquires a padding image (PIN) in which the image shrink generated in the super-resolution device 200 is pre-compensated to the input image (IN) in advance, and the obtained padding image (PIN) is used in the super-resolution device By transferring to 200, the super-resolution device 200 may obtain an output image OUT of a required size by performing a specified operation without a padding operation for compensating for image shrink in units of each convolutional layer.

FIG. 7 is a diagram for explaining an operation of a conventional pre-processing padding technique and an integrated pre-processing padding technique of the present embodiment by comparison.

In FIG. 7, (a) shows the existing pre-processing padding technique, and (b) shows the integrated pre-processing padding technique according to the present embodiment.

As shown in (a) of FIG. 7 , as the conventional super-resolution device performs padding for each convolutional layer, the padding process for adjusting the size of the feature map among a plurality of sequentially connected convolutional layers is performed a plurality of times. In order to perform a convolution operation on a feature map of a size changed due to the padding process, the positional relationship between the pixels of the feature map whose size is not changed and the pixels of the extended padding area should always be considered. This is a problem due to a limitation in the number of operators provided along with a problem due to a method in which an input image or a feature map is stored in the memory of a super-resolution device implemented in hardware. As such, in order to configure the super-resolution device as hardware to perform calculations in consideration of the positional relationship between the pixels of the feature map and the pixels of the extended padding area, the design must be very complicated, as well as a section in which a bottleneck may occur. It is highly likely that the operation speed will slow down as the number of is increased.

In contrast, in the present embodiment shown in (b), the super-resolution device does not need to perform padding by applying the pre-integrated padding image (PIN) in the pre-processing unit 100, so memory that may occur due to the padding area It is only necessary to perform the specified operation without changing the storage method or operation order. Accordingly, the hardware structure can be very simple, and the operation speed can be improved.

Here, for convenience of explanation, the pre-processing apparatus 100 and the super-resolution apparatus 200 are separated and illustrated as separate apparatuses, but the pre-processing apparatus 100 of this embodiment may be included in the super-resolution apparatus 200 and configured. .

8 is a graph comparing performance changes of a super-resolution device according to a padding method.

8, (a) to (c) show the results of comparing the average peak signal-to-noise ratio (PSNR) of the output image OUT obtained using different data sets, (d) ) to (f) show the results of comparing the structural similarity (Structure Similarity Index: SSIM). And in each of (a) to (f), 1 indicates a conventional method of performing pre-processing padding in units of convolutional layers, and 2 to 5 indicate a zero padding technique, an edge copy technique, a boundary copy technique, and a boundary copy technique in the integrated padding technique, respectively. The boundary mirror technique is shown.

Referring to FIG. 8 , it can be seen that the edge copy technique, the boundary copy technique, and the boundary mirror technique exhibit better performance than the zero padding technique. It can be seen that also shows similar performance.

Therefore, the pre-processing apparatus according to the present embodiment may be applied even when the super-resolution apparatus is implemented in software, and the pre-processing apparatus may also be implemented in software. However, the pre-processing device of this embodiment has the advantage of being able to maintain performance while greatly improving the hardware efficiency of the super-resolution device when the super-resolution device is implemented in hardware than when it is implemented in software.

9 shows a pre-processing method for a super-resolution device according to an embodiment of the present invention.

When the pre-processing method of FIG. 9 is described with reference to FIGS. 4 to 6 , first, the size of the integrated padding area to expand the input image IN in consideration of the size reduced due to image shrink in the super-resolution device 200 ( ΔL) is calculated (S10). Here, the size of the integrated padding area is the number (α) of the convolutional layer having a filter of size greater than 1 × 1 in the super-resolution device 200 implemented as an artificial neural network having N convolutional layers and the number of convolutional layers in each convolutional layer. It can be calculated by checking the length of the included filter (K _n ), the stride size (s), and the input image length (L _in ) and the output image length (L _out ).

And when the size (ΔL) of the integrated padding area is calculated, according to the calculated size (ΔL) of the integrated padding area, half the size (ΔL/2) of the integrated padding size (ΔL) for the outer omnidirectional direction in the input image (IN) , and integrated padding is performed at once (S20).

Thereafter, pixel values are filled in the pixels of the padding area extended in the input image IN according to a predetermined filling method ( S30 ).

In this case, pixel values of the padding area may be filled according to one of an edge copy technique, a boundary copy technique, and a boundary mirror technique. In the edge copy technique, the padding area is filled in the form of repeatedly pasting the pixel values of pixels located in one row and one column located at the edge of the input image IN to the corresponding pixel area.

And in the boundary copy technique, the pixel values of pixels located in rows and columns of half the size (ΔL/2) of the integrated padding size (ΔL) located at the edge of the input image IN are copied to the corresponding pixel area as the same pattern. Fill the padding area in the pasted format.

Finally, in the boundary mirror technique, the pixel values of pixels located in rows and columns of half the size (ΔL/2) of the integrated padding size (ΔL) located at the edge of the input image (IN) are copied and copied to the corresponding pixel area. Fills the padding area in the form of inverting and pasting the pixel values to form a pattern symmetrical to the corresponding pixel area.

In this case, in the edge copy technique, the boundary copy technique, and the boundary mirror technique, a pixel value is filled with 0 for a pixel in a corner area where a pixel value is not filled.

When the padding image PIN is obtained by filling pixel values in all pixels of the padding area extended to the outside of the input image IN, the obtained padding image PIN is transmitted to the super-resolution device (S40). Accordingly, the super-resolution device does not consider the sequential size reduction due to image shrink that occurs depending on the size of the filter in a plurality of convolution layers, and performs a predetermined operation including convolution operation to obtain an upscaled super-resolution image. have.

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is only exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: preprocessor 110: integrated padding size calculator
120: integrated padding unit 130: padding area filling unit
200: super-resolution device

Claims (14)

As a pre-processing device for a super-resolution device that is implemented as an artificial neural network including a plurality of filters of a predetermined size and sequentially connected N convolutional layers to perform image upscaling,
an integrated padding size calculation unit for calculating a size of an integrated padding area indicating an overall size in which an image input to the super-resolution device is reduced due to image shrink of a convolutional layer;
an integrated padding unit for performing integrated padding by expanding at once to half the size of the integrated padding area in all directions along the periphery of the input image to be upscaled; and
A padding area filling unit that fills a pixel value of each pixel of a padding area that is an area extended by the integrated padding in a predetermined manner;
The integrated padding size calculation unit
The number of convolutional layers with filters of size greater than 1 × 1 (α) and the length of filters included in each convolutional layer (K _i ) and input image length (L _in ) and output image length (L _out ) based on the formula

A preprocessor for a super-resolution device that calculates the size of the integrated padding area according to .
According to claim 1, wherein the padding area filling part
A preprocessor for a super-resolution device that first fills a corresponding area of the padding area with pixel values of a predetermined number of rows and columns of each edge of the input image, and fills a non-corresponding area with a pixel value of zero. The method of claim 2, wherein the padding area filling part
A preprocessor for a super-resolution device that copies pixel values of pixels positioned in one row and one column positioned at each edge of the input image, and repeatedly fills the padding area corresponding to the copied row and column pixel values. The method of claim 2, wherein the padding area filling part
A super-resolution device that copies rows and columns of sizes corresponding to the sizes in the padding area at each edge of the input image, and fills the padding areas with pixel values of rows and columns of sizes corresponding to the sizes in the copied padding area. preprocessor for The method of claim 2, wherein the padding area filling part
At each edge of the input image, a row and column of a size corresponding to the size are copied to the padding area, and pixel values of rows and columns of the size corresponding to the size are copied to the padding area so that the pixel values are symmetrical in the corresponding pixel area. Preprocessor for super-resolution devices with inverted filling. delete According to claim 1, wherein the super-resolution device is
A preprocessor for a super-resolution device implemented in hardware including a plurality of operators and memory. As a pre-processing method for a super-resolution device that is implemented as an artificial neural network including a plurality of filters of a predetermined size and sequentially connected N convolutional layers to perform image upscaling,
calculating a size of an integrated padding area indicating an overall size in which an image input to the super-resolution device is reduced due to image shrink of a convolutional layer;
performing integrated padding by expanding at one time to half the size of the integrated padding area in all directions along the periphery of the upscaling target input image; and
Filling in a pixel value of each pixel of a padding area that is an area extended by the integrated padding in a predetermined manner;
The step of calculating the size of the integrated padding area is
The number of convolutional layers with filters of size greater than 1 × 1 (α) and the length of filters included in each convolutional layer (K _i ) and input image length (L _in ) and output image length (L _out ) based on the formula

A preprocessing method for a super-resolution device for calculating the size of the integrated padding area according to .
The method of claim 8, wherein the filling of the pixel value comprises:
filling a corresponding area of the padding area with a predetermined number of row and column pixel values of each edge of the input image; and
A pre-processing method for a super-resolution device comprising the step of filling non-corresponding regions with a pixel value of zero. 10. The method of claim 9, wherein filling the corresponding region comprises:
copying pixel values of pixels positioned in one row and one column positioned at each edge of the input image; and
A preprocessing method for super-resolution devices, comprising the step of repeatedly filling corresponding padding regions with pixel values of copied rows and columns. 10. The method of claim 9, wherein filling the corresponding region comprises:
copying rows and columns having a size corresponding to the size in a padding area at each edge of the input image; and
A preprocessing method for a super-resolution device, comprising: filling the copied padding region with pixel values of rows and columns having a size corresponding to the size of the padding region. 10. The method of claim 9, wherein filling the corresponding region comprises:
copying rows and columns having a size corresponding to the size in a padding area at each edge of the input image; and
A preprocessing method for a super-resolution device, comprising the step of filling the copied padding area by inverting it so that pixel values of rows and columns having a size corresponding to the size become a symmetrical pattern in the corresponding pixel area. delete According to claim 8, wherein the super-resolution device is
A preprocessing method for a super-resolution device implemented in hardware including a plurality of operators and memory.

Images