KR20210141304A - Image processing apparatus and operating method for the same - Google Patents

Abstract

The present invention relates to an image processing apparatus. The apparatus includes: a memory for storing one or more instructions; and a processor for executing the one or more instructions stored in the memory. The processor uses one or more neural networks to extract n pieces of first feature information from a first image based on positions of pixels included in the first image, and generates n pieces of second feature information by performing a convolution operation on each of the n pieces of first feature information and n kernels. The second image is an image from which a compression artifact included in the first image is removed.

Description

Image processing apparatus and operating method for the same

Various embodiments relate to an image processing apparatus for removing artifacts from an image using a neural network, and an operating method thereof.

As data traffic increases exponentially with the development of computer technology, artificial intelligence has become an important trend driving future innovation. Since artificial intelligence is a method that imitates human thinking, it can be applied to virtually all industries infinitely. Representative technologies of artificial intelligence include pattern recognition, machine learning, expert systems, neural networks, and natural language processing.

A neural network models the characteristics of human biological nerve cells by mathematical expressions, and uses an algorithm that mimics the ability of learning that humans have. Through this algorithm, the neural network can generate a mapping between input data and output data, and the ability to generate this mapping can be expressed as the learning ability of the neural network. In addition, the neural network has a generalization ability to generate correct output data for input data that has not been used for learning, based on the learned result.

In a conventional convolutional neural network (CNN), it is common to use the same kernel for all pixels in an input image. Accordingly, when an image is processed using an existing convolutional neural network, compression artifacts (eg, blocking artifacts or ringing artifacts) appearing at block boundaries cannot be effectively removed. have.

In addition, a method of removing artifacts by using different filters according to rules at the block boundary is difficult to implement using a convolutional neural network, and is not suitable for parallel processing.

Various embodiments may provide an image processing apparatus capable of effectively reducing compression artifacts using a convolutional neural network and an operating method thereof.

An image processing apparatus according to an embodiment includes a memory that stores one or more instructions, and a processor that executes the one or more instructions stored in the memory, wherein the processor uses one or more neural networks to generate a first image. Based on the positions of the included pixels, n pieces of first feature information are extracted from the first image, and a convolution operation is performed between each of the n pieces of first feature information and each of the n kernels. Second feature information may be generated, and a second image may be generated based on the n pieces of second feature information, and the second image may be an image from which compression artifacts included in the first image are removed.

The processor according to an embodiment may divide the first image into blocks having a preset size, and divide pixels having the same position in each of the blocks into the same group to extract the n pieces of first feature information .

Each of the blocks according to an embodiment may include the n pixels, and n may be determined based on a preset size of the blocks.

The processor according to an embodiment divides the second image into blocks having the preset size, and places pixel values included in one second characteristic information among the second characteristic information at the same location in each of the blocks. It can be determined by the pixel values of .

The processor according to an embodiment may extract the n pieces of first feature information by performing a convolution operation between the first image and the second kernel by applying a stride size L.

The number n of the first characteristic information according to an embodiment may be determined based on the stride size L.

The n kernels according to an embodiment may be different kernels.

The size of the first image and the second image according to an embodiment may be the same.

According to an embodiment, the method of operating an image processing apparatus performing image processing using one or more neural networks includes n pieces of first feature information from a first image based on positions of pixels included in a first image. extracting , generating n pieces of second feature information by performing a convolution operation with each of the n pieces of first feature information and each of the n kernels, and adding to the n pieces of second feature information based on the method, generating a second image, wherein the second image may be an image from which compression artifacts included in the first image are removed.

An image processing apparatus according to an exemplary embodiment may primarily extract feature information by grouping pixels having similar characteristics based on positions within a block of pixels included in an image, and then add the extracted feature information to the extracted feature information. By performing convolution by applying different kernels, compression artifacts can be effectively removed.

The image processing apparatus according to an embodiment may perform image processing for removing compression artifacts in parallel by using the artifact removal network.

1 is a diagram illustrating a method of an image processing apparatus processing an image using an artifact removal network according to an exemplary embodiment.
2 is a diagram illustrating one convolutional layer according to an embodiment.
3 is a reference diagram for describing an operation of a pixel sampling unit according to an exemplary embodiment.
4 is a diagram illustrating an example of a pixel sampling period according to an embodiment.
5 is a diagram referenced to describe an operation of a convolution unit according to an embodiment.
6 is a reference diagram for describing an operation of a pixel combining unit according to an exemplary embodiment.
7 is a diagram illustrating one convolutional layer of FIG. 1 according to another embodiment.
FIG. 8 is a diagram referenced to describe an operation performed in the convolution layer of FIG. 7 .
9 is a diagram referenced to describe an operation of a feature extractor performing a convolution operation according to an embodiment.
10 is a flowchart illustrating a method of operating an image processing apparatus according to an exemplary embodiment.
11 is a block diagram illustrating a configuration of an image processing apparatus according to an exemplary embodiment.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram illustrating a method of an image processing apparatus processing an image using an artifact removal network according to an exemplary embodiment.

Referring to FIG. 1 , an artifact removal network 30 according to an embodiment may receive an input image 10 and process the input image 10 to generate an output image 20 .

According to an embodiment, the input image 10 may include compression artifacts. Compression artifacts are artifacts generated in the process of compressing an image, and may include blocking artifacts, ringing artifacts, and the like. In general, compression artifacts occur in the process of compressing and decompressing an image or video.

The image or video compression may be performed in units of blocks by dividing pixels included in the image or video into a plurality of blocks such that one block includes a predetermined number of pixels. Accordingly, the compression artifact is mainly generated at the boundary of the block, and is generated with a certain period.

The image processing apparatus 100 according to an embodiment may generate an output image 20 from which artifacts are removed by image processing the input image 10 using an Artifact Removal Network 30 . have.

The artifact neural network 30 according to an embodiment may include one or more convolutional layers 210, 220, ..., 290, and in each of the convolutional layers, pixel sampling, convolution operation, and pixel synthesis. etc. may be performed.

Image processing performed in each of the convolutional layers will be described in detail with reference to the drawings below.

2 is a diagram illustrating one convolutional layer according to an embodiment.

Referring to FIG. 2 , one convolutional layer 200 according to an embodiment may include a pixel sampling unit 300 , a convolution unit 400 , and a pixel combining unit 500 . One convolutional layer 200 may be any one of the first to N-th convolutional layers 210 , 220 , ..., 290 of FIG. 1 . For example, each of the first to Nth convolutional layers 210 , 220 , ..., 290 may include a pixel sampling unit 300 , a convolution unit 400 , and a pixel combining unit 500 . can The convolutional layer 200 according to an embodiment may include a structure in which a first image F_in is received and a second image F_out is output, and the first image F_in and the second image F_out are included. ) may be the same size, but is not limited thereto.

Hereinafter, an operation performed in one convolutional layer will be described in detail with reference to FIGS. 3 to 6 .

3 is a reference diagram for describing an operation of a pixel sampling unit according to an exemplary embodiment.

The pixel sampler 300 according to an embodiment may generate n sampling features by grouping pixels based on position information of pixels included in the first image F_in. In this case, the first image may be the input image 10 of FIG. 1 or feature information output from a previous convolutional layer. For example, when the current convolution layer 200 is the first convolution layer Conv_1 , the first image may be the input image 10 , and the current convolution layer 200 includes the second to N-th convolutions. In the case of one of the convolution layers Conv_2, ..., Conv_N, the first image F_in may be feature information output from a previous convolution layer. Alternatively, it may be feature information in which a predetermined operation or processing is performed on feature information output from a previous convolutional layer. However, the present invention is not limited thereto.

The first image F_in according to an embodiment has a size of W x H x Cin. In this case, W denotes the width of the first image F_in, H denotes the height of the first image F_in, and Cin denotes the number of channels of the first image F_in.

As shown in Equation 1, the pixel sampling unit 300 may generate a plurality of sampling features by sampling pixels included in the first image with a period L in the width direction and the height direction, respectively. For example, the pixel sampling unit 300 may divide the first image F_in into blocks having a preset size (L x L) and sample pixels having the same position in each of the blocks as the same group.

[Equation 1]

F_in _u,v = F_in[u:L:W-1, v:L:H-1, 0:1:C-1]

In Equation 1, F_in _u,v denotes a sampling characteristic, and F_in denotes a first image. F_in[] represents an operation of sampling pixels of the first image F_in in a width direction, a height direction, and a channel direction. u:L:W-1 means that pixels are sampled in a period of L starting from the pixel positioned at the u point in the width direction of the first image F_in and up to the pixel positioned at the W-1 point. v:L:H-1 means that pixels are sampled with a period of L starting from the pixel located at the point v in the height direction of the first image F_in and up to the pixel located at the point H-1. 0:1:C-1 means that all pixels are sampled from the pixel positioned at point 0 to the pixel positioned at point C-1 in the channel direction of the first image F_in. That is, among the pixels included in the first image F_in, a position in a width direction and a position in a height direction are the same, and pixels located in different channels may be sampled into the same group.

Hereinafter, a case in which one channel image (first channel image, 310 ) included in the first image F_in has a size of 16×16 and L=8 is taken as an example, and a method of generating a plurality of sampling features to be explained.

The first image F_in may be divided into blocks having a size of 8×8, and the pixel sampling unit 300 may sample pixels having the same position in each of the blocks as a group. For example, when the first channel image 310 is divided into blocks having a size of 8 x 8, the first channel image 310 is divided into four blocks (first to fourth blocks B1, B2, B3, B4)). Also, if pixels included in the first channel image 310 of the first image _{F_in} are indexed from P _{0,0,0 to P 15,15,0 in order according to positions, P 0,0,0} (P0), P _0,8,0 (P1), P _8,0,0 (P2) _, P _8,8,0 (P3) has the same position in each of the first to fourth blocks B1, B2, B3, and B4. The pixel sampling unit 300 receives P _0,0,0 (P0), P _0,8,0 (P1), P _8,0,0 (P2) _, P _8,8,0 (P3) The first group can be sampled. In addition, since the sampling period is 1 in the channel direction, the pixels (P _0,0,1 , P _{0,8, P 0,8, 1} , P _8,0,1, P _8,8,1 ,..., P _0,0,C-1 , P _0,8,C-1 , P _{8,0,C-1 ,} P _8,8,C-1 ) may be sampled as the first group. _{The pixel sampling unit 300 may generate a first sampling feature (F_in 0,0} ) based on the pixels sampled into the first group, and the first sampling feature (F_in _0,0 ) is 2 X 2 XC has the size of

Also, in the first channel image 310 in the same manner, the pixel sampling unit 300 may perform P _1,0,0 ( P4), P _9,0,0 (P5), P _1,8,0 (P6) _{, and} P _9,8,0 (P7) are sampled into the second group, and second to C-1 channel images In the same manner as in the first channel image 310 for , pixels P _1,0,1 , P _9,0,1 , P _{1,8,1 ,} P _9,8,1 , ... , P _1,0,C-1 , P _9,0,C-1 , P _{1,8,C-1 ,} P _9,8,C-1 ) may be sampled as a second group. The pixel sampling unit 300 may generate a second sampling characteristic F_in _1,0 based on the pixels sampled into the second group, and the second sampling characteristic F_in _1,0 is 2 X 2 XC has the size of

In addition, in the first channel image 310 in the same manner, the pixel sampling unit 300 may have the same position in each of the first to fourth blocks B1, B2, B3, and B4 P _{7, 7, 0} (P8), P _7,15,0 (P9), P _15,7,0 (P10) _, P _15,15,0 (P11) _{is sampled into the 64th group, and the pixels P 7,7,1} , P _{7,15,1 at the} same position as the first channel image 310 also for the second to C-1 channel images. , P _15,7,1, P _15,15,1 ,..., P _7,7,C-1 , P _{7,15,C-1 , P 15,7,C-1} , P _{15,15 ,C-1} ) may be sampled as the 64th group. The pixel sampling unit 300 may generate a 64th sampling feature based on pixels sampled into the 64th group, and the 64th sampling feature F_in _7,7 has a size of 2 X 2 XC.

In the above description, for convenience of description, W=H=16 and L=8 have been described as examples, but the present invention is not limited thereto. When sampling is performed by the method described above, the pixel sampling unit 300 generates L x L sampling features (F_in _0,0 , F_in _1,0 ,...,F_in _L-1,L-1 )). and each of the sampling features may have a size _{of W F} x H _{F x Cin.} In this case, W _F may be determined based on the width (W) of the first image and the sampling period L in the width direction, and H _F is the height (H) of the first image and the sampling period L in the height direction. can be decided.

4 is a diagram illustrating an example of a pixel sampling period according to an embodiment.

4 illustrates an 8×8 block included in the first image F_in according to an exemplary embodiment.

Referring to FIG. 4 , the pixel sampling unit 300 according to an exemplary embodiment samples the first image F_in at a period of Lw=4 in the width direction and Lh=4 in the height direction to obtain 4×4 sampling features. (F_in _0,0 , F_in _1,0 ,..., F_in _3,3 ), each of the sampling features (F_in _0,0 , F_in _1,0 ,..., F_in _3,3 ) may have a size of W/4 x H/4 x Cin.

In addition, the pixel sampling unit 300 samples the first image F_in with a period of Lw=4 in the width direction and Lh=1 in the height direction, so that 4 x 1 sampling features F_in _0,0 , F_in _{1 , 0} ,..., F_in _3,0 ) can be created. In this case, each of the sampling features may have a size of W/4 x H x Cin.

In addition, the pixel sampling unit 300 samples the first image F_in with a period of Lw=8 in the width direction and Lh=2 in the height direction, so that 8 x 2 sampling features F_in _0,0 , F_in _{1 , 0} ,..., F_in _7,1 ) can be created. In this case, each of the sampling features may have a size of W/8 x H/2 x Cin.

In addition, the pixel sampling unit 300 samples the first image F_in with a period of Lw=8 in the width direction and Lh=1 in the height direction, so that 8 x 1 sampling features F_in _0,0 , F_in _{1 , 0} ,..., F_in _8,0 ) can be created. In this case, each of the sampling features may have a size of W/8 x H x Cin.

The pixel sampling unit 300 according to an exemplary embodiment may be configured based on the complexity of the image processing system, memory, whether line processing is performed, whether deinterlacing is performed, encoding information (eg, compression parameter, etc.), etc. , the sampling period in the width direction and the height direction may be determined. In addition, the sampling period according to an exemplary embodiment is not limited to the sampling period illustrated in FIG. 4 , and may include various sampling periods.

5 is a diagram referenced to describe an operation of a convolution unit according to an embodiment.

Referring to FIG. 5 , the convolution unit 400 may perform convolution (group convolution) with each of n sampling features generated by the pixel sampling unit 300 and each of n kernels, The solution operation can be expressed by Equation (2).

[Equation 2]

F_out _u,v =Conv2D(input=F_in _u,v , kernel=K _u,v , stride=1)

In Equation 2, F_in _u,v represents one of n sampling features, K _u,v represents a kernel corresponding to the input sampling feature, and the size of the stride represents 1. If the size of the stride is 1, it means that the convolution operation is performed while moving pixels included in the input sampling feature one by one.

5 and Equation 2, different kernels may be applied to each of the n sampling features. For example, the convolution unit 400 is the first sampling characterized by applying the first kernel (K _0,0) to _(0,0 F_in), the first sampling features (F_in _0,0) with the first kernel ( By performing a convolution operation with K _0,0 ), the first output feature F_out _0,0 may be generated.

Also, the first kernel K _0,0 may include Cout sub-kernels, and one sub-kernel may have a size _{of W K} x H _{K x Cin.} The number of channels (Cin) of one sub-kernel may be the same as the number of channels (Cin) of the sampling feature.

Also, the convolution unit 400 is the second sampling characterized by applying a second kernel (K _1,0) to _(1,0 F_in), the second sampling features (F_in _1,0) with the second kernel (K _{1 ,0} ), the second output feature F_out _1,0 may be generated. In addition, the convolution unit 400 applies the n _{-th kernel (K L-1,L-1 ) to the n-th sampling feature (F_in L-1,L-1} ), and the n-th sampling feature (F_in _{L-1) ,L-1} ) and the n _{-th kernel K L-1,L-1} are convolved to generate an n-th output feature F_out _L-1,L-1. In addition, each of the second to nth kernels may include Cout sub-kernels, and one sub-kernel may have a size _{of W K} x H _{K x Cin.}

At this time, the width and height of each of the features output as a result of the convolution operation (F_out _0,0 , F_out _1,0 ,..., F_out _L-1,L-1 ) are the same as the sampling features, and The number of channels may be determined based on the number of sub-kernels included in one kernel. For example, if each of the sampling features according to an embodiment _{has a size of W F} x H _F x Cin and the number of sub-kernels included in one kernel is Cout, each of the output n features is W _F x It may have a size of _{H F x Cout.}

According to an embodiment, the n sampling features are features sampled for each position in a block of pixels, and one sampling feature may be information sampled between pixels having a similar probability of occurrence of an artifact. Also, as described with reference to FIG. 5 , different kernels are applied to each of the n sampling features, so that different kernels are applied according to the probability of occurrence of artifacts. Accordingly, it is possible to effectively remove compression artifacts mainly occurring at block boundaries.

6 is a reference diagram for describing an operation of a pixel combining unit according to an exemplary embodiment.

Referring to FIG. 6 , the pixel synthesizing unit 500 includes n features (F_out _0,0, F_out _{1,0, ...,} F_out _L-1,L-1 ) output from the convolution unit 400 . may be synthesized to generate a second image F_out. In this case, the width and height of the second image F_out are the same as those of the first image F_in, and the number of channels of the second image F_out is the number of n features F_out _{0,0 output from the convolution unit 400 . ,} F_out _{1,0, ...,} F_out _L-1,L-1 ) may be determined to be the same as the number of channels Cout of each.

The pixel synthesizing unit 500 may generate the second image F_out based on a period in which the pixel sampling unit 300 samples the pixels. As shown in Equation 3, the pixel synthesizing unit 500 includes n features (F_out _0,0, F_out _{1,0, ...,} F_out _L-1,L-1 ) output from the convolution unit 400 . Based on , the second image F_out may be generated.

[Equation 3]

F_out[u:L:W-1, v:L:H-1, 0:1:Cout-1]=F_out _u,v

For example, when the first image F_in is sampled with a period of L in the width direction and a period of L in the height direction by the pixel sampling unit 300 , the pixel combining unit 500 sets the second image F_out to L It is divided into blocks of size x L, and _{the pixel values included in each of the output features (F_out 0,0,} F_out _{1,0, ...,} F_out _L-1,L-1 ) are located at the same position in each of the blocks. It can be determined by pixel values of the pixels.

Hereinafter, for convenience of description, each of the output features F_out _0,0, F_out _{1,0, ...,} F_out _L-1,L-1 has a size of 2 X 2 X Cout, and the second An image has a size of 16 X 16 X Cout and a method of synthesizing output features will be described by taking the case of L=8 as an example.

In this case, since the method of synthesizing features in each of the channels is the same, the description will be made based on a single channel image. Referring to FIG. 6 , the first channel images F_out ₀ included in each of the first to nth output features F_out _0,0, F_out _{1,0, ...,} F_out _{L-1,L-1 . ,0,0,} F_out _{1,0,0, ...,} F_out _L-1,L-1,0 ) has a size of 2 x 2, and the first channel image ( 610) has a size of 16 x 16. Also, since L=8, the first channel image 610 of the second image F_out may be divided into 8×8 blocks (first to fourth blocks).

The pixel synthesizer 500 combines pixel values included in the first channel image F_out _0,0,0 of the first output feature F_out _0,0 to the first channel image 610 of the second image F_out. It may be determined as pixel values of the pixels P0, P1, P2, and P3 having a position (0, 0) in each of the first to fourth blocks B1, B2, B3, and B4 included in . In addition, pixel values included in the first channel image F_out _1,0,0 of the second output feature F_out _1,0 are applied to the first to fourth blocks B1 and B2 of the first channel image 610 . , B3, B4) may be determined as pixel values of pixels P4, P5, P6, and P7 having positions (0, 1), respectively. In addition, pixel values included in the first channel image F_out _{L-1,L-1,0 of} the n-th output feature F_out _L-1,L-1 are applied to the first to first channel images 610 of the first channel image 610 . It may be determined as pixel values of the pixels P8, P9, P10, and P11 having positions (7, 7) in each of the fourth blocks B1, B2, B3, and B4. As such, the pixel synthesizing unit 500 includes the first channel image included in each of _{the first to nth output features F_out 0,0,} F_out _{1,0, ...,} F_out _L-1,L-1. The first channel image ( F_out) of the second image F_out using pixel values of the fields F_out _{0,0,0, F_out 1,0,0, ...,} F_out _L-1,L-1,0 610) can be created. Also, second to Cout channel images of the second image F_out may be generated in the same manner.

Meanwhile, at least one of the pixel sampling unit 300 , the convolution unit 400 , and the pixel synthesis unit 500 illustrated and described with reference to FIGS. 2 to 6 is manufactured in the form of a hardware chip to be installed in the image processing apparatus 100 . can be mounted. For example, at least one of the pixel sampling unit 300, the convolution unit 400, and the pixel synthesis unit 500 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or It may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted on the image processing apparatus 100 according to an embodiment.

Also, at least one of the pixel sampling unit 300 , the convolution unit 400 , and the pixel synthesis unit 500 may be implemented as a software module. When at least one of the pixel sampling unit 300, the convolution unit 400, and the pixel synthesis unit 500 is implemented as a software module (or a program module including an instruction), the software module is a computer It may be stored in a readable non-transitory computer readable medium that can be read by a user. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

7 is a diagram illustrating one convolutional layer of FIG. 1 according to another embodiment.

Referring to FIG. 7 , one convolutional layer 700 according to another embodiment may include a feature extraction unit 710 , a convolution unit 720 , and a pixel synthesis unit 730 . Since the convolution unit 720 and the pixel combining unit 730 of FIG. 7 are the same as the convolution unit 400 and the pixel combining unit 500 of FIG. 2 , a detailed description thereof will be omitted.

Also, the convolutional layer 700 of FIG. 7 may be any one of the first to Nth convolutional layers 210 , 220 , ..., 290 of FIG. 1 .

Hereinafter, an operation performed in the convolutional layer of FIG. 7 will be described in detail with reference to FIGS. 8 and 9 .

FIG. 8 is a diagram referenced to describe an operation performed in the convolution layer of FIG. 7 .

Referring to FIG. 8 , the feature extraction unit 710 according to an embodiment includes a first image F_in and a kernel P _0,0 , P _1,0 ,...,P _L-1,L-1 . By performing convolution with , it is possible to generate n hidden features.

The first image F_in according to an embodiment has a size of W x H x Cin. In this case, W denotes the width of the first image F_in, H denotes the height of the first image F_in, and Cin denotes the number of channels of the first image F_in.

The feature extraction unit 710 is a convolution between the first image F_in and the kernel P _0,0 , P _1,0 ,...,P _{L-1,L-1 as shown in Equation 4 below.} You can perform a solution operation.

[Equation 4]

F_hidden _u,v =Conv2D(input=F_in[u:1:W-1, v:1:H-1,0:1:Cin-1], kernel=Pu,v, stride=L)

In Equation 4, F_hidden _u,v represents a hidden feature extracted through a convolution operation, and Conv2D (input, kernel, stride = L) represents that a convolution operation is performed on input and kernel with stride size L. In this case, F_in represents the first image, and F_in[] represents an operation of sampling pixels to be included in the input image, among pixels included in the first image F_in. u:1:W-1 means that sampling is performed by moving one pixel at a time from the pixel positioned at the u point in the width direction of the first image F_in to the pixel positioned at the W-1 point. 1 In the width direction of the image F_in, all pixels from the pixel located at the u point to the pixel located at the W-1 point are included in the input image.

In addition, v:1:H-1 means sampling by moving one pixel at a time from the pixel located at the point v in the height direction of the first image F_in to the pixel located at the point H-1. , all pixels from the pixel positioned at the point v in the height direction of the first image F_in to the pixel positioned at the point H-1 are included in the input image. Also, 0:1:C-1 means sampling all pixels from a pixel located at point 0 to a pixel located at point C-1 in the channel direction of the first image.

When the feature extraction unit 710 performs a convolution operation between the input image (input) and the kernel (kernel), data (pixel values) that are the target of the convolution operation in the input image (input) is the size of the stride It can be scanned while moving as much as possible. For example, when the size of the stride is L, the feature extractor 710 may perform a convolution operation with the kernel while moving data by L pixels in the input image.

A method in which the feature extraction unit 710 performs a convolution operation to generate a hidden feature will be described in detail with reference to FIG. 9 .

9 is a diagram referenced to describe an operation of a feature extractor performing a convolution operation according to an embodiment.

In FIG. 9 , for convenience of explanation, a method of generating a plurality of hidden features is illustrated by taking the case where one channel image included in the first image F_in has a size of 16 x 16 and L = 8 as an example. to explain

Referring to FIG. 9 , the input data 910 represents a first channel image 910 of the first image F_in, and the feature extractor 710 includes an upper left 8×8 area 915 of the input data 910 . ), a convolution operation may be performed by applying the _{kernel 920, P 0,0} to pixels included in the . That is, by multiplying and summing pixel values included in the upper left 8 x 8 region 915 by weight values included in the kernel 920 , one pixel value 931 mapped to the upper left 8 x 8 region is generated. can

In addition, pixel values included in the 8×8 region 925 moved by 8 pixels from the upper left 8×8 region 915 of the input data 910 in the width direction (right) and included in the kernel 920 By multiplying and summing the weight values, one pixel value 932 mapped to the 8×8 area 925 may be generated.

In the same manner, the object of the convolution operation in the input data 910 is moved from left to right (in the width direction) and from the top to the bottom (in the height direction) by 8 pixels, and included in the kernel 920 By multiplying and summing the weighted values, pixel values (eg, a third pixel value and a fourth pixel value) may be generated. Accordingly, in response to the input data 910 , a 2×2 hidden feature F_hidden _0,0,0 may be output.

In FIG. 9 , for convenience of explanation, only one channel image of the first image F_in has been described, but convolution may be performed on each of the other channel images in the same manner as described in FIG. 9 .

In addition, for convenience of explanation, in FIG. 9 , W=H=16 and L=8 have been described as examples, but the present invention is not limited thereto. Hidden features (F_hidden _0,0 , _{F_hidden 1,0} ,..., _{F_hidden L-1,L-1} ) can be generated, and each of the hidden features has _{a size of W F} x H _F x Cin. can have In this case, W _F may be determined based on the width (W) of the first image and the stride size L in the width direction, and H _F is the height (H) of the first image and the stride size L in the height direction. can be decided.

The hidden features F_hidden _0,0 , _{F_hidden 1,0} ,..., _{F_hidden L-1,L-1} according to an embodiment include the first image F_in and the kernel P _0,0 , P _1,0 ,...,P _L-1,L-1 ), the features are generated by performing a convolution operation, and may be information in which features of neighboring pixels of a target pixel are also reflected.

Again, referring to FIG. 8 , the convolution unit 720 includes the n hidden features F_hidden _0,0 , _{F_hidden 1,0} ,..., _{F_hidden L-1 generated by the feature extraction unit 710 . , L-1} ) and the kernels can be convolved, and the convolution operation can be expressed by Equation (5).

[Equation 5]

F_out _u,v =Conv2D(input=F_hidden _u,v , kernel=K _u,v , stride=1)

In Equation 5, F_hidden _u,v represents one of n hidden features, K _u,v represents a kernel corresponding to the input hidden feature, and the size of the stride is 1.

For example, a convolution unit 720, a first hidden characterized by applying the first kernel (K _0,0) to _(0,0 F_hidden), the first hidden features (F_hidden _0,0) with the first kernel ( By performing a convolution operation with K _0,0 ), the first output feature F_out _0,0 may be generated. Also, the first kernel K _0,0 may include C _out sub-kernels, and one sub-kernel may have a size _{of W K} x H _K x C _{in .} The number of channels of one sub-kernel may be equal to the number of channels (Cin) of the hidden feature.

Also, the convolution unit 720, a second hidden characterized by applying a second kernel (K _1,0) to _(1,0 F_hidden), the second hidden features (F_hidden _1,0) with the second kernel (K _{1 ,0} ), the second output feature F_out _1,0 may be generated. In addition, the convolution unit 720 applies the n _{-th kernel (K L-1,L-1 ) to the n-th hidden feature (F_hidden L-1,L-1} ), and the n-th hidden feature (F_hidden _{L-1) ,L-1} ) and the n _{-th kernel (K L-1,L-1} ) are convolved to generate an n-th output feature (F_out _L-1,L-1). In addition, each of the second to nth kernels K _1,0 ,..., K _L-1,L-1 may include Cout sub-kernels, and one sub-kernel is W _K x H _K x Cin.

At this time, the width and height of each of the features output as a result of the convolution operation (F_out _0,0 , F_out _1,0 ,..., F_out _L-1,L-1 ) are the same as the hidden feature, and the channel of the output feature The number may be determined based on the number of sub-kernels included in one kernel. For example, if each of the hidden features according to an embodiment _{has a size of W F} x H _F x C _in and the number of sub-kernels included in one kernel is Cout, each of the output N features is W _F It may have a size of x H _{F x Cout.}

Also, the pixel synthesizer 730 according to an embodiment may synthesize the n features output from the convolution unit 720 to generate the second image F_out. The pixel combining unit 730 performs the convolution performed by the feature extracting unit 710 to generate the hidden features F_hidden _0,0 , _{F_hidden 1,0} ,..., _{F_hidden L-1,L-1 ).} The second image F_out may be generated based on the size L of the stride applied to the solution operation. The pixel synthesizer 730 may generate the second image F_out based on Equation 3 .

For example, the pixel synthesizer 730 performs the first image F_in and the kernel P _0,0 , P _1,0 ,...,P _L-1,L-1 in the feature extraction unit 710 . When performing a convolution operation with and, if the stride size is L, the second image F_out is divided into blocks of size L x L, and output features F_out _0,0, F_out _{1,0, .. .,} F_out _{L-1, L-1} ) may be determined as pixel values of pixels having the same position in each of the blocks. Since this has been described in detail with reference to FIG. 6 , a detailed description thereof will be omitted.

Meanwhile, at least one of the feature extraction unit 710 , the convolution unit 720 , and the pixel synthesis unit 730 illustrated and described in FIGS. 7 to 9 may be manufactured in the form of a hardware chip and mounted in an image processing apparatus. have. For example, at least one of the feature extraction unit 710, the convolution unit 720, and the pixel synthesis unit 730 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), Alternatively, it may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted on the image processing apparatus 100 according to an embodiment.

In addition, at least one of the feature extraction unit 710 , the convolution unit 720 , and the pixel synthesis unit 730 may be implemented as a software module. When at least one of the feature extraction unit 710 , the convolution unit 720 , and the pixel synthesis unit 730 is implemented as a software module (or a program module including instructions), the software module is It may be stored in a readable non-transitory computer readable medium that can be read by a user. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

10 is a flowchart illustrating a method of operating an image processing apparatus according to an exemplary embodiment.

Referring to FIG. 10 , the image processing apparatus 100 according to an embodiment may extract n pieces of first characteristic information from the first image based on positions of pixels included in the first image ( S1010 ).

For example, the image processing apparatus 100 may generate n sampling features by grouping pixels based on position information of pixels included in the first image. The image processing apparatus 100 may generate n sampling features by dividing the first image into blocks of a preset size (L x L) and sampling pixels having the same position in each of the blocks into the same group. In this case, the preset size (sampling period in the width direction and the height direction) of the block may be variously set. Since the method of generating the n sampling features has been described in detail with reference to FIG. 3 , the same description will be omitted.

Alternatively, the image processing apparatus 100 may generate n hidden features by applying a stride size L and performing a convolution operation between the first image and kernels. Since the method of generating the n hidden features has been described in detail with reference to FIGS. 8 and 9 , the same description will be omitted.

The image processing apparatus 100 according to an embodiment may generate n pieces of second characteristic information by performing a convolution operation on each of the n pieces of first characteristic information and each of the n kernels ( S1020 ).

For example, the image processing apparatus 100 generates n output features by performing convolution (group convolution) between each of the n sampling features generated in step S1010 and each of the n kernels. can do. In this case, each of the n kernels may include Cout (the number of channels of the output feature) sub-kernels, and one sub-kernel may _{have a size of W K} x H _K x Cin (the number of channels of the sampling feature). have.

Alternatively, the image processing apparatus 100 may generate n output features by performing convolution (group convolution) between each of the n hidden features generated in step S1020 and each of the n kernels. have. In this case, each of the n kernels may include Cout (the number of channels of the output feature) sub-kernels, and one sub-kernel may _{have a size of W K} x H _K x Cin (the number of channels of the hidden feature). have.

Since the method of performing the convolution operation has been described in detail with reference to FIG. 5 , the same description will be omitted.

The image processing apparatus 100 according to an embodiment may generate a second image based on n pieces of second characteristic information (S1030).

For example, the image processing apparatus 100 may generate the second image based on the sampling period L in the width and height directions used when sampling pixels included in the first image in step S1010 ( S1010 ). have.

Alternatively, the image processing apparatus 100 may generate the second image based on the stride size L applied to the convolution operation performed to generate the hidden features in operation S1010 ( S1010 ).

The image processing apparatus 100 divides the second image into blocks having a size of L x L, and sets pixel values included in each of the n output features generated in operation S1020 ( S1020 ) to pixels having the same position in each of the blocks. can be determined by the pixel values of

Since the method of generating the second image has been described in detail with reference to FIG. 6 , the same description will be omitted.

11 is a block diagram illustrating a configuration of an image processing apparatus according to an exemplary embodiment.

Referring to FIG. 11 , the image processing apparatus 100 according to an embodiment may include a processor 120 and a memory 130 .

The processor 120 according to an embodiment may control the image processing apparatus 100 as a whole. The processor 120 according to an embodiment may execute one or more programs stored in the memory 130 .

The memory 130 according to an embodiment may store various data, programs, or applications for driving and controlling the image processing apparatus 100 . A program stored in the memory 130 may include one or more instructions. A program (one or more instructions) or an application stored in the memory 130 may be executed by the processor 120 .

The processor 120 according to an embodiment may include at least one of a central processing unit (CPU), a graphic processing unit (GPU), and a video processing unit (VPU). Alternatively, according to an embodiment, it may be implemented in the form of a system on chip (SoC) in which at least one of a CPU, a GPU, and a VPU is integrated. Alternatively, the processor ( ) may further include a Neural Processing Unit (NPU).

The processor 120 according to an embodiment may generate a second image from which compression artifacts are removed from the first image including compression artifacts by using an artifact removal network including one or more convolutional neural networks.

For example, the processor 120 may perform at least one of the operations of the pixel sampling unit 300 , the convolution unit 400 , and the pixel synthesis unit 500 illustrated and described with reference to FIGS. 2 to 6 . , at least one of the operations of the feature extraction unit 710 , the convolution unit 720 , and the pixel synthesis unit 930 illustrated and described with reference to FIGS. 7 to 9 may be performed. The processor 120 may extract n pieces of first characteristic information based on positions of pixels included in the first image. For example, the processor 120 may divide the first image into blocks of a preset size (L x L) and extract pixels having the same position from each of the blocks as one sampling feature. Accordingly, n (= L x L) sampling features can be generated. Alternatively, the processor 120 may generate n hidden features by applying the stride size L and performing a convolution operation between the first image and the kernels.

Also, the processor 120 may generate n pieces of second characteristic information by performing a convolution operation with each of the n pieces of first characteristic information and each of the n kernels.

Also, the processor 120 may generate a second image from which compression artifacts are removed, based on the n pieces of second characteristic information. The processor 120 may divide the second image into blocks having a size of L x L, and determine pixel values included in each of the generated n pieces of second characteristic information as pixel values of pixels having the same position in each of the blocks. have.

Meanwhile, the artifact removal network 30 according to an embodiment may be a network trained by a server or an external device. The external device may train the de-artifacting network 30 based on the training data. In this case, the training data may include a plurality of data sets including image data including compression artifacts and image data from which compression artifacts are removed.

The server or the external device may determine weight values included in kernels used in each of a plurality of convolutional layers included in the de-artifacting network 30 . For example, the server or external device may determine the weight values in a direction that minimizes the difference (loss information) between the image data generated by the de-artifacting network 30 and the image data from which compression artifacts are removed as training data. .

The image processing apparatus 100 according to an embodiment may receive the trained artifact removal network 30 from a server or an external device and store it in the memory 130 . For example, the memory 130 may store the structure and weight values of the de-artifact network 30 according to an embodiment, and the processor 120 uses the weight values stored in the memory 130, according to an embodiment. A second image from which compression artifacts are removed may be generated from the first image according to .

Meanwhile, a block diagram of the image processing apparatus 100 illustrated in FIG. 11 is a block diagram for an exemplary embodiment. Each component of the block diagram may be integrated, added, or omitted according to specifications of the image processing apparatus 100 that are actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components as needed. In addition, the function performed in each block is for describing the embodiments, and the specific operation or device does not limit the scope of the present invention.

The method of operating an image processing apparatus according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Also, the image processing apparatus and the method of operating the image processing apparatus according to the disclosed embodiments may be included in a computer program product and provided. Computer program products may be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, computer program products may include products (eg, downloadable apps) in the form of S/W programs distributed electronically through manufacturers of electronic devices or electronic markets (eg, Google Play Store, App Store). have. For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product, in a system consisting of a server and a client device, may include a storage medium of the server or a storage medium of the client device. Alternatively, when there is a third device (eg, a smart phone) that is communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the client device or the third device, or transmitted from the third device to the client device.

In this case, one of the server, the client device and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a client device, and a third device may execute a computer program product to distribute the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control a client device communicatively connected with the server to perform the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also included in the scope of the present invention. belongs to

Claims (17)

In the image processing apparatus,
a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor, using one or more neural networks,
Based on the positions of pixels included in the first image, n pieces of first feature information are extracted from the first image, and a convolution operation is performed between each of the n pieces of first feature information and each of the n kernels. By doing so, n pieces of second characteristic information are generated, and a second image is generated based on the n pieces of second characteristic information,
The second image is an image from which compression artifacts included in the first image are removed. According to claim 1,
The processor is
and dividing the first image into blocks having a preset size, and dividing pixels having the same position in each of the blocks into the same group to extract the n pieces of first characteristic information. 3. The method of claim 2,
each of the blocks includes the n pixels,
The n is determined based on a preset size of the blocks. 3. The method of claim 2,
The processor is
dividing the second image into blocks having the preset size;
and determining pixel values included in one piece of second characteristic information among the second characteristic information as pixel values at the same position in each of the blocks. According to claim 1,
The processor is
An image processing apparatus for extracting the n pieces of first feature information by performing a convolution operation between the first image and a second kernel by applying a stride size L. 6. The method of claim 5,
The image processing apparatus, wherein the number n of the first characteristic information is determined based on the stride size L. According to claim 1,
The n kernels are different kernels from each other. According to claim 1,
The image processing apparatus of claim 1 , wherein the size of the first image and the size of the second image are the same. A method of operating an image processing apparatus for performing image processing using one or more neural networks, the method comprising:
extracting n pieces of first feature information from the first image based on positions of pixels included in the first image;
generating n pieces of second feature information by performing a convolution operation with each of the n pieces of first feature information and each of the n pieces of kernels; and
generating a second image based on the n pieces of second characteristic information;
The second image is an image from which compression artifacts included in the first image are removed. 10. The method of claim 9,
The step of extracting the n pieces of first characteristic information includes:
and extracting the n pieces of first characteristic information by dividing the first image into blocks having a preset size and dividing pixels having the same position in each of the blocks into the same group Way. 11. The method of claim 10,
each of the blocks includes the n pixels,
and n is determined based on a preset size of the blocks. 11. The method of claim 10,
The generating of the second image comprises:
dividing the second image into blocks having the preset size, and determining pixel values included in one piece of second characteristic information among the second characteristic information as pixel values at the same position in each of the blocks; Including, an operating method of an image processing apparatus. 10. The method of claim 9,
The step of extracting the n pieces of first characteristic information includes:
and extracting the n pieces of first feature information by performing a convolution operation between the first image and the second kernel by applying a stride size L. 14. The method of claim 13,
The method of operating an image processing apparatus, wherein the number n of the first characteristic information is determined based on the stride size L. 10. The method of claim 9,
The n kernels are different kernels from each other, the operating method of the image processing apparatus. 10. The method of claim 9,
and sizes of the first image and the second image are the same. A computer program product comprising one or more computer-readable recording media in which a program for performing the method of claim 9 is stored.

Images