KR20230157220A - Image processing apparatus and operating method for the same - Google Patents

Abstract

An image processing device that processes an image using one or more neural networks according to an embodiment includes a memory that stores one or more instructions, and a first feature based on the first image by executing one or more instructions stored in the memory. Obtain data, perform first image processing on the first feature data, obtain second feature data corresponding to first areas including a first number of pixels, and third feature based on the first image. Obtain data, perform second image processing on the third feature data, obtain fourth feature data corresponding to second areas including a second number of pixels greater than the first number, and obtain second feature data. and at least one processor that generates a second image based on the first and fourth feature data.

Description

Image processing apparatus and operating method for the same}

Various embodiments relate to an image processing device that processes images using a neural network and a method of operating the same.

As data traffic increases exponentially with the development of computer technology, artificial intelligence has become an important trend leading future innovation. Because artificial intelligence imitates human thinking, it can be applied virtually to all industries. Representative technologies of artificial intelligence include pattern recognition, machine learning, expert systems, neural networks, and natural language processing.

A neural network models the characteristics of human biological nerve cells using mathematical expressions and uses an algorithm that imitates the learning ability of humans. Through this algorithm, the neural network can create a mapping between input data and output data, and the ability to create this mapping can be expressed as the learning ability of the neural network. In addition, the neural network has a generalization ability to generate correct output data for input data that was not used for learning, based on the learned results.

Neural networks can be used in image processing, and in particular, deep neural networks (DNNs) can be used to perform image processing to remove noise or artifacts from images or increase the resolution of images.

An image processing device according to an embodiment may process images using one or more neural networks.

An image processing device according to an embodiment may include a memory that stores one or more instructions and at least one processor that executes the one or more instructions.

At least one processor according to an embodiment may acquire first feature data based on the first image by executing the one or more instructions stored in the memory.

At least one processor according to an embodiment performs first image processing on the first feature data by executing the one or more instructions stored in the memory, thereby corresponding to first areas including a first number of pixels. Second characteristic data that can be obtained can be obtained.

At least one processor according to an embodiment may acquire third characteristic data based on the first image by executing the one or more instructions stored in the memory.

At least one processor according to an embodiment performs a second image processing on the third characteristic data by executing the one or more instructions stored in the memory, and includes a second number of pixels greater than the first number. Fourth feature data corresponding to the second areas may be obtained.

At least one processor according to an embodiment may generate a second image based on the second feature data and the fourth feature data by executing the one or more instructions stored in the memory.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment may include acquiring first feature data based on a first image.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment includes performing first image processing on the first feature data to create a first region including a first number of pixels. It may include obtaining second feature data corresponding to the features.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment may include acquiring third feature data based on the first image.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment includes performing a second image processing on the third feature data to produce a second number of pixels greater than the first number. It may include acquiring fourth feature data corresponding to second areas including .

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment includes generating a second image based on the second feature data and the fourth feature data. It can be included.

A computer-readable recording medium according to an embodiment includes at least one instruction for performing, by a computer, a method of operating an image processing device that processes an image using one or more neural networks according to an embodiment. You can save the program.

FIG. 1 is a diagram illustrating an operation of an image processing device processing an image using an image processing network, according to an embodiment.
Figure 2 is a diagram showing a second feature extraction network according to an embodiment.
Figure 3 is a diagram showing a transform block according to one embodiment.
Figure 4 is a diagram showing a first transform layer according to one embodiment.
Figure 5 is a diagram showing a self-attention operation performed in a first self-attention module according to an embodiment.
Figure 6 is a diagram showing a first self-attention module according to an embodiment.
Figure 7 is a diagram showing a multi-perceptron module according to an embodiment.
Figure 8 is a diagram showing a second transform layer according to one embodiment.
FIG. 9 is a diagram illustrating a self-attention operation performed in a second self-attention module according to an embodiment.
Figure 10 is a diagram showing a second self-attention module according to an embodiment.
Figure 11 is a diagram showing a second self-attention module according to an embodiment.
Figure 12 is a diagram showing a transform block according to an embodiment.
Figure 13 is a diagram showing a transform block according to an embodiment.
Figure 14 is a flowchart showing a method of operating an image processing device according to an embodiment.
Figure 15 is a block diagram showing the configuration of an image processing device according to an embodiment.

The terms used in this specification will be briefly explained, and the present invention will be described in detail.

The terms used in the present invention are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

When it is said that a part "includes" a certain element throughout the specification, this means that, unless specifically stated to the contrary, it does not exclude other elements but may further include other elements. In addition, terms such as "... unit" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. .

Below, with reference to the attached drawings, embodiments will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

FIG. 1 is a diagram illustrating an operation of an image processing device processing an image using an image processing network, according to an embodiment.

Referring to FIG. 1, the image processing network 103 according to an embodiment may receive a first image 101 and process the first image 101 to generate a second image 102. At this time, the first image 101 may be an image containing noise or artifacts, and may be a low-resolution image or low-quality image. The image processing device 100 uses the image processing network 103 to perform denoising to remove noise while maintaining the detailed edges and texture of the first image 101, thereby producing a second image. (102) can be generated. Additionally, the second image 102 may be a higher resolution image than the first image 101. Additionally, the second image 102 may be an image with improved image quality than the first image 101. However, it is not limited to this.

The image processing network 103 according to one embodiment may include one or more neural networks. For example, the image processing network 103 may include a first feature extraction network 200, a second feature extraction network 300, and an image restoration network 400. However, it is not limited to this.

The first feature extraction network 200 is a network that maps the image space in which the first image 101 (input image) exists to a feature space, and includes one or more convolutional neural networks (CNN). can do. In addition, the second feature extraction network 300 generates higher-dimensional features than the features extracted from the first feature extraction network 200, based on the features extracted from the first feature extraction network 200. It may be an extracting network.

Additionally, the image restoration network 400 may be a network that obtains the second image 102 (output image) using high-dimensional features. Image restoration network 400 may include one or more convolutional neural networks (CNNs). The image restoration network 400 according to an embodiment may perform up-sampling to generate a second image with higher resolution than the first image.

However, it is not limited to this, and the image processing network 103 according to one embodiment may include various neural networks.

Hereinafter, an image processing network according to an embodiment will be described in detail with reference to the drawings.

Figure 2 is a diagram showing a second feature extraction network according to an embodiment.

Referring to FIG. 2, the second feature extraction network 300 according to one embodiment may include one or more transform blocks 310, a convolution layer 320, and a summation layer 330.

Each of the transform blocks 310 according to an embodiment contains information about surrounding pixels adjacent to each of the pixels included in the input first feature data and each of predetermined unit areas included in the input first feature data. Second feature data can be extracted using information about adjacent surrounding areas. The conversion block will be described in detail with reference to FIG. 3.

In the convolution layer 320 according to one embodiment, a convolution operation may be performed between input data (or input information) input to the convolution layer 320 and a kernel included in the convolution layer 320. At this time, the input data input to the convolution layer 320 may be result data output after an operation is performed in the transform blocks 310. In addition, although the second feature extraction network 300 is shown in FIG. 2 as including one convolution layer 320, it is not limited to this and may include two or more convolution layers.

In the summing layer 330 according to one embodiment, a summation operation may be performed for each element of the input data (x1) input to the second feature extraction network 300 and the output data output from the convolution layer 320. Output data (y1) output from the summation layer 330 may be input to the image restoration network 400 of FIG. 1. However, it is not limited to this.

Figure 3 is a diagram showing a transform block according to one embodiment.

The transform block 311 shown in FIG. 3 may be any one of the transform blocks 310 in FIG. 2 . Referring to FIG. 3, the transform block 311 according to one embodiment may include one or more first residual transform blocks and one or more second residual transform blocks.

The first residual transform block 350 according to one embodiment may be referred to as an intra residual transform block (aRTB), and the second residual transform block 370 may be referred to as an inter residual transform block (aRTB). It may be referred to as inter Residual Transformer Block (eRTB).

Additionally, the transform block 311 according to one embodiment may include a structure in which the first residual transform block 350 and the second residual transform block 370 are arranged at an intersection. However, the present invention is not limited to this, and the transform block 311 may include a structure in which the first residual transform block 350 and the second residual transform block 370 are arranged in parallel.

The first residual transform block 350 may include one or more first transform layers 351, a convolution layer 352, and a summing layer 353. The first conversion layer will be described in detail with reference to FIG. 4.

In the convolution layer 352 according to one embodiment, a convolution operation may be performed between input data (or input information) input to the convolution layer 352 and a kernel included in the convolution layer 352. At this time, the input data input to the convolution layer 352 may be result data output after an operation is performed in the first transformation layers 351. In addition, in FIG. 3, the first residual transform block 350 is shown as including one convolution layer 352, but it is not limited to this, and the first residual transform block 350 includes two or more convolution layers. It may also contain layers.

In the summing layer 353 according to one embodiment, an element-by-element summation operation may be performed on the input data input to the first residual transform block 350 and the output data output from the convolution layer 352. Data output from the summing layer 353 may be input to the second residual transform block 370. However, it is not limited to this.

Additionally, referring to FIG. 3, the second residual transform block 370 may include one or more second transform layers 371, a convolution layer 372, and a summation layer 373. Here, the second transform layer will be described in detail with reference to FIG. 8.

In the convolution layer 372 according to one embodiment, a convolution operation may be performed between input data (or input information) input to the convolution layer 372 and a kernel included in the convolution layer 372. At this time, the input data input to the convolution layer 372 may be result data output after an operation is performed in the second transformation layers 371. In addition, in FIG. 3, the second residual transform block 370 is shown as including one convolution layer 372, but it is not limited to this, and the second residual transform block 370 includes two or more convolution layers. It may also contain layers.

In the summing layer 373 according to one embodiment, an element-by-element summation operation may be performed on the input data input to the second residual transform block 370 and the output data output from the convolution layer 372. Data output from the summing layer 373 may be input to the first residual transform block located after the second residual transform block 370. However, it is not limited to this.

Figure 4 is a diagram showing a first transform layer according to one embodiment.

The first transform layer 410 shown in FIG. 4 may be one of the first transform layers 351 of FIG. 3 . Referring to FIG. 4, the first transformation layer 410 according to one embodiment includes a first normalization layer 411, a first self-attention module 412, a first summation layer 413, and a second normalization layer 414. ), a multi-layer perceptron (MLP) module 415, and a second summation layer 416.

The first normalization layer 411 according to one embodiment may normalize input data (x2) input to the first transformation layer 410. For example, the first normalization layer 411 may normalize the input data (x2) so that the sum of the input data input to the first transformation layer 410 is 1. However, it is not limited to this, and input data can be normalized using various normalization methods. Normalized input data may be input to the first self-attention module 412.

The first self-attention module 412 according to an embodiment performs a self-attention operation on the first feature data (e.g., normalized input data) input to the first self-attention module 412, and calculates the first number Second feature data corresponding to the first areas including pixels may be obtained. At this time, the first number may be one pixel.

Here, the attention operation acquires association information (e.g., similarity information) between query data (query, Q) and key data (key, K), obtains a weight based on the association information, and applies the weight to the key data. This refers to an operation that reflects on the value data (V) mapped to (K) and performs a weighted sum on the value data (V) with the weight reflected.

At this time, the attention operation performed based on query data (Q), key data (K), and value data (V) obtained from the same input data may be referred to as a self-attention operation.

The self-attention operation performed in the first self-attention module 412 according to an embodiment will be described in detail with reference to FIGS. 5 and 6.

Figure 5 is a diagram showing a self-attention operation performed in a first self-attention module according to an embodiment.

The first self-attention module 412 according to one embodiment may be referred to as an intra Multi-head Self-Attention (MSA) module.

Referring to FIG. 5, the first self-attention module 412 generates query data (Q), key data (K), and value based on the first input data 510 input to the first self-attention module 412. Data (V) can be obtained.

For example, the size of the first input data 510 may be W x H, and the number of channels may be C. In FIG. 5 , for convenience of explanation, it will be described as an example that the first input data 510 includes one channel (C=1).

The first self-attention module 412 may perform self-attention processing on pixels included in the first input data 510 in units of patches having a size of M x M. For example, the first self-attention module 412 may perform self-attention processing in units of M ² pixels included in one patch. In Figure 5, for convenience of explanation, the self-attention operation will be explained based on the pixels (x ₁ , x ₂ , ..., x _K ) included in one patch. Here K is M ² . Referring to FIG. 5, the pixels (x 1 _, x 2) are calculated by multiplying the pixels (x ₁ , x ₂ , ..., x _K ) included _in one patch and the first weight matrix (W _Q1 ) , ..., x _K ), respectively, and corresponding query data 521 may be obtained. _In addition, each of the pixels (x ₁ , x ₂ , ..., x _K ) is calculated through a multiplication operation between the pixels (x _{1 ,} x ₂ , ..., Corresponding key data 522 may be obtained. _In addition, each of the pixels ₍ x ₁ , x ₂ , ..., x _K ) is calculated through a multiplication operation between the pixels (x ₁ , x ₂ , ..., Corresponding value data 523 may be obtained.

Relevance data (E, 530) can be obtained through an element-by-element multiplication operation of the key data 522 and the query data 521. For example, correlation data (E, 530) can be calculated by Equation 1 below.

In Equation 1, e _ij represents the element at the (i, j) position in the correlation data (E, 530), and q _i represents query data corresponding to the ith pixel (xi) among the query data 521. , k _j represents key data corresponding to the jth pixel (xj) among the key data 522.

The first self-attention module 412 may obtain weight data (A, 540) by applying a softmax function to the correlation data (E, 530). For example, weight data (A, 540) can be calculated by Equation 2 below.

In Equation 2, a _ij represents the element at the (i, j) position in the weight data (A, 540), and e _ij represents the element at the (i, j) position in the correlation data (E, 530).

The first self-attention module 412 performs a weighted sum of the weight data (A, 540) and the value data 523, thereby generating the first self-attention module corresponding to the pixels (x ₁ , x ₂ , ..., x _K ). 1 Output data 550 can be obtained. For example, the first output data 550 may be calculated by Equation 3 below.

In Equation 3, y _i represents a feature value corresponding to pixel x _i included in the pixels (x ₁ , x ₂ , ..., x _K ) in the first output data 550.

Figure 6 is a diagram showing a first self-attention module according to an embodiment.

Referring to FIG. 6, the first self-attention module 412 according to one embodiment may include three linear layers 611, 612, and 613 configured in parallel. The three linear layers 611, 612, and 613 may be fully connected layers.

For example, the product corresponding to the first input data (x, 510) is obtained through a multiplication operation between the first input data (x, 510) and the first weight matrix (W _Q1 ) included in the first linear layer 611. Query data (Q) may be obtained. Additionally, key data (K) may be obtained through a multiplication operation between the first input data (x, 510) and the second weight matrix (W _K1 ) included in the second linear layer 612. Additionally, value data (V) may be obtained through a multiplication operation between the first input data (x, 510) and the third weight matrix (W _V1 ) included in the third linear layer 613.

The first self-attention module 412 can obtain the first correlation data (E1) through an element-by-element multiplication operation of the data (K ^T ) and the query data (Q) by applying a transpose function to the key data (K). .

The first self-attention module 412 may obtain second correlation data (E2) by adding the position bias (B) to the first correlation data (E1) and applying a window mask.

Here, the position bias (B) can be determined by the following equation.

Here, B[] represents the position bias to be applied to the first correlation data, and d(x _i , x _j ) may mean the distance between pixel x _i and pixel x _j . Additionally, B _train [d(x _i , x _j )] is a value determined through training of a neural network and may be a previously stored value.

Meanwhile, reflection padding may be applied to the first input data (x, 510) input to the first self-attention module 412. Reflection padding is used when the size of the first input data (x, 510) is not a multiple of the patch size, for example, the width W of the first input data (x, 510) is not a multiple of the patch size M, or the first This means that if the height H of the input data (x, 510) is not a multiple of the patch size M, padding is performed so that the size of the first input data is a multiple of the patch size. For example, if the size (resolution) of the first input data (x, 510) is 126 x 127 and the patch size M = 8, the first self-attention module 412 performs reflection padding to The size (resolution) of data (x, 510) can be set to 128 x 128.

Additionally, by applying a window mask, the effect of shifting the first input data (x, 510) can be achieved. For example, the first self-attention module 412 may shift the partitioning position of the patch by applying a window mask. By shifting the position where patches are divided, patches can be configured more diversely, and as the configuration of patches containing one pixel becomes more diverse, pixels adjacent to that pixel can be configured more diversely.

The first self-attention module 412 may obtain weight data (A) by applying the softmax function to the second correlation data (E2).

The first self-attention module 412 may obtain first output data (y) by performing a weighted sum of weight data (A) and value data (V).

Referring again to FIG. 4 , the first output data (y) output from the first self-attention module 412 may be output to the first summation layer 413. In the first summing layer 413, an element-by-element summation operation is performed on the first output data (y) output from the first self-attention module 412 and the input data (x2) input to the first conversion layer 410. You can.

The second normalization layer 414 may normalize the second output data output from the first summing layer 413. Normalized data may be input into the multi-perceptron module 415. The multi-perceptron module 415 will be described in detail with reference to FIG. 7.

Figure 7 is a diagram showing a multi-perceptron module according to an embodiment.

Referring to FIG. 7 , the multi-perceptron module 415 according to one embodiment may include a first linear layer 710, a normalization layer 720, and a second linear layer 730.

In the first linear layer 710, third output data may be obtained through a multiplication operation between data input to the first linear layer 710 and the first weight matrix included in the first linear layer 710. The third output data may be input to the normalization layer 720. The normalization layer 720 can normalize data input to the normalization layer 720. Normalized data may be input to the second linear layer 720.

In the second linear layer 720, fourth output data may be obtained through a multiplication operation between data input to the second linear layer 720 and a second weight matrix included in the second linear layer 720.

Meanwhile, in FIG. 7, the multi-perceptron module 415 is shown as including two linear layers, but it is not limited to this and may include three or more linear layers.

Referring again to FIG. 4, the fourth output data obtained from the multi-perceptron module 415 may be output to the second summation layer 416. In the second summation layer 416, an element-by-element sum operation may be performed on the fourth output data output from the multi-perceptron module 415 and the second output data output from the first summation layer 413.

Figure 8 is a diagram showing a second transform layer according to one embodiment.

The second transform layer 810 shown in FIG. 8 may be one of the second transform layers 371 of FIG. 3 . Referring to FIG. 8, the second transform layer 810 according to one embodiment includes a first normalization layer 811, a second self-attention module 812, a first summation layer 813, and a second normalization layer 814. ), a multi-perceptron (MLP) module 815, and a second summation layer 816.

The first normalization layer 811 according to one embodiment may normalize input data (x3) input to the second transformation layer 810. For example, the first normalization layer 811 may normalize the input data so that the sum of the input data input to the second transformation layer 810 is 1. However, it is not limited to this, and input data can be normalized using various normalization methods. Normalized input data may be input to the second self-attention module 812.

The second self-attention module 812 according to an embodiment performs a self-attention operation on the third feature data (e.g., normalized input data) input to the second self-attention module 812 to obtain a second number. Third feature data corresponding to the second areas including pixels may be obtained. At this time, the first number may be larger than the first number described in FIG. 5, the second area is larger than the first area, and each of the second areas may include a plurality of pixels.

The self-attention operation performed in the second self-attention module 812 according to an embodiment will be described in detail with reference to FIGS. 9 and 10.

FIG. 9 is a diagram illustrating a self-attention operation performed in a second self-attention module according to an embodiment.

The second self-attention module 812 according to one embodiment may be referred to as an inter Multi-head Self-Attention (MSA) module.

Referring to FIG. 9, the second self-attention module 812 generates query data (Q), key data (K), and value based on the second input data 910 input to the second self-attention module 812. Data (V) can be obtained.

For example, the size of the second input data 910 may be W x H, and the number of channels may be C. In FIG. 9 , for convenience of explanation, it will be described as an example that the second input data 910 includes one channel (C=1). If the second input data 910 is divided into regions (patches) including a predetermined number (eg, M ² ) of pixels, the number of patches included in one channel may be N.

The second self-attention module 812 according to an embodiment may obtain feature information corresponding to each of the patches (P ₁ , P ₂ , ..., P _N ).

For example, through the product operation (W _Q2 ) of the patches (920, P ₁ , P ₂ , ..., P _N ) included in the second input data 910 and the first weight matrix, the second input data Query data 921 corresponding to each of the patches 920, P ₁ , P ₂ , ..., P _N included in 910 may be obtained.

In addition, the second input data 910 is calculated by multiplying the patches 920, P ₁ , P ₂ , ..., P _N included in the second input data 910 and the second weight matrix W _K2 . ), key data 922 corresponding to each of the patches 920, P ₁ , P ₂ , ..., P _N may be obtained.

In addition, the second input data 910 is calculated through a multiplication operation between the patches 920, P ₁ , P ₂ , ..., P _N included in the second input data 910 and the third weight matrix W _V2 . ), value data 923 corresponding to each of the patches 920, P ₁ , P ₂ , ..., P _N may be obtained.

Relevance data (E, 930) may be obtained through an element-by-element multiplication operation of the key data 922 and the query data 921. Since this has been explained in Equation 1 of FIG. 5, the same description will be omitted.

By applying the softmax function to the correlation data (E, 930), weight data (A, 940) can be obtained. Since this has been explained in Equation 2 of FIG. 5, the same explanation will be omitted.

The second self-attention module 812 may obtain second output data 950 by performing a weighted sum of the weight data (A, 940) and the value data 923. Since this has been explained in Equation 3 of FIG. 5, the same description will be omitted.

Figure 10 is a diagram showing a second self-attention module according to an embodiment.

Referring to FIG. 10 , the second self-attention module 812 according to one embodiment may include a first reshape layer 1010. In the transformation layer 1010, third input data can be obtained by reshaping the second input data so that pixels included in the second input data 910 (x) are grouped with pixels included in the same patch. . At this time, the third input data may be divided into patches 920 of FIG. 9.

The second self-attention module 812 according to one embodiment may include three linear layers 1021, 1022, and 1023 configured in parallel. The three linear layers 1021, 1022, and 1023 may be fully connected layers.

For example, query data (Q') corresponding to the third input data can be obtained through a multiplication operation between the third input data and the first weight matrix (W _Q2 ) included in the first linear layer 1021. there is. Additionally, key data K' may be obtained through a multiplication operation between the third input data and the second weight matrix W _K2 included in the second linear layer 1022. Additionally, value data (V') may be obtained through a multiplication operation between the third input data and the third weight matrix (W _V2 ) included in the third linear layer 1023.

The second self-attention module 812 generates the first correlation data (E1') through an element-by-element multiplication operation of the data ( ^K'T ) obtained by applying the transpose function to the key data (K') and the query data (Q'). It can be obtained.

The second self-attention module 812 may obtain second correlation data (E2') by adding the position bias (B') to the first correlation data (E1') and applying a window mask.

Here, the position bias (B') can be determined by Equation 5 as follows.

Here, B'[] represents the position bias to be applied to the first correlation data, and d(P _i , P _j ) may mean the distance between patch P _i and patch P _j . Additionally, B' _train [d(P _i , P _j )] is a value determined through training of a neural network and may be a previously stored value.

Additionally, by applying a window mask, the effect of shifting the second input data (x, 910) can be achieved. For example, the second self-attention module 812 may shift the partitioning position of the patch by applying a window mask. By shifting the position where patches are divided, patches can be configured in various ways. Accordingly, the configuration of patches adjacent to one patch may also be diversified.

The second self-attention module 812 may obtain weight data (A') by applying the softmax function to the second correlation data (E2').

The second self-attention module 812 may obtain second output data (y') by performing a weighted sum of the weight data (A') and the value data (V').

At this time, the second output data (y') may be divided into patches, and the second output data (y') is transformed into the third output data (y) in the second reshape layer 1030. It can be. The third output data (y) may be divided into pixels rather than patches.

Referring again to FIG. 8, the third output data (y) output from the second self-attention module 812 may be output to the first summation layer 813. In the first summing layer 813, an element-by-element summation operation is performed on the third output data (y) output from the second self-attention module 812 and the input data (x3) input to the second conversion layer 810. You can.

The second normalization layer 814 may normalize the fourth output data output from the first summing layer 813. Normalized data may be input into the multi-perceptron module 815. Since the multi-perceptron module 815 has been described in detail in FIG. 7, the same description will be omitted.

The fifth output data obtained from the multi-perceptron module 815 may be output to the second summation layer 816. In the second summation layer 816, an element-by-element summation operation may be performed on the fifth output data output from the multi-perceptron module 815 and the fourth output data output from the first summation layer 813.

Figure 11 is a diagram showing a second self-attention module according to an embodiment.

Referring to FIG. 11, the second self-attention module 812 according to one embodiment may include three linear layers 1111, 1112, and 1113 configured in parallel. The three linear layers 1111, 1112, and 1113 may be fully connected layers.

For example, query data (Q1') corresponding to the second input data is obtained through a multiplication operation between the second input data (910, x) and the first weight matrix (W _Q3 ) included in the first linear layer (1111). ) can be obtained. Additionally, key data K1' may be obtained through a multiplication operation between the second input data and the second weight matrix W _K3 included in the second linear layer 1112. Additionally, value data V1' may be obtained through a multiplication operation between the second input data and the third weight matrix W _V3 included in the third linear layer 1113.

The second self-attention module 812 according to one embodiment may include first to third deformation layers 1121, 1122, and 1123. For example, the first transformation layer 1121 is a structure connected to the first linear layer 1111 and transforms the first query data (Q1') output from the first linear layer 1111 to generate second query data ( Q') can be obtained. For example, the second query data (Q') may be in the form of grouping query values corresponding to pixels included in the first query data (Q1') with query values corresponding to pixels included in the same patch. You can. That is, the second query data (Q') may be divided into query data corresponding to patches.

In addition, the second transformation layer 1122 is connected to the second linear layer 1112 and transforms the first key data (K1') output from the second linear layer (1112) into second key data (K'). ) can be obtained. For example, the second key data K' may be in the form of grouping key values corresponding to pixels included in the first key data K1' with key values corresponding to pixels included in the same patch. You can. That is, the second key data K' may be divided into key data corresponding to patches.

In addition, the third transformation layer 1123 is connected to the third linear layer 1113 and transforms the first value data (V1') output from the third linear layer (1113) into second value data (V'). ) can be obtained. For example, the second value data (V') may be in the form of grouping value values corresponding to pixels included in the first value data (V1') with value values corresponding to pixels included in the same patch. You can. That is, the second value data (V') may be divided into value data corresponding to patches.

The second self- ^attention module 812 generates first correlation data ( E1') can be obtained.

The second self-attention module 812 may obtain second correlation data (E2') by adding the position bias (B') to the first correlation data (E1') and applying a window mask.

The second self-attention module 812 may obtain weight data (A') by applying the softmax function to the second correlation data (E2').

The second self-attention module 812 may obtain second output data (y') by performing a weighted sum of the weight data (A') and the value data (V').

At this time, the second output data (y') may be divided into patches, and the second output data (y') is transformed into the third output data (y) in the fourth reshape layer 1130. It can be. The third output data (y) may be divided into pixels rather than patches.

In the case of the second self-attention module 812 of FIG. 10, the size of the second input data (x) may be H x W x C, and is input to the first to third linear layers 1021, 1022, and 1023. The size of the modified third input data may be N x M ² x C. At this time, N represents the number of patches, M ² represents the size of one patch, C represents the number of channels, and H x W is equal to N x M ² . Accordingly, the number of parameters of the weight matrix included in each of the first to third linear layers 1021, 1022, and 1023 may be M ² C x M ² C.

On the other hand, in the case of the second self-attention module 812 of FIG. 11, the size of the second input data input to the first to third linear layers 1111, 1112, and 1113 is H x W x C, accordingly , the number of parameters of the weight matrix included in each of the first to third linear layers 1111, 1112, and 1113 may be C x C.

Therefore, as shown in FIG. 11, in the case where the second input data is operated on linear layers and then transformed to be grouped into data corresponding to patches, as shown in FIG. 10, the second input data is transformed to be grouped into patches. Afterwards, the amount of computation and the number of computation parameters can be reduced compared to the case of computation in linear layers. Accordingly, the second self-attention module of FIG. 11 can perform image processing using a smaller amount of calculation and calculation parameters while maintaining the performance of image processing (inter-self attention).

Figure 12 is a diagram showing a transform block according to an embodiment.

The transform block 1200 shown in FIG. 12 may be one of the transform blocks 310 of FIG. 2 .

Referring to FIG. 12, the transform block 1200 according to one embodiment includes a first residual transform block 1210, a second residual transform block 1220, a connection layer 1230, a normalization layer 1240, and a linear transform block 1200. It may include a layer 1250 and a summing layer 1260.

The transform block 1200 according to one embodiment may include a structure in which the first residual transform block 1210 and the second residual transform block 1220 are connected in parallel.

The first residual transform block 1210 according to one embodiment may be referred to as an intra residual transform block (aRTB), and the second residual transform block 1220 may be referred to as an inter residual transform block (aRTB). It may be referred to as inter Residual Transformer Block (eRTB). The first residual transform block 1210 corresponds to the first residual transform block 350 in FIG. 3, and the second residual transform block 1220 corresponds to the second residual transform block 370 in FIG. 3. It can be. Accordingly, the first transform layer 410 included in the first residual transform block 350, the second residual transform block 370, and the first residual transform block 350 shown and explained in FIGS. 3 to 11 ), the second transform layer 810 included in the second residual transform block 370 can be equally applied to the first residual transform block 1210 and the second residual transform block 1220 of FIG. 12, , the same description will be omitted.

The first input data (X1) input to the transform block 1200 according to one embodiment may be input to the first residual transform block 1210 and the second residual transform block 1220. The first output data output from the first residual transform block 1210 and the second output data output from the second residual transform block 1220 may be concatenated in the connection layer 1230. For example, the connection layer 1230 may output third output data obtained by connecting the first and second output data input to the connection layer 1230 in the channel direction to the normalization layer 1240. The third output data may be input to the normalization layer 1240 and normalized, and the normalized fourth output data may be input to the linear layer 1250.

In the linear layer 1250, the fifth output data may be obtained through a multiplication operation between the fourth output data input to the linear layer 1250 and the weight matrix included in the linear layer 1250. The fifth output data may be input to the summation layer 1260. Additionally, the first input data (X1) input to the transform block 1200 may be input to the summing layer 1260.

In the summation layer 1260, a summation operation for each element of the fifth output data and the first input data input to the summation layer 1260 may be performed.

The transform block 1200 according to an embodiment includes a first residual transform block 1210, a second residual transform block 1220, a connection layer 1230, a normalization layer 1240, a linear layer 1250, and a summation layer. The module 1201 including the layer 1260 may include a structure in which the module 1201 is repeatedly arranged in series. However, it is not limited to this.

Figure 13 is a diagram showing a transform block according to an embodiment.

The transform block 1300 shown in FIG. 13 may be one of the transform blocks 310 of FIG. 2 .

Referring to FIG. 13, the transform block 1300 according to an embodiment includes a first residual transform block 1310, a second residual transform block 1320, a first normalization layer 1321, and a second normalization layer ( 1322), first linear layer (1331), second linear layer (1332), first attention layer (1341), second attention layer (1342), third linear layer (1351), fourth linear layer (1352) , may include a first summation layer (1361), a second summation layer (1362), a connection layer (1370), a third normalization layer (1380), a fifth linear layer (1390), and a third summation layer (1395). there is.

The transform block 1300 according to one embodiment may include a structure in which the first residual transform block 1310 and the second residual transform block 1320 are connected in parallel.

The first residual transform block 1310 according to one embodiment may be referred to as an intra residual transform block (aRTB), and the second residual transform block 1320 may be referred to as an inter residual transform block (aRTB). It may be referred to as inter Residual Transformer Block (eRTB). The first residual transform block 1310 corresponds to the first residual transform block 350 in FIG. 3, and the second residual transform block 1320 corresponds to the second residual transform block 370 in FIG. 3. It can be. Accordingly, the first transform layer 410 included in the first residual transform block 350, the second residual transform block 370, and the first residual transform block 350 shown and explained in FIGS. 3 to 11 ), the second transform layer 810 included in the second residual transform block 370 can be equally applied to the first residual transform block 1310 and the second residual transform block 1320 of FIG. 13, , the same description will be omitted.

The first input data (X1) input to the transform block 1300 according to one embodiment may be input to the first residual transform block 1310 and the second residual transform block 1320. The first output data output from the first residual transform block 1310 may be input to the first normalization layer 1321 and normalized, and the normalized second output data may be input to the first linear layer 1331. there is.

In the first linear layer 1331, third output data is obtained through a multiplication operation between the first normalized data input to the first linear layer 1331 and the first weight matrix included in the first linear layer 1331. You can.

The third output data can be used as an attention map to attend to the fourth output data output from the second residual transform block 1320. For example, the third output data and the fourth output data may be input to the second attention layer 1342, and in the second attention layer 1342, an element-by-element multiplication operation of the third output data and the fourth output data is performed. It can be done.

Additionally, the fourth output data output from the second residual transform block 1320 may be input to the second normalization layer 1322 and normalized, and the normalized fifth output data may be input to the second linear layer 1332. It can be.

In the second linear layer 1332, the sixth output data is obtained through a multiplication operation between the fifth output data input to the second linear layer 1332 and the second weight matrix included in the second linear layer 1332. You can.

The sixth output data can be used as an attention map to attend to the first output data output from the first residual transform block 1310. For example, the first output data and the sixth output data may be input to the first attention layer 1341, and in the first attention layer 1341, an element-by-element multiplication operation of the first output data and the sixth output data is performed. It can be done.

The seventh output data output from the first attention layer 1341 may be input to the third linear layer 1351.

In the third linear layer 1351, the eighth output data is obtained through a multiplication operation between the seventh output data input to the third linear layer 1351 and the third weight matrix included in the third linear layer 1351. You can. The eighth output data may be input to the first summation layer 1361. Additionally, the first output data output from the first residual transform block 1310 may also be input to the first summation layer 1361. In the first summation layer 1361, an element-by-element sum operation may be performed on the eighth output data and the first output data input to the first summation layer 1361.

Additionally, the ninth output data output from the second attention layer 1342 may be input to the fourth linear layer 1352.

In the fourth linear layer 1352, the 10th output data is obtained through a multiplication operation between the 9th output data input to the 4th linear layer 1362 and the 4th weight matrix included in the 4th linear layer 1352. You can. The tenth output data may be input to the second summation layer 1362. Additionally, the fourth output data output from the second residual transform block 1320 may also be input to the second summation layer 1362. In the second summation layer 1362, an element-by-element summation operation may be performed on the 10th output data and the fourth output data input to the second summation layer 1362.

The 11th output data output from the first summation layer 1361 may be input to the connection layer 1370.

The twelfth output data output from the second summation layer 1362 may be input to the connection layer 1370. The connection layer 1370 may output the 13th output data by connecting the 11th output data and the 12th output data in the channel direction to the third normalization layer 1380. The 13th output data may be input to the third normalization layer 1380 and normalized, and the normalized 14th output data may be input to the fifth linear layer 1390.

In the fifth linear layer 1390, the 15th output data is obtained through a multiplication operation between the 14th output data input to the 5th linear layer 1390 and the 5th weight matrix included in the 5th linear layer 1390. You can. The 15th output data may be input to the third summation layer 1395. Additionally, the first input data (X1) input to the transform block 1300 may be input to the third summation layer 1395.

In the third summation layer 1395, the 16th output data can be obtained by performing an element-wise sum operation of the 15th output data input to the third summation layer 1395 and the first input data (X1).

The transform block 1300 according to an embodiment includes a first residual transform block 1310, a second residual transform block 1320, a first normalization layer 1321, a second normalization layer 1322, and a first linear transform block. Layer 1331, second linear layer 1332, first attention layer 1341, second attention layer 1342, third linear layer 1351, fourth linear layer 1352, first summation layer ( 1361), a second summation layer 1362, a connection layer 1370, a third normalization layer 1380, a fifth linear layer 1390, and a third summation layer 1395 are connected in series. It may contain repetitively arranged structures. However, it is not limited to this.

Figure 14 is a flowchart showing a method of operating an image processing device according to an embodiment.

The image processing device 100 according to an embodiment may acquire first feature data based on the first image using one or more neural networks (S1410).

For example, the image processing apparatus 100 may extract first feature data corresponding to the first image using one or more convolutional neural networks.

The image processing apparatus 100 according to an embodiment may perform first image processing on first feature data to obtain second feature data (S1420).

For example, the image processing apparatus 100 may obtain second feature data corresponding to first areas including the first number of pixels from the first feature data. The image processing apparatus 100 may obtain second feature data corresponding to each of the first areas based on information on surrounding areas for each of the first areas.

The image processing device 100 may obtain second feature data by performing self-attention on the first feature data.

The image processing apparatus 100 according to an embodiment may obtain second feature data corresponding to first feature data using the first self-attention module 412 shown and described in FIGS. 5 and 6. Since the operations performed in the first self-attention module 412 have been described in detail in FIGS. 5 and 6, the same description will be omitted.

The image processing device 100 according to an embodiment may acquire third feature data based on the first image (S1430).

For example, the image processing apparatus 100 may extract third feature data corresponding to the first image using one or more convolutional neural networks. Alternatively, the image processing device 100 may acquire third feature data based on the second feature data acquired in step S1420. However, it is not limited to this.

The image processing apparatus 100 according to one embodiment may perform second image processing on the third feature data to obtain fourth feature data (S1440).

For example, the image processing apparatus 100 may obtain fourth feature data corresponding to second areas including a second number of pixels greater than the first number from the third feature data. The image processing apparatus 100 may obtain fourth feature data corresponding to each of the second areas based on information on surrounding areas for each of the second areas.

The image processing device 100 may obtain fourth feature data by performing self-attention on the third feature data.

The image processing apparatus 100 according to an embodiment may acquire fourth feature data corresponding to third feature data using the second self-attention module 812 shown and described in FIGS. 9 to 11 . Since the operations performed in the second self-attention module 812 have been described in detail in FIGS. 9 to 11, the same description will be omitted.

The image processing device 100 according to an embodiment may generate a second image based on the second feature data and the fourth feature data (S1450).

For example, the second feature data may be feature data based on information about surrounding pixels adjacent to each pixel included in the first image, and the fourth feature data may be feature data based on a predetermined unit area included in the first image. It may be feature data based on information about surrounding areas (patches) adjacent to each of the fields (patches). Accordingly, the image processing device 100 according to an embodiment uses both information about adjacent surrounding pixels (local information) and information about adjacent surrounding areas (non-local information) to generate a second image. can be created.

Specifically, the image processing device 100 according to an embodiment may extract feature data from the second feature extraction network 300, receive the extracted feature data, and use the image restoration network 400 to 2 Images can be acquired. At this time, the second feature extraction network 300 includes a module for acquiring second feature data (e.g., a first self-attention module) and a module for obtaining fourth feature data (e.g., a second self-attention module). may include.

The second image according to one embodiment may be a higher-resolution image than the first image, and may be an image with improved image quality than the first image by removing artifacts, noise, etc. from the first image.

Figure 15 is a block diagram showing the configuration of an image processing device according to an embodiment.

The image processing device 100 of FIG. 15 may be a device that performs image processing using the image processing network 103. The image processing network 103 according to one embodiment may include one or more neural networks. For example, the image processing network 103 may include a first feature extraction network 200, a second feature extraction network 300, and an image restoration network 400. However, it is not limited to this.

Referring to FIG. 15 , the image processing device 100 according to an embodiment may include a processor 110, a memory 120, and a display 130.

The processor 110 according to one embodiment may generally control the image processing device 100. The processor 110 according to one embodiment may execute one or more programs stored in the memory 120.

The memory 120 according to one embodiment may store various data, programs, or applications for driving and controlling the image processing device 100. A program stored in memory 120 may include one or more instructions. A program (one or more instructions) or application stored in the memory 120 may be executed by the processor 110.

The processor 110 according to one embodiment may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Video Processing Unit (VPU). Alternatively, depending on the embodiment, it may be implemented in the form of a SoC (System On Chip) integrating at least one of CPU, GPU, and VPU. Alternatively, the processor 110 may further include a Neural Processing Unit (NPU).

The processor 110 according to one embodiment may generate a second image by processing the first image using one or more neural networks. For example, the processor 110 may use the image processing network 103 to remove noise from the first image and generate a second image that has undergone denoising to maintain detailed edge processing and texture. Alternatively, the processor 110 may use the image processing network 103 to generate a second image with higher resolution than the first image.

The processor 110 according to one embodiment may use the first feature extraction network 200 to obtain first feature data of the first image.

The processor 110 according to one embodiment may obtain second feature data (deep features) of the first image using the second feature extraction network 300 of FIG. 2. Since the structure and operation of the second feature extraction network 300 of FIG. 2 have been described in detail in FIGS. 3 to 14, detailed description will be omitted.

In particular, the processor 110 according to one embodiment uses the first self-attention module 412 included in the second feature extraction network 300 to select neighboring pixels adjacent to each of the pixels included in the first image. Feature data based on information about can be obtained. Since the structure and operation of the first self-attention module 412 are explained in detail in FIGS. 5 and 6, detailed description will be omitted.

Additionally, the processor 110 according to one embodiment uses the second self-attention module 812 included in the second feature extraction network 300 to select each of the predetermined unit regions (patches) included in the first image. Feature data based on information about surrounding areas (patches) adjacent to can be obtained. Since the structure and operation of the second self-attention module 812 are explained in detail in FIGS. 9 to 11, detailed description will be omitted.

In addition, the processor 100 according to one embodiment receives data extracted from the second feature extraction network 300 using the image restoration network 400, and uses the image restoration network 400 to generate a second image. can be obtained.

Meanwhile, the image processing network 103 according to one embodiment may be a network trained by a server or an external device. An external device can train the image processing network 103 based on training data. At this time, the training data may include a plurality of data sets including image data containing noise and image data from which the noise is removed while edge characteristics or texture characteristics are preserved.

The server or external device may determine the parameter values included in the kernels used in each of the plurality of convolutional layers included in the image processing network 103 and the parameter values included in the weight matrices used in each of the linear layers. there is. For example, the server or external device sets parameters in a direction to minimize the difference (loss information) of the image data (training data) that preserves edge characteristics while removing the image data and noise generated by the image processing network 103. Values can be determined.

The image processing device 100 according to one embodiment may receive the trained image processing network 103 from a server or an external device and store it in the memory 120 . For example, the memory 120 may store the structure and parameter values of the image processing network 103 according to an embodiment, and the processor 110 may use the parameter values stored in the memory 120, according to an embodiment. While noise is removed from the first image according to , a second image in which edge characteristics are preserved can be generated.

The display 130 according to one embodiment generates a driving signal by converting image signals, data signals, OSD signals, and control signals processed by the processor 110. The display 130 may be implemented as a PDP, LCD, OLED, flexible display, etc., and may also be implemented as a 3D display. Additionally, the display 130 can be configured as a touch screen and used as an input device in addition to an output device.

The display 130 according to one embodiment may display a second image that has been image processed using the image processing network 103.

Meanwhile, the block diagram of the image processing device 100 shown in FIG. 15 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the image processing device 100 that is actually implemented. That is, as needed, two or more components may be combined into one component, or one component may be subdivided into two or more components. In addition, the functions performed by each block are for explaining the embodiments, and the specific operations or devices do not limit the scope of the present invention.

An image processing device according to an embodiment may process images using one or more neural networks.

An image processing device according to an embodiment may include a memory that stores one or more instructions, and at least one processor that executes the one or more instructions stored in the memory.

The at least one processor may acquire first feature data based on the first image by executing the one or more instructions.

The at least one processor may perform first image processing on the first feature data by executing the one or more instructions to obtain second feature data corresponding to first areas including a first number of pixels. You can.

The at least one processor may obtain third feature data based on the first image by executing the one or more instructions.

The at least one processor performs second image processing on the third characteristic data by executing the one or more instructions, thereby producing second images corresponding to second areas including a second number of pixels greater than the first number. 4 Feature data can be obtained.

The at least one processor may generate a second image based on the second feature data and the fourth feature data by executing the one or more instructions.

The first image processing and the second image processing include self-attention.

The at least one processor may execute the one or more instructions to obtain the second feature data corresponding to each of the first areas, based on information on surrounding areas for each of the first areas.

The at least one processor may execute the one or more instructions to obtain the fourth characteristic data corresponding to each of the second areas based on information on surrounding areas for each of the second areas.

The first number is 1, and each of the first areas may include one pixel.

An image processing device according to an embodiment is based on information about surrounding pixels adjacent to each of the pixels included in the first image and information about surrounding areas adjacent to each of predetermined unit areas included in the first image. Thus, by processing the first image, the second image can be generated. Accordingly, the performance of image processing according to one embodiment can be improved compared to existing image processing technologies. For example, the degree of improvement in image quality or removal of noise of the generated second image may increase compared to the image processed by existing image processing technology.

The at least one processor executes the one or more instructions to generate query data, key data, and values respectively corresponding to the first areas included in the first characteristic data. Data can be obtained.

The at least one processor may obtain a weight matrix based on the query data and the key data by executing the one or more instructions.

The at least one processor may acquire the second feature data based on the value data and the weight matrix by executing the one or more instructions.

The at least one processor may obtain a correlation matrix based on the query data and the key data by executing the one or more instructions stored in the memory 120.

The at least one processor, by executing the one or more instructions, applies a positional bias based on the size of the first image and the size of the images used to train the one or more neural networks to the correlation matrix to determine the weights. You can obtain a matrix.

The at least one processor may convert the third feature data divided into third areas including the first number of pixels to be divided into the second areas by executing the one or more instructions.

The at least one processor may acquire the fourth feature data by executing the one or more instructions and performing the second image processing on each of the second areas.

By executing the one or more instructions, the at least one processor generates first query data and first key data respectively corresponding to third areas including the first number of pixels included in the third characteristic data. and first value data can be obtained.

The at least one processor, by executing the one or more instructions, groups each of the first query data, first key data, and first value data to correspond to each of the second areas, thereby Second query data, second key data, and second value data corresponding to the areas may be obtained.

The at least one processor may obtain a weight matrix based on the second query data and the second key data by executing the one or more instructions.

The at least one processor may acquire the fourth feature data based on the second value data and the weight matrix by executing the one or more instructions.

Third feature data according to one embodiment may be obtained from the second feature data.

One or more neural networks according to one embodiment may include one or more convolutional neural networks.

The at least one processor may extract the first feature data from the first image using the one or more convolutional neural networks by executing the one or more instructions.

The at least one processor may acquire fifth characteristic data based on the second characteristic data and the fourth characteristic data by executing the one or more instructions.

The at least one processor may acquire the second image from the fifth feature data by executing the one or more instructions and using the one or more convolutional neural networks.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment may include acquiring first feature data based on a first image.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment includes performing first image processing on the first feature data to create a first region including a first number of pixels. It may include obtaining second feature data corresponding to the features.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment may include acquiring third feature data based on the first image.

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment includes performing a second image processing on the third feature data to produce a second number of pixels greater than the first number. It may include acquiring fourth feature data corresponding to second areas including .

A method of operating an image processing device that processes an image using one or more neural networks according to an embodiment includes generating a second image based on the second feature data and the fourth feature data. It can be included.

The first image processing and the second image processing according to one embodiment may include self-attention.

Obtaining the second feature data may include obtaining the second feature data corresponding to each of the first regions based on information on surrounding regions for each of the first regions. .

Obtaining the fourth feature data may include acquiring the fourth feature data corresponding to each of the second regions based on information on surrounding regions for each of the second regions. .

The first number is 1, and each of the first areas may include one pixel.

The step of acquiring the second feature data includes acquiring query data, key data, and value data corresponding to the first areas included in the first feature data. May include steps.

Obtaining the second feature data may include obtaining a weight matrix based on the query data and the key data.

Obtaining the second feature data may include obtaining the second feature data based on the value data and the weight matrix.

The step of obtaining a weight matrix based on the query data and the key data includes obtaining a correlation matrix based on the query data and the key data, and determining the size of the first image and the one or more The method may include obtaining the weight matrix by applying a positional bias based on the size of images used for training neural networks to the correlation matrix.

Obtaining the fourth feature data may include converting the third feature data divided into third regions including the first number of pixels to be divided into the second regions.

Obtaining the fourth feature data may include obtaining the fourth feature data by performing the second image processing on each of the second regions.

The step of acquiring the fourth feature data includes first query data, first key data, and first query data, respectively, corresponding to third areas including the first number of pixels included in the third feature data. It may include acquiring value data.

The step of acquiring the fourth characteristic data includes grouping each of the first query data, first key data, and first value data to correspond to each of the second areas, thereby grouping each of the first query data, first key data, and first value data to correspond to each of the second areas. It may include obtaining second query data, second key data, and second value data.

Obtaining the fourth feature data may include obtaining a weight matrix based on the second query data and the second key data.

Obtaining the fourth feature data may include acquiring the fourth feature data based on the second value data and the weight matrix.

The third feature data may be obtained from the second feature data.

The one or more neural networks may include one or more convolutional neural networks.

Obtaining the first feature data may include extracting the first feature data from the first image using the one or more convolutional neural networks.

Generating the second image may include acquiring fifth feature data based on the second feature data and the fourth feature data.

Generating the second image may include obtaining the second image from the fifth feature data using the one or more convolutional neural networks.

A method of operating an image processing device according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

Additionally, the image processing device and the method of operating the image processing device according to the disclosed embodiments may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers.

A computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, a computer program product may include a product in the form of a S/W program (e.g., a downloadable app) distributed electronically by the manufacturer of an electronic device or through an electronic marketplace (e.g., Google Play Store, App Store). there is. For electronic distribution, at least part of the S/W program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a manufacturer's server, an electronic market server, or a relay server's storage medium that temporarily stores the SW program.

A computer program product, in a system comprised of a server and a client device, may include a storage medium of a server or a storage medium of a client device. Alternatively, if there is a third device (e.g., a smartphone) in communication connection with the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself, which is transmitted from a server to a client device or a third device, or from a third device to a client device.

In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a client device, and a third device may execute the computer program product and perform the methods according to the disclosed embodiments in a distributed manner.

For example, a server (eg, a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server and control a client device connected to the server to perform the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also included in the scope of the present invention. belongs to

Claims (23)

In an image processing device that processes images using one or more neural networks,
a memory 120 that stores one or more instructions; and
By executing the one or more instructions stored in the memory,
Obtain first feature data based on the first image,
Performing first image processing on the first feature data to obtain second feature data corresponding to first areas including a first number of pixels,
Obtaining third feature data based on the first image,
Performing second image processing on the third feature data to obtain fourth feature data corresponding to second areas including a second number of pixels greater than the first number,
An image processing device comprising at least one processor 110 that generates a second image based on the second feature data and the fourth feature data. According to paragraph 1,
The first image processing and the second image processing include self-attention. According to claim 1 or 2,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
Based on information on surrounding areas for each of the first areas, obtain the second feature data corresponding to each of the first areas,
An image processing device that acquires the fourth characteristic data corresponding to each of the second areas based on information on surrounding areas for each of the second areas. According to any one of claims 1 to 3,
The first number is one, and each of the first areas includes one pixel. According to any one of claims 1 to 4,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
Obtaining query data, key data, and value data corresponding to the first areas included in the first feature data, respectively,
Based on the query data and the key data, obtain a weight matrix,
An image processing device that obtains the second feature data based on the value data and the weight matrix. According to clause 5,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
Based on the query data and the key data, a correlation matrix is obtained, and a position bias based on the size of the first image and the size of the images used for training the one or more neural networks is applied to the correlation matrix. An image processing device that obtains the weight matrix by doing so. According to any one of claims 1 to 6,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
Converting the third feature data divided into third areas including the first number of pixels to be divided into the second areas,
An image processing device that obtains the fourth feature data by performing the second image processing on each of the second areas. According to any one of claims 1 to 7,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
Obtaining first query data, first key data, and first value data respectively corresponding to third areas including the first number of pixels included in the third feature data,
By grouping each of the first query data, first key data, and first value data to correspond to each of the second areas, second query data and second key data corresponding to the second areas and obtain second value data,
Based on the second query data and the second key data, obtain a weight matrix,
An image processing device that acquires the fourth feature data based on the second value data and the weight matrix. According to any one of claims 1 to 8,
The third feature data is obtained from the second feature data. According to any one of claims 1 to 9,
The one or more neural networks include one or more convolutional neural networks,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
An image processing device that extracts the first feature data from the first image using the one or more convolutional neural networks. According to any one of claims 1 to 10,
The one or more neural networks include one or more convolutional neural networks,
The at least one processor 110 executes the one or more instructions stored in the memory 120,
Based on the second characteristic data and the fourth characteristic data, obtain fifth characteristic data,
An image processing device that obtains the second image from the fifth feature data using the one or more convolutional neural networks. In a method of operating an image processing device that processes images using one or more neural networks,
Obtaining first feature data based on the first image;
performing first image processing on the first feature data to obtain second feature data corresponding to first areas including a first number of pixels;
Obtaining third feature data based on the first image;
performing second image processing on the third feature data to obtain fourth feature data corresponding to second areas including a second number of pixels greater than the first number; and
A method of operating an image processing device, comprising generating a second image based on the second feature data and the fourth feature data. According to clause 12,
The first image processing and the second image processing include self-attention. According to claim 12 or 13,
The step of acquiring the second feature data is,
Based on information on surrounding areas for each of the first areas, obtaining the second feature data corresponding to each of the first areas,
The step of acquiring the fourth characteristic data is,
A method of operating an image processing apparatus, comprising acquiring the fourth characteristic data corresponding to each of the second areas based on information on surrounding areas for each of the second areas. According to any one of claims 12 to 14,
The first number is one, and each of the first areas includes one pixel. According to any one of claims 12 to 15,
The step of acquiring the second feature data is,
Obtaining query data, key data, and value data respectively corresponding to the first areas included in the first characteristic data;
Obtaining a weight matrix based on the query data and the key data; and
A method of operating an image processing device, comprising acquiring the second feature data based on the value data and the weight matrix. According to clause 16,
The step of obtaining a weight matrix based on the query data and the key data includes:
Based on the query data and the key data, a correlation matrix is obtained, and a position bias based on the size of the first image and the size of the images used for training the one or more neural networks is applied to the correlation matrix. A method of operating an image processing device, comprising obtaining the weight matrix by doing so. According to any one of claims 12 to 17,
The step of acquiring the fourth characteristic data is,
converting the third feature data divided into third areas including the first number of pixels to be divided into the second areas; and
A method of operating an image processing apparatus, comprising obtaining the fourth feature data by performing the second image processing on each of the second areas. According to any one of claims 12 to 18,
The step of acquiring the fourth characteristic data is,
Obtaining first query data, first key data, and first value data respectively corresponding to third areas including the first number of pixels included in the third feature data;
By grouping each of the first query data, first key data, and first value data to correspond to each of the second areas, second query data and second key data corresponding to the second areas and acquiring second value data;
Obtaining a weight matrix based on the second query data and the second key data; and
A method of operating an image processing device, comprising acquiring the fourth feature data based on the second value data and the weight matrix. According to any one of claims 12 to 19,
A method of operating an image processing device, wherein the third feature data is obtained from the second feature data. According to any one of claims 12 to 20,
The one or more neural networks include one or more convolutional neural networks,
The step of acquiring the first feature data is,
A method of operating an image processing apparatus, comprising extracting the first feature data from the first image using the one or more convolutional neural networks. According to any one of claims 12 to 21,
The one or more neural networks include one or more convolutional neural networks,
The step of generating the second image is,
Obtaining fifth feature data based on the second feature data and the fourth feature data; and
A method of operating an image processing device, comprising obtaining the second image from the fifth feature data using the one or more convolutional neural networks. One or more computer-readable recording media storing a program for performing the method of any one of claims 12 to 22.

Images