KR102599753B1 - YUV Image Processing Method and System Using A Neural Network Composed Of Dual-Path Blocks - Google Patents

Abstract

A YUV image processing method and device using the YUV data format and utilizing a neural network composed of dual path blocks suitable for the YUV data format are disclosed. The YUV image processing method of the present invention includes the steps of a neural network inputting and processing image data in YUV format to generate a luma feature map and a chroma feature map; The dual-path block of the neural network includes extracting features from input image data; And the neural network includes performing a task based on the extracted features, wherein the dual path block is composed of multiple layers, and two paths each composed of two branches are configured in parallel, and each branch is configured in parallel. It is characterized by learning to extract Luma (Y) feature information and Chroma (UV) feature information separately, and to exchange and combine the extracted information. According to the present invention, an efficient and effective image processing method can be provided by utilizing a neural network composed of dual-path blocks that use YUV image data as an input to the neural network.

Description

YUV image processing method and device using a neural network composed of dual-path blocks {YUV Image Processing Method and System Using A Neural Network Composed Of Dual-Path Blocks}

The present invention relates to a YUV image processing method and device using a neural network composed of dual-path blocks. More specifically, it uses the YUV image data format, which has relatively less data compared to the RGB format, and uses the YUV image data format. This relates to a YUV image processing method and device using a neural network composed of suitable dual-path blocks.

Recently, deep learning technology using neural networks in the image processing field has been attracting attention for its amazing performance. Neural networks have the ability to learn complex features from data, allowing the development of highly effective image processing techniques.

Deep learning technology has been developed with a focus on improving the model's inference accuracy and operation speed in a balanced manner. For example, making models deeper and more complex improves inference accuracy but reduces operation speed. On the other hand, lowering the image resolution makes the model run faster, but makes inferences more inaccurate. By combining these factors well, the trade-off between accuracy and speed of neural networks can be improved.

Generally, the RGB format, a standard color expression method, is used as input image data for a neural network. However, when using RGB data as input to a model, you may face the following problems:

First of all, RGB data can represent a variety of colors by expressing the three colors of red, green, and blue with values between 0 and 255, but it has the disadvantage of requiring large memory usage.

In addition, because processes such as YUV transformation, discrete cosine transformation, and quantization are used to efficiently store RGB data, there is a disadvantage that additional costs for decoding are incurred during the learning or inference process.

Republic of Korea Patent Publication No. 10-2023-0013989 Republic of Korea Patent No. 2234097 Republic of Korea Patent No. 2200496

The present invention was developed in consideration of the above-mentioned problems, and its purpose is to provide a YUV image processing method and device using a neural network composed of dual-path blocks that use relatively less data compared to the RGB format.

The YUV image processing device of the present invention for solving the above problems includes YUV image data consisting of a luma (Y) component, which is a brightness component, and a chroma (UV) component, which is a color difference; and an input unit that receives and processes the YUV image data to separately input and process the luma component and chroma component, a feature detection unit that extracts features from a dual path block using the luma component and chroma component input from the input unit, and the dual path block. It includes a neural network composed of a task execution unit that performs a task based on features extracted from the path block, wherein the dual path block is composed of multiple layers, and two paths each composed of two branches are configured in parallel, respectively. The branch of is learned to separately extract luma (Y) feature information and chroma (UV) feature information, and exchange and combine the extracted information.

The YUV image processing method of the present invention for solving the above other problems includes the steps of inputting and processing image data in YUV format, which consists of a luma (Y) component and a chroma (UV) component, in a neural network; Extracting features from luma (Y) and chroma (UV) components input to a dual-path block of the neural network; And the neural network includes performing a task based on the extracted features, wherein the dual path block is composed of multiple layers, and two paths each composed of two branches are configured in parallel, and each branch is configured in parallel. It is characterized by learning to extract luma (Y) feature information and chroma (UV) feature information separately, and to exchange and combine the extracted information.

In the present invention, learning can be facilitated by adding a skip connection between the dual path blocks stacked in multiple layers.

In the present invention, the step of extracting features from an image input as a dual-path block involves extracting the luma component (Y) from a convolutional layer. Generating an enhanced luma feature map by inputting it into an enhanced luma component extraction branch including; The chroma component (UV) is a convolution layer. Generating an enhanced chroma feature map by inputting it into an enhanced chroma component feature extraction branch including; Inputting the input luma component (Y) into an exchanged luma component extraction branch to generate an exchanged luma feature map and connecting it to the extraction result of the enhanced chroma extraction branch; Inputting the input chroma component (UV) into a swapped chroma component feature extraction branch to generate a swapped chroma feature map and connecting it to the output result of the enhanced luma feature extraction branch; and combining the connected feature maps.

In the present invention, the generation of the exchange luma feature map involves combining the input luma component (Y) with a pooling layer and a convolution layer. Using , a scaled exchange luma feature map can be generated.

In the present invention, the generation of the exchanged chroma feature map involves converting the input chroma component (UV) into a convolution layer. and Nearest Neighbor Interpolation layer (up) can be used to form a scaled exchange chroma feature map.

In the present invention, the convolutional layers configured in parallel , , , and can be performed by being integrated into one convolutional layer at the time of execution.

In the present invention, in the step of combining the connected feature maps, the connected feature maps are each combined by an element-wise sum operation, and batch normalization (BN) and activation function are performed. By passing through it, a richer feature map can be output.

According to the present invention having the configuration described above, an efficient and effective image processing method can be provided by utilizing a neural network composed of dual-path blocks that use YUV image data as an input to the neural network.

There are two major advantages when using YUV image data as input instead of RGB image data:

First, since the process of converting YUV format to RGB format can be omitted during the image decoding process, image processing speed can be accelerated.

Additionally, the YUV 4:2:0 format requires four times less data for color expression than before. Therefore, memory consumption during learning and inference can be small and computation speed can be fast.

Additionally, the performance of the neural network can be improved by using a dual-path block, which is a form suitable for processing YUV image data.

Figure 1 is a block diagram showing a YUV image processing device using a neural network composed of dual path blocks according to an embodiment of the present invention.
Figure 2 is a flowchart showing a YUV image processing method of a neural network using a dual path block according to an embodiment of the present invention.
Figure 3 is a flowchart explaining the operation of a dual-path block according to an embodiment of the present invention.
Figure 4 is a diagram for explaining the actual operation process of a convolution layer according to an embodiment of the present invention.

Hereinafter, a YUV image processing method and device using a neural network composed of dual path blocks according to a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

The present invention provides an efficient and effective deep learning-based image processing method. To this end, we use image data in the YUV format, which has relatively less data than the RGB format, and propose a neural network block structure composed of dual-path blocks suitable for processing the YUV image data format.

Specifically, by using image data in YUV 4:2:0 format as input, memory usage and calculation amount can be reduced, and good inference performance can be obtained by implementing a dual-path block suitable for image processing in the YUV format. suggests a method.

Additionally, an advantage of using YUV image data is that the image decoding process can be simplified. In one embodiment, the decoding process of a JPEG compressed image includes converting from YUV format to RGB format. When using a YUV image directly, the conversion process can be omitted, which can achieve the effect of slightly reducing the amount of calculation and reducing latency in the execution of actual applications.

Figure 1 is a block diagram showing a YUV image processing device using a neural network composed of dual path blocks according to an embodiment of the present invention.

Referring to FIG. 1, a YUV image processing device using a neural network consists of YUV image data 10 and a neural network 50 that processes the image data.

The YUV image data 10 consists of a luma (Y) component, which is the brightness, and a chroma (UV) component, which is the color difference.

The neural network 50 includes an input unit 100 that receives and processes the YUV image data 10 to separate the luma component and the chroma component, and outputs the luma component and the chroma component input from the input unit 100 through a dual path. It includes a feature detection unit 200 that extracts features from the block 210, and a task execution unit 300 that performs a task based on the features extracted from the dual path block 210.

The dual path block 210 is composed of two paths each consisting of two branches in parallel, so that each branch separately extracts luma (Y) feature information and chroma (UV) feature information, and the extracted information Learned to exchange and combine them.

Figure 2 is a flowchart showing a YUV image processing method of a neural network using a dual path block according to an embodiment of the present invention.

Referring to Figures 1 and 2, the neural network 50 inputs (S100) and processes (S110) image data in YUV format, using a dual-path block in the input image data. It includes extracting features (S200) and performing a task based on the extracted features (S300). In the present invention, image data in YUV format is divided into Y (luma) component and UV (chroma) component.

The step of inputting (S100) and processing the image (S110) involves using a convolution layer with a stride of 2 and a kernel size of 7×7, a batch normalization layer, and ReLU activation. It includes an activation function and a max pooling layer with a stride of 2 and a kernel size of 3×3. After passing this step, the sizes of the luma (Y) component and chroma (UV) component are each reduced by 4 times. For example, when an image of size 32×32 is input, an 8×8 luma feature map and a 4×4 chroma feature map are created.

In the step (S200) of extracting features from the input image using a dual-path block (210), the input luma component and chroma component pass through the dual-path block (210) connected in series. The information contained in the map becomes richer. The operation of the dual path block 210 will be described later.

As the neural network 50 is made deeper by stacking multiple layers of the dual path blocks 210, inference performance can be improved.

Additionally, learning can be facilitated by adding a skip connection (211) between the dual path blocks 210 stacked in multiple layers.

In the step of performing a specific task based on the extracted features (S300), the extracted feature map passes through a head (310, task-specific head) appropriate for the task to be performed. The neural network of the present invention can be used in various image processing tasks. For example, image class classification, face recognition, autonomous driving, etc.

Figure 3 is a flowchart explaining the operation of the dual path block 210 according to an embodiment of the present invention.

As described above, image data in YUV format is separated into luma component (Y) and chroma component (UV) and processed (S100, S110).

First, the dual path block 210 inputs the luma component (Y) to the convolution-based enhanced luma component feature extraction branch (S210).

The enhanced luma component feature extraction branch (S210) is a convolutional layer with a kernel size of 3×3. Includes. The input luma component (Y) is As it passes through, the information contained in the luma feature map is strengthened and an enhanced luma feature map is created.

Next, the chroma component (UV) is input to the convolution-based enhanced chroma component feature extraction branch (S220).

The enhanced chroma component feature extraction branch (S220) is a convolutional layer with a kernel size of 3 × 3, Includes. The input chroma component (UV) is As it passes through, the information contained in the chroma feature map is strengthened and an enhanced chroma feature map is created.

Next, the input luma component (Y) is input to the exchanged luma component feature extraction branch (S230) to generate an exchanged luma feature map, and is connected to the extraction result of the enhanced chroma component feature extraction branch (S220). For this purpose, an average pooling layer with a stride of 2 and a kernel size of 2 × 2, and a convolution layer with a kernel size of 3 × 3 are used. Includes. The input luma component (Y) is adjusted to the same size as the enhanced chroma feature map through the pooling layer, and then Information to be transmitted to the enhanced chroma feature map is extracted.

Next, the input chroma component (UV) is input to the exchanged chroma component feature extraction branch (S240) to generate an exchanged chroma component, and connected to the output result of the enhanced luma component feature extraction branch (S210). For this purpose, a convolutional layer with a kernel size of 3×3 and a Nearest Neighbor Interpolation layer (up). The input chroma component (UV) is, Information to be transmitted to the enhanced luma feature map is extracted through , and its size is adjusted so that it can be connected to the enhanced luma feature map through the nearest neighbor interpolation layer (up).

Next, the connected feature maps are combined (S250, S260). In the process of combining connected feature maps, luma feature information and chroma feature information can become more diverse.

Combination of connected feature maps includes element-wise sum operation, batch normalization (BN), and ReLU function (ReLU). The connected feature maps are combined by a sum operation for each element, and pass through batch normalization (BN) and ReLU activation function (ReLU) to output a richer combined feature map.

Here, the parallel convolutional layers are , , , and can be performed by being integrated into one convolutional layer at the time of execution.

In general, the convolution operation within a neural network converts a 4-dimensional feature map tensor of size (B, iC, iH, iW) and a 4-dimensional weight kernel tensor of size (oC, iC, k, k) into a 2-dimensional matrix, respectively. and is optimized by performing GEMM (GEneral Matrix Multiplications) operations. The process of converting a 4-dimensional feature map tensor of size (B, iC, iH, iW) into a 2-dimensional matrix of size (iC × k × k, B × oH × W) is called im2col, and if this process is used appropriately, The luma feature map and chroma feature map can be combined into a two-dimensional matrix of size (iC_C×k×k + iC_L×k×k, 1.5×B×oH×oW). also, , , , and The weight kernel tensors can be simply combined to create a two-dimensional weight kernel matrix of size (oC_C + oC_L, iC_C×k×k + iC_L×k×k), so four convolutions can be performed simultaneously with just one GEMM operation. You can.

Here, B is the batch size, iH, iW are the height and width of the input feature map, iC is the number of channels of the input feature map, k is the weight kernel size, oH, oW are the height and width of the output feature map, oC. means the number of channels of the output feature map. The subscripts L and C represent luma and chroma, respectively.

Figure 4 is a diagram for explaining the actual operation process of a convolution layer according to an embodiment of the present invention.

Referring to Figure 4, output feature maps are generated through a single GEMM operation of the weight kernel and input feature maps.

Referring again to FIG. 1, the feature map extracted through the multi-layer dual path block 200 passes through a task-specific head depending on the task to be performed. The neural network 50 of the present invention can be used for various image processing tasks, such as image class classification, face recognition, and autonomous driving.

Below, as an example, an image class classification task using the present invention will be described.

To verify the effectiveness of using the neural network 50 of the present invention containing YUV image data 10 and dual path blocks 200, it is compared with the famous network ResNet-18. In order to have a similar size, the neural network of the present invention is composed of 16-layer dual path blocks. In each block, the number of channels for luma and chroma feature maps is 32 each.

The neural network of the present invention has 11.1M learnable parameters, a similar size to ResNet-18, which has 11.2M parameters. However, the amount of calculations (FLOPs) of the neural network of the present invention that inputs YUV image data is 48,3M, which is 30% smaller than the 70,5M of ResNet-18 that inputs RGB image data, and therefore operates more efficiently. You can.

Additionally, YUV image data consumes less cost than RGB image data in the image decoding process, so the latency of the entire execution process may be lower.

Specifically, the effectiveness of the neural network of the present invention can be verified using the CIFAR-10 dataset consisting of 10 classes and 6,000 images for each class. The dataset consists of 50,000 images for training and 10,000 images for evaluation, and all images have a size of 32×32.

For a fair comparison, all learning hyperparameter settings are the same.

We learn for 200 epochs using image data for training, and measure the accuracy of how accurate the inference results for the image data for evaluation are. The batch size is 128, the loss function is cross-entropy, and the optimizer uses SGD (Stochastic Gradient Descent). The initial learning rate is set to 0.01, and a schedule that decreases by a factor of 10 at 134 and 178 epochs is used.

For the evaluation dataset, the inference accuracy is 89% for the neural network of the present invention and 86% for ResNet-18, so the present invention can achieve better performance.

The present invention described above is not limited to the above-described drawings and detailed description, but can be modified and modified in various ways by those skilled in the art without departing from the spirit and scope of the present invention as set forth in the claims below. Of course, it is also included within the scope of the present invention.

10: YUV image data 50: Neural network
100: input unit 200: feature detection unit
210: Dual path block 211: Skip connection
300: Work performance unit 310: Head

Claims (7)

delete The neural network includes inputting and processing image data in YUV format, which consists of a luma (Y) component and a chroma (UV) component;
Extracting features from luma (Y) and chroma (UV) components input to a dual-path block of the neural network; and
The neural network includes performing a task based on the extracted features,
The dual path block is composed of multiple layers, and two paths each composed of two branches are configured in parallel, so that each branch separately extracts luma (Y) feature information and chroma (UV) feature information, and Learned to exchange and combine extracted information.
The step of extracting features from the image input to the dual-path block is,
The luma component (Y) is a convolution layer. Generating an enhanced luma feature map by inputting the enhanced luma component into a feature extraction branch including;
The chroma component (UV) is a convolution layer. Generating an enhanced chroma feature map by inputting it into an enhanced chroma component feature extraction branch including;
Inputting the input luma component (Y) into a swapped luma component feature extraction branch to generate a swapped luma feature map and connecting it to the extraction result of the enhanced chroma component feature extraction branch;
Inputting the input chroma component (UV) into a swapped chroma component feature extraction branch to generate a swapped chroma feature map and connecting it to the output result of the enhanced luma component feature extraction branch; and
A YUV image processing method comprising combining the connected feature maps. According to paragraph 2,
A YUV image processing method characterized in that learning can be facilitated by adding a skip connection between the dual path blocks stacked in multiple layers. delete According to paragraph 2,
The generation of the exchange luma feature map is,
The input luma component (Y) is divided into a pooling layer and a convolution layer. Using , a scaled exchange luma feature map is generated,
The generation of the exchange chroma feature map is,
The input chroma component (UV) is converted to a convolution layer. A YUV image processing method characterized in that a resized chroma feature map is generated using a Nearest Neighbor Interpolation layer (up). According to clause 5,
The above convolutional layers constructed in parallel , , , and is a YUV image processing method characterized in that it is integrated into one convolutional layer when executed. According to paragraph 2,
The step of combining the connected feature maps is,
A YUV image processing method characterized in that the connected feature maps are combined by an element-wise sum operation, pass through batch normalization (BN) and an activation function, and output a combined feature map. .