KR20220162548A - Artificial intelligence learning method and apparatus for noise removal of image - Google Patents

Abstract

The present invention relates to an image learning apparatus for noise removal which can increase image recognition performance. The image learning apparatus for noise removal comprises: a first continuous feature value extraction module (CFE1) performing continuous convolution operations twice on an input image to continuously extract a first feature map and a second feature map; a deep feature map extraction module (DFE) extracting deep feature maps based on the second feature map, and concatenating the deep feature maps to extract a third feature map; a global shortcut concatenation module (GSC) adding the first feature map and the third feature map to extract a fourth feature map; a second continuous feature value extraction module (CFE2) performing continuous convolution operations on the fourth feature map to extract a fifth feature map and a sixth feature map; and a global residual mapping (GRM) subtracting an output value of the second continuous feature value extraction module from the input image to generate a correction image.

Description

Artificial intelligence learning method and apparatus for noise removal of image {Artificial intelligence learning method and apparatus for noise removal of image}

The present invention relates to an artificial intelligence learning method and apparatus for denoising images, and to improve image recognition speed and image recognition performance.

CCTV helps a lot in physical security, safety and crime prevention. Also, the dash cam is an essential part of the vehicle. Moreover, in autonomous vehicles, camera sensors are important because they act as the driver's eyes, sensing the environment, including the road and objects around them. Since digital imaging devices are common in everyday life, interest has been focused on improving image quality. However, the presence of artifacts in the image leads to image damage and degrades image quality. In particular, unpredictable artifacts such as raindrops degrade the performance of computer vision such as object detection and face detection. For this reason, it is necessary to develop an algorithm that removes unnecessary artifacts. In this paper, to secure visibility in bad weather, we deal with a single image rain removal problem.

An object of the present invention is to provide an image learning method and apparatus for removing noise capable of improving image recognition performance.

An object of the present invention is to provide an image learning method and apparatus for removing noise capable of increasing image recognition speed.

It relates to an image learning apparatus for denoising according to the present invention, and extracts a first continuous feature value by performing two consecutive convolution operations on an input image to continuously extract a first feature map and a second feature map. module (CFE1); a deep feature map extraction module (DFE) extracting deep feature maps based on the second feature map and extracting a third feature map by concatenating the deep feature maps; a wide area shortcut connection module (GSC) for outputting a fourth feature map by adding the first feature map and the third feature map; a second continuous feature value extraction module (CFE2) extracting fifth and sixth feature maps by performing continuous convolution operations on the fourth feature map; and a wide area afterimage mapping module (GRM) generating a corrected image by subtracting an output value of the second continuous feature value extraction module from the input image.

The present invention can improve image recognition processing speed while reducing training time.

According to the present invention, the image recognition quality can be improved by hierarchically extracting feature values.

1 is a diagram illustrating an artificial intelligence learning system for removing noise from an image according to an embodiment of the present invention.
2 is a flowchart illustrating an artificial intelligence learning method for denoising an image according to an embodiment of the present invention.
3 is a diagram showing a deep feature map extraction module according to an embodiment of the present invention.
4 and 5 are diagrams showing images used for experimental results.

Advantages and features of this specification, and methods of achieving them, will become clear with reference to embodiments described below in detail in conjunction with the accompanying drawings. However, this specification is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments make the disclosure of this specification complete, and the common knowledge in the technical field to which this specification belongs. It is provided to fully inform the owner of the scope of the invention, and this specification is only defined by the scope of the claims.

Each feature of the various embodiments of the present specification can be partially or entirely combined or combined with each other, technically various interlocking and driving are possible, and each embodiment can be implemented independently of each other or can be implemented together in an association relationship. may be

1 is a diagram illustrating an artificial intelligence learning system for removing noise from an image according to an embodiment of the present invention.

Referring to FIG. 1, the artificial intelligence learning system for denoising an image according to an embodiment of the present invention includes a first continuous feature value extraction module 100, a deep feature map extraction module 200, and a wide area shortcut connection module 300. ), a second continuous feature value extraction module 400, and a wide area afterimage mapping module 500.

The first continuous feature extraction module (hereinafter referred to as CFE) 100 continuously extracts a first feature map and a second feature map by performing two consecutive convolution operations on the input image.

A deep feature extraction module (hereinafter, DFE) 200 extracts deep feature maps based on the second feature map and extracts a third feature map by concatenating the deep feature maps. Through this, the DFE 200 may remove noise such as rain streaks from the input image IMG.

The Global Shortcut Connection (GSC) 300 adds the first feature map and the third feature map and outputs a fourth feature map.

The second CFE 400 extracts fifth and sixth feature maps by performing continuous convolution operations on the fourth feature map.

The Global Residual Mapping (GRM) 500 generates a corrected image by subtracting the output value of the second continuous feature value extraction module from the input image.

2 is a flowchart illustrating an artificial intelligence learning method for denoising an image according to an embodiment of the present invention.

Referring to FIG. 2, in the first step (S210) of the artificial intelligence learning method for denoising an image, the first CFE 100 performs two consecutive convolution operations on the input image to obtain a first characteristic The map and the second feature map are continuously extracted. To this end, the first CFE 100 includes first and second convolutional layers 110 and 120 .

The first convolution layer 110 extracts a first feature map F1 by performing a convolution operation on the input image IMG. In this specification, an operation for extracting the first feature map F1 by the first convolutional layer 110 is defined as in [Equation 1] below.

[Equation 1]

The second convolution layer 120 extracts a second feature map F2 by performing a convolution operation on the first feature map F1. In this specification, an operation for extracting the second feature map F2 by the second convolutional layer 120 is defined as in Equation 2 below.

[Equation 2]

The number of first feature maps F1 and the number of second feature maps F2 may be different from each other.

In a second step S220, the DFEs 200 extract the deep feature maps DF1 to DF5 based on the second feature map F2, and concatenate the deep feature maps DF1 to DF5. to extract the third feature map F3.

To this end, the DFE 200 includes a plurality of light residual blocks (hereinafter referred to as LRBs) LRB1 to LRBn (n is a natural number) and a 1×1 convolution layer 230 .

The LRBs LRB1 to LRBn receive the second feature map F2 and extract in-depth features. The first LRB (LRB1) receives the second feature map (F2) and extracts the first deep feature map (DF1). The i-th (i is a natural number greater than or equal to 2 and less than or equal to n) LRB receives the (i-1)th deep feature map (DF[i-1]) output by the (i-1)th LRB, and receives the ith deep feature map ( DFi) output.

To this end, each of the LRBs is composed of a plurality of convolution modules including a rectified linear unit (hereinafter, ReLU) and a 3×3 convolution layer, as shown in FIG. 3 .

In addition, the concatenation layer combines the deep feature maps output from each of the LRBs in a concatenation method, and provides the combined result as a 1×1 convolution layer.

The 1×1 convolution layer adjusts the number of third feature maps F3 finally output by the DFE 200 .

In this specification, the process of calculating the third feature map (F3) is defined as the following [Equation 3].

[Equation 3]

은 제3 특징맵(F3)의 개수를 조절하기 위해서 사용되는 1×1 컨볼루션 레이어를 지칭한다. At this time,

denotes a 1×1 convolution layer used to adjust the number of third feature maps F3.

DFE not only extracts features hierarchically, but also connects the deep feature maps generated in all layers by the depth connection used in DFE.

)를 각 계층의 출력에 연결하여 필요한 최소 계층을 제(i+1) LRB의 입력으로 연결한다. 본 명세서에서 LRB들 간의 연결은 [수학식 4]로 정의하기로 한다.Since the outputs of all layers are connected, the L-layer network has "{L(L+1)}/2" connections, increasing the learning parameter. To compensate for the increase in the learning parameter, the i th LRB provides initial information (

) to the output of each layer to connect the minimum required layer to the input of the (i+1)th LRB. In this specification, the connection between LRBs is defined by [Equation 4].

[Equation 4]

는 LRB의 3×3 컨볼루션 레이어와 ReLU를 결합한 복합 함수(composite function)이다.

는

과 초기 정보(

) 간의 연결(concatenation)을 의미한다. LRB는 연산의 효율성을 위해서 심층 특징맵의 개수를 줄이는 것이 중요하고, 따라서 LRB들의 출력에 해당하는 "

"는 1×1 컨볼루션 레이어의 입력에 해당한다. At this time,

is a composite function that combines the 3×3 convolutional layer of LRB and ReLU.

Is

and initial information (

) means the concatenation between them. For LRB, it is important to reduce the number of deep feature maps for operational efficiency, and therefore, the output of LRBs

" corresponds to the input of the 1×1 convolutional layer.

In addition, in order to prevent gradient vanishing, a shortcut connection is used. The shortest connection may be defined by the following [Equation 5].

[Equation 5]

Gradient vanishing refers to a phenomenon in which the differential operation increases as the depth of the network increases, and as a result, the differential value decreases and the size of the weight that affects the output decreases. In the present specification, the gradient vanishing phenomenon can be improved by using the shortest connection method.

The loss function of LRB may be defined as in [Equation 6] below.

[Equation 6]

는 유클리드 거리 손실(Euclidean distance loss)을 지칭하며, 두 픽셀 간의 거리를 측정하기 위해서 다음과 같은 [수학식 7]을 바탕으로 산출될 수 있다.At this time,

Refers to Euclidean distance loss, and can be calculated based on the following [Equation 7] to measure the distance between two pixels.

[Equation 7]

는 훈련을 위한 지상 실측(ground truth) 내의 패치 이미지을 지칭하며,

는 LRB로부터의 평가 패치 이미지를 지칭한다. 또한, SSIM은 최대값이 1이며 크기가 클 수록 두 이미지 간의 유사도가 높은 것을 나타낸다. 따라서,

은 1-SSIM으로 정의될 수 있다.In this case, R is a residual function, and n refers to the number of patch images. I input,

denotes a patch image in the ground truth for training,

denotes the evaluation patch image from the LRB. In addition, SSIM has a maximum value of 1, and the larger the size, the higher the similarity between the two images. therefore,

may be defined as 1-SSIM.

In a third step S230, the GSC 300 adds the first feature map F1 and the third feature map F3 to output a fourth feature map.

In a fourth step (S240), the second CFE 400 extracts fifth and sixth feature maps F5 and Ic by performing continuous convolution operations on the fourth feature map F4. The second CFE 400 may have the same structure as the first CFE 100.

In a fifth step (S250), the GRM 500 generates a corrected image by subtracting the output value of the second continuous feature value extraction module from the input image. Through this, the GRM 500 may generate a corrected image from which noise such as rain streaks are removed from the input image.

Examination results using the artificial intelligence learning device according to the present invention are as follows. 4 is a diagram showing images used for experimental results. FIG. 4(a) is a diagram showing an image including a virtual rain streak, and FIG. 4(b) is a diagram showing an image from which the rain streak is removed. Figure 4 (c) is a diagram showing the original image.

Each of the images shown in FIG. 4 was selected as a 16×16 patch unit, and 128 patches were randomly selected within the patch unit and trained using the artificial intelligence learning apparatus according to the present invention. The experiment environment was conducted on a PC using Intel® Core™ i7-1070K and 2080 Super GPU using tensorflow 2 framework.

Peak Signal to Noise Ratio (PSNR) and SSIM indicators were used to quantitatively evaluate image quality.

As shown in [Table 1] below, PSNR between the rain streak image and the original image without rain streak is 18.722 and SSIM is 0.7140. And the PSNR between the image with rain streaks removed and the original image without rain streaks is 31.158, and the SSIM is 0.9363.

[Table 1]

As can be seen from the experimental results shown in [Table 1], noise such as rain streaks was well removed from the image (a) of FIG. 4, and the results shown in (b) of FIG. It can be seen that the image from which the rain streaks are removed is similar to the original image shown in FIG. 4(c). That is, according to an embodiment of the present invention, rain streaks can be effectively removed from an image including rain streaks.

5 is a diagram showing noise removal simulation results according to an embodiment of the present invention.

5(a) is an image including virtual snow. Figure 5 (b) is an image in which eyes are removed from Figure 5 (a) based on an embodiment of the present invention. 5(c) is an original image without adding a virtual eye.

Referring to FIG. 5 , according to an embodiment of the present invention, it can be seen that noise such as snow is also efficiently removed. That is, according to an embodiment of the present invention, noise or artifacts such as rain and snow can also be efficiently removed.

Through the above description, those skilled in the art will know that various changes and modifications are possible without departing from the technical spirit of the present specification. Therefore, the technical scope of the present specification is not limited to the contents described in the detailed description of the specification, but should be determined by the claims.

Claims (11)

a first continuous feature value extraction module (CFE1) for continuously extracting a first feature map and a second feature map by performing two consecutive convolution operations on the input image;
a deep feature map extraction module (DFE) extracting deep feature maps based on the second feature map and extracting a third feature map by concatenating the deep feature maps;
a wide area shortcut connection module (GSC) outputting a fourth feature map by adding the first feature map and the third feature map;
a second continuous feature value extraction module (CFE2) extracting fifth and sixth feature maps by performing continuous convolution operations on the fourth feature map; and
and a global afterimage mapping module (GRM) generating a corrected image by subtracting the output value of the second continuous feature extracting module from the input image.
According to claim 1,
The first continuous feature value extraction module
a first convolution layer extracting the first feature map by performing a convolution operation on the input image; and
and a second convolution layer for extracting the second feature map by performing a convolution operation on the first feature map.
According to claim 1,
The deep feature map extraction module
lightweight afterimage blocks (LRBs) that detect noise in an image based on a deep neural network and output the deep feature map; and
and a 1×1 convolution layer for adjusting the number of the deep feature maps output by the lightweight afterimage blocks.
According to claim 3,
Among the lightweight afterimage blocks, the ith (i is a natural number) lightweight afterimage block,
Denoising, characterized in that by receiving the (i-1)th deep feature map output by the (i-1)th lightweight afterimage block, and outputting the deep feature map to the (i+1)th lightweight afterimage block. image learning device for
According to claim 4,
The i-th lightweight afterimage block is composed of a plurality of convolution modules including a rectification linear unit and a 3×3 convolution layer,
Each of the rectified linear units and 3 × 3 convolutional layers,
Each of the convolution modules combines a deep feature map output by the (i-1)th lightweight afterimage block and a feature output by the previous convolution module.
According to claim 5,
The deep feature map extraction module
and a concatenation layer combining each of the deep feature maps output by the plurality of lightweight afterimage blocks by a concatenation method and providing them as the 1×1 convolution layer. image learning device for
According to claim 1,
The deep feature map extraction module
An image learning device for denoising, characterized in that for outputting the deep feature map based on a loss function defined as the sum of a Euclidean distance loss function and an SSIM loss function.
sequentially extracting a first feature map and a second feature map by performing two consecutive convolution operations on the input image;
extracting deep feature maps based on the second feature map and extracting a third feature map by concatenating the deep feature maps;
outputting a fourth feature map by adding the first feature map and the third feature map;
extracting fifth and sixth feature maps by performing continuous convolution operations on the fourth feature map; and
and generating a correction image by subtracting an output value of the second continuous feature value extraction module from the input image.
According to claim 8,
The step of continuously extracting the second feature map
extracting the first feature map by performing a convolution operation on the input image; and
An image learning method for denoising, comprising the step of extracting the second feature map by performing a convolution operation on the first feature map.
According to claim 8,
Extracting the third feature map
Detecting noise in the input image based on a deep neural network, sequentially outputting a plurality of deep feature maps, and adjusting the number of the deep feature maps based on a 1×1 convolution layer. learning method.
According to claim 10,
Each i-th deep feature map is extracted step by step based on a rectified linear unit and a 3×3 convolution layer, and the j-th deep feature map is generated by combining the initial deep feature map and the j-1-th deep feature map. Image learning method for noise removal, characterized in that.

Images