KR20220040025A - Thermal image reconstruction device and method based on deep learning - Google Patents

Abstract

The present invention relates to a deep learning-based thermal image reconstruction apparatus and method. According to one embodiment of the present invention, the deep learning-based thermal image reconstruction device and method can easily detect an object by reconstructing a low-resolution thermal image and a thermal image of a blur phenomenon through deep learning.

Description

DEEP LEARNING BASED THERMAL IMAGE RECONSTRUCTION DEVICE AND METHOD

The present invention relates to an apparatus and method for reconstructing a thermal image, and more particularly, to a thermal image reconstruction based on deep learning that can easily perform object detection by restoring a low resolution and blur phenomenon of a thermal image acquired with a camera using deep learning It relates to an apparatus and method.

Recently, as the demand for thermal imaging technology is rapidly increasing due to COVID-19, various technologies related to thermal imaging are being developed.

Unlike conventional visible light cameras, thermal imaging cameras that take thermal images capture wavelengths in a specific region (8-12 μm) so that they can be viewed as images. Thermal imaging cameras are used in various fields, such as detecting the condition and internal defects of industrial equipment or large buildings, detecting fire, and detecting heat of the human body.

However, thermal images taken with thermal imaging cameras have problems such as human motion, camera shake, and out-of-focus due to the characteristics of the thermal image, and the characteristic blur of thermal images occurs. Circumstances that cannot be obtained arise. In addition, there is a problem in that it is necessary to spend a lot of money in order to purchase a thermal imaging camera capable of taking a high-resolution thermal image.

Background art of the present invention is disclosed in Korean Patent Registration No. 10-2127014.

The present invention provides a deep learning-based thermal image reconstruction apparatus and method for reconstructing a low-resolution thermal image to a high resolution through deep learning learning.

The present invention provides an apparatus and method for reconstructing a thermal image based on deep learning that can easily detect an object by restoring a thermal image in which a blur phenomenon exists through deep learning learning.

According to one aspect of the present invention, a deep learning-based thermal image reconstruction apparatus is provided.

A deep learning-based thermal image reconstruction apparatus according to an embodiment of the present invention includes an input unit for receiving thermal image information acquired with a thermal camera, a color conversion unit for color-converting the received thermal image into a color corresponding to temperature, and a color-converted column When the resolution of the image is less than a preset reference value, a high-resolution restoration unit for performing high-resolution restoration may be included.

According to another aspect of the present invention, a deep learning-based thermal image reconstruction method is provided.

A deep learning-based thermal image reconstruction method according to an embodiment of the present invention includes the steps of receiving thermal image information acquired with a thermal camera, color-converting the received thermal image into a color corresponding to temperature, and the color-converted thermal image. The method may include performing high-resolution restoration when the resolution is less than a preset reference value.

According to an embodiment of the present invention, a deep learning-based thermal image reconstruction apparatus and method may restore a low-resolution thermal image to a high resolution through deep learning learning.

According to an embodiment of the present invention, an apparatus and method for reconstructing a thermal image based on deep learning can easily detect an object by reconstructing a thermal image having a blur phenomenon through deep learning learning.

1 to 12 are diagrams for explaining a deep learning-based thermal image reconstruction apparatus according to an embodiment of the present invention.
13 to 20 are diagrams illustrating restoration performance of a deep learning-based thermal image reconstruction apparatus according to an embodiment of the present invention.
21 is a view for explaining a deep learning-based thermal image reconstruction method according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Also, the expressions "a" and "a", "a" and "a", as used in this specification and claims, should generally be construed to mean "one or more" unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do it with

1 to 8 are diagrams for explaining a deep learning-based thermal image reconstruction apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , the deep learning-based thermal image reconstruction apparatus 10 includes an input unit 110 , a color conversion unit 120 , a high-resolution restoration unit 130 , an image division unit 140 , a blur restoration unit 150 and It includes an image coupling unit 160 .

The input unit 110 receives thermal image information required in the process of performing high-resolution restoration. For example, the input unit 110 may receive a black-and-white thermal image or a color thermal image acquired by a thermal camera.

When the acquired thermal image is a black-and-white thermal image, the color conversion unit 120 converts it to color and expresses it as a color corresponding to the temperature of the measured thermal image.

Referring to FIG. 2 , the color conversion unit 120 converts 0 to 255 pixels (a total of 256 pixels) of a black-and-white column image by matching colors. For example, in the black-and-white column image input to the input unit 110 , a pixel value of 255 of an area having the highest column appears as white, and a pixel value of 0 as a pixel value of an area with the lowest column appears as black. The color conversion unit 120 converts the black-and-white column image input to the input unit 110 into color, so that 255, the pixel value of the region with the highest column, is red (255,0,0), and the pixel value of the region with the lowest column. 0 is expressed in blue (0,0,255). That is, the color conversion unit 120 converts a 1-channel thermal image into a 3-channel thermal image.

Referring to FIG. 3 , Peak Signal-to-Noise Ratio (PSNR), Signal-to-Noise Ratio (SNR), and Structural Similarity Index Map (SSIM) values in the black-and-white space (gray) are lower than those of the color space (RGB). The values were measured, and the Mean Square Error (MSE) value was measured to be higher than that of the color space (RGB, etc.). Here, PSNR (Peak Signal-to-Noise Ratio), SNR (Signal-to-Noise Ratio), and MSE (Mean Square Error) are indicators for evaluating loss information on image quality, and SSIM (Structural Similarity Index Map) is an indicator for evaluating visual quality. For PSNR (Peak Signal-to-Noise Ratio), SNR (Signal-to-Noise Ratio), and SSIM (Structural Similarity Index Map) values, the higher the measured value, the higher the image accuracy, and the MSE (Mean Square Error) value is measured The lower the value, the higher the image accuracy.

It can be confirmed that the color thermal image converted to a color corresponding to the temperature of the thermal image by the color conversion unit 120 through the numerical value of the thermal image index implements a thermal image restoration performance of higher quality than that of the black-and-white thermal image.

Referring back to FIG. 1 , the high-resolution restoration unit 130 restores the thermal image color-converted by the color transformation unit 120 . Specifically, the high-resolution restoration unit 130 performs high-resolution restoration when the resolution of the color-converted thermal image is less than a preset reference value. Here, the high-resolution restoration unit 130 performs resolution restoration using a high-resolution reconstruction generative adversarial neural network (SRRGAN) as shown in FIG. 4 .

The high-resolution reconstructed generative adversarial neural network (SRRGAN) is configured so that the Generator Network and the Discriminator Network compete with each other while learning. The generator network plays a role in generating data from an input image, and the discriminator network distinguishes whether an image given as an input is data generated through the generator network or an actual image. perform the role

Referring to FIG. 5 , a generator network of a high-resolution reconstructed generative adversarial neural network (SRRGAN) generates high-resolution thermal image data by inputting a low-resolution thermal image. Thereafter, the discriminator network outputs a high-resolution thermal image through a process of determining whether the input image is an actual image 1 or a high-resolution image 0 generated through a generator network.

6 to 8 show the structures of a generator network and a discriminator network of a high-resolution reconstructed generative adversarial neural network (SRRGAN).

Referring to FIG. 6 , a generator network is composed of a plurality of layers. Specifically, the Generator Network includes 17 layers, among which conv2d_1 to conv2d_5 layers have a filter size of 9x9, and the remaining layers have a filter size of 3x3. have

In this case, the structure of the res_block layer further includes conv2d_1, prelu, convd_2, and add layers as shown in FIG. 7 . In addition, the generator network has a stride of 1x1 and a padding of 1x1.

Referring to FIG. 8, the discriminator network of the high resolution reconstruction generative adversarial neural network (SRRGAN) includes 9 layers, of which 1 stride in the case of conv_block_1, conv_block_3 and conv_block_6 layers. (stride) and has a padding of 1x1. In addition, the remaining convolution blocks, conv_block_2, conv_block_4, and conv_block_5 layers, have 2 strides and 0x0 padding. All layers from conv_block_1 to conv_block_6 have a filter size of 3x3.

Referring back to FIG. 1 , the image division unit 140 receives the thermal image from the color conversion unit 120 and converts it into a color corresponding to the temperature of the thermal image when the resolution of the thermal image is greater than or equal to a preset reference value. The entire image can be divided into regions of a certain size. Also, the image dividing unit 140 may receive the high-resolution thermal image restored from the high-resolution restoration unit 130 and divide it into regions of a predetermined size. Here, the image dividing unit 140 may determine the blur of the reconstructed high-resolution thermal image using a preset reference value, receive only the thermal image of the blur, and divide it into regions of a predetermined size.

For example, the image dividing unit 140 generates a thermal image by dividing a 360*360 thermal image into 60*60 dimensions as shown in FIG. 9 . If learning is performed without dividing the entire image converted into color by a certain size, a problem occurs in that many features included in the entire image cannot be learned, thereby deteriorating the blur restoration performance of the thermal image. Accordingly, the image dividing unit 140 according to an embodiment of the present invention may obtain a high-quality thermal image by dividing the entire image into predetermined sizes before blur restoration.

The blur restoration unit 150 restores the blur of the thermal image divided by the image division unit 140 into a predetermined size. Specifically, the blur restoration unit 150 performs blur restoration using a deblur convolutional neural network (DeblurCNN).

Deblur Convolutional Neural Network (DeblurCNN) is a type of artificial neural network used to analyze visual images. It is characterized by processing the blur by extracting features from the image.

Referring to FIG. 10 , the deblur convolutional neural network (DeblurCNN) is composed of a generator network and a discriminator network like the high-resolution reconstructed generative adversarial neural network (SRRGAN) described above. A generator network of a deblur convolutional neural network (DeblurCNN) generates thermal image data in which blur is restored by inputting a thermal image of the blur phenomenon. Thereafter, the discriminator network determines whether the input image is an actual image 1 or a blur restored image 0 generated through a generator network.

The deblur convolutional neural network (DeblurCNN) performs blur restoration by inputting an image divided into a predetermined size by the image dividing unit 140 as shown in FIG. 11 . Thereafter, the blur restoration unit 150 transmits the restored images to the image combining unit 160 through a deblur convolutional neural network (DeblurCNN).

Referring to FIG. 12 , a generator network of a deblur convolutional neural network (DeblurCNN) includes 12 layers, and all layers have a filter size of 3x3. In addition, the generator network has a stride of 1x1 and a padding of 1x1.

The structure of the generator network res_block layer of the deblur convolutional neural network (DeblurCNN) and the structure of the discriminator network are the high-resolution reconstructed generative adversarial neural networks (SRRGAN) of FIGS. 7 and 8 described above. The structure of the generator network res_block layer and the discriminator network of the res_block are the same.

Referring back to FIG. 1 , the image combining unit 160 receives images restored from the blur restoration unit 150 through a deblur convolutional neural network (DeblurCNN). Thereafter, the image combining unit 160 combines the divided reconstructed images and regenerates them into one entire image.

13 to 20 are diagrams illustrating restoration performance of a deep learning-based thermal image reconstruction apparatus according to an embodiment of the present invention.

13 and 14 , a reconstructed thermal image (Proposed method) according to an embodiment of the present invention includes an existing high-resolution restoration technique (Bicubic, SRCNN, SRGAN, PULSE) and an existing blur restoration technique (DeblurCNN, DeblurGAN, It can be seen that DeblurCGAN-v2) exhibits more effective restoration performance than the thermal image reconstructed. The thermal image reconstructed through color conversion in the color conversion unit 200 has the most similar resolution and blur restoration effect to the original compared to other restoration techniques.

Referring to FIG. 15 , the restored thermal image (Proposed method) according to an embodiment of the present invention has better restoration performance than the thermal image restored through the existing high-resolution method by comparing SSIM (Structural Similarity Index Map) measurement values. can be checked

As a result of measuring Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Map (SSIM) values using six pieces of data, the PSNR (Proposed method) of a reconstructed thermal image according to an embodiment of the present invention ( The maximum value of Peak Signal-to-Noise Ratio) was 22.99, and the maximum value of SSIM (Structural Similarity Index Map) was measured to be 0.9760. The measured SSIM (Structural Similarity Index Map) value was higher than that of the thermal image reconstructed through other high-resolution restoration techniques, but the PSNR (Peak Signal-to-Noise Ratio) value is the thermal image restored through other high-resolution restoration techniques. A relatively low value was measured compared to . In the case of PSNR (Peak Signal-to-Noise Ratio), it is not possible to accurately evaluate the difference and similarity of visual quality seen by humans, and there are cases in which the accuracy is unclear. Accordingly, in the present invention, the quality of a thermal image is relatively evaluated by comparing SSIM (Structural Similarity Index Map) values.

Referring to FIG. 16 , as a result of measuring a Structural Similarity Index Map (SSIM) value using six pieces of data, a Structural Similarity Index Map (SSIM) maximum of a reconstructed thermal image (Proposed method) according to an embodiment of the present invention The value was determined to be 0.9873. Through this, it can be confirmed that the reconstructed thermal image (Proposed method) according to an embodiment of the present invention has better blur restoration performance than other restoration techniques by comparing SSIM (Structural Similarity Index Map) measurement values.

17 and 18 are other examples for comparing a thermal image reconstructed according to an embodiment of the present invention with a thermal image reconstructed by a conventional high-resolution restoration technique and a conventional blur restoration technique. 13 and 14, the reconstructed thermal image (Proposed method) according to an embodiment of the present invention includes an existing high-resolution restoration technique (Bicubic, SRCNN, SRGAN, PULSE) and an existing blur restoration technique (DeblurCNN, DeblurGAN and DeblurCGAN-v2) show more effective reconstruction performance than the reconstructed thermal image.

19 and 20 , a reconstructed thermal image (Proposed method) according to an embodiment of the present invention includes Recall, Precision, GlobalACC, F1 and Check the restoration performance by comparing Jaccard values.

Here, Recall represents the ratio of the detection result to the actual correct answer, and represents the ratio of how closely the precision detector detects the correct answer. F1 is the harmonic mean value of Precision and Recall results, and Jaccard represents the calculated similarity of thermal images.

Referring to FIG. 19 , the Recall, Precision, GlobalACC, F1 and Jaccard values of the reconstructed thermal image (Proposed method) according to an embodiment of the present invention are Recall of the existing high-resolution restoration techniques (Bicubic, SRCNN, SRGAN, PULSE), Higher values than Precision, GlobalACC, F1 and Jaccard were measured, and referring to FIG. 20 , the Recall, GlobalACC, F1 and Jaccard values of the reconstructed thermal image (Proposed method) according to an embodiment of the present invention restore the existing blur. Methods (DeblurCNN, DeblurGAN, DeblurCGAN-v2) had higher values than Recall, GlobalACC, F1 and Jaccard. Through this, it can be confirmed that the restored thermal image (Proposed method) according to an embodiment of the present invention has superior resolution and blur restoration performance than other restoration techniques.

21 is a diagram for explaining a deep learning-based thermal image reconstruction method according to an embodiment of the present invention.

Referring to FIG. 21 , in step S2101 , the deep learning-based thermal image reconstruction apparatus 10 receives thermal image information required in the process of performing high-resolution reconstruction. For example, the deep learning-based thermal image reconstruction apparatus 10 may receive a black-and-white thermal image or a color thermal image.

In step S2103, the deep learning-based thermal image reconstruction apparatus 10 converts the received monochrome thermal image into color. Specifically, the deep learning-based thermal image reconstruction apparatus 10 converts 0 to 255 pixels (a total of 256 pixels) of a black-and-white thermal image by color matching. For example, in the black-and-white thermal image input to the deep learning-based thermal image reconstruction apparatus 10 , a pixel value of 255, which is a pixel value of a region having the highest column, appears as white, and a pixel value of 0, which is a pixel value of a region with the lowest column, appears as black. The deep learning-based thermal image reconstruction apparatus 10 converts the input black-and-white thermal image into color, so that 255, the pixel value of the region with the highest column, is red (255,0,0), and the pixel value of the region with the lowest column, 0 is expressed in blue (0,0,255).

In step S2105, when the resolution of the color-converted thermal image is less than a preset reference value, the deep learning-based thermal image reconstruction apparatus 10 moves to step S2107 to perform high-resolution restoration. Here, the deep learning-based thermal image reconstruction apparatus 10 performs resolution reconstruction using a high-resolution reconstruction generative adversarial neural network (SRRGAN).

In step S2109, the deep learning-based thermal image reconstruction apparatus 10 determines the blur of the reconstructed high-resolution thermal image by using a preset reference value, and if it is a blur, moves to step S2111.

Also, if the resolution of the column image converted in color in step S2105 is less than a preset reference value, the process moves to step S2111.

In step S2111, the deep learning-based thermal image reconstruction apparatus 10 divides the received thermal image into predetermined sizes.

For example, the deep learning-based thermal image reconstruction apparatus 10 divides a thermal image of a size of 360*360 into a size of 60*60. If learning is performed without dividing the entire image converted into color by a certain size, a problem occurs in that many features included in the entire image cannot be learned, thereby deteriorating the blur restoration performance of the thermal image. Accordingly, the deep learning-based thermal image reconstruction apparatus 10 according to an embodiment of the present invention may obtain a high-quality thermal image by dividing the entire image into predetermined sizes before blur restoration.

In step S2113, the deep learning-based thermal image reconstruction apparatus 10 performs thermal image blur restoration using a deblur convolutional neural network (DeblurCNN).

In step S2115, the deep learning-based thermal image reconstruction apparatus 10 receives the reconstructed thermal image, combines the divided reconstructed images, and regenerates it into one full image.

In the above, even though it has been described that all components constituting the embodiment of the present invention are combined or operated as one, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more.

Although acts are shown in a particular order in the drawings, it should not be understood that the acts must be performed in the specific order or sequential order shown, or that all illustrated acts must be performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is

So far, the present invention has been focused on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential subway characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (10)

In the deep learning-based thermal image reconstruction apparatus,
an input unit for receiving thermal image information acquired by a thermal camera;
a color conversion unit for color-converting the received thermal image into a color corresponding to a temperature; and
A deep learning-based thermal image reconstruction apparatus comprising a high-resolution restoration unit that performs high-resolution restoration when the resolution of the color-converted thermal image is less than a preset reference value.
The method of claim 1,
an image dividing unit that receives the thermal image and divides it into regions of a predetermined size when the resolution of the color-converted thermal image is greater than or equal to a preset reference value;
a blur restoration unit that restores the blur of the image divided into the region of a certain size; and
Deep learning-based thermal image reconstruction apparatus further comprising an image combining unit that combines the thermal images restored from the blur from the blur restoration unit and regenerates them into one whole image.
The method of claim 1,
an image division unit that receives a high-resolution reconstructed thermal image and divides it into regions of a predetermined size;
a blur restoration unit that restores the blur of the image divided into the region of a certain size; and
Deep learning-based thermal image reconstruction apparatus further comprising an image combining unit that combines the thermal images restored from the blur from the blur restoration unit and regenerates them into one whole image.
4. The method of claim 2 or 3,
The blur restoration unit is a deep learning-based thermal image reconstruction apparatus for performing blur restoration using a deblur convolutional neural network (DeblurCNN).
The method of claim 1,
The high-resolution reconstruction unit is a deep learning-based thermal image reconstruction apparatus for performing high-resolution reconstruction using a super-resolution reconstruction generative adversarial neural network (SRRGAN).
In the deep learning-based thermal image reconstruction method,
Receiving thermal image information acquired by a thermal camera;
color-converting the received thermal image into a color corresponding to a temperature; and
and performing high-resolution restoration when the resolution of the color-converted thermal image is less than a preset reference value.
7. The method of claim 6,
receiving the column image and dividing the column image into regions of a predetermined size when the resolution of the color-converted column image is greater than or equal to a preset reference value;
performing blur restoration on the image divided into the region of the predetermined size; and
Deep learning-based thermal image reconstruction method further comprising the step of combining the reconstructed thermal images to regenerate into one whole image.
7. The method of claim 6,
receiving the high-resolution reconstructed thermal image and dividing it into regions of a predetermined size;
performing blur restoration of the image divided into the area of the predetermined size; and
Deep learning-based thermal image reconstruction method further comprising the step of combining the reconstructed thermal images to regenerate into one whole image.
9. The method according to claim 7 or 8,
The step of performing blur restoration of the image divided into the area of a certain size is
A deep learning-based thermal image reconstruction method that performs blur restoration using a deblur convolutional neural network (DeblurCNN).
According to claim 6,
The step of performing high-resolution restoration when the resolution of the color-converted thermal image is less than a preset reference value is a deep learning-based thermal image reconstruction method of performing high-resolution restoration using a super-resolution reconstruction generative adversarial neural network (SRRGAN).

Images